Table of contents
ARTIFICIAL INTELLIGENCE & SOCIETY
PART 18
In 2013, Paul Zilly was in court in Barron County, Wisconsin, for stealing a push lawnmower and some tools. The prosecutor and his lawyer had agreed on a plea deal: a year in county jail and supervision after that.[1]
Then the judge looked at Zilly’s risk scores. A program called COMPAS had rated him high risk for future violent crime.[1]
“When I look at the risk assessment,” the judge said, “it is about as bad as it could be.”[1]
He threw out the plea deal. He sentenced Zilly to two years in state prison instead.[1]
On appeal, the man who built COMPAS testified that he did not design it for sentencing. The judge then cut the sentence to 18 months and said that without the score, he would likely have given one year, six months.[1]
That is six months of a man’s life tied to a score whose calculations the company does not disclose.[1]
Last week in this series I wrote about software screening job applications before a person reads them. Courts and prisons have been doing a version of this for years. The stakes are a lot higher than a job.
So I want to look at how these scores work, what the research found, and what a person in the system can actually do about one.
Direct Answer
A criminal risk assessment is a scoring tool. It takes a person’s answers to a questionnaire, plus the criminal record, and gives a rating. The rating is usually low, medium, or high. It is a guess at how likely the person is to be arrested again.[2] Judges, pretrial staff, probation agents, and prison officials use these scores. They help decide bail, sentences, and supervision. In the federal system, they also help decide who can earn extra time off.[1][6]
The research on them is mixed. ProPublica found in 2016 that COMPAS was right about 61% of the time, and that Black defendants who did not reoffend were labeled higher risk almost twice as often as white defendants who did not reoffend.[2]
The company disputed that, but it did not share its calculations.[1]
A 2018 study found people with no criminal justice training predicted reoffending about as well as COMPAS did.[3]
In Wisconsin, the state Supreme Court ruled in State v. Loomis (2016) that judges may consider a COMPAS score. But the score cannot decide whether a person goes to prison or how long the sentence is. The judge has to explain the other reasons for the sentence.[4]
What a risk score is
COMPAS stands for Correctional Offender Management Profiling for Alternative Sanctions. It was built by a company then called Northpointe, now equivant.[1][3] Its scores come from up to 137 questions. Some are answered by the person; others are pulled from criminal records. Race is not one of them.[1]
The questions include things like “Was one of your parents ever sent to jail or prison?” and “How many of your friends/acquaintances are taking drugs illegally?”[1] The tool gives a score from 1 to 10. A 1 to 4 is low, 5 to 7 is medium, and 8 to 10 is high.[2]
Is this AI? I want to be careful with that word. COMPAS is not a chatbot, and it is not the kind of AI people mean when they talk about ChatGPT. The company does not publish how it weighs its questions.[1] When two Dartmouth researchers tested it, they found a simple formula using only a person’s age and number of prior convictions predicted as well as COMPAS did with all of its questions.[3]
I’m including it in a series about AI because the question is the same one I keep coming back to. When software hands an institution a number about a person, who checks the number, and who is responsible for what happens next?
The scores are based on group data. In plain terms, the tool says that people with answers like yours were arrested again at a certain rate. It cannot say what you will do. Wisconsin’s Supreme Court made that exact point in Loomis.[4]
What the research found
ProPublica got COMPAS scores for more than 7,000 people arrested in Broward County, Florida, in 2013 and 2014, then checked who was charged with a new crime within two years.[1]
Here are the main results:
- Of the people COMPAS said were likely to reoffend, 61% were arrested again within two years.[1]
- The violent crime score was right 20% of the time.[1]
- Black defendants who did not reoffend were labeled higher risk 45% of the time. For white defendants who did not reoffend, it was 23%.[2]
- White defendants who did reoffend were labeled lower risk 48% of the time, compared with 28% for Black defendants.[2]
Northpointe disputed the analysis and defended the tool’s accuracy.[1] As the Dartmouth researchers summed it up, the company argued the score is fair by a different measure. A high-risk score means about the same chance of rearrest whether the person is Black or white.[3] Other researchers have shown a problem with that. When two groups are rearrested at different rates, a score cannot meet both definitions of fair at once.[3]
So people can look at the same numbers and disagree about whether the tool is biased.
The Dartmouth study asked a different question. Julia Dressel and Hany Farid recruited people online. Most had little or no criminal justice background. Each one read short descriptions of real defendants and guessed who would reoffend. Pooled together, the group was right 67% of the time. COMPAS was right 65.2% of the time on the same defendants.[3] They also found the people in the study made the same kind of racial error COMPAS made, even when they were not told the defendant’s race.[3]
There’s one more limit. These tools are tested against rearrest, not against whether someone actually committed a crime. The Dartmouth researchers pointed out that Black people are almost four times as likely as white people to be arrested for drug offenses, so arrest data carries its own gaps into the score.[3]
What Wisconsin’s Supreme Court said
Wisconsin has used COMPAS more than most states. ProPublica reported that the Department of Corrections rolled it out statewide in 2012 and used it at each step from sentencing to parole. Once a person was convicted of a felony, the score was attached to the presentence report the judge got.[1]
Eric Loomis was sentenced in La Crosse County in 2013. The judge said COMPAS had identified him as “an individual who is at high risk to the community.”[1] Loomis argued that using a score with secret methods violated his right to due process.[5]
The Wisconsin Supreme Court ruled against him in July 2016, but it set limits.[4] A sentencing court may consider a COMPAS score. The score may not be used to decide whether a person is incarcerated or how severe the sentence is. It also may not be the deciding factor in whether someone can be supervised in the community. The judge has to explain the other factors that support the sentence.[4]
The court also required any presentence report with a COMPAS score to carry written warnings.[4] Those warnings say:
- The company keeps secret how the factors are weighed and how the scores are calculated.
- The scores are based on group data, so they can point to groups of high-risk people, not a particular high-risk person.
- Some studies have raised questions about whether the scores classify minority defendants as higher risk.
- The tool compares people to a national sample, and no study had yet tested it on a Wisconsin population.
A Harvard Law Review note on the case argued that the warnings probably would not change much, because judges were not told how strong the criticisms were and had no way to check the tool themselves.[5] That is an argument, not a finding, and it is from 2017.
I think the note raises a fair concern, but I don’t know how Wisconsin judges use these scores today.
The federal version
The federal system uses its own tool.
The First Step Act of 2018 required the Bureau of Prisons to build a risk assessment, which became PATTERN.[6]
PATTERN puts people in four levels: minimum, low, medium, and high. People with eligible offenses can earn up to 10 days of time credit per month for programming. People in the minimum or low levels whose risk does not go up over two assessments in a row can earn 5 more days per month.[6] Those credits can move someone to a halfway house or home confinement sooner.[7] So in the federal system, the score affects how much time a person can earn off.
The Justice Department has to review PATTERN every year.[6]
The August 2024 review found it predicts rearrest well overall. But it also found the general tool overpredicts risk for Black, Hispanic, and Asian men and women compared with white people. The gap for Black men and women had gotten worse.[6] The Justice Department’s own report called that disparity “the Department’s leading concern” about the tool.[7]
The same review found that people can lower their PATTERN level while they are locked up. The people who lowered it had the lowest rearrest rates.[6] To me that is the most useful part of the federal tool. It can show change, and change is the whole point of programming.
Where this touches my own story
I was first convicted of a felony possession of marijuana charge at 19, and I spent 15 years in and out of the system after that. I wrote about the 2019 stretch in my memoir.[8]
That year, after a relapse on probation, my probation agent told me, “It’s MSDF or revocation of your probation.”[8]
She told me she didn’t think I needed to be locked up that long. She said there was a shorter program at MSDF, but when they forwarded my paperwork, I was assigned the longer one. The paperwork said 60 to 120 days, or until I finished the program.[8]
So what was in that paperwork? I don’t know, and I don’t know whether a risk score was part of it. The memoir doesn’t say, and I’m not going to claim it was. What I do know is that the next eight months of my life came down to paperwork, and the agent who gave me the news didn’t think I needed that long.
I did my time for my own choices, and I still think that.
I’m not arguing that people shouldn’t face consequences. I’m saying that when a number helps decide those consequences, the person it’s about should be able to see what went into it and check it.
What you can check if a score is in your case
This is not legal advice. If you have a case, your lawyer is the person to ask, and the rules are different from state to state.
- Ask whether a risk tool was used. In Wisconsin, a COMPAS score in a presentence report has to come with the written warnings the Supreme Court required.[4] If you see those warnings, a score is in there.
- Check the inputs. The Loomis court said a person can check whether the questions and answers on the report are accurate, because they come from the person’s own answers and public records.[4] Wrong criminal history, a charge that was dismissed, or an answer you did not give are things your lawyer can raise.
- Ask how the judge used it. Under Loomis, the score cannot decide whether you go to prison or how long, and the judge has to explain the other reasons for the sentence.[4] If a judge leans on the score, that is something your lawyer can look at.
- In federal prison, know your level. Your PATTERN level affects how many time credits you can earn, and it can change with programming and good conduct.[6] Ask your case manager what your level is and when you’ll be reassessed.
The question underneath
The people who sell and use these tools have a fair point. Judges and parole boards made these calls on instinct for a long time, and instinct carries bias too.[1] A score that’s tested, published, and checked could make decisions more consistent.
But a tool can only be checked if people can see how it works. COMPAS’s method is a trade secret.[4] PATTERN’s items and scoring are published, and the government reviews it every year.[6]
For me, the second kind of system is a lot easier to defend than the first.
If a score is going to follow someone into a courtroom, the person should be able to see it. They should be able to check what went into it and argue with it. And a judge should still have to explain the other reasons for the decision, the way Wisconsin’s court required.
Frequently Asked Questions
What is a criminal risk assessment?
It is a scoring tool that uses a questionnaire and a person’s criminal record to rate how likely the person is to be arrested again, usually as low, medium, or high. Courts, probation, and prisons use the ratings to help with bail, sentencing, supervision, and release decisions.[1][2]
Is COMPAS biased?
It depends on how you measure fairness. ProPublica found that Black defendants who did not reoffend were labeled higher risk 45% of the time, compared with 23% for white defendants.[2] The company says the scores are fair because a given score means about the same rearrest rate for both groups.[3] Researchers have shown both kinds of fairness cannot be met at once when groups are rearrested at different rates.[3]
Can a judge in Wisconsin sentence someone based on a COMPAS score?
A judge may consider it, but under State v. Loomis the score cannot be used to decide whether a person is incarcerated or how severe the sentence is, and the judge has to explain the other factors that support the sentence.[4]
How accurate are risk assessment tools?
In ProPublica’s Broward County data, COMPAS was right about 61% of the time for any new arrest and 20% of the time for violent crime.[1] In a 2018 study, a group of untrained people predicted rearrest about as well as COMPAS did.[3] The federal PATTERN tool tested better overall in the Justice Department’s 2024 review, but it still overpredicts risk for some racial and ethnic groups.[6]
What is PATTERN in the First Step Act?
PATTERN is the federal Bureau of Prisons’ risk tool. It places people in minimum, low, medium, or high risk. People with eligible offenses earn up to 10 days of time credit a month for programming, and those rated minimum or low can earn 5 more.[6]
References
- Angwin, Julia, Jeff Larson, Surya Mattu, and Lauren Kirchner. “Machine Bias.” ProPublica, May 23, 2016. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
- Larson, Jeff, Surya Mattu, Lauren Kirchner, and Julia Angwin. “How We Analyzed the COMPAS Recidivism Algorithm.” ProPublica, May 23, 2016. https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm
- Dressel, Julia, and Hany Farid. “The accuracy, fairness, and limits of predicting recidivism.” Science Advances 4, no. 1 (2018): eaao5580. https://www.science.org/doi/10.1126/sciadv.aao5580
- Supreme Court of Wisconsin. State v. Loomis, 2016 WI 68, No. 2015AP157-CR, opinion filed July 13, 2016. https://www.wicourts.gov/sc/opinion/DisplayDocument.pdf?content=pdf&seqNo=171690
- Harvard Law Review. “State v. Loomis: Wisconsin Supreme Court Requires Warning Before Use of Algorithmic Risk Assessments in Sentencing.” Vol. 130, March 10, 2017. https://harvardlawreview.org/print/vol-130/state-v-loomis/
- National Institute of Justice. “2023 Review and Revalidation of the First Step Act Risk Assessment Tool.” U.S. Department of Justice, August 2024. https://www.ojp.gov/pdffiles1/nij/309264.pdf
- U.S. Department of Justice. “First Step Act Annual Report.” June 2024. https://www.bop.gov/inmates/fsa/docs/first-step-act-annual-report-june-2024.pdf
- Drasen, Andrew. A Vision of Hope. Memoir. https://www.avisionofhopebook.com/books/memoir