Illinois DCFS Uses Augintel to Uncover Bias
The Office of Racial Equity Practice at DCFS could track investigations, placements, and permanency outcomes.
Learn how Maryland DHS uses AI to analyze the case notes their workforce was already writing →

At APHSA ISM in Spokane this year, the thing I noticed more than anything else is that the conversations around AI are dramatically changing. Where in years past the discussion was about how agencies could consider AI or how they might implement AI, the whole of the space has accepted that AI isn’t the future — AI is now. This led to more sophisticated questions than in years past.
One question came up more than any other: How accurate is the model?
A few years ago, we rarely heard it. Agencies wanted to know what AI could do, not how well it did it. So the fact that accuracy is now the first question is major progress.
However, it’s only the first piece of an AI evaluation.
Many of the agency leaders we talked with are learning, often through their own pilots, that accuracy alone doesn’t tell them whether their tools and models will work. It only opens up more questions. Making sense of accuracy requires a much deeper and richer understanding.
An accuracy score is only as meaningful as the data it was measured on. A model tested on a small, clean sample, or on cases from another state with different policies, can score well and still struggle with your caseload. The first follow-up to any accuracy claim should be: accurate on what?
Accuracy combines every right and wrong answer into one number, but in child welfare not all errors are equal.
A model can have a strong overall accuracy score and still miss a meaningful share of the cases you care about most, especially when those topics are relatively rare. Which tradeoff is acceptable depends on how the tool will be used, so agencies should decide that before they look at the numbers.
“AI” covers different kinds of models, and they need to be evaluated differently.
Traditional machine learning models typically classify or score: is this allegation type present or not? You can measure them against labeled cases with precision and recall, and adjust thresholds to favor one over the other depending on the importance of the signal.
Large language models generate language: summaries, extracted details, answers to questions. Evaluating them is less straightforward. There may be more than one acceptable answer. Results can shift depending on how the model is instructed and what context it’s given. And when an LLM is wrong, it can still sound fluent and confident.
Agencies don’t need to become data scientists. But knowing which kind of model they’re looking at, and what evaluation looks like for each, helps them ask the right questions.
The best illustration came from Arizona’s session on AI accuracy.
The team set an 80% accuracy benchmark for identifying allegation types in narrative text and worked toward 90%. They didn’t stop at the number. They looked closely at where the model and human reviewers disagreed and asked why.
One answer involved domestic violence. Under the agency’s definition, a child’s presence during domestic violence can meet the threshold for abuse, even if the child isn’t physically harmed. The model didn’t capture that. A higher accuracy score wouldn’t have revealed the gap. Studying the disagreements did.
That is the shift I’d want every agency to make: from “Is it accurate?” to “Where is it accurate, where isn’t it, and why?”
The domestic violence example shows that the hardest errors aren’t technical. They’re gaps between how a model reads language and how child welfare policy and practice define risk. Closing those gaps takes people who know the work, not just people who know the models.
If the only answer you get is an accuracy number, keep asking.
The most encouraging thing about ISM this year was watching agencies move past AI as a novelty. They’re asking harder questions about performance, implementation, and oversight, and whether a tool actually fits the work.