At APHSA ISM in Spokane this year, the thing I noticed more than anything else is that the conversations around AI are dramatically changing. Where in years past the discussion was about how agencies could consider AI or how they might implement AI, the whole of the space has accepted that AI isn’t the future — AI is now. This led to more sophisticated questions than in years past.

One question came up more than any other: How accurate is the model?  

A few years ago, we rarely heard it. Agencies wanted to know what AI could do, not how well it did it. So the fact that accuracy is now the first question is major progress.

However, it’s only the first piece of an AI evaluation.  

Evaluating AI Accuracy in Child Welfare

Many of the agency leaders we talked with are learning, often through their own pilots, that accuracy alone doesn’t tell them whether their tools and models will work. It only opens up more questions. Making sense of accuracy requires a much deeper and richer understanding.

1. Evaluate the sample set

An accuracy score is only as meaningful as the data it was measured on. A model tested on a small, clean sample, or on cases from another state with different policies, can score well and still struggle with your caseload. The first follow-up to any accuracy claim should be: accurate on what?

2. Evaluate precision and recall

Accuracy combines every right and wrong answer into one number, but in child welfare not all errors are equal.

• Precision tells you how often the model is right when it flags something. Low precision means caseworkers spend time on false alarms.

• Recall tells you what proportion of relevant cases the model successfully identifies. Low recall means real concerns get missed.

A model can have a strong overall accuracy score and still miss a meaningful share of the cases you care about most, especially when those topics are relatively rare. Which tradeoff is acceptable depends on how the tool will be used, so agencies should decide that before they look at the numbers.

3. Evaluate how the technology itself works

“AI” covers different kinds of models, and they need to be evaluated differently.

Traditional machine learning models typically classify or score: is this allegation type present or not? You can measure them against labeled cases with precision and recall, and adjust thresholds to favor one over the other depending on the importance of the signal.  

Large language models generate language: summaries, extracted details, answers to questions. Evaluating them is less straightforward. There may be more than one acceptable answer. Results can shift depending on how the model is instructed and what context it’s given. And when an LLM is wrong, it can still sound fluent and confident.

Evaluating an LLM means checking whether:

• its output is complete,

• faithful to the source record,  

• and consistent with your agency’s definitions, not just whether it matches a single label.

Agencies don’t need to become data scientists. But knowing which kind of model they’re looking at, and what evaluation looks like for each, helps them ask the right questions.

What this looks like in practice

The best illustration came from Arizona’s session on AI accuracy.

The team set an 80% accuracy benchmark for identifying allegation types in narrative text and worked toward 90%. They didn’t stop at the number. They looked closely at where the model and human reviewers disagreed and asked why.

One answer involved domestic violence. Under the agency’s definition, a child’s presence during domestic violence can meet the threshold for abuse, even if the child isn’t physically harmed. The model didn’t capture that. A higher accuracy score wouldn’t have revealed the gap. Studying the disagreements did.

That is the shift I’d want every agency to make: from “Is it accurate?” to “Where is it accurate, where isn’t it, and why?”

Why domain expertise matters

The domestic violence example shows that the hardest errors aren’t technical. They’re gaps between how a model reads language and how child welfare policy and practice define risk. Closing those gaps takes people who know the work, not just people who know the models.

Questions to ask any AI vendor (including us)

• What data was the model tested on, and how similar is it to our population?

• What are the precision and recall for the specific task we’re using it for?

• Is this a traditional machine learning model, a large language model, or both, and how was each evaluated?

• Where does the model disagree with experienced caseworkers, and why?

• How will human oversight work, and how will performance be monitored after launch?

If the only answer you get is an accuracy number, keep asking.

A field that’s growing up

The most encouraging thing about ISM this year was watching agencies move past AI as a novelty. They’re asking harder questions about performance, implementation, and oversight, and whether a tool actually fits the work.  

‍

About Jon: 

Jon Schuster leads product at Augintel, helping shape the technology and experiences that enable health and human services organizations to turn complex case records into actionable insights. He works at the intersection of product strategy, technology, and customer needs to build solutions that help agencies make better-informed decisions and better support the people and families they serve.

‍

‍