99.86% Agreement: Measuring AI Grading Against Clinical Educator Decisions
1 min read
Across 36,616 clinical skills checkoff evaluations, AI grading agreed with educator-reviewed outcomes 99.86% of the time, and flagged most of the rare disagreements as Low Confidence.
Clinical skills checkoffs only scale if automated grading holds up under educator scrutiny. We recently analyzed 36,616 clinical skills checkoff evaluations to measure how closely AI grading aligns with clinical educator decisions.
The results were striking.
What the data showed
- 99.86% overall agreement between AI-generated scores and educator-reviewed outcomes across more than 36,000 evaluations
- 99.95% agreement on evaluations classified as High Confidence by the HealthTasks.ai grading engine
- Only 51 evaluations showed meaningful divergence. Of those, 64.7% had already been flagged by the system as Low Confidence, accurately identifying cases that warranted educator review
In other words: when the model is confident, it almost always matches the educator. When it is not, the system usually says so before a grade is finalized.
Where disagreements occur
Most often, they involve nuanced situations where a student verbally explains a step rather than physically demonstrates it on video.
For example, a student may correctly describe priming IV tubing or explain APGAR scoring criteria, but not perform the action during the recorded assessment. In these cases, the AI applies a strict demonstration-based score, while educators may choose to award credit based on demonstrated understanding.
That gap is not a failure of the rubric so much as a gray area in how programs interpret evidence of competency. The important finding is that HealthTasks.ai recognizes these edge cases instead of forcing a silent pass or fail.
Why confidence scoring matters
By identifying lower-confidence evaluations before they are finalized, HealthTasks.ai enables institutions to focus educator attention only where human judgment adds value.
Faculty do not need to re-watch every submission to trust the process. They can spend review time on the small set of ambiguous performances, while the vast majority of clear demonstrations move through with consistent, audit-ready scoring.
What this means for programs
Educators can automate the vast majority of skills checkoff grading while maintaining oversight, consistency, and educational standards.
Automated first-pass grading is only useful if programs can defend it. This analysis shows that alignment with educator judgment is high at scale, and that the remaining disagreements concentrate in situations the system already flags for review.