Faculty Agreement with Vision AI Skills Checkoffs

Time savings answer whether AI skills checkoffs are operationally worth it. They do not answer whether the grades hold up when faculty can review the same items.

We added that analysis to the Vision AI Skills Checkoffs ROI case study with South Florida College of Nursing. Through October 13, 2026, faculty-reviewed item scores show Cohen's κ = 0.99 in production and Cohen's κ = 0.91 where educators scored. We are also adding a built-in Cohen's kappa calculator to HealthTasks so programs can measure inter-rater reliability on their own checkoff data.

What we measured

The unit of analysis is each rubric item on a completed submission. Two submissions still processing were excluded. No production data was changed. Items are graded 0 (unsatisfactory) or 1 (satisfactory).

Educators can intervene on every checkoff. When they leave the AI score in place, that is a real production decision, not a gap in the study. When they score, that is a second rater on the same item. Both views matter, so we report both.

Both kappas

• 3,330 completed submissions (January 25–October 13, 2026)
• 45,597 rubric-item grades (1,041 distinct items across 76 templates)
• 99.75% operational agreement, Cohen's κ = 0.99 (faculty could review every item)
• 99.09% dual-rated agreement, Cohen's κ = 0.91 (12,770 educator-scored items)
• 936 submissions with any educator item score
• 45 submissions with any item overturn

κ = 0.99 is how the workflow runs: AI first pass, faculty able to change any grade. κ = 0.91 is how the two raters compare when faculty do score. Both sit in the almost-perfect range on a 0/1 scale (Landis and Koch).

Educator scoring is all-or-nothing in practice: 935 of 936 reviewed submissions have an educator score on every item.

Divergence of interpretation

Every dual-rated disagreement went the same direction. Educators upgraded an AI unsatisfactory mark to satisfactory on 116 items. They never downgraded an AI satisfactory mark.

Among dual-rated items where AI marked unsatisfactory, educators upgraded 116 of 733 (15.8%). In the dual-rated set, faculty never marked a step unsatisfactory after the AI marked it satisfactory.

That is a divergence of interpretation, not a split vote. Faculty and Vision AI differ on whether a step was shown, usually on documentation, reporting, verbalization, or holistic criteria. When they differ, faculty award credit the AI did not. They never take credit away from an AI satisfactory.

Overturns are also rare and clustered. 51 of 76 templates have 100% operational agreement. When any flip happens, it concentrates: 45 submissions, mean 2.58 items flipped, maximum 14 in one submission. 97 of 1,041 rubric items have at least one disagreement; 944 have none. Those items are the abstract steps that are easy to miss on video, not the core psychomotor sequence.

Kappa in the platform

Faculty, simulation directors, and accreditation reviewers ask whether two raters agree beyond chance on the same skill. Spreadsheets should not be the only way to answer that.

We are adding a built-in Cohen's kappa calculator to HealthTasks. Programs will be able to measure inter-rater reliability on educator-versus-AI item scores inside the skills checkoff workflow they already use.

Read the full case study

The ROI snapshot from the first 60 days is unchanged: 1,403 graded checkoffs, more than 200 faculty hours saved. The new section is the reliability finding at production volume.

Read the updated Vision AI Skills Checkoffs ROI case study, or book a demo to see skills checkoffs and the kappa calculator in the platform.