Agentic intelligence
for healthcare education.

AI vs. Instructor Grade Agreement

Based on an analysis of the skills checkoff data, there is a strong baseline of agreement between the automated AI evaluations and your instructors, with adjustments occurring under predictable circumstances.

Below is the detailed breakdown of rater consistency, instructor adjustments, and rater variability based on 286 comparable steps where both an AI score and an instructor score were recorded across the institution's checkoffs.

Key Findings

1. High Overall Agreement

• Agreement Rate: 94.8% (271 out of 286 steps matched perfectly).
• Disagreement Rate: 5.2% (15 out of 286 steps were adjusted by instructors).

This indicates that the AI grading is highly consistent with instructor standards, serving as a reliable pre-grading baseline.

2. Systematic Upward Bias (No Downward Overturns)

• Overturned Upwards (AI 0 → Instructor 1): 100.0% of all disagreements (15 steps).
• Overturned Downwards (AI 1 → Instructor 0): 0.0% of disagreements (0 steps).

When instructors adjust an AI grade, they always grant credit where the AI was conservative. The AI acts as a conservative, strict evaluator, and instructors step in to correct false negatives or extend the benefit of the doubt to students. No instructor has had to downgrade an AI-passed step.

3. High Correlation with AI Confidence Flags

The AI generates a confidence rating (high or low) for its assessments. Instructors are significantly more likely to adjust grades when the AI itself flags a step as “low confidence”:

• High-Confidence AI Steps: 1.7% disagreement rate (only 4 adjustments out of 235 steps).
• Low-Confidence AI Steps: 21.6% disagreement rate (11 adjustments out of 51 steps).

This demonstrates that the AI's confidence flag is highly reliable. Instructors are 12.7 times more likely to adjust a grade when the AI flags a step as low confidence. This indicates that the confidence flag is a prime indicator for when manual review is actually necessary.

Standard I-A: Faculty and Staff Assessment of Program Outcomes

The program maintains an ongoing, systematic, and evidence-based process through which faculty and staff assess and evaluate achievement of identified program outcomes. This process is documented in the institution's CNEA Standard 1 Compliance Report, which details the framework, responsibilities, and continuous improvement cycle used during the 2024-2025 academic year.

Systematic Evaluation Plan (SEP)

The program utilizes a centralized SEP to track achievement of two categories of outcomes:

• Student Learning Outcomes (SLOs): Clinical competence, critical thinking, and professional ethics.
• Program Outcomes (POs): Graduation rates, NCLEX-RN pass rates, and employer satisfaction.

Key data points are collected on a defined schedule with assigned responsibility and specified evidence sources, as outlined below:

Evidence-Based Assessment Cycle

Faculty engage in a structured four-stage cycle to ensure continuous improvement:

• Data Collection: Quantitative scores are gathered from standardized testing (HESI/ATI), and qualitative feedback is obtained from the Program Advisory Board.
• Analysis: Faculty meetings are held every second Tuesday to review aggregate student performance against the 80% benchmark for all core competencies.
• Evaluation: Current data are compared against historical trends to identify curriculum gaps.
• Action/Change: The program implements revised teaching strategies or clinical rotations based on identified weaknesses.

Closing the Loop: Recent Program Improvements

The assessment process has directly informed curricular change. During the 2024 mid-year evaluation, data indicated a 12% drop in Pharmacological Math proficiency. In response, the program integrated mandatory dosage calculation modules into Level 1 and Level 2 clinical preparation. End-of-year assessments subsequently showed a 15% increase in math competency scores, restoring performance to the program benchmark.

Faculty and Staff Engagement

The systematic evaluation process involves both faculty and staff through the following structures:

• Annual Retreat: Dedicated to the comprehensive review of the SEP and CNEA standards.
• Curriculum Committee: Conducts monthly reviews of course-level outcomes to ensure alignment with overarching program goals.
• Staff Integration: Admissions and advising staff provide data on retention triggers, which are integrated into the systematic review of student success.

Through these mechanisms, faculty and staff collectively participate in an ongoing, evidence-based evaluation process that directly supports the achievement of identified program outcomes.