Evaluating a healthcare AI model before deployment means proving, with your own work and your own standards, that the model does the job it was bought for, fails in ways you can see, and never crosses the lines that matter: inventing a finding, exposing a patient, taking an irreversible action without a person. Vendor demos do not establish any of that. The checklist below does. It applies to clinical tools and to the billing, coding, authorization, and documentation AI that most practices will deploy first, and it is written so that a practice administrator, a compliance officer, or a health-system AI committee can run it without a data science team.
What the Rules Expect in 2026
No single regulation tells a practice how to evaluate an AI model, but several now set expectations that an evaluation record should meet.
| Source | What it says | What it means for your evaluation |
|---|---|---|
| ONC HTI-1 rule (certified health IT) | Since January 1, 2025, certified EHRs must expose “source attributes” for the decision support interventions their developers ship, including 31 attributes for predictive models: how the model was developed, validated, and monitored. | Ask your EHR vendor for the source attributes of any predictive tool it supplies, and hold third-party tools to the same disclosure even though the rule does not reach them. |
| FDA (AI-enabled device software) | AI functions that meet the device definition are regulated; the FDA maintains a public list of authorized AI-enabled devices and has published guidance on lifecycle management and predetermined change control plans. | Know whether the tool is a regulated device. Most billing, coding, and administrative AI is not, which means the evaluation burden is entirely yours. |
| NIST AI Risk Management Framework | A voluntary framework organized around four functions: govern, map, measure, manage, with a Generative AI Profile added in 2024. | Your pre-deployment evaluation is the “measure” function. Document it so it can be shown to a board or an auditor. |
| Joint Commission and CHAI guidance (September 2025) | Seven elements for responsible use of AI in healthcare organizations: governance, patient privacy and transparency, data security, quality and safety monitoring, safety-event reporting, bias assessment, and staff training; playbooks and a voluntary certification are to follow. | Keep the evaluation record in the shape of those elements. It will be what accreditation surveyors eventually ask for. |
| Payer-side AI laws | California’s SB 1120, in force since January 1, 2025, requires a licensed physician to make medical-necessity determinations and makes the algorithm open to audit; the 2026 CMS prior authorization rule requires a specific reason for every denial. | The standard you hold payers to is the standard your own AI should meet: a human decision, an auditable record, a stated reason. |
| HIPAA | Protected health information may be used and disclosed only as permitted; business associate agreements, the minimum-necessary standard, and the de-identification standards all apply to AI vendors. | Trace the PHI path before anything else. An accurate model with an undocumented data path is a breach waiting to be reported. |
The Ten-Point Checklist
Run it as a project with an owner and a written result for each point. A point without a document is not done.
1. Write down the job, not the model
- Name the task in operational terms. “Suggest ICD-10 and CPT codes from the completed note for review by the coder” is a job. “AI-powered coding” is not.
- List every decision the model will touch and mark which are irreversible: a submitted claim, a sent message, a changed order, an authorization request.
- State who acts on the output today and what they would need to see to trust it.
2. Decide what “correct” means and who says so
- Separate verifiable outcomes from judgment calls. Coverage, code validity, claim edits, arithmetic, and deadlines have right answers. Appeal quality and note quality need rubrics.
- Write the rubrics with the practitioners who do the work, and name the critical errors: an unsupported code, a fabricated finding, PHI to the wrong party, a missed deadline. A critical error fails the task regardless of the rest.
- Score critical errors separately from accuracy. A 95% accurate model that commits them is not deployable, and a blended score hides that.
3. Build the test set from your own work, and keep it private
- Use your payers, your specialties, your templates. A model’s published benchmark score says nothing about your Medicaid managed care plan or your orthopedic authorizations.
- Hold the set out. Never share it with the vendor, and never let it be used for tuning. If a vendor wants examples, give different ones.
- Include the ugly tail: secondary payers, retroactive eligibility, pended authorizations, notes with contradictions, and the denial reasons that actually hit your practice.
- Size it to the decision. A few hundred stratified cases will expose failure modes; a demo’s twenty cases will not.
4. Test the whole workflow, not the chat
- Evaluate the model with the tools it will use in production: the EHR, the portal, the clearinghouse, the eligibility lookup, by API, agent tool calls, or the screen itself.
- Look for the failure modes of multi-step work: loops, acting on the wrong patient, stale data, stopping early, declaring success without verifying the end state.
- Check the end state in the system of record, not in the model’s own summary of what it did.
5. Measure behavior under ambiguity and missing information
- Include cases where the right answer is “ask” or “stop.” A model that guesses confidently when the plan is unclear or the note is incomplete is the dangerous one.
- Record calibration: when the model expresses confidence, is it right at the rate it claims?
- Grade the reasoning on ambiguous cases, not just the final action. Two valid paths may exist; the question is whether the model saw both.
6. Follow the PHI
- Map every hop: from your system to the vendor, to any model provider, to any subprocessor, and back. Get it in writing with the business associate agreement.
- Confirm whether your data trains anyone’s model and whether it is retained, for how long, and where.
- Check the logs: prompts and outputs that contain PHI are PHI, and they need the same controls as the chart.
- Prefer architectures that keep PHI inside a perimeter, where the models run under the practice’s own credentials and nothing leaves for processing.
7. Require a human approval gate and a decision record
- No irreversible action without a named person approving it: claim submission, outbound messages, order changes, authorization requests.
- Log every suggestion with the model and version that produced it, what the person did with it, and when. That record is what an auditor, a payer, or a plaintiff’s attorney will ask for.
- Make the gate real. An approval that is one click on a pre-checked box is not a review; measure how long approvals take and how often suggestions are changed.
8. Run shadow mode before production
- Run the model beside the staff who do the work today, on live cases, without letting it act, for a defined period.
- Compare every output with the human decision, review every disagreement, and classify them: model wrong, human wrong, both defensible.
- Report the agreement rate, the critical errors, and the disagreement reasons before anyone decides to let the model act.
9. Set the monitoring plan before go-live
- Choose the metrics now: for billing AI, clean-claim rate, denial rate by payer, coding changes on review, and appeal success; for documentation AI, edit distance on review and addendum rates.
- Watch for drift when the vendor updates the model, when payer rules change, and when your templates change. Each is a re-evaluation trigger.
- Define the rollback: who can turn the model off, how fast, and what the manual process is.
10. Document it like you will have to defend it
- Keep an evaluation report with the task definition, the test set’s source and size, the scoring method and rubrics, results with critical errors broken out, the PHI path, the shadow-mode results, and the model and version with dates.
- Ask the vendor for its own evidence in the same shape, and for the HTI-1 source attributes if the tool ships inside a certified EHR.
- Version the report and re-issue it on every trigger from point nine. The record is the compliance program.
Three Places Evaluations Go Wrong
- Demo-grade test sets. Twenty clean cases chosen by the vendor prove nothing. The test set has to be yours, private, and ugly.
- Accuracy without critical-error analysis. The headline number hides the one failure that matters. Count the critical errors first and the accuracy second.
- One-time evaluation. Models change, payer rules change, templates change. An evaluation without a monitoring plan and re-run triggers is a snapshot of a system that no longer exists.
A Note on Billing AI Specifically
Billing, coding, and authorization AI is the easiest healthcare AI to evaluate well, because the answers exist. A claim pays or it does not. A code is supported by the note or it is not. An appeal is filed in the window or it is late. Insist on measurement against adjudication outcomes and coder adjudication, not just on agreement with whatever the vendor’s own grader says. That is the standard Canis AI is held to inside our own billing operation: every suggestion logged to a decision record, a human approving every submission, and the models trained and graded in environments built from real revenue-cycle work before they touch a chart. Our guide to healthcare AI training environments explains how that grading works and how to judge an environment’s quality.
Frequently Asked Questions
How do you evaluate an AI model in healthcare?
Define the job and the decisions the model will touch, decide what correct means and who judges it, build a private test set from your own workflows, run the whole tool-using workflow rather than a chat window, measure behavior under ambiguity and missing information, trace where protected health information goes, require a human approval gate with a decision record, run shadow mode beside staff, set the monitoring plan before go-live, and document all of it with versions and dates.
What is a critical error in healthcare AI evaluation?
An action that fails a task on its own regardless of everything else the model got right: a fabricated clinical finding, an unsupported code on a claim, protected health information disclosed to the wrong party, a missed filing deadline, or an irreversible action taken without the required approval. Critical errors are counted separately from accuracy because a 95% accurate model that commits them is not deployable.
What is shadow mode for healthcare AI?
Running the model on live work beside the staff who do it today, without letting it act. Its outputs are compared with the human decisions over a defined period, every disagreement is reviewed, and the agreement rate, the critical errors, and the reasons for disagreement are reported before anyone decides to let it act.
What evaluation evidence should a healthcare AI vendor provide?
A written description of the task the model was evaluated on, the test set’s source and size, the scoring method including rubrics and graders, results broken out by critical errors, performance on your payers or specialties if the vendor claims it, the PHI data path with subprocessors, the model and version evaluated with dates, and, for predictive decision support inside certified EHRs, the source attributes the ONC HTI-1 rule requires.
How often should a deployed healthcare AI model be re-evaluated?
Continuously on monitored metrics, and formally whenever the model version changes, the payer rules or clinical guidelines it depends on change, or performance drifts past a threshold you set before go-live. Quarterly is a reasonable floor for a formal re-run of the held-out test set.
Conclusion
A healthcare AI model earns deployment the way a new hire does: by doing your work, under observation, to your standard, with someone checking. Write the job down, decide what correct means, test on your own cases with the real tools, count the critical errors, follow the PHI, keep a person on every irreversible action, run it in the shadow of your staff, and keep the record. Everything on this list is a document you can produce when someone asks how you knew the model was safe to use.
For help evaluating a billing, coding, or documentation model against your own claims, contact us at management@cclbilling.com or call (845) 579-2737.
Sources: ONC, Decision Support Interventions Criterion Resource Guide (HTI-1); FDA, Artificial Intelligence-Enabled Medical Devices; NIST AI Risk Management Framework; Coalition for Health AI, Joint Commission and CHAI Release Initial Guidance (September 2025); National Law Review, California Limits Health Plan Use of AI in Utilization Management (SB 1120); HHS, Guidance on De-identification of Protected Health Information.