A clinical AI feature competes with a simple alternative: the clinician ignores it and looks the information up themselves. That comparison, not model accuracy, decides whether the feature gets used. If checking the suggestion takes more attention than doing the task unaided, the suggestion is noise even when it is correct.
Call that verification cost. It covers everything between the output appearing and the clinician deciding whether to trust it: reading it, finding what it rests on, judging what is missing, and working out what acting would commit them to. How much of it the clinician pays is largely an interface decision. Not all of it. An unmapped lab code raises the cost whatever the screen looks like.
One thing to hold onto while reading the rest. The target is not zero.
What a clinician needs before acting
Four things, in this order. A complete statement rather than a forming one. The evidence it rests on, filtered to what this user is permitted to see. An honest account of what was missing or contradictory. And an accounting of what acting would commit them to, which decides both how much friction the accept control deserves and whether the suggestion earns an interruption at all.
The order is not arbitrary. A system that gets the fourth right and the first wrong has built a fast path to a recommendation nobody can check.
Complete output, not partial output
Consumer chat products stream tokens to hide latency. Clinical output inverts that. A line reading hold the anticoagulant before it completes into hold the anticoagulant only if platelets are below 50 has already been read by someone moving between rooms, and the correction arrives after the decision.
Hold the generation and commit it in one piece. If the wait needs feedback, show progress on retrieval instead: which records are being read, how many came back. This has a real cost. A retrieval and generation pass over a chart can run into double-digit seconds, and the clinician spends every one of them watching a panel do nothing.
Buffering buys one thing back. Before anything renders, you can compare numeric values in the response against the retrieved context. This is less straightforward than it sounds. Creatinine arrives as 1.2 mg/dL from one source and 106 µmol/L from another, weights come in kilograms and pounds, and a model writing 1.2 for a stored 1.24 is not hallucinating. Normalise units and tolerate rounding first. A check that fires on everything gets switched off in week two. It also breaks on values the model legitimately derived, such as creatinine clearance or body surface area, which need an allowlist of their own.
Evidence, filtered by permission
Every claim in the output should resolve to a source the clinician can open: this result, drawn at this time; this note, written by this person on this date. A footnote at the end of a response fails a specific way. It leaves the reader to map claims back to sources after reading, and the mapping is the expensive part.
Two things decide whether provenance is usable. List the individual records, not a count: "2 lab results" cannot be opened, a named result with its timestamp can. Then show what was not consulted alongside what was. The record a clinician would reach for next is the one whose absence changes the answer.
The permission model breaks in a less obvious place. SMART on FHIR scopes describe resource types and, since version 2, search parameters: patient/Observation.rs conveys read and search on observations, and a ?category= constraint narrows it further. None of that describes sensitivity. A scope granting access to DocumentReference says nothing about whether a note carries a restricted confidentiality label or sits behind break-the-glass.
The second gap is architectural and produces the same bug. Retrieval indexes get built once through Backend Services with system/ scopes, which see everything, then queried on behalf of users who cannot. Resolve every citation against the reading user's own token before rendering it. The alternative is a reference that opens into a 403.
Honest uncertainty, as explicit states
Abstention tends to get built as a single fallback: answer, or say nothing useful. Four states carry more, and each maps to a different user action.
Supported. The records needed were retrieved and the output follows from them.
Incomplete. Something material was missing. Name it. A renal dosing question with no recent creatinine is not a low-confidence answer, it is a different answer, and the gap is the useful output.
Conflicting. Two sources disagree. An allergy in the problem list contradicted by a recent note is a finding worth surfacing on its own. No summarising sentence should paper over it.
Unsupported. Retrieval returned nothing relevant. Say what was searched.
Two cases hide inside the incomplete state. A value can be genuinely absent, and FHIR has a vocabulary for that in dataAbsentReason, separating not-asked from masked from unknown. Whether a given feed populates it is worth establishing early, since where it does not, absence reaches you as an empty result set and nothing else. The second case is a value present while its code never mapped to LOINC or RxNorm. The number and its units survive intact. What you lose is the ability to identify the test reliably or compare it across sources. That is not missing data. It is data you cannot safely reason over, and a model handed it will reason anyway.
Splitting these apart changes the interface from a confidence percentage nobody calibrates into four states a clinician can act on differently.
What acting costs
Now the part that cuts against everything above. Lowering verification cost has a failure mode of its own. Automation bias is the tendency to accept a system's recommendation without the scrutiny the same claim would get from a colleague, and a well-designed interface makes it worse. Polish reads as authority. Every technique here buys trust, and trust spent on a wrong answer is the expensive kind.
So the goal was never minimum friction. Grade it by what acting costs. Informational summaries that reorganise what is already in the chart can be shown freely. Suggestions that change a plan need their evidence visible before the accept control is reachable. Anything that writes to the record or triggers an order needs a confirmation step restating what will change. Tie friction to consequence, not to model confidence. Confidence is the thing you are least able to calibrate. This is where interface design for enterprise AI parts company with consumer product work, where the reflex is to remove steps wherever possible.
If the suggestion surfaces inside the EHR rather than beside it, CDS Hooks already models this. A card's indicator is a required field carrying info, warning or critical. Use it. No host application understands a severity scheme you invented.
Interruption and alert hierarchy
Consequence decides the other half of the fourth requirement: not just how hard the accept control should be to reach, but whether the suggestion should arrive uninvited at all. Clinicians dismiss interruptions quickly, and the base rate of useful ones is low enough to justify it. A 2006 JAMIA review of seventeen studies put drug safety alert override rates between 49 and 96 percent. AI suggestions dropped into that stream inherit the reflex, and inherit it fast.
Treat interruption as a budget. Decide which conditions earn one, route the rest to a surface the user visits by choice, and keep the threshold configurable per deployment, since a surgical floor and an outpatient clinic do not share a tolerance. Per-user budgets are harder than they look where workstations are shared and cover arrangements are common. The identity in the token is frequently not the person at the keyboard.
One event record for audit and evaluation
None of the four requirements above is verifiable in a design review. One event row per suggestion is what makes them measurable after release.
Write what was shown, which records grounded it, which model and prompt version produced it, the state it was shown in, who saw it, and what happened next. Compliance needs one half of that row and evaluation the other. Build them separately and you get two logs that each answer half a question.
The row reaches across the data layer, the permission model and the interface, which puts it with the clinical software engineering rather than with the model work, and makes it expensive to retrofit once all three are in production.
Be careful using dismissals as training signal. When override rates sit where the literature puts them, most dismissals are a reflex and not a judgment, and training on them teaches the model to match interface fatigue. Weight overrides by whether the user engaged first: opened the evidence, expanded a source, edited the text. CDS Hooks 2.0 added overrideReasons as a structured place to record why a card was rejected. It is optional, it is easy to leave empty, and an override log without it cannot be read back later.
Where this leaves you
None of this needs a better model. It needs the interface to answer, unprompted, the questions a clinician would otherwise go and answer alone: is this complete, what is it based on, what is missing, and what does acting commit me to.
The goal is not the lowest possible verification cost. A system where every suggestion was free to accept would be the most dangerous thing you could ship. The goal is to spend it where it changes an outcome, near zero for a summary that reorganises the chart, deliberate and unskippable for anything that writes an order. The four questions are what let you tell those apart before the clinician has to.
Comments
Loading comments…