A health AI passed the US Medical Licensing Exam with a perfect score. If you work with language models, your first reaction should not be amazement. It should be a question: what does that number actually tell us about whether the thing is safe to put in front of a sick person at 2am?
I have spent enough time around model evaluation to be suspicious of clean benchmark numbers, and a 100% on a professional exam is about as clean as they come. It is also, on its own, close to meaningless as a measure of clinical usefulness. That gap between benchmark and reality is the whole interesting problem, and it is worth walking through, because it applies to far more than medicine.
Why exams are the easy part for a language model
Medical licensing exams are multiple choice. They are written, standardized, and drawn from a finite, well-documented body of knowledge that appears all over the training data. That is the exact shape of the task a large language model is built to crush.
Think about what the USMLE actually measures for a human. It is a proxy. We cannot directly test whether someone will be a good doctor, so we test whether they can recall and apply medical knowledge under exam conditions, and we treat that as correlated with competence. For a person, that correlation is reasonable, because a human who memorized the material also went through years of supervised clinical training that the exam does not measure but assumes.
A model has no such hidden training. It has exam-shaped knowledge and nothing underneath it. So when it scores 100%, it has demonstrated exactly one thing: it is extremely good at the part of medicine that looks like a test. The parts that do not look like a test, which is most of them, are untouched by that number.
The benchmark-reality gap, stated plainly
Here is the failure mode that should worry anyone deploying these systems. A model can ace the exam and still be dangerous in production, because the exam does not test the things that break in production.
Consider what an exam question gives you. A clean, complete vignette: a 34-year-old presents with these five symptoms, these lab values, this history. Pick the diagnosis. Real patients do not arrive as clean vignettes. They give you three symptoms, forget the fourth, describe the fifth wrong, and leave out the medication that changes everything. The skill of extracting a usable picture from a messy, incomplete, sometimes contradictory human account is the actual job, and no multiple-choice exam tests it.
Then there is the calibration problem, which is the one I find genuinely hard. A good clinician knows when they do not know, and escalates. A language model, by default, does not have reliable access to its own uncertainty. It can be confidently, fluently wrong in exactly the register that sounds most trustworthy. On an exam, confidence costs nothing. In a clinical setting, a confident wrong answer is the whole risk.
What a benchmark score would need to be paired with
None of this means the exam result is worthless. It means it is one data point that needs company. If I were evaluating a clinical model for real use, the exam score is where I would start, not where I would stop. The questions I would actually want answered:
How does it perform on incomplete or contradictory inputs, not clean vignettes?
Is it calibrated, and does it escalate or refuse when the case exceeds its competence?
What is its behavior on the long tail of rare presentations, where the training data thins out?
Does it hold up on adversarial or emotionally loaded phrasing, the way real patients talk?
Crucially, what does it do at the boundary between safe-to-handle and needs-a-human?
That last one is where clinical AI lives or dies. The value is not in answering the hard question. It is in correctly recognizing which questions it should not answer at all, and handing those off. A model that scores 100% on knowledge and 100% on knowing its limits would be interesting. A model that scores 100% on knowledge alone is a very well-read intern with no supervision, which is not a comforting thing to give autonomy to.
Where this is actually being deployed
The reason this is not just an abstract eval discussion is that these systems are already in front of patients, and the sensible deployments have understood the point above. They use the model for what it is good at, and they wrap it in structure for everything else.
A concrete example is the health assistant behind August, which reports a 100% USMLE result but does not present that as a licence to replace a doctor. The interesting engineering is in what surrounds the model: it is scoped to common, low-risk complaints, it is built to recognize red-flag symptoms and push those toward in-person care, and it operates alongside licensed clinicians rather than instead of them.
You can see the same philosophy in how the product is bounded. Its online urgent care service handles a defined list of routine conditions, the kind where a clear description and a photo genuinely are enough to act, and it routes anything outside that list to a human. That scoping is not a limitation bolted on for legal reasons. It is the actual safety architecture, and it matters far more than the headline score.
The lesson generalizes past medicine
If you build with LLMs in any high-stakes domain, the pattern here is the one to internalize. Impressive benchmark performance tells you a model has capability. It tells you almost nothing about whether a system built around that model is safe, because safety lives in the scoping, the calibration, the escalation paths, and the human oversight, none of which the benchmark measures.
The benchmark answers it. The system design answers it, and when. Those are different questions, and conflating them is how you ship something that demos beautifully and fails quietly. A 100% score is a fine reason to keep investigating. It is a terrible reason to stop.
So what should you take from a perfect score?
Treat it the way you would treat any model facing any evaluation: as evidence of raw capability and a prompt to ask harder questions, not as a finish line. The exam is the part that was always going to be easy for a system like this. The hard part, the messy-input, know-your-limits, escalate-correctly part, is the part that does not fit on a scorecard, and it is the only part that determines whether the thing is actually good.
The next time you see a clinical AI announce a perfect board score, do not ask how it did it. Ask what it does when it does not know. That answer, not the benchmark, is the one worth paying attention to.
Comments
Loading comments…