A handwriting pipeline that returns clean JSON on every page probably isn't working. It's failing silently, and silent failure tends to be the expensive kind. The models sitting inside most document workflows today will look at a smudged date, fail to read it, and hand back something anyway. A benchmark published in April 2026 put seventeen frontier and open-source models against forty-nine real, redacted maternity records and recorded the behaviour without much drama: when a field can't be interpreted, the model doesn't flag uncertainty. It returns a blank, or the closest plausible value, or something it made up. Three outcomes, one shared property. Downstream, all three look identical to data.
That is the part worth sitting with, and accuracy scores don't capture it. Validation passes. The pipeline reports success. Whatever number ends up on your dashboard describes the average page, and no page in your batch is the average page.
Reading the Fifteen Percent
The strongest models in that benchmark reached roughly 85% median accuracy across those forms. Eighty-five percent reads like a passing grade right up until you ask which fifteen. Free-text fields ran a word error rate of 0.50 at best, meaning about half the words in a handwritten note came back wrong or missing. Hallucination rates landed between 6 and 23 percent depending on the model. None of that is visible in the output, because the output is a well-formed object with every key present and every value the right type.
There's a second thing buried in those numbers that deserves more airtime. The same benchmark reported strong weighted scores and weak macro scores across nearly every model, which is a technical way of saying the systems did well on the common values and badly on the rare ones. In a clinical registry the rare values are the entire point. Nobody builds a pregnancy exposure registry to count the uncomplicated deliveries. If your evaluation is dominated by empty fields and routine answers, it will report health while the pipeline mangles precisely the records you built it to find. Anyone running an extraction eval should probably look at the per-class breakdown before the headline figure, because the headline figure is usually the null class wearing a disguise.
Vendors understand this better than their marketing usually lets on. Apryse, whose engineering write-up on handwritten data capture spends more of its length on recognition mechanics and correction workflows than on accuracy claims, sits closer to the real shape of the job than much of the category does.
Confidence Belongs to Fields, Not Documents
So build for doubt rather than for accuracy. Every field needs its own confidence value. A document-level average is close to useless in practice, because it smears a 0.97 on a printed patient name across a 0.44 on a handwritten dosage and reports the result as fine. Google's documentation for its custom extractors describes exactly this granularity, scoring each extracted entity between zero and one so that manual review can be triggered on individual entities rather than whole documents, with the illustrative case being whether a string reads "Hello, world!" or something that merely looks like it.
A caution before you lean on those numbers too hard. Confidence scores are themselves model outputs, and their reliability varies. A calibration benchmark released in August 2026 built 1,346 degraded document variants and ran more than seventy thousand entity-level evaluations, and its authors report that calibration quality ranges from near-perfect to severely overconfident depending on which model you picked. Two systems can both hand you a 0.8 and mean entirely different things by it. Their finding that per-model post-hoc correction rescales those values without disturbing the ranking is worth knowing, because it means a miscalibrated model can still be useful for routing even when its raw numbers can't be trusted as probabilities.
Practical version: run a calibration check on your own corpus before you trust any threshold, plotting observed accuracy against reported confidence in ten buckets. If the 0.9 bucket is only 70% correct, your threshold is a decoration. And whatever you decide, store the confidence value in the warehouse next to the value it describes. A column of dosages with no sibling column of confidences throws away the only evidence you had about which ones to doubt later.
Who Actually Owns the Threshold
Once you have per-field scores, somebody has to pick a cutoff, and this is where teams reach for a tuning exercise when they should be reaching for a meeting. Google's guidance for its own document splitters is unusually blunt about it: the threshold should be derived from historical error rates at given confidence levels, and it calls the choice a business decision about tolerance for errors rather than a modelling one. Correct, and slightly uncomfortable. The person who owns that number arguably should be whoever absorbs the cost of a wrong value, and that person is rarely anywhere near the repository.
There's arithmetic attached to that decision, and it's usually the one nobody has run. If a pipeline handles two thousand forms a week and the threshold sends 18% to review, that's three hundred and sixty manual checks, which is somebody's whole job. Move the threshold to catch more errors and the queue doubles. Move it the other way and the errors go into the warehouse instead. Whoever signs off on that trade is making a claim about how much a wrong record costs, whether or not they realise they are making it. Write the number down. A threshold nobody can defend in a sentence is one that will drift the first time the backlog gets embarrassing.
One habit worth building early: log every field that fell below threshold, including the ones you auto-accepted anyway during a busy period. Six months later that log is the only thing that lets you back-test a different cutoff against real outcomes instead of arguing from intuition.
Routing, Fallbacks, and the Silent Empty String
The architecture that follows from all this is unglamorous. A watcher, a router, a fallback path, a queue. One widely-read walkthrough of building an automated PDF extraction pipeline in Python routes each incoming file by checking whether the embedded text layer returns anything substantial and sending everything else to OCR. That decision costs about six lines. Getting it wrong costs you every scanned page in the batch without a single exception being raised, because a PDF with no text layer politely returns an empty string.
Two things worth doing here that rarely make the tutorial. Keep the original page image and the raw model response alongside the parsed object, because reprocessing a corpus with a better model in eighteen months is trivial if you kept them and archaeology if you didn't. And treat every schema change as a data migration with a version stamp on each row, since a field that meant one thing in March and another in September will silently poison any longitudinal query that spans both.
The Review Queue Has a Ceiling
Now a partial retraction. Everything above argues that the escalation path matters more than the recognition accuracy, and that is mostly true and occasionally very wrong. Put a review queue behind a model that reads dosages at 60% and you haven't built a safety net, you've built a job no human can actually do. Reviewers rubber-stamp when the queue is long. They rubber-stamp faster when the model is usually right, because usually-right is exactly the condition that teaches you to stop looking.
This isn't a hunch about lazy staff. A systematic review in the Journal of the American Medical Informatics Association screened nearly fourteen thousand papers and analysed seventy-four studies on automation bias, the tendency to over-rely on automated output, and found workload, task complexity and time constraint among the environmental factors that reliably make it worse. Those three describe most document-processing operations at month-end. The same review points at mitigations that translate directly: emphasising reviewer accountability, attaching updated confidence levels to the output the reviewer sees, and presenting information rather than a recommendation. A queue that shows the cropped image region next to the proposed value, with the confidence displayed, is a meaningfully different instrument from one that shows a filled form and an Approve button.
Accuracy is what buys you a review queue a person can actually clear at the end of a shift.
What the Archivists Worked Out First
There's a version of this problem that predates every model in that benchmark. The Library of Congress runs a crowdsourced transcription programme where volunteers key handwritten pages from the collections, and its guidance for reviewing a completed transcription is stricter than what most production pipelines enforce. Read the whole thing line by line against the image. Preserve the line breaks so the next person can. And if you submitted the transcription, you aren't allowed to review it.
The programme's own data paper goes further, describing a consensus model in which at least two volunteers must agree before a page counts as complete, with staff specialists spot-checking afterwards. The quality analyses in that paper are instructive in a way accuracy leaderboards aren't. A close review of one Civil War diary logged 703 character-level errors across roughly 152,000 characters, most of them trivial expansions and typos. The largest single error was a page left untranscribed and marked complete. Even a two-person consensus system with expert oversight produced one silent empty field, which should tell you something about how hard this failure mode is to design out.
The detail worth stealing, though, is elsewhere in that analysis: parts of some pages were judged so damaged or ambiguous that no definitive reading could be given by volunteers or specialists, and those were excluded from the error rate rather than guessed at. Ground truth for handwriting is not a fixed thing sitting behind the image waiting to be recovered. Two competent people reading the same nineteenth-century scrawl will sometimes disagree, and the archival answer to that is a documented process and an honest "illegible", not a better reader. Pipelines that treat extraction as a lookup with a guaranteed correct answer are importing an assumption that the people who have done this longest abandoned decades ago.
The Schema Is the Business Logic
Meanwhile the schema is doing more work than the model, and gets a fraction of the attention. In that maternity benchmark, constraining the output space at the schema level, handing the model an enumerated list of permitted values instead of describing the rules in prose, lifted macro precision and recall by more than 60%. Writing the same rules as narrative instruction moved those metrics by two to five points. The lesson seems fairly clear. Whether you define the shape with Pydantic models or with plain JSON Schema, that definition is where your business rules actually live. A field typed as a free string is a decision to accept anything at all. Most teams make that decision by accident, in a hurry, on a Tuesday.
Two refinements that pay for themselves. Give every enumerated field an explicit "unreadable" member rather than letting the model fall back to null, because null already means "the box was empty" and collapsing those two states destroys the distinction you most need. And where a field has a real-world constraint, encode it as a validator rather than a comment: a gestational age of 62 weeks should fail loudly at parse time, not sit in a table until someone runs a distribution plot next quarter.
Shorthand, Dialects, and Local Convention
None of this is what the demo shows. Demos show one clean form and one tidy object. The actual work of turning documents into structured rows and columns is header detection, column alignment heuristics, fuzzy matching on field names that drift between sources, and a great deal of code that exists because somebody wrote "NAD" in a box that expected a tick.
The maternity benchmark catalogued exactly this. A degree symbol pressed into service to mean "no". Bare digits standing in for categories the form never defined anywhere. Dates written vertically down a margin because there was no room to write them across. One condition column had staff writing the word "No" into boxes the schema expected to be left blank, so an absence of disease arrived as a string where the parser wanted nothing at all. Clinical staff invented their own encoding under time pressure, each site drifting into its own dialect.
Which suggests a step most teams skip: before writing a line of extraction code, sit with fifty real pages and a domain expert and write down the local conventions. That document is worth more than the model choice. Any pipeline that meets those pages without it is not extracting data so much as transcribing a language it hasn't been taught.
Ship the Doubt
If you take one thing into your next sprint, make it this: add a confidence column beside every extracted value, and give your schema a way to say "unreadable". Both are an afternoon's work. Together they convert a system that cannot fail into one that can, which is the only kind worth trusting in production.
Then go and ask the question that rarely comes up at kickoff. When your pipeline meets a page like the ones above, what do you want it to do? Not what should it do in principle, in the design document, in the architecture diagram someone drew in the second week. What does the code you have already written do, today, at three in the morning, on the batch nobody is watching, when the API times out on the retry and the fallback path fires?
Go and find out. Pull twenty low-confidence records from last month and check them by hand against the source images. Most teams who run that exercise discover something they'd rather not have known.
Because the honest answer for most systems is that they will fill the field. The object will validate. The row will land in the warehouse, and it will sit there looking exactly like the rows that are right, with no visual difference, no flag, no annotation, nothing to distinguish a value that was read from a value that was guessed. Somewhere tonight a number that nobody ever actually read is being averaged into a report, and the report will be correct to two decimal places.
Comments
Loading comments…