You’ve got the right spine already — Michael’s exception review, with confidence scoring and the deterministic regex/parser checks — so the real question isn’t whether to flag a doc, it’s how to flag few enough that a human isn’t back to checking all of them. That’s exactly where a single confidence score quietly fails: LLM confidence is badly calibrated. A model will report 0.95 on a PO code it hallucinated, so “review the low-confidence ones” still leaves you eyeballing almost everything — the exact 100% you’re trying to get out of. A few mechanisms that actually cut the load, given your confirmed scope (po_code, supplier, date — no line items)
1. Run those regex checks per field, not per document. Michael’s deterministic checks do more work applied field by field: po_code against your PO pattern, supplier against a known-vendor list, date parsed into a real date, each with its own pass/fail. A doc marked “85% confident” tells your reviewer nothing; a doc where po_code and supplier pass but the date didn’t parse tells them the one cell to fix. Field-level status is what turns a read into a glance.
2. Verify against the PDF page, not the extracted text. This is the one aimed straight at your pain. On non-standard layouts, a lot of misreads start in the text-extraction layer, not in the model’s reading — so a checker that only sees the extracted text inherits the same bad text and confidently agrees with it. Add a second pass that sends the PDF page image plus the proposed JSON to a vision model — ideally a different model family than the extractor — with one job: “read this page, read this JSON, list only the fields that disagree.” A model checking its own output tends to rubber-stamp it; a different one actually catches the misread. A human then only touches docs where the verifier flags a disagreement or a hard check failed — that’s what moves you toward ~10% reviewed instead of 100%.
3. Anchor every value to the source — and treat a miss as a signal. This extends Michael’s parser check with one twist: require each extracted value to appear as an exact substring of the document text. The catch, given #2: you’re matching against the extracted text, so a value the bad text layer mangled or dropped will fail this check — which is fine. A miss isn’t a bug to smooth over; it’s a second reason to route the doc to the verifier. A po_code that isn’t verbatim on the page is either hallucinated or lost in extraction, and both belong in the review queue.
4. Give it a memory of past corrections. This is the part that makes it improve instead of staying flat. Every time a human fixes a field, store the corrected example in a Make Data Store keyed by sender domain. On the next PO from that domain, retrieve the last few corrected examples and drop them into the extractor’s prompt. Recurring suppliers — most of your volume — converge on correct within a handful of documents and fall out of the review queue; new or rare senders still route to a human. That’s the difference between a one-off scenario and something that gets more accurate the longer it runs.
5. Measure per supplier. Keep 10–20 hand-verified docs per top sender as a small eval set and re-run it whenever you change the prompt or swap models. Review load almost always concentrates in two or three messy senders — tracking accuracy per supplier shows you which ones, so you spend effort on the right vendor instead of guessing.
The closest thing I’ve shipped to this is a US healthcare intake flow — field-level validation on every extracted value, live under compliance sign-off — built specifically to get humans down to the exceptions and keep them there. If the verifier pass or the supplier-memory loop is worth digging into, my notes on both are at priyanshukumar.co. Even if you build it yourself, the field-level checks plus the vision verifier are the two that’ll actually move your review number.
What is your goal?
What is the problem & what have you tried?