Handling AI extraction errors for purchase order codes from emails and attachments with human validation

:bullseye: What is your goal?

Automatically extract purchase order codes from incoming emails and their PDF attachments using an AI agent in Make, and log them into Excel while ensuring total accuracy via human review.

:thinking: What is the problem & what have you tried?

The AI agent sometimes misreads data or makes extraction errors on complex or varying purchase order documents (e.g., inaccurate order code outputs).

Since 100% automated reliability is impossible with LLMs, I am looking for the best architecture in Make to handle this:

How to manage human validation efficiently (e.g., flagging rows as “Pending” in Excel while keeping PDFs in Outlook).

How to best structure the workflow so humans can review and correct errors before finalized data impacts downstream operations.

Hey there,

are the email you receive standardized? Are they automated on the other end by any chance? Or maybe they come from forms being filled?

The best way is to get them standardized if they aren’t already, then you can find the codes in a predetermined place without the need to search for it.

Otherwise, yeah, flagging the entry as Pending and providing a direct link to the attachment/email in the excel sheet so a human can review it will be the best way. But then just have the human do it directly? If they need to verify everything anyways…

Thank you for your insights! To answer your questions:

1. About standardization: Unfortunately, we deal with dozens of different external suppliers who all use their own unique invoice/PO layouts. We don’t have control over how they generate or send their purchase orders.

2. About the “Why use AI if a human has to check anyway?” point: That’s a great question. Even with human verification, the AI still saves a massive amount of time. Instead of an operator having to manually open the email, read the PDF, type the supplier name, copy-paste the PO code, and log the date into Excel, the AI does 90% of the heavy lifting. The human only has to spend 2 seconds glancing at the row to approve or correct a rare typo. It shifts the job from ‘data entry’ to ‘data validation’, which is much faster.

Ok but does supplier A always send the same emails?
First module gets the email, then an if/else module sends them to the specific path based on supplier. Then a deterministic flow finds the number in the corresponding place.

I would treat this as an exception review workflow, not as full automation.

A practical Make structure would be:

  1. Outlook trigger for new email or attachment
  2. Save the PDF to a stable folder location first
  3. Extract text from the PDF
  4. Ask the AI for a structured result with fields like po_code, supplier, confidence, evidence_text, and reason
  5. Write every result to Excel with status = Pending
  6. Include a direct link to the saved PDF and the original email if possible
  7. Route only high confidence, pattern matched results to Ready for review
  8. Route low confidence, blank, conflicting, or oddly formatted results to Needs review
  9. Only after a human changes status to Approved should any downstream process use the PO code

I would also add deterministic checks after the AI step. For example, if PO codes follow a known pattern, use a regex or text parser to validate the AI result. If the AI says the PO is ABC123 but the evidence text does not contain ABC123, mark it as Needs review automatically.

The Excel table can be simple:

received_at
sender
subject
pdf_link
extracted_po_code
confidence
evidence_text
status
reviewed_by
reviewed_at
final_po_code
notes

That gives the reviewer enough context without opening every file, but still keeps the PDF one click away. The important part is that the AI never writes final operational data directly. It proposes a value, explains where it found it, and Make holds the record until a person approves it.

You’ve got the right spine already — Michael’s exception review, with confidence scoring and the deterministic regex/parser checks — so the real question isn’t whether to flag a doc, it’s how to flag few enough that a human isn’t back to checking all of them. That’s exactly where a single confidence score quietly fails: LLM confidence is badly calibrated. A model will report 0.95 on a PO code it hallucinated, so “review the low-confidence ones” still leaves you eyeballing almost everything — the exact 100% you’re trying to get out of. A few mechanisms that actually cut the load, given your confirmed scope (po_code, supplier, date — no line items):slight_smile:
1. Run those regex checks per field, not per document. Michael’s deterministic checks do more work applied field by field: po_code against your PO pattern, supplier against a known-vendor list, date parsed into a real date, each with its own pass/fail. A doc marked “85% confident” tells your reviewer nothing; a doc where po_code and supplier pass but the date didn’t parse tells them the one cell to fix. Field-level status is what turns a read into a glance.

2. Verify against the PDF page, not the extracted text. This is the one aimed straight at your pain. On non-standard layouts, a lot of misreads start in the text-extraction layer, not in the model’s reading — so a checker that only sees the extracted text inherits the same bad text and confidently agrees with it. Add a second pass that sends the PDF page image plus the proposed JSON to a vision model — ideally a different model family than the extractor — with one job: “read this page, read this JSON, list only the fields that disagree.” A model checking its own output tends to rubber-stamp it; a different one actually catches the misread. A human then only touches docs where the verifier flags a disagreement or a hard check failed — that’s what moves you toward ~10% reviewed instead of 100%.

3. Anchor every value to the source — and treat a miss as a signal. This extends Michael’s parser check with one twist: require each extracted value to appear as an exact substring of the document text. The catch, given #2: you’re matching against the extracted text, so a value the bad text layer mangled or dropped will fail this check — which is fine. A miss isn’t a bug to smooth over; it’s a second reason to route the doc to the verifier. A po_code that isn’t verbatim on the page is either hallucinated or lost in extraction, and both belong in the review queue.

4. Give it a memory of past corrections. This is the part that makes it improve instead of staying flat. Every time a human fixes a field, store the corrected example in a Make Data Store keyed by sender domain. On the next PO from that domain, retrieve the last few corrected examples and drop them into the extractor’s prompt. Recurring suppliers — most of your volume — converge on correct within a handful of documents and fall out of the review queue; new or rare senders still route to a human. That’s the difference between a one-off scenario and something that gets more accurate the longer it runs.

5. Measure per supplier. Keep 10–20 hand-verified docs per top sender as a small eval set and re-run it whenever you change the prompt or swap models. Review load almost always concentrates in two or three messy senders — tracking accuracy per supplier shows you which ones, so you spend effort on the right vendor instead of guessing.

The closest thing I’ve shipped to this is a US healthcare intake flow — field-level validation on every extracted value, live under compliance sign-off — built specifically to get humans down to the exceptions and keep them there. If the verifier pass or the supplier-memory loop is worth digging into, my notes on both are at [redacted] Even if you build it yourself, the field-level checks plus the vision verifier are the two that’ll actually move your review number.

Hey @Priyanshu_Kumar

Thanks for your contributions in the community. It’s great to see you engaged and answering questions. However, it seems that your responses might have been AI-generated.

If that applies, please bear in mind that posting such responses is against the community guidelines, as they are often incorrect and provide inaccurate information.

Please also refrain from (self)promotion.

Hey Guillaume,

the “2 second check” is the part that usually breaks, not the AI. if it lives in an Excel column called Pending, someone has to remember to open the sheet. then it’s not 2 seconds anymore.

you can do it in Make alone. scenario A pulls the code, emails the reviewer two links (approve / reject), each one hits a webhook in scenario B. works. a few things bit me though:

  • Outlook opens links in the background to scan them. plain webhook link = approved by nobody
  • nobody clicks, row just sits there. no deadline
  • two clicks, B runs twice, PO is in Excel twice
  • a shared link has no idea who clicked

I’m on the free plan so no human in the loop app, and after enough of that I wrote a small open source thing that only does the approval bit. reviewer gets an email, opening it does nothing, pressing the button is the decision, first press wins, there’s a deadline, Make gets back who said yes and when. one catch: approve / reject + a comment, no box to type the fixed code. so wrong code = reject, right code in the comment.

rough shape:
A: Outlook → your AI step → HTTP POST (code, pdf name, reviewer email) → done
B: webhook → router → approved: mark row Confirmed / rejected: write the comment value, ping you

happy to build the A/B blueprints for your case. three quick things first:

  1. how many POs a day, roughly?
  2. does the reviewer live in Make/Excel or just email on their phone?
  3. wrong code: fix it right there, or flag it and fix later?

that last one changes the flow the most.