Hello!
This is exactly the right direction, so it’s good to hear it.
Where I’m at right now:
It is a decision engine, not a search flow. I moved away from Pinecone/semantic search for the final product decision. I extracted the tech sheets into a structured spec file (product codes, size options, coverage rules, accessories) that’s separate from the rules that decide which product to pick. So updating a product spec never touches the decision logic.
Every decision now returns a rule trace, not just the answer. For each item needed, the output includes the selected product plus a step-by-step log of why: which rule fired, what assumptions were made, what was inferred vs. given directly by the customer.
There’s a “needs review” flag. It fires whenever an input is missing, an assumption had to be made, or two options were a close call. Those get flagged for a manual check instead of the system just guessing.
I ran the “previous quotes” test you suggested — checked it against 3 real past quotes so far, product-line by product-line against what was actually delivered. Two are at 100% match; the third is 8/10 lines correct (the 2 gaps turned out to be a data version mismatch on my side, not a rule error).
Here’s an anonymized example of exactly the kind of break you’re describing:
Customer input: one room, ~37 m² gross area, one of my install types (the one where the product goes fully embedded rather than surface-mounted), one finish type.
Expected output (from the real past quote): 1x large-capacity unit of Product Type A.
Initial wrong output: 2x smaller units of Product Type A (system split it into two).
Root cause, found via the trace: my rule was sizing off the wrong area basis (a “usable” area field that was empty and defaulted) at a density target that didn’t match how this product is actually sized in practice. The correct rule sizes off the gross area directly, at a lower density band, and prefers one slightly-under-density unit over splitting into two when it’s close. Once I fixed the area basis and the density target, it matched the real quote exactly.
That trace made it obvious in under a minute this wasn’t a product-data problem or an extraction problem — it was two rule parameters that didn’t match real practice.
Next step is wiring this into Make — there’s a native Code App module that runs Python directly in a scenario now, so no external API/server needed. But open to your suggestions. The review flag will route anything uncertain to a manual check before the quote is finalized.