We test this product harder than most people would bother testing a tool they’re not selling. Not because we enjoy finding our own bugs, but because “the AI actually knows your business” is a claim that’s worthless if nobody’s checked whether it’s true under real pressure.
One question kept failing. Not almost-right, not vague, a flat “I don’t have that.”
The question, and how often it failed
“Propose a hybrid service package, grounded in what has actually worked.”
We asked some version of this across three separate AlphaForge simulation runs, a full Law Firm vertical test, and a Landscaping vertical test. Every single time, on the old retrieval method: some version of “I don’t have Vault entries or records showing which combinations have worked.”
The frustrating part wasn’t that it failed. It’s that the information genuinely wasn’t missing. The Vault had the general pricing entry. It had the specific outcome entry. It just never connected them, because the retrieval method underneath was doing live keyword matching on every single message, up to forty separate database scans per question, and keyword matching doesn’t know that “sod pallets” and “pallet coverage” are talking about the same thing.
What we actually changed
Not what the Vault stores. How it gets searched.
Every Vault entry now gets embedded, converted into a numeric representation of its actual meaning, at the moment it’s written, not at the moment someone asks a question about it. Retrieval became a single nearest-neighbour lookup against that, instead of scanning the whole Vault from scratch on every message. It’s the same underlying database, Postgres, running the same instance we already had, using an extension built for exactly this. No new infrastructure category. A different way of finding what was already there.
Getting there took real, unglamorous engineering most of these posts don’t dwell on: confirming which AI providers even offer an embeddings endpoint (not all of them do), a tenant-aware design so each company’s own configured provider gets used rather than one platform-wide key, and a genuinely picky bug where one provider’s embeddings needed explicit normalisation before distance comparisons would even be mathematically valid. We also benchmarked five model candidates on a real question before picking one, and found something worth knowing on its own: the fastest option beat the slowest by ten to fifteen times, with zero difference in answer quality on the exact numeric question we tested it against.
The re-test
Same tenant. Same question. Old method, then new method, back to back.
The old answer: “I don’t have Vault entries or records showing which combinations have worked.”
The new answer, confirmed via logs to have genuinely used the new retrieval path, not a coincidence: a real, specific proposal, named services, a real client referenced by name, a computed total value, reasoning tied to how that company’s own work actually flows. Not a subtly better version of the old non-answer. A completely different outcome, on a question that had failed identically, every time, for the entire testing program up to that point.
What we’re not going to pretend
The honest cost showed up immediately, so we’re reporting it, not smoothing it over. Backfilling embeddings for an already-large Vault, over a thousand entries, took the better part of five minutes, uncomfortably close to a hard request-timeout limit. At a genuinely mature Vault, several thousand entries deep, that same one-shot backfill would likely fail partway through. We know that now because we measured it, not because we’re guessing. It’s a real, flagged follow-up, not something we shipped and quietly hoped nobody would hit.
The number underneath all of this
This single before-and-after is one test inside a much larger program: roughly twenty-five simulated years of business activity across several industries, over nine thousand real Vault entries generated along the way, eleven real bugs found and fixed, this retrieval gap among them. Total real API spend for the entire multi-month program: $9.57.
We’d rather show you the question that failed five times in a row, and exactly what it took to fix it, than lead with a headline number and hope nobody asks what’s actually behind it.