Every migration failure we’ve ever seen in other software comes from the same place: treating “clean the data first” as the customer’s problem instead of the product’s. We didn’t want to build that. So early on, we hit a real one ourselves, and it taught us something we didn’t expect: sometimes the right answer to “should the AI handle this” is no.
The migration that taught us the difference
We had a real legacy CRM export to bring in: 349 companies, 491 notes, most of it call-log history written in Slovenian, 393 contacts. The obvious move would have been running it through Neo, our AI-assisted import, the same one that reads a messy spreadsheet and proposes clean records from it.
We decided not to. The source data was already structured, already resolved, already correct. Running it through an AI pass wouldn’t have added anything except risk: a model paraphrasing or summarising real Slovenian call notes, quietly losing precision in text that needed to stay exactly as written. So we built a second, separate, deliberately AI-free path: exact, deterministic copying, matched by legacy record ID, preserving original timestamps, no language model anywhere in it.
Building that path surfaced real gaps we wouldn’t have found otherwise: a form field that had been silently dropped by the backend for who knows how long because the database column behind it never existed, contacts that had been getting jammed into free-text notes for lack of a proper place to live. Fixed both, for good, not just for this one migration.
That’s the actual shape of the Migration Engine now: Neo for the honest mess, throw us your worst spreadsheet and we’ll tell you plainly what we could and couldn’t make sense of, and a quiet, exact-copy path for the data that’s already trustworthy and just needs to arrive intact. Two different problems. We stopped pretending they were one.
Getting to something worth testing at all
None of the testing below matters if there’s nothing real underneath it. Honestly, part of how we got here started by accident. Screen recordings kept getting put off, mostly plain reluctance to sit down and record something properly, so the actual workaround was just watching the AI click through the app live and taking notes on what happened. That turned out to be one of the best debugging methods we stumbled into the whole project, not a clever plan, a byproduct of not wanting to do something else. A few real features exist today because a bug got caught exactly that way.
The bigger build followed a similar shape, less by grand design than by necessity: one real piece at a time, each one tested end to end before the next one started, because a project this size doesn’t stay sane any other way. What actually surprised us is that the baseline test suite ended up predicting things nobody asked it to predict. Add a new feature months later, and Velvet would sometimes catch that some adjacent piece had been sitting half-finished the whole time, quietly, waiting to be noticed. Not because anyone involved is especially sharp. A test that actually clicks the button and checks the real result just has a way of finding things a person skimming code wouldn’t.
Velvet
Somewhere along the way, our own end-to-end testing suite got a name instead of just being “the tests.” We call her Velvet. Roughly five thousand lines of Playwright, across twelve named suites, smoke, regression, the Vault, the AI Village, action-bus flows, hardcore edge cases, stress, among others, run against five separate synthetic seeded companies. She drives the real, running application, backend and frontend both live, not a mocked stand-in for either.
Inside the named regression suites, AI calls are mocked to deterministic responses, on purpose, not as a shortcut. At real volume, genuine AI calls hit browser connection limits and produce failures that have nothing to do with whether the product actually works. Mocking there isolates one honest question, does the plumbing work, from a different one, is the AI’s actual answer any good, which gets tested separately, for real.
The discipline underneath every layer
Five rules apply everywhere testing happens in this project, not just in Velvet:
Lock a prediction in writing before running the test, so a pass can’t quietly get redefined afterward as whatever happened to occur. A test that only confirms a button exists isn’t a test; it has to click the button, wait for the real response, including real AI latency, and check the actual resulting state. Every feature gets tried doing what it’s supposed to do, and tried the way a real, sometimes careless, sometimes adversarial person would actually use it wrong. When a “failure” turns out to be a bug in the test script rather than the product, that gets reported too, not quietly patched and forgotten. And a fix isn’t done when the code compiles; it’s done when the exact scenario that broke has been re-run for real and passed.
Beyond Velvet: seven layers, not one
Velvet is one layer of seven. Real, unmocked AI verification runs separately, a one-off script against the live app with genuine model calls, then cleaned up afterward, specifically to catch the class of bug that only shows up because a real model’s answer varies run to run. Long-horizon simulation runs simulated business histories, a month, a year, five years, against real tenants with live AI calls throughout, asking the same canonical questions at multiple points to check whether answers actually got sharper as history accumulated, not just whether they came back at all. A deliberate adversarial layer, eight hostile tests we call the Obstacle Course, tries to break the product’s honesty and safety claims directly: prompt injection, cross-tenant data leakage, conflicting instructions, deletion resilience, real financial reconciliation, duplicate-account handling. Backend unit and integration tests cover the Sacred Spine and document ingestion end to end. Guided walkthroughs test the actual UI at real screen sizes, desktop and a genuine mobile viewport, with screenshots as evidence, not a checklist tick. Even the presentational trailer videos doubled as a testing layer: each one is a real script hitting the real live app, and every moment that looked slightly off during review got individually traced to either a real bug or a script-timing issue, never assumed to be one without checking.
Across all of it, roughly twenty-five simulated years of business activity, several industries, over nine thousand real Vault entries generated along the way, eleven real bugs found and fixed. Total real API spend for the entire program: $9.57.
What we’re not going to claim
We haven’t tested enterprise scale, thousands of concurrent tenants at once. We haven’t stress-tested what happens to AI cost under genuinely heavy, open-ended daily chat, as opposed to the scripted business activity most of this testing was built around. During the work above, we found a real scaling risk in our own embeddings backfill, a large Vault’s one-time catch-up getting uncomfortably close to a request timeout, and we’re telling you that instead of quietly patching it and saying nothing.
Nothing here is perfect, because nothing is. What we can actually stand behind is that we looked, hard, at our own claims before making them to you, and we’re telling you what we found either way. That’s the whole discipline. We’ll keep doing it.