How We Keep Quality High When AI Builds at Speed
Speed in AI-augmented delivery comes from Claude writing and testing the code. Quality comes from layers, each designed to catch a different class of fault, most of them automatic. In a change we measured end to end, twelve layers each caught something different, from contradictory requirements before the build to a regression between two stories during QA.
The worry about AI-written code is rarely that it is badly written. It is that there is so much of it, produced so quickly, that nobody can read it all closely enough to trust it. Line-by-line review does not scale to the pace a model works at, so quality has to come from somewhere else.
We recently measured one change closely enough to show where it comes from in our delivery. Two complex, linked user stories on one of the booking platforms we build went from open questions to two merged pull requests in 15 hours 35 minutes. The full account is in Anatomy of an AI Change. This post follows the checks.
What are the layers?
Each layer is built to catch something the others might miss, and the cheap, fast checks run before the slow, thorough ones. This is what each one did in the change we measured:
| Layer | How it works | What it did in this change |
|---|---|---|
| Planning | The planning skill checks acceptance criteria against the story’s thread and the code, and raises questions rather than guessing | Found contradictory criteria, a security gap, and a migration risk almost a week before the build |
| Design | Rules are read before each layer is written, and affected paths are read end to end | Caught a display threshold that would have fed into whether an item could be booked |
| At the edit | A hook checks each code edit against the project’s non-negotiables | Ran on every code edit |
| Build gates | Release build, unit tests, type-checking and linting, acceptance criteria coverage, pattern check, and feature flag readiness | Every criterion covered by a named test; feature flags confirmed on test and staging |
| Independent review | A separate model reads the full diff against the rules | Found line endings changed in two files, which were fixed |
| Expert code review | An experienced engineer reviews every pull request, including its alignment with the architecture | Required before either pull request could merge |
| Guard tests | Around two dozen test classes that fail the build on a known rule breach | Proved the new write sits behind the scope guard |
| Depth of testing | Unit, database integration through the real HTTP stack, and browser tests | Around 50 new tests for the first story and a handful for the second |
| Regression | Broad database regressions run before merge, although CI only runs them after | Two runs of several hundred tests each, with no failures |
| QA | The QA skill runs the smoke tests alongside the story’s tests and maps results to acceptance criteria | Caught a regression between the two stories |
| After merge | Validation on the main branch | Caught a broken admin test fixture |
| People | Test case review, a manual session on the test environment, and sign-off | Test cases reviewed; the manual session still to come at the time of writing |
Across the session, the unit suite of about 2,400 tests ran twice with no failures, and the combined smoke and story tests, around 160 of them, passed in full. Our guide to AI test quality, mutation testing, and evals covers how we check that tests like these test the right things, rather than simply agreeing with the code.
What does QA catch that the build does not?
Interactions between changes. During QA on the second story, a smoke test from an earlier, unrelated story failed. It passed on its own, and a rerun on a used database added two unrelated failures, so the first theories did not hold. A fresh database with the test files run in a fixed order made it fail every time.
The cause sat between the two stories. The first had introduced a new kind of history entry, the second story’s test created one, and the older test did not recognise it. It took about ten minutes from the first failure to proof, and seven minutes to fix once the person driving the session had chosen to fix it rather than raise a bug.
The build gates had run the story’s own tests and the smoke tests, and neither combination exposed it. When a fault gets past a layer like this, we fix it and then change the layer that should have caught it. Several of the rules and guard tests in our harness started that way.
Which layers still need people?
Most of the layers are automatic, but the ones that decide whether the work is right are not:
- Planning raises questions, but a person with the authority to decide has to answer them.
- Code review of every pull request is done by an experienced engineer, who also checks that the change fits the architecture.
- Test case review needs someone who knows whether the cases test the right things.
- The manual session is where a QA analyst and a business analyst use the feature and judge whether it makes sense. No acceptance criterion fully describes that, so we would not try to automate it.
- Sign-off stays with people who have seen the feature working on the test environment.
We cover why these roles matter more with AI, not less, in why people still decide the outcome of AI delivery.
How can you check a supplier’s approach to quality?
You will not usually see a supplier’s pipeline, but you can ask for evidence that is easy to show if the layers exist:
- Traceability: ask to see how the acceptance criteria for a recent story are tied to named tests.
- What runs before merge: ask which checks fail the build, and which only report.
- What got through last: ask for a recent example of a fault that reached QA or production, and what they changed as a result.
- Sign-off: ask who signs a story off, and what they look at before they do.
A supplier with layered checks will answer these with examples. Our post on checking code against acceptance criteria in CI/CD with AI shows one of those layers in more detail.