How We Keep Quality High When AI Builds at Speed
Claude writes and tests code quickly, so quality depends on knowing which checks a machine can settle and which need a person. In a change we measured, automated gates proved exactly what each test asserts. An engineer, a QA analyst, and a business analyst (BA) judged the design, the usability, and the intent. The requirements stayed in Azure DevOps, where the tooling can read them but not change them, and our MCP server tied every test and result back to them.
In our experience, the worry about AI-written code is volume: a model produces plausible code faster than anyone can read it closely. Line-by-line review does not scale to that pace, so quality has to come from somewhere else.
We measured one change closely enough to show where it comes from in our delivery. Two complex, linked user stories on one of the booking platforms we build went from open questions to two merged pull requests in 15 hours 35 minutes. The full account is in Anatomy of an AI Change. This post follows the checks, and the people behind them.
Twelve layers checked that change. Most of them are automatic, and they are fast and tireless at proving what they were built to prove. The rest belong to three people: the engineer who drives the session, the QA analyst who owns how the work is tested, and the BA who owns what the work is meant to do. In our experience, quality depends on being clear about which questions each layer can answer, and which it cannot.
Who owns each of the twelve layers?
Each layer is built to catch something the others might miss, and the cheap, fast checks run before the slow, thorough ones. Figure 1 shows who does what at each stage.
This is what each layer did in the change we measured:
| Layer | Who | How it works | What it did in this change |
|---|---|---|---|
| Planning | Claude, then the BA and product owner | Checks acceptance criteria against the story’s comments and the code, and asks rather than guesses | Found contradictory criteria, a security gap, and a migration risk almost a week before the build |
| Design | Claude | Reads the project’s rules before each layer is written, and traces affected paths end to end | Caught a display threshold that would have fed into whether an item could be booked |
| At the edit | Automatic | A hook checks each code edit against the project’s patterns | Ran on every code edit |
| Build gates | Automatic | Release build, unit tests, type-checking and linting, acceptance criteria coverage, pattern check, and feature flag readiness | Every criterion covered by a named test; feature flags confirmed on test and staging |
| Independent review | A separate model | Reviews the full diff against the project’s pattern rules | Found line endings changed in two files, which were fixed |
| Guard tests | Automatic | Around two dozen test classes that fail the build when a known rule is broken | Proved the new write sits behind the scope guard |
| Depth of testing | Claude, directed by the engineer | Unit, database integration through the real HTTP stack, and browser tests | Around 50 new tests for the first story and a handful for the second |
| Regression | Automatic | Broad database regressions run before merge, although CI only runs them after | Two runs of several hundred tests each, with no failures |
| QA run | Claude | Runs the story’s tests and the smoke tests, and maps each result to an acceptance criterion | Caught a regression between the two stories |
| Expert code review | The engineer | An experienced engineer reviews every pull request, including its fit with the architecture | Required before either pull request could merge |
| After merge | Automatic | Validation on the main branch | Caught a broken admin test fixture |
| Manual session and sign-off | The QA analyst and the BA | The tester’s pass on the test environment, and a judgement on whether the feature makes sense | Draft test cases reviewed in the session; the manual session still to come at the time of writing |
Across the session, the unit suite of about 2,400 tests ran twice with no failures. The combined smoke and story tests, around 160 of them, passed in full. Our guide to AI test quality, mutation testing, and evals covers how we check that tests like these test the right things, rather than simply agreeing with the code.
Where can automated signals be trusted?
Every automated check proves something precise:
- A build proves the code compiles and fits its types.
- The acceptance criteria coverage gate proves every acceptance criterion (AC) has at least one test tagged with the story and criterion number. CI fails the pull request if one does not.
- A guard test proves a known rule has not been broken. On this project, one fails the build if runtime code runs SQL text against a table instead of going through a stored procedure. Its list of exceptions can only shrink.
- A regression run proves that what used to pass still passes.
We trust those signals for exactly what they assert, and the harness is written to say so. Its QA skill states that ‘a green targeted selection proves the ACs it asserts, not that the feature is usable’. The fix skill notes that a green regression test ‘proves the specific assertion it makes, not that the surface looks right’. Even the coverage gate has a limit, which the build skill spells out: ‘a tag match is not coverage’.
| Signal | What it proves | What it cannot tell you |
|---|---|---|
| Build and type checks | The code compiles and fits its types | Whether it does the right thing |
| Coverage gate | Every acceptance criterion has a tagged test | Whether that test asserts the scenario that matters |
| Guard tests | A known rule has not been broken | Whether a new kind of mistake has been made |
| Automated tests | The behaviour each test asserts holds, and keeps holding | Whether the feature is usable, clear, or what the business meant |
| Independent review | The diff follows the project’s pattern rules | Whether the design is right for the system |
| QA mapping | Which criteria passed, failed, or have no coverage | Whether the criteria themselves are right |
Why does knowing the boundary matter?
Being precise about this changes how people spend their time. If a green pipeline is treated as sign-off, nobody checks what it cannot see. If it is not trusted at all, testers repeat by hand what the machine has already proved, and run out of attention before they reach the cases that need them.
We have seen the second happen on this project. For one acceptance criterion, a tester worked through four manual cases by hand. A browser test, a unit test of the business rule, and a database integration test already asserted three of them, but the list gave the tester no way to know. The harness now records which manual cases an automated test proves, so the tester’s attention goes where the automation is weakest.
Where is human analysis critical?
Some questions no automated check can answer, however many tests run. In our delivery they belong to three people, and each owns a different kind of judgement. We cover why these roles matter more with AI in why people still decide the outcome of AI delivery.
The business analyst owns what the work is meant to do
The BA, working with the client’s product owner, owns each story’s description and acceptance criteria. Planning reads every story against its comments, its attachments, and the existing code, then raises what it finds as questions. It separates decisions for us from questions for the client, and a person decides which questions go to the client. In the change we measured, three such questions changed what was built.
The BA’s judgement returns at the end, in the manual session. There, the BA and the QA analyst use the feature and ask whether it does what the business needed. The client then runs user acceptance testing (UAT). Small changes the client asks for in UAT come back through analysis, and the criteria are only rewritten once the change is approved.
The engineer owns the design and the code
The engineer drives the session and answers Claude’s questions as they arise. The build skill’s rule is to pause on ambiguity and ask, rather than guess. Before handing over to QA, the engineer walks through the story’s acceptance criteria on their own machine, so the automated run is not the first time anyone has seen the feature work.
Every pull request then gets an expert code review. The engineer checks the change against the system’s architecture and asks whether the design is right, which no pattern rule can settle. GitHub’s branch rules enforce it: a pull request cannot merge until it has an approving review and the required automated checks have passed. CodeQL also scans the codebase for security issues on a daily schedule.
The automated checks run before the review, so the reviewer can spend their attention on design and intent rather than on formatting or missing tests.
The QA analyst owns how the work is tested
Our QA analyst designed the approach to test cases that the harness follows, and wrote the specification for the skill that drafts them. They sign off every draft manual test case before it joins the story’s test suite, and the skill will not add one without that approval. They also contribute browser tests alongside the engineers.
The QA analyst runs the tester’s pass on the test environment, which is the only step that signs a case off. They also run exploratory testing. A skill prepares short, time-boxed charters aimed at the riskiest recent changes, so the tester spends the session looking for problems nobody has written a case for. As the skill puts it, ‘scripted testing answers the questions somebody already thought of’. An exploratory finding becomes a bug, a new case, or a task, and never a verdict on an acceptance criterion.
How do we decide which tests the engineer writes and which the QA analyst designs?
Every acceptance criterion gets both kinds of test, and each kind has a different job.
The engineer’s tests prove the code
The automated tests are the engineer’s own proof that the code does what each criterion says. Claude writes them during the build, under the engineer’s direction:
- At least one browser test for every acceptance criterion. This is the preferred form on this project, because it exercises the whole stack.
- A database integration test for every stored procedure that is added or changed.
- Unit tests only where there is real logic that benefits from testing in isolation.
Every test carries the story and criterion number, and that tag is what the coverage gate reads.
The QA analyst’s cases challenge it
The manual test cases are designed to challenge the work. Following the QA analyst’s approach, each criterion is split into separate cases for the happy path, the negative path, and the boundaries. A two-sided threshold gets a case just inside it and a case just outside it. The negative and boundary cases deliberately cover ground the automated tests do not.
The cases are drafted at the end of the build rather than at planning. By then the criteria have been through the planning questions and any ambiguities the build paused on. The cases therefore describe what was actually built, which is what the tester will run. Our post on generating test cases from acceptance criteria with AI covers the drafting in more detail.
Which manual cases does an automated test already prove?
The engineer makes that call, once, at the end of the build. They have just written both the test and the case, so they are best placed to judge whether the test asserts the same scenario. Where it does, the case is bound to the test in Azure DevOps by name.
The rule is to bind on the scenario, not on the criterion. Every criterion has a tagged test, but that test may cover the happy path and nothing else. The build skill is blunt about the risk: ‘An over-eager binding is worse than none, because it retires a manual check that was doing real work.’
What always stays with the tester?
Three kinds of check go to the tester whatever the automated tests say:
- Subjective judgement, such as whether something is clearly labelled or easy to find.
- A real external system, such as an email arriving or a payment sandbox.
- A real device, where the criterion is about a physical interaction rather than a screen size.
Presentation is checked separately, by an automated sweep of the admin screens that reports issues without blocking the work. Figure 2 shows how a manual case finds its place.
What does the manual session look like?
Once a pull request has merged, a skill prepares the tester’s session. It first checks that the change has reached the test environment, that its feature flag is on there, and that the data each check needs exists. It then sorts every manual case into one of four lanes:
| Lane | What it means | Signs the case off? |
|---|---|---|
| Covered | A named automated test asserts this case’s scenario and passed on the merge commit | No. It proves the code, not the deployment |
| Pre-verified | Claude has driven the scenario on a local copy of the system | No. It finds defects early but proves nothing about the test environment |
| Tester | The tester’s own pass on the test environment | Yes, and it is the only lane that does |
| Blocked | The case cannot run yet, because a dependency, the data, or the environment is missing | No, and it is never counted as a pass |
Every case ends in the tester’s lane. The other lanes decide how much attention it needs there. A case proven by named tests and pre-verified locally is a glance, while a case with neither is a careful read. The skill’s own summary is that ‘the value here is the triage, not the clicking’.
The tester approves the plan, or moves cases between lanes, before anything is recorded, and their outcomes are never assumed. The BA joins the QA analyst for the session, and together they judge whether the feature makes sense to the people who will use it. No acceptance criterion fully describes that, so we would not try to automate it.
Both stories in the measured change were merged but not yet signed off when we wrote this up. A change is not done until people have used it on the test environment.
Why do the requirements live in a separate system?
In our delivery, the stories and their acceptance criteria live in the work-item system, which on this project is Azure DevOps. The code repository holds a snapshot of the criteria for the engineer to work from, and Azure DevOps stays the source of truth.
The delivery skills that plan, build, and test the work never write a story’s description or acceptance criteria, not even to append to them. If one finds a criterion that is wrong, untestable, or ambiguous, it raises that as a finding for the BA and product owner. The harness documentation explains why: ‘acceptance criteria that the delivery tooling is free to rewrite stop being an independent check on delivery.’
That independence matters more with AI. A model that can edit the criteria, the code, and the tests can make all three agree, and then every check passes. Keeping the criteria in a system the tooling can read but not change means the work is always measured against something it did not write. The QA skill calls the criteria ‘the only independent check in the workflow’.
A separate system also suits the people who own the requirements. BAs, delivery managers, and the client work in Azure DevOps rather than in the repository. Its history keeps every earlier version of a story, and its comments hold the conversation that shaped it. We tried keeping specifications as files in the repository and moved away from it, for reasons we set out in why we’ve moved away from OpenSpec.
How does the MCP server tie it together?
Claude reaches Azure DevOps through our own Model Context Protocol (MCP) server, which gives an AI model a set of tools it can call. Its tools use delivery language, such as work items, test suites, and test cases, rather than raw API calls. The server applies our conventions, so a skill states what it wants to record and the tool records it in the right shape. We explained why we built it that way in running Claude Code against Azure DevOps.
Every skill in the harness reads and writes the delivery record through that server. Writes that shape the record, such as tasks, test cases, and bugs, wait for a person to approve them. Across one story, the server ties the work together like this:
- Planning reads the story, its acceptance criteria, every comment and attachment, and its parent feature. Once a person approves the plan, it creates the engineering, QA, and human tasks under the story.
- The build records what changed on the story, along with the model used, the tokens consumed, and the AI cost.
- Test case drafting creates each manual case in the story’s test suite. Each case is linked to the story and to the criterion it tests, and carries a verbatim copy of that criterion, so a later change to the criterion shows.
- Binding records, on each manual case, the name of the automated test that proves it.
- The QA run posts a result for every acceptance criterion on the story. A story cannot move to testing until its manual cases exist.
- The manual session records the tester’s outcome for every case on the work item.
With those links in place, anyone can start from an acceptance criterion and follow it to the tests that prove it, the manual cases that challenge it, and the results recorded against it. Coverage becomes a property of the work item, checked as the work is done, rather than a report someone assembles afterwards.
What does QA catch that the build does not?
Interactions between changes. During QA on the second story, a smoke test from an earlier, unrelated story failed. It passed on its own, and a rerun on a used database added two unrelated failures, so the first theories did not hold. A fresh database with the test files run in a fixed order made it fail every time.
The cause sat between the two stories. The first had introduced a new kind of history entry, the second story’s test created one, and the older test did not recognise it. It took about ten minutes from the first failure to proof. The fix took seven minutes once the engineer had chosen to fix it rather than raise a bug.
The build gates had run the story’s own tests and the smoke tests, and neither combination exposed it. When a fault gets past a layer like this, we fix it and then change the layer that should have caught it. Several of the rules and guard tests in our harness started that way.
How can you check a supplier’s approach to quality?
You will not usually see a supplier’s pipeline, but you can ask for evidence that is easy to show if the layers exist:
- Traceability: ask to see how the acceptance criteria for a recent story are tied to named tests and manual cases.
- Who can change the requirements: ask whether the tools that build and test the work can edit the acceptance criteria.
- Automated or manual: ask how they decide which checks stay manual, and how a tester knows which cases automation already proves.
- What runs before merge: ask which checks fail the build, which only report, and whether a merge needs an approving review.
- What got through last: ask for a recent fault that reached QA or production, and what they changed as a result.
- Sign-off: ask who signs a story off, and what they look at before they do.
A supplier with layered checks will answer these with examples. Our post on checking code against acceptance criteria in CI/CD with AI shows one of those layers in more detail.