AI Test Quality: Mutation Testing and Evals

This guide goes deep on one specific sensor from our maintainability sensors guide: the one that checks whether your tests actually test anything. It matters more in the age of coding agents, because agents are very good at producing tests that look thorough and are not.

Why does test coverage lie about AI-written tests?

Coverage tells you a line of code executed during a test. It does not tell you that any assertion checked what that line did. The two are easy to confuse, and the gap between them is where false confidence lives.

AI coding agents make the gap worse. Asked to add tests, an agent will happily produce a suite that exercises every branch and asserts almost nothing meaningful, or asserts something trivially true. The coverage number climbs, the suite goes green, and a reviewer glancing at the percentage concludes the code is well tested. It is not.

This is not a hypothetical. Birgitta Böckeler’s 27 May 2026 article on maintainability sensors describes finding high-coverage mappers whose assertions did not actually verify the mapping. The coverage was real. The testing was not.

What is mutation testing, and how does it catch the gap?

Mutation testing measures whether your tests detect changes to the code, which is the thing coverage cannot see. It works by deliberately introducing small faults and checking whether the suite notices.

The mechanism is simple:

• Make a small change to the code under test, for example flip a > to a >=, or delete a statement.
• Rerun the affected tests.
• If a test fails, it “killed” the mutant. Good: a test was actually checking that behaviour.
• If no test fails, the mutant “survived”. That code ran during testing but nothing asserted against it.

The output is a list of surviving mutants, and that list is precise and actionable. Each survivor names a specific behaviour your suite claims to cover but does not. Unlike a coverage percentage, it points at exactly what to fix.

How do you run mutation testing on .NET without it being slow?

Run it incrementally and place it in the post-integration tier, not on every commit. Mutation testing is more expensive than ordinary test runs because it reruns tests many times, so the cost has to be managed rather than ignored.

On .NET we use Stryker.NET. The practical configuration:

• Incremental runs so only changed code is mutated, which keeps each run proportionate to the size of the change.
• Post-integration or scheduled cadence, alongside other expensive sensors, rather than in the tight commit loop.
• A target on the surviving-mutant list, not on a single headline score, so the team fixes real gaps instead of gaming a number.

JavaScript and TypeScript teams have an equivalent in StrykerJS, so the same discipline carries to the React side of a stack. The sequencing principle from the harness engineering guide applies: cheap checks every commit, expensive checks on a cadence.

What about correctness beyond unit tests?

Some correctness questions cannot be expressed as a deterministic assertion at all. For these, a semantic eval adds a probabilistic, often model-based check. The Generative Programmer’s 29 May 2026 article on the missing quality layer names this as one of the controls teams most often lack.

Semantic evals are useful in three places in particular:

• Generated tests, to judge whether the assertions are meaningful, not just present.
• Policy and business logic, where correctness depends on intent that is awkward to encode in fixtures.
• Tool-using workflows, where the agent chains calls and the right answer is a behaviour, not a value.

Frameworks such as DeepEval provide structure for writing these evaluations. Research techniques like semantic entropy, which flags unreliable model output by measuring disagreement across sampled responses, point at where a generated answer is least trustworthy. The detail matters less than the role: an eval is a judgement-based check for correctness that deterministic tests cannot reach.

When is a semantic eval worth the cost?

Reach for a semantic eval only when a deterministic check cannot answer the question, and always run it downstream of the cheap checks. Evals are slower, cost tokens, and are non-deterministic, so they earn their place only where judgement is genuinely required.

The heuristic is the same one that governs inferential sensors generally: a control that produces the same answer twice should be a computational check instead. If a semantic eval keeps returning a stable verdict, promote it to a deterministic assertion and save the cost. Keep evals for the questions that really do need interpretation.

How do you combine coverage, mutation testing, and evals into a quality gate?

Layer the three by cost and certainty, so the cheapest, most certain checks gate first and the expensive, probabilistic ones run last.

A workable gate for a .NET and React system:

• Every commit: the test suite plus coverage, as a fast baseline. Coverage stays useful as a floor, just not as proof of quality.
• Post-integration: Stryker.NET mutation testing, incremental, to expose weak assertions.
• Scheduled or targeted: semantic evals on generated tests, policy logic, and high-risk workflows, grounded in the deterministic results above.

The combination is what matters. Coverage shows reach, mutation testing shows whether the reach is real, and evals cover the correctness that neither can express. No single layer is sufficient on its own.

Why should a CTO care about AI test quality?

Because a high-coverage AI-generated test suite can quietly overstate how well a system is tested, and that is a delivery and assurance risk. Defects ship behind a reassuring number, and the evidence trail says quality is higher than it is.

This is sharper now that AI authors most of the code. Our Q1 2026 AI Velocity Report records 84% of code as AI-authored. At that ratio, taking test suites at face value is a real exposure. Mutation testing and targeted evals turn test quality from an assumption into a measurement, which is what both code review and regulatory assurance depend on. We make the wider case for not trusting unsupervised AI output in our blog post on why we do not let AI ship code unsupervised.

How does Talk Think Do test AI-written code?

We treat test quality as something to measure, not assume, and the sensor layer reflects that:

• Stryker.NET mutation testing runs as a post-integration sensor, so weak assertions surface before release.
• Coverage remains a baseline floor, never the headline quality signal.
• Every AI-authored change passes senior engineer review and ISTQB-qualified QA validation, inside our ISO 27001-certified framework.

The result is that high AI authorship and high test quality coexist, because the tests themselves are tested. For the full sensor picture this fits into, read the maintainability sensors guide and the harness engineering guide.

To review the test quality of an AI-augmented codebase, book a free consultation.