98% code coverage sounds bulletproof. It is not.
A developer recently documented what happens when you stress-test AI-generated unit tests using mutation testing, an older software quality technique where you deliberately inject small bugs into your code and check whether your test suite catches them. Think of it as a quality audit for your tests themselves: if a test still passes after you break the code it is supposed to protect, that test is not doing its job.
The results were telling. Starting from a codebase with 98% line coverage (meaning the tests executed nearly every line of code), mutation testing revealed that 10 out of 40 intentionally introduced bugs went completely undetected. That is a 25% miss rate hiding behind a near-perfect coverage number.
The core problem is what the author calls "tautological" tests. AI coding assistants tend to generate tests that look correct, hit the right lines, and pass reliably, but do not actually verify meaningful behavior. A test might call a function and check that it returns something without confirming it returns the right thing. Coverage metrics cannot distinguish between a test that validates logic and one that just exercises code paths.
The Process That Worked
The approach used a multi-stage workflow. First, the AI wrote tests using test-driven development practices. Then, in a completely separate AI session (to prevent the model from "remembering" its own test logic and gaming the results), mutation testing was run against the codebase. The separation matters: if the same session writes tests and evaluates mutations, it can unconsciously bias toward catching its own patterns.
After identifying the 10 uncaught mutations, the developer fed those specific gaps back to the AI to generate targeted fixes, then re-ran the mutation suite to confirm improvement.
A Practical Takeaway for AI-Assisted Development
The argument here is not that AI-generated tests are bad. It is that coverage percentages alone are a terrible proxy for test quality, and this has always been true, even with human-written tests. Mutation testing has existed for decades but was historically expensive because humans had to review each mutation manually. AI makes the economics work: generating, running, and analyzing dozens of mutations costs minutes instead of hours.
For developers using tools like Cursor, Claude Code, or GitHub Copilot to generate tests, adding a mutation testing pass is a cheap way to catch the gaps that coverage reports hide. Tools like mutmut (Python), Stryker (JavaScript/TypeScript), and pitest (Java) can run mutation suites against any existing test suite with minimal setup.
The 98%-coverage-but-25%-miss-rate finding is a useful number to keep in your head the next time an AI assistant tells you your tests all pass.