Agents Can't Test the Software They Can Write

Give a frontier coding agent a detailed specification of a CLI application and it builds an implementation from scratch that passes up to 93% of the original tests. Give it the same specification and ask for a test suite, and even the best agent's tests catch just over half of the bugs.

That is the central finding of TestSuiteBench, our new benchmark for generating complete test suites from natural-language specifications. It covers 165 open-source command-line applications, each with a detailed specification, a reference implementation, a reference test suite, and on average 74 injected bugs that each violate exactly one requirement. Across six frontier agents, the generated tests are largely valid, but the strongest suites catch 55.5% of the bugs and four of the six agents stay below 40%.

‍

‍

Why test suites from specifications

Agents are increasingly capable of building software with little supervision. This makes building trust in the implementation the key limitation. When an agent re-implements existing software, the original test suite can do that. For new software, the only statement of what was wanted is the specification, so the agent has to turn it into tests itself.

Existing benchmarks don't measure this. Code-generation benchmarks have scaled to full applications, while test-generation benchmarks, including our previous SWT-Bench, are still scoped to a single issue, file or scenario, and give the agent the implementation to test against.

How we built TestSuiteBench

Measuring how strong a test suite is takes two ingredients: a specification precise enough that "correct" is well defined, and bugs that each violate specific requirements, so that catching a bug shows that the suite checks that requirement.

Starting point. We build on ProgramBench: 200 open-source command-line tools in Rust, Go and C/C++, each with a reference executable and an extensive test suite of pure input-output checks. After removing tests that need network access or excessive resources, or that are flaky, 165 applications remain.

Specifications from tests, not docs. Developer documentation turned out to be far too thin to test against (more on that below). Instead, an agent writes a specification that describes every behaviour the reference test suite checks. The median specification has about 5,600 words.

Mutants that each break one requirement. We link every specification section to the tests that assert it, and select target tests so that every section is covered evenly (solved as a min-cost flow problem). For each target test, an agent modifies the reference implementation to break its logic so that it violates the corresponding requirement and fails that test. We then refine the candidates to isolate the violated behaviour: 73% of resulting mutants are killed by exactly their target test. Each application ends up with a median of 67 mutants, over 12,000 in total.

Specifications refined until they suffice. We then stress-test each specification. Agents generate both an implementation and a test suite from it, and we classify every failure as either a generation mistake or a defect in the specification: ambiguous, incomplete or contradictory. We fix the specification defects and repeat until fewer than 5% of failures trace back to the specification, followed by a final manual review.

‍

‍

Metrics. A generated suite's validity is the share of its tests that pass on the reference implementation. Its mutation score is the share of mutants it kills, counting only valid tests, so a suite can't score points by failing on everything. For implementations that agents generate from the same specification, we report the implementation pass rate: the share of reference tests they pass. Agents run in sandboxed containers without network access or search tools, with a six-hour time limit.

Results

We evaluated six agents, each with its vendor-recommended harness at default reasoning effort.

‍

Table 1. Main results at default reasoning effort. Mutation score, test validity, cost and time refer to test generation. Implementation pass rate is the share of reference tests passed by an implementation the same agent built from the same specification.

‍

Valid, but weak

For the four strongest agents, 88–96% of generated tests pass on the correct implementation, and the implementations they build from the same specification pass 86–93% of the reference tests. The specification is clear and complete enough to build the software almost correctly. Yet those same agents' test suites catch only 38–56% of the bugs.

That combination is the problem. Valid tests make a suite look healthy, because everything is green on correct code. A suite that doesn't distinguish correct from incorrect behaviour gives the impression of well-tested software without the protection.

It's not the specification

If the specification were the bottleneck, letting the agent run the reference implementation should help, since it can then settle any remaining ambiguity by observation. For GPT-5.6 Sol, access to the reference executable raises validity from 94.4% to 98.4%, but the mutation score barely moves, from 37.7% to 38.1%. The agents know what the software should do. What they don't do is write the tests that would notice when it doesn't.

Why suites miss bugs

Two patterns account for much of the gap.

First, the suites are too small. Claude Opus 5 comes within 9% of the reference suite's size, but the other agents write far fewer tests, and GPT-5.6 Sol's suites are about five times smaller than the reference. Covering every requirement takes many tests: applications where agents write more tests, or where there are fewer reference tests to match, score higher.

‍

‍

Second, suites test the main path and skip the input combinations that expose a violated requirement. In the example below, the specification says that ast-grep's run command accepts multiple paths and returns all matching files. The mutant silently ignores the second path. Opus's test passes two files and catches it. Sol's test only ever passes one file, and the bug survives.

‍

‍

Agents share blind spots

Combining agents helps less than you might hope. Pooling test suites across Opus, Sol and Gemini only raises the share caught from 54% for Opus alone to 59%.

‍

‍

Real documentation makes it much worse

Our specifications are deliberately detailed. On 30 applications with extensive official documentation, we replaced our specification with the developers' own docs. GPT-5.6 Sol's mutation score dropped from 25.5% to 4.0%. It wrote 40% fewer tests, fewer of them were valid, and the documentation caused three times as many underspecification failures per test.

What this means for teams running agents

A green test suite written by an agent tells you the code agrees with the tests. It doesn't tell you the code does what you intended. On TestSuiteBench, the best agent writes tests that pass on correct code 96% of the time and still miss 45% of the bugs. Both numbers describe the same suites, and only the first one shows up in CI.

A few practical consequences:

• Judge test suites by what they catch, not by what they pass. Pass rate and coverage reward valid-but-weak tests. Mutation testing, even on a sample of requirements, measures what you actually care about.
• Don't treat an agent's tests as the review. Until test generation closes this gap, something other than the agent's own suite has to check that implemented and intended behaviour agree, requirement by requirement. That is where review effort should go.
• Invest in the specification. Replacing developer docs with a detailed specification took the mutation score from 4% to 25.5% in our experiment. The specification is the only place where intended behaviour is written down.

Measuring agent and especially review quality against real engineering outcomes is the problem we work on at LogicStar.

Limitations

• Mutants stand in for test strength. Each mutant violates a single requirement by design, so the set doesn't follow the distribution of real-world faults. Mutants also target only a sample of requirements, but since agents can't know which ones, only suites that cover the whole specification score well in expectation.
• Our specifications are more detailed than most teams'. They are derived from extensive test suites and refined until they suffice for both implementation and testing. That favours the agents, so our results probably overestimate how well agents test in practice.
• The scope is command-line applications in compiled languages (Rust, Go and C/C++). Agents only ever see the specification, so the pipeline should extend to interpreted languages such as Python and TypeScript; we leave that to future work.

Paper, data and code

• Paper: TestSuiteBench: Coding Agents Struggle to Generate Strong Test Suites from Specifications (arXiv:2610.13065)
• Leaderboard: testsuitebench.com
• Dataset: Hugging Face
• Code: github.com/eth-sri/testsuitebench

To cite:

@misc{egashira2026testsuitebenchcodingagentsstruggle,
      title={TestSuiteBench: Coding Agents Struggle to Generate Strong Test Suites from Specifications},
      author={Kazuki Egashira and Niels Mündler-Sasahara and Mark Niklas Müller and Martin Vechev},
      year={2026},
      eprint={2610.13065},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2610.13065},
}

By Mark Niklas Müller. Work done with Kazuki Egashira, Niels Mündler-Sasahara and Martin Vechev at the SRI Lab, ETH Zürich.