Coding Agents Find Different Bugs Than Maintainers Fix

Coding agents can fix reported issues and build entire features from scratch. But can they find the bugs that matter in a repository without being told where to look? We built PROBE, a benchmark for proactive bug discovery, to find out.

Ask a frontier coding agent to audit a repository in one go, and it finds only a handful of bugs across hundreds of files. Have it work through the repository one focused pass at a time, and it can return thousands of bug reports, most of them describing real defects. Yet a single run of the best model we tested still misses 79% of the bugs maintainers go on to fix.

That is the central finding of PROBE. We snapshot 20 open-source repositories and collect 2,457 bugs that were already present in those snapshots and that maintainers fixed over the following 18 months; that 79% is measured against this set. To check whether agent findings are real defects, a panel of coding agents, validated against blind human grading, tries to reproduce a sample of them and confirms 85% of those it can assess conclusively.

Agents find real bugs, but mostly not the ones maintainers fix.

Why bug discovery needs its own benchmark

Fixing a reported bug and finding an unreported one are different tasks. A bug report gives an agent a symptom, an expected behaviour and often a starting point in the code. In an open-ended audit, the agent has to work all of that out itself, and decide which of the many possible problems are worth reporting.

Existing benchmarks cover parts of this problem. Review and detection benchmarks such as SWR-Bench and Active-SWE point the agent to a proposed patch or to files known to contain a flaw. Security benchmarks such as CyberGym and BountyBench consider only vulnerabilities. SWE-sweep, developed concurrently with PROBE, comes closest: agents search whole repositories without hints, but only bugs they both find and fix count. That makes evaluation more expensive and prevents measuring either capability in isolation. None measures discovery on its own: whether an agent auditing a whole repository finds the bugs maintainers go on to fix.

Measuring discovery is hard because nobody knows every bug a repository contains. Maintainers do reveal some of them, though, by fixing them over time, and those are bugs that mattered enough to fix. PROBE uses this record as its reference for open-ended bug discovery.

How we built PROBE

Snapshots. We snapshot 20 actively developed Python, JavaScript and TypeScript repositories on 1 January 2025, totalling 1.1 million lines of code. They span domains from SQL parsing (SQLGlot) and PDF manipulation (pypdf) to HTML rendering (WeasyPrint), UI compilation (Svelte) and workflow orchestration (Prefect).

Reference bugs from maintainer fixes. We analyse all 13,254 commits made in the 18 months after the snapshot and identify 4,296 bug fixes. From these, we extract the distinct defects each fix addresses, map the affected code back to the snapshot, and have a coding agent with the full revision history confirm that each bug was already present then. After deduplication, 2,457 reference bugs remain.

An isolated audit. The agent under evaluation audits the snapshot in a container with dependencies pinned to their versions at the snapshot date, so it can run the code as it was then. Revision history and the reference bugs are withheld, and network access is disabled so the agent cannot look up later fixes.

Validated matching. A finding matches a reference bug if fixing the root cause of either would also fix the other. An LLM-based matcher applies this criterion with about 99% precision and 98% recall on expert-labelled pairs.

Metrics. Recall is the share of reference bugs recovered by the agent. Historical precision is the share of agent findings that match a reference bug. F1 is the harmonic mean of recall and historical precision. An unmatched finding can still be a real bug: the reference contains only bugs maintainers fixed within our observation window.

Results

Thorough audits still miss most bugs

Simply asking an agent to audit an entire repository produces few findings: fewer than 100 across all 20 repositories in our single-session runs with Claude Sonnet 5 and GPT-5.6 Luna. Recall stays below 1%.

To make the search more thorough, we split each repository into focused audit passes: the most complex files get a pass of their own, and the rest are grouped by directory. Each pass receives its target files, a description of the project's architecture, public API and supported inputs, and snippets of callers and callees in other files. It can still explore the whole repository. Across the benchmark, 740 passes cover 4,468 source files, and every model receives the same passes and context.

This raises recall roughly tenfold and outperforms prior bug finders. Hound, a security audit agent, reaches less than half our harness's recall with the same model, at 8.6 times the cost. Even so, the best single-run configuration on the full benchmark misses roughly four in five reference bugs.

Model	Reasoning effort	Recall	Historical precision	F1	Findings	API cost
DeepSeek V4.1 Flash	High	15.2%	8.1%	10.5%	5,019	$73
GPT-5.6 Sol	High	20.8%	6.7%	10.2%	8,285	$1,185
GPT-5.6 Luna	Extra-high	17.3%	7.1%	10.1%	6,636	$131
GLM 5.3 Flash	High	15.8%	7.0%	9.7%	6,029	$120
GPT-5.6 Sol	Medium	16.9%	6.7%	9.6%	6,675	$678
GPT-5.4	Medium	11.0%	6.9%	8.5%	4,204	$655
GPT-5.6 Terra	Medium	9.1%	7.6%	8.3%	3,093	$224
Gemini 3.5 Flash	Medium	9.8%	6.4%	7.8%	3,867	$493
Muse Spark 1.3	High	15.9%	4.8%	7.4%	8,372	$164
GPT-5.6 Luna	Medium	7.6%	7.0%	7.3%	2,774	$15
Claude Sonnet 5	Medium	7.9%	5.8%	6.7%	3,472	$421

Table 1. Nine models on the full benchmark, including additional reasoning levels for Sol and Luna, sorted by descending F1. Luna at medium effort is averaged over ten runs and Muse Spark over five. Costs are per run across all 20 repositories at public API prices.

For cost reasons, Claude Opus 5 is evaluated on a five-repository subset, where it reaches 21.2% recall against 24.0% for Sol at high effort.

More reasoning, more findings

Increasing reasoning effort recovers more reference bugs. Luna's recall rises from 5.3% at low effort to 17.3% at extra-high effort, while its cost rises from $8 to $131. Sol's recall rises from 11.4% at low effort to 20.8% at high effort, with cost increasing from $276 to $1,185.

Historical precision barely moves. Across configurations, recall tracks the total number of findings reported, with a Pearson correlation of 0.94. More compute expands the audit's output without making that output match maintainer fixes more closely.

The most expensive model is also not always the strongest choice. Luna at extra-high effort strictly dominates Sol at medium effort in both recall and historical precision for less than a fifth of the cost. Model choice and reasoning budget need to be evaluated together.

Valid, but not what maintainers fix

The best single run reaches at most 8.1% historical precision. That does not mean the remaining findings are false alarms. Our reference set is incomplete. A finder recovering every bug maintainers would fix within five to ten years would score only 26–47% historical precision on the reference set mined over 18 months.

To assess validity independently of historical matches, we sample 100 findings from each of nine models and use a separate panel of coding agents to investigate each finding. Across conclusive verdicts, the panel confirms 85% as valid, including 84% of findings that do not match the reference set. Validity, however, varies by model, from 64% for Claude Sonnet 5 to 95% for GPT-5.4 and GPT-5.6 Sol.

We also manually grade 180 findings, blind to the generating model and the panel's verdict. Where both reach a conclusive verdict (131 findings), they agree 93% of the time. Every disagreement is a finding the panel confirms and our grader rejects, so the panel is somewhat lenient: on those 131 findings, our grader confirms 79% against the panel's 85%. Either way, most unmatched findings are real bugs.

It's not just the reference window

Maintainers may simply not have reached these bugs yet: bugs fixed after our 18-month window cannot match. We model the rate at which maintainers fix snapshot bugs and, assuming the observed pace continues, estimate that they would fix roughly 5,200–5,900 within five years and 7,200–9,400 within ten. Our 2,457 reference bugs are 26–47% of those totals, which is where the historical-precision estimate above comes from.

Agents show no detectable preference for earlier- or later-fixed reference bugs. If that pattern continues beyond the observation window, extrapolating historical precision from around 7% to the estimated ten-year pool raises it to only 21–27%. Combined with the panel's 85% validity estimate, more than two-thirds of valid agent findings would remain unfixed for at least a decade.

While this is only a model-based extrapolation, it suggests that the short window alone does not explain the whole mismatch.

Different kinds of bugs

We classify agent findings and reference bugs into 33 bug types. Agents disproportionately report local faults, such as incorrect arithmetic, comparisons and handling of degenerate values. Maintainer fixes more often address missing support for an input variant or errors in the public API.

Bug type	Share of maintainer-fixed bugs	Mean share of agent findings
Wrong arithmetic or comparison	9.9%	18.5%
Unhandled degenerate value	9.1%	13.9%
Missing branch for a supported input variant	18.0%	7.7%
Missing dispatch or lookup-table entry	5.1%	0.7%
Incorrect public type declaration	4.5%	1.1%

Table 2. Selected bug types. Agent shares are averaged over the evaluated configurations.

One reference bug in WeasyPrint illustrates a supported behaviour that was silently ignored. An SVG image could set image-rendering: pixelated, but the renderer always passed auto to the drawing code, so scaling the image could blur it despite the requested mode. The maintainer fix forwards the image's rendering setting to the drawing operation.

No configuration we evaluated found this bug. Spotting it requires connecting a setting the code accepts to the output a user expects to see, rather than checking the code's local logic.

Environments and rendered output matter

The differences also appear in what is needed to trigger or observe a bug.

Trigger or observation requirement	Maintainer-fixed bugs	Agent findings
A particular environment	10.8%	4.7%
Interaction or inspection of output	3.8%	0.7%
A particular input	74.3%	83.9%

Table 3. Selected trigger categories; the remaining categories concern ordinary execution and particular timing.

The evaluated agents primarily read code, while maintainers more often fix failures that surface in a particular environment or when someone looks at the output. Agents that run software through its supported workflows and inspect the results might close part of this gap, though we have not tested that.

Agents share blind spots

The mismatch is consistent across models, whose bug-type distributions resemble each other's more than the maintainers'. It also persists within bug types: comparing recall within each type, repeated Luna runs overlap with each other a median of 6.3 times more than with the human reference set.

The overlap extends across models. Of the reference bugs that none of nine Luna runs find, other models find only 5% on average. Of those that all nine runs find, other models find 87% on average. If models picked bugs at random within each bug type, both figures would be 12–17%.

Agents are drawn towards the same defects, and they miss the same ones.

Repeated runs find different bugs

Rerunning the same configuration gives similar scores but different findings. On average, only 57% of one Luna run's findings appear in another run, and 63% for Muse Spark. Combining runs exploits this. The union of five Muse Spark runs reaches 26.7% recall, above any single-run configuration on the full benchmark, at a combined cost of $821. Keeping only the findings that all ten Luna runs report raises historical precision to 10.5%. Findings an agent reports consistently are more likely to be ones maintainers fix.

What this means for teams running agents

A repository audit can produce many technically valid reports while leaving most of the bugs maintainers later fix undiscovered.

• Measure recall against real engineering outcomes. Use historical fixes to check whether an audit would have caught the problems that later needed repair, not just whether its findings are valid.
• Evaluate model choice and reasoning effort together. More reasoning can expand coverage, and a cheaper model with more reasoning can outperform a more expensive configuration. Compare performance at a given budget.
• Run audits more than once. A single run reports only a sample of what an agent can find. Pooling runs raises recall, and findings that recur across runs are more likely to be ones maintainers would fix.
• Exercise supported behaviour. Include realistic environments, documented input variants and inspection of rendered outputs when evaluating an audit. These are areas where maintainer fixes differ from the findings of the agents we tested.

Agents already find many real defects that do not appear in subsequent maintainer fixes. Finding the bugs that drive actual maintenance work remains the harder problem. PROBE gives us a way to measure how relevant an agent's findings are, not only whether they are valid.

Paper and benchmark

• Paper: PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix
• Benchmark and results: probebench.com

@misc{paina2026probe,
  title  = {PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix},
  author = {Marco Paina and Mark Niklas Müller and Niels Mündler-Sasahara and Martin Vechev},
  year   = {2026},
  eprint = {2610.35987},
  archivePrefix = {arXiv},
  primaryClass = {cs.SE},
  url    = {https://arxiv.org/abs/2610.35987}
}

By Mark Niklas Müller. Work done with Marco Paina at LogicStar, and with Niels Mündler-Sasahara and Martin Vechev at the SRI Lab, ETH Zürich.