June 12, 2026 / 3 min read
Code agents are great at investigating and fixing known bugs. But are they good at finding the important ones across a real codebase?
To answer this question, we ran a case study comparing Claude Code to LogicStar’s bug finder on a snapshot of our own codebase.
Key result: LogicStar found 2x more unique high-impact bugs than Claude Code with LogicStar-generated rules, at less than one-third the cost per unique high-impact bug.
We compared three different bug finders:
In all three settings, we used our bug validation engine to confirm whether the found bugs were true positives and assess their severity. This means the results are comparable across all settings. In our product, all bugs go through this validation before they are shown to customers.
To assess which approach works best, we considered six metrics:
LogicStar surfaced 40 issues across 43% of source files. After validation, 28 were confirmed as real bugs. Of those, 8 were high impact and 20 were medium impact. The remaining 12 surfaced issues were filtered out as false positives.
The scan used multiple specialized sub-agents and cost $47 in token spend.
Claude Code, in contrast, surfaced only 10 issues, 7 of which were bugs. None of them were high impact and only 3 were medium impact, with the rest being so low impact that we would filter them out instead of showing them to customers. It explored only 7% of the codebase, leading to a low cost of $5.
This is not unexpected. Code agents are not well suited for open, codebase-spanning tasks. They were designed and trained for concrete tasks that require locating the right part of the codebase as context and then executing. Bug finding requires scanning through most of the codebase and looking for anomalies while understanding the relevant context.
To help Claude Code, we prompted it with 20 of our Bug Finding Rules. These describe failure modes specific to the codebase under review, including the areas of the codebase where they are likely to appear. This significantly increased the depth of the scan, covering 29% of files and surfacing 166 issues at a cost of $78.
After validation and deduplication, 78 were unique real bugs, 4 of which were unique high-impact bugs.
LogicStar learns from your feedback, both explicit and implicit. It observes which bugs you actually end up fixing and focuses on showing you more bugs of these types.
LogicStar not only filters out false positives, but also remembers what slipped through and got designated as false positive by you. It can avoid repeatedly showing the same false positives, while a standalone Claude Code scan does not retain this product-level feedback memory by default.
Vanilla code agents are not enough for production-grade open-ended bug finding. They cover only a small part of the codebase and find few high-impact issues.
To unlock better performance, code agents need to be instructed to look for specific patterns in the right parts of the codebase. However, they still find many false positives and only surface a small fraction of high-impact bugs. This results in a lot of noise to dig through and comes at a higher cost compared to more optimized solutions.
LogicStar’s specialized bug finding system produced the most high-impact bugs with the least noise, covering the largest part of the codebase at moderate cost. In addition, it comes with the validation layer required to filter out false positives and the memory layer to learn from feedback and avoid repeatedly showing the same false positives.
Blog
A coding agent can turn a ticket into a green pull request in minutes. We split the merge decision into narrow checks, let fixed rules decide, set the strictness by the blast radius of a change, and bring in a person only where the agent gets stuck.
We ran three coding agents through four gateways to see what a gateway controls, what it measures correctly and where it misleads you, from budgets that overshoot to telemetry that leaks prompts and data residency set hop by hop.
We compared a stronger model, a different agent, answers from the author and a read-only investigation before any code. On 19 fresh tickets, the investigation went ahead on 9 of 10 wanted changes and held back 7 of 9 requests an author had already declined.
With the team behind the first AI for code.
Backed by frontier coding agent research with ETH Zurich.
Cookie settings
By clicking 'Accept all cookies', you agree to storing cookies on your device to enhance site navigation, analyze site usage and assist in our marketing efforts as outlined in our cookie policy.