Claude Code Can Find Bugs. LogicStar Finds the Ones That Matter.

Code agents are great at investigating and fixing known bugs. But are they good at finding the important ones across a real codebase?

To answer this question, we ran a case study comparing Claude Code to LogicStar’s bug finder on a snapshot of our own codebase.

Key result: LogicStar found 2x more unique high-impact bugs than Claude Code with LogicStar-generated rules, at less than one-third the cost per unique high-impact bug.

Setup

We compared three different bug finders:

• LogicStar’s internal bug finding engine
• Vanilla Claude Code using the standard code-review skill
• Claude Code prompted with codebase knowledge and Bug Finding Rules provided by LogicStar

In all three settings, we used our bug validation engine to confirm whether the found bugs were true positives and assess their severity. This means the results are comparable across all settings. In our product, all bugs go through this validation before they are shown to customers.

To assess which approach works best, we considered six metrics:

1. File coverage: what percentage of source files the approach inspected
2. Surfaced issues: how many potential issues the approach reported
3. Unique validated bugs: how many surfaced issues were confirmed as real bugs after deduplication
4. Unique high-impact bugs: how many unique validated bugs were classified as high impact
5. Total cost: token spend for one scan
6. Cost per unique high-impact bug: token spend divided by unique high-impact bugs found

Results

LogicStar surfaced 40 issues across 43% of source files. After validation, 28 were confirmed as real bugs. Of those, 8 were high impact and 20 were medium impact. The remaining 12 surfaced issues were filtered out as false positives.

The scan used multiple specialized sub-agents and cost $47 in token spend.

Claude Code, in contrast, surfaced only 10 issues, 7 of which were bugs. None of them were high impact and only 3 were medium impact, with the rest being so low impact that we would filter them out instead of showing them to customers. It explored only 7% of the codebase, leading to a low cost of $5.

This is not unexpected. Code agents are not well suited for open, codebase-spanning tasks. They were designed and trained for concrete tasks that require locating the right part of the codebase as context and then executing. Bug finding requires scanning through most of the codebase and looking for anomalies while understanding the relevant context.

To help Claude Code, we prompted it with 20 of our Bug Finding Rules. These describe failure modes specific to the codebase under review, including the areas of the codebase where they are likely to appear. This significantly increased the depth of the scan, covering 29% of files and surfacing 166 issues at a cost of $78.

After validation and deduplication, 78 were unique real bugs, 4 of which were unique high-impact bugs.

Comparison

What the Metrics Do Not Capture

LogicStar learns from your feedback, both explicit and implicit. It observes which bugs you actually end up fixing and focuses on showing you more bugs of these types.

LogicStar not only filters out false positives, but also remembers what slipped through and got designated as false positive by you. It can avoid repeatedly showing the same false positives, while a standalone Claude Code scan does not retain this product-level feedback memory by default.

Conclusion

Vanilla code agents are not enough for production-grade open-ended bug finding. They cover only a small part of the codebase and find few high-impact issues.

To unlock better performance, code agents need to be instructed to look for specific patterns in the right parts of the codebase. However, they still find many false positives and only surface a small fraction of high-impact bugs. This results in a lot of noise to dig through and comes at a higher cost compared to more optimized solutions.

LogicStar’s specialized bug finding system produced the most high-impact bugs with the least noise, covering the largest part of the codebase at moderate cost. In addition, it comes with the validation layer required to filter out false positives and the memory layer to learn from feedback and avoid repeatedly showing the same false positives.

Try the no-login LogicStar demo.

‍