June 5, 2026 / 1 min read
.png)
AI coding agents are improving rapidly.
But writing code is only a small part of software engineering.
The harder questions are:
At LogicStar, we believe the future of software engineering will be determined by answering these questions, not by generating more code.
That's why we spend significant effort studying where agents fail.
Over the last several years, our team has built a series of benchmarks, each focused on a different weakness of software engineering agents.
Failure mode: Action bias.
Agents often modify code even when the correct action is to do nothing. FixedBench studies whether agents can distinguish between code that is broken and code that is already correct.
Failure mode: Verification.
Can agents reproduce real-world bugs and generate tests that prove a fix actually works?
Failure mode: Security.
Can agents build backend systems that are not only functional but secure?
Failure mode: Repository-scale refactoring.
Can agents perform large-scale code transformations while preserving behavior and maintainability?
Failure mode: Evaluation.
How do we automatically generate realistic software engineering tasks that accurately measure agent performance?
Failure mode: Context overload.
Do repository-level instruction files actually improve outcomes, or do they simply add more context without improving understanding?
Across all six benchmarks, we found the same pattern.
Agents are increasingly capable of writing code.
But software maintenance requires much more than code generation.
It requires investigation.
Verification.
Prioritization.
Architectural understanding.
And evidence.
This observation became the foundation of LogicStar.
Rather than treating maintenance as a code-generation problem, we treat it as a software understanding problem.
LogicStar correlates production signals, customer reports, code structure, historical changes, and runtime behavior to identify what actually matters.
Every issue is investigated.
Every fix is validated.
Every recommendation is grounded in evidence.
The result is not an agent that simply writes code.
It is a system designed around the known failure modes of software engineering agents.
Because the future of autonomous software engineering will not be decided by who generates the most code.
It will be decided by who makes the best decisions.
Blog
A coding agent can turn a ticket into a green pull request in minutes. We split the merge decision into narrow checks, let fixed rules decide, set the strictness by the blast radius of a change, and bring in a person only where the agent gets stuck.
We ran three coding agents through four gateways to see what a gateway controls, what it measures correctly and where it misleads you, from budgets that overshoot to telemetry that leaks prompts and data residency set hop by hop.
We compared a stronger model, a different agent, answers from the author and a read-only investigation before any code. On 19 fresh tickets, the investigation went ahead on 9 of 10 wanted changes and held back 7 of 9 requests an author had already declined.
With the team behind the first AI for code.
Backed by frontier coding agent research with ETH Zurich.
Cookie settings
By clicking 'Accept all cookies', you agree to storing cookies on your device to enhance site navigation, analyze site usage and assist in our marketing efforts as outlined in our cookie policy.