November 6, 2026 / 6 min read
At LogicStar we run a software factory on our own code: coding agents pick up tickets from our issue tracker and work them without anyone watching, and a person reviews the pull requests they open. In the first week of November 2026 we measured what it did with tickets from our own platform, including tickets made from findings of the review bot that checks our pull requests.
Four of those tickets turned into pull requests that passed CI and our review bot. Nobody wanted any of them. Each one asked for something a person had already answered: a fix that was already in, or a change the team had decided against. The answers were written down in replies on old pull requests, and none of the agents read them.
We wanted to know what would have caught this, so we compared the options: a stronger model, a different coding agent, a simulated author answering the agent's questions, and a read-only investigation before any code is written. This post covers what we measured, why it matters more when agents work without anyone watching, and the controls we keep on the work once it starts.
When you run a coding agent yourself, you are the gatekeeper. You know the history, you remember the thread where the team said no, and you stop before you write the prompt.
A software factory takes that person out of the loop on purpose. Tickets come from many people, agents work them unattended, and the first person to see the result is a reviewer looking at a finished pull request. A wrong pull request costs review time, and one that looks right can get merged. Whatever a person would check in their head before starting has to become a step the factory runs on every ticket.
The main coding agents now pick up tasks where people already discuss and assign work: an issue, a tracker or a chat thread. Mention an agent or assign it a task and it starts working; when it is done, a person reviews the result.
Reads past review threads before it starts: not documented for any of the five.
From each vendor's documentation, November 2026
Some of them can show a plan first, and some ask questions in the thread. None of the five documents reading past review threads before it starts. What happens between the ticket and the code is left to the agent and to whoever wrote the ticket.
With several ways in, we keep one record per piece of work: the issue. A chat thread or a document can start a task, but it files an issue and links back to it. Questions from the factory go to the issue and mention its author, and the answers come back there. Runs, questions, holds and cost all sit on the issue, so nobody has to watch a second queue.
We took ten hard tickets from our own platform and ran them through three setups: a stronger model, a different coding agent, and a simulated author who answered the agent's questions. Each setup solved the same two tickets. Retrying didn't change that: a third attempt after two failures never produced a correct patch.
On small, clear tickets the same factory did well. Issues became pull requests for $0.16 to $0.69 each, in about ten minutes, and when a reviewer left a comment the factory fixed it in one more round.
OpenAI saw the same pattern at a larger scale. When it audited 138 SWE-bench Verified problems that its o3 model could not solve reliably, 59.4% had flawed tests or problem descriptions, which made them extremely difficult or impossible "even for the most capable model or human to solve".
To get more tickets, we turned six findings from our review bot into issues. Five of them asked for a change that a person had already answered in the same review thread. One was already fixed in the pull request the finding came from. Three were explicit decisions to keep the code as it was ("keeping this as is, deliberately"). One was a design choice its author had defended.
Four of the five became pull requests that passed CI and the review bot. The agents followed their instructions: they read the ticket and the code, and the decision was in neither. It was in a reply on an old pull request.
Teams keep decisions in four places: docs in the repository, comments in the code, review threads and people's heads. Coding agents read the first two.
Other teams see this too. In a study of 600 agent-written pull requests that maintainers did not merge, 23% duplicated work that already existed, and another 4% were features the maintainers said they did not want.
We added a step between the ticket and the code. A read-only agent run, which can't change anything, writes a short brief:
TicketRetry the upload when the storage call times out
Proceed hands the brief to the coding agent. Ask and stop post the brief on the issue and wait for a person. A stop never closes the issue; a person decides.
The history lookup has to follow lines, not line numbers. The review comment behind one of our tickets was on line 327 of the file at the time. The same code is on line 160 today, so a lookup of line 327 no longer finds it. The lookup has to follow each line back through every commit.
export async function exportRows(account) { const rows = await loadRows(account); log.info('export started', { account }); const started = Date.now(); metrics.count('export'); if (!account.active) { throw new InactiveAccount(account); } const format = account.format ?? 'csv'; const limit = EXPORT_RATE_LIMIT; await limiter.wait(account, limit); return toCsv(rows, format);}We first ran the investigation on the 20 tickets that motivated it, with the links to the review threads removed and the history cut off at each ticket's starting commit, so it couldn't see answers from later.
All five unwanted tickets stopped, and each brief named the decision or the earlier fix. An investigation took three to five minutes and cost $0.12 to $0.32.
It also asked a lot: two of the five wanted tickets got a question. Both were fair, but a factory that questions half its tickets will be ignored.
Asking does help when the answer is missing. In controlled tests on deliberately vague tickets, agents that could ask a simulated user did up to 74% better than agents that couldn't, and the models rarely asked on their own.
So we split the decision. The investigation now blocks only when the ticket contradicts a recorded decision (a policy comment, a design document or a person's reply in review) or when the work is already done. For any other open detail it picks a reasonable default, states it as an assumption with the line it rests on, and proceeds. The assumptions go into the pull request for the reviewer.
Then the real test: 19 tickets the investigation was never shaped around, built from the team's own merged work and frozen before the run. Ten were changes the team wanted, written from the problem only. Nine were suggestions an author had explicitly declined in review, written as plain requests.
It went ahead on 9 of the 10 wanted changes and held back 7 of the 9 declined ones, each time citing the author's reply. The two misses had the author's reply on a different line from the code the investigation looked at, so the lookup now also reads the other review threads of the pull request it finds.
On the original 20, the same version went ahead on all five wanted tickets and still held four of the five unwanted ones. The fifth showed a new failure: it treated a partial existing fix as permission to build the rest. A partial fix now produces a question instead.
Once a ticket is cleared, a few plain controls keep the work in check. Each one is there because something went wrong without it.
All of this is one repository and about 40 tickets, and a person graded every reason the investigation gave. The last two fixes, reading the pull request's other review threads and asking on a partial fix, have not been re-measured yet; that is the next run. Everything measured here came in through the issue tracker, and the chat and document front doors are the part we are building now.
If agents work tickets in your team without anyone watching:
Blog
A coding agent can turn a ticket into a green pull request in minutes. We split the merge decision into narrow checks, let fixed rules decide, set the strictness by the blast radius of a change, and bring in a person only where the agent gets stuck.
We ran three coding agents through four gateways to see what a gateway controls, what it measures correctly and where it misleads you, from budgets that overshoot to telemetry that leaks prompts and data residency set hop by hop.
We ran OpenSpec, Spec Kit, Superpowers and Claude Code plan mode on a real ticket, then planted drift and contradictions. For one feature we couldn't separate the code, and plan mode cost about half. What differs most is the record each leaves for the next agent.
With the team behind the first AI for code.
Backed by frontier coding agent research with ETH Zurich.
Cookie settings
By clicking 'Accept all cookies', you agree to storing cookies on your device to enhance site navigation, analyze site usage and assist in our marketing efforts as outlined in our cookie policy.