How a Software Factory Earns the Right to Merge

A coding agent can turn a ticket into a pull request with green tests in a few minutes. Deciding which of those pull requests deserve to be merged is the hard part. Tests only check what someone thought to test. When you iterate with agents, not every case gets a test, and not everything can be easily expressed as one. Asking a person to review every agent pull request buries them in extra work and takes away the fun part of the job: solving problems. A person's time is better spent on the few pull requests that actually need their judgement.

So we set out to find which changes coming out of a software factory, where agents work tickets unattended, need a developer's review, and to do it without swamping developers with extra work. This post covers what we learned: which checks we kept and why, what already exists, and what is still open.

In short

• Specialists, not a generalist. Many narrow agents beat one agent that decides everything. Fixed rules, not an agent, make the call.
• Lean on deterministic checks. Tests, types and scanners run first. Narrow agents fill the gaps.
• Vary the policy with the blast radius of a change: strict where a bug would reach customers or their data, lenient on dev tools and docs.
• Bring in a person where the agent gets stuck: when it disagrees with a finding, or when a blocking problem survives the agent's self-improvement loop.
• No single product covers it all, and each specialist can be a short prompt for Claude or Codex.

What problem are we trying to solve?

The software factory turns a ticket into a code change. Our task is to verify that change and decide, without a person reading every line, whether it is ready to merge as is, needs another round of fixes from the agent, or needs a person to look at it. Verification gathers the evidence; the merge gate reads it and decides.

To make that decision, we set four goals:

1. Solve what was asked. The change does what the ticket asks.
2. Break nothing. No failing tests, no behaviour changed beside the task.
3. Hold the quality bar. Reuse what already exists, and keep the code easy to maintain.
4. Don't annoy reviewers unnecessarily. Limit human review time to real problems.

What we recommend: narrow questions, fixed rules

• Impact assessment. Looks at where in the codebase the change lands and estimates how far a bug could reach and how hard it would hit users. The bigger the impact, the stricter the rules.
• CI and fixed checks. Deterministic checks: tests, type checks, linters, and scanners for known-bad patterns, leaked secrets and vulnerable dependencies.
• Narrow checks. Read-only agents, each answering one question about the change: is the ticket resolved, are the tests honest, does everything belong to the task, is it secure, is it maintainable. More in the next sections.
• AI review tools. A general review of the whole change, as a last pass. Tools for this are plentiful: Codex review, Greptile, CodeRabbit and many more.
• Rule table. Fixed rules combine the findings with the impact to decide what happens next. No LLM decides here, so the same findings always give the same outcome, and no agent can pass or block a change on its own.
• The loop. Blocking findings go back to the coding agent, which gets a few rounds to fix them; only what it cannot resolve goes to a person.

Impact assessment: how strict should the gate be?

Not every change needs the same level of checking. A fix in a developer tool and a change to how customer data is stored should not face the same rules: one fixed set is either too strict for the first or too permissive for the second.

So before any check runs, the impact assessment estimates the blast radius of the change: if it shipped with a bug, how far could that bug reach, and how hard would it hit users? The answer sets how many findings the agent may resolve without a person. A change with a large blast radius carries more risk, so more of its findings must be fixed and it is more likely to reach a person. A change with a small one is blocked only on severe issues. That keeps human attention where a mistake would hurt, which is the point of the whole gate.

We have not found existing work that does this. The obvious alternative, merging only when no AI reviewer has anything left to say, is too strict: in PROBE we found that much of what AI reviewers raise is not what human maintainers go on to fix.

The narrow checks

Each narrow check is a read-only agent that sees the ticket and the code, never the coding agent's reasoning or its own description of the change.

On historical pull requests, the general AI reviewers we tried held back good and bad changes at about the same rate. Why several narrow checks instead of one general reviewer? Each one can be tuned on its own. In one long prompt, changing one task shifts the others, and each gets only a share of the model's attention. Checks can also be added or dropped without touching the rest.

We still run a general AI review as the last step, to cover what the narrow checks do not ask about.

Ticket Resolved Check

Writes acceptance criteria from the ticket alone, before seeing the change, then checks the change against each one.

• Why: a change is only worth merging if it solves the problem it was asked to solve.
• Challenges: tickets are written with very different precision and granularity. Vague ones leave room for interpretation, so whether a change resolves them depends on where you draw the line.
• Existing: CodeRabbit's linked-issue check and Qodo Merge's ticket compliance check.
• What we tried: three checks, on correct and on wrong fixes: Qodo Merge's ticket compliance check, a Claude check that asks directly whether the ticket is resolved, and a Claude check that first lists the ticket's requirements and then checks each one.

Our recommendation: work from the ticket's requirements. The two checks that worked from requirements, Qodo Merge and ours, performed about the same: they caught most wrong fixes and always gave a verdict, at the cost of blocking a few more good fixes. That trade is worth it: a good fix blocked by mistake costs the factory one more round, while a wrong fix that slips through reaches users and needs to be fixed later. Asking directly, without a requirement list, flagged less overall, which puts it on the worse side of that trade. A requirement list also shows the agent exactly what is missing. Use Qodo Merge, or build your own check if you want to tune it to your team's tickets.

Test Integrity Check

Asks whether tests were skipped, loosened or rewritten to fit the agent's own change, instead of guarding against regressions.

• Why: passing tests only mean something if they check the desired behaviour. Researchers showed with ImpossibleBench that when the tests cannot be passed honestly, agents tend to bend them instead, so that the change appears to pass everything. Letting agents retry after a failure made this more common, and a verification loop is exactly such a retry loop.
• Challenges: some test changes are legitimate, because the ticket changes the behaviour or interfaces. So this check needs to tell a required update from a weakened assertion, and that gets harder as the changes grow.
• Existing: none that we know of.

Our recommendation: an agent check that reads the test changes against the ticket. We ran it on real changes and on bad copies of them, where a test was bent on purpose. It caught most of the bent tests and raised no false alarms on the real changes. The check misses more on very large test diffs, which matches what the ImpossibleBench researchers found.

Scope Check

Asks whether every part of the diff belongs to the task, or whether the agent changed behaviour nobody asked for.

• Why: agents like to tidy up on the way: they rename things, refactor nearby code or change behaviour nobody asked for. Every extra change is something a reviewer does not expect and the tests may not cover. Spotify, which runs coding agents unattended on its own codebase, reports that going beyond the task is the most common reason its automated reviewer stops a change.
• Challenges: a harmless extra, such as a rename or a small refactor, should not weigh like an unrequested behaviour change. Drawing that line is the hard part, and an extra change hidden inside code the task already touches is easy to miss. Sometimes the best fix is a larger refactor first, with the new behaviour built on top. The agent should be free to propose it, but such a change should not be merged without a person's approval.
• Existing: no good solution that we know of. One option is a size limit on changes, but that blocks good big fixes and lets small bad ones through.

Our recommendation: an agent check that groups the diff into parts that belong together and checks whether each part traces back to the ticket. A harmless extra goes back to the agent as a note; a behaviour change nobody asked for, or a large refactor, is flagged for a person to decide. It caught nearly all scope creep we planted and raised few false alarms on real pull requests. It misses more when the extra change hides inside code the task already touches.

Security Check

Looks for security weaknesses the change introduces.

• Why: every change needs it, and an agent's change has to meet the same bar as a person's.
• Challenges: false alarms on code that only looks risky, and holes made of missing code, which are hard to see in a diff.
• Existing: static scanners such as CodeQL, Semgrep and Snyk, and AI security reviews such as Claude Code's and many more.
• What we tried: Claude Code's security review, Codex with a security prompt, and our own agent check in three prompt variants. We ran them on real changes and on bad copies with a vulnerability planted on purpose.

Our recommendation: don't reinvent this. Use the established tools: static scanners such as Semgrep or CodeQL for known patterns, secret scanning such as Gitleaks, and dependency scanning such as OSV-Scanner or Snyk. On top, add an off-the-shelf AI security review such as Claude Code's for what pattern rules cannot see. A custom agent can be tuned to look for specific kinds of bugs, but in our tests all approaches scored about the same.

Maintainability Check

Looks for duplicates, needless layers, debug leftovers and code that does not fit the repository, and rates how much each one matters.

• Why: agents add code more readily than they reuse it. Each change can look fine on its own while the codebase slowly gets worse.
• Challenges: maintainability is partly taste, and "fits the repository" depends on the repository. Without a rating of how much each finding matters, real problems drown in nitpicks. Metric-based scores have the opposite problem: they need a threshold, and where to set it is hard to justify.
• Existing: SonarQube, CodeScene and complexity metrics such as lizard, plus the style comments of general AI reviewers.

Our recommendation: an agent check that looks for duplicated code, needless layers, leftovers and code that does not fit the repository, and rates how much each finding matters. Serious ones block; the rest go back to the agent for self-improvement. We dropped the metric tools, as there was no threshold on their scores separating good fixes from bad ones.

Potential Extra Checks

The set of checks can be extended as needed. We did not add the ones below because they did not apply directly to the projects we tested on, but depending on your project and its tasks they can be useful:

• QA agent: drives the app and checks that the changed flows work and render correctly. Existing: QA agents such as QA.tech, or a coding agent such as Claude Code driving the browser through Playwright's MCP server.
• Dependency risk: checks every package the agent adds, including names the model invented and an attacker has since registered. Existing: Socket, Snyk.
• Architecture: checks the change against the documented design and specs. Existing: CodeRabbit and Qodo Merge can both be pointed at your design docs with custom instructions.

This list is not complete: there are many more cases we did not mention, and every codebase will have its own checks worth adding.

What is still open

The gate is more deterministic than a single judge, but not fully: the rules are fixed, while the findings come from AI checks that can vary between runs. The next step is to evaluate each component on more data, across more projects and kinds of tickets.

What already looks clear: trust in an agent's change doesn't come from one smart reviewer. It comes from many small questions, rules that don't change their mind, and a person who only sees what really needs them.