November 5, 2026 / 6 min read
Higher quality, better tested code follows.
We tested OpenSpec 1.16.0, GitHub's Spec Kit 1.3.0 and the Superpowers 7.0.2 plugin against plan mode in Claude Code 2.1.317 as the baseline.
specs/.propose writes a proposal, a design, tasks and spec deltas, and apply builds them. archive merges the deltas into living specs and moves the rest to an archive.implement builds them, and converge checks the code against the spec and adds tasks for any gap. Each feature keeps its folder.Each system was tested against the same series of tasks: a full feature (ingesting historically faithful PR review comments into our data platform, a change that took us +1,444/−69 lines), a small follow-up feature, a bug fix, and 5 drift and contradiction scenarios.
To keep instructions consistent, a Claude session played the engineer from a fixed answer sheet, in addition to running the feature through OpenSpec and Spec Kit by hand. We scored each build against 16 gotchas we found while implementing and reviewing the real change ourselves. We counted only the gotchas that could occur in a build's design and let our full CI and code review run on every build of the full feature.
Plan mode built the ticket for $11.08, against $19.30–26.41 for the others, at API rates. All six builds kept every constraint the ticket set and passed CI. Beyond that, one ticket doesn't separate them: all missed 3 to 6 gotchas and received 1 to 7 review findings. The two Spec Kit builds had both the fewest findings and the only P0. What did change the code was review inside the workflow. Superpowers' reviewer and Spec Kit's converge caught a real defect each. A spec system thus mostly improves code through the review it enables, not through the spec itself.
To review specs, chat is great for co-development. In a factory, however, review happens asynchronously, and for that a document is a much better choice. Reading every word the spec systems produce (up to 16,000 for our feature) isn't realistic. What works better is skimming the requirements, decisions and risks, then digging deeper where something looks off or is particularly tricky. For that, a good structure and skimmable headings matter. Here OpenSpec excels with descriptive titles, while Spec Kit and Superpowers mostly number and name sections, with one useful exception in Superpowers: "Calls made by Claude (check these)".
In a software factory, an agent working on a new task doesn't inherently know which requirements the system already has to meet. That is a key role of the spec, and more precisely of the permanent record it leaves behind when a task is done. That works best as a short statement of how the system is supposed to behave now, not a history of how it got there. Only OpenSpec builds this split in: each change merges into living specs, and the details of how we got there are archived. After a ticket is done, a later change only has to read a 1,805-word spec instead of all 7,562 words written along the way. Spec Kit keeps a folder per feature, 16,015 words for ours. Plan mode's spec is the shortest, at 681 words, but also captures the fewest of the gotchas to look out for, 3 of 16 against 6–10 elsewhere.
The planted bug shows the spec at work. All four restored the guard, but only OpenSpec asked whether the report meant a bug or new behaviour, because its spec and the latest commit disagreed. Superpowers fixed it fastest without opening its spec at all.
After implementing the feature, we tested whether the spec system notices contradictions with three requests that each reverse an earlier rule: use poll time as push time, let analysis fetch from GitHub, and backfill from project exports.
Of 36 runs, 28 stopped and asked, 7 flagged the conflict and carried on, and 1 never noticed. 26 cited hand-written repo docs. Plan mode stopped in 9 of 9 by finding each rule in a written document: its own short spec, a design-decision doc or an older spec. What catches a contradiction is a decision written down where the agent looks, whichever tool wrote it. OpenSpec's one miss shows the other side: the rule sat in an older spec not in OpenSpec's format, and its search output was cut off before reaching it. Spec Kit's 6 all noticed the conflict, but its specify step writes the spec before it can ask, so each had already rewritten or reversed the rule. Superpowers' one asked why we wanted the change but had already decided to replace the old rule.
On a throwaway branch, we made the code and the spec disagree in one of two ways: a commit that changed behaviour against the spec, or a new requirement in the spec with no code behind it. Each run then got the same messages in order: an unrelated one-line edit a few lines from the drift, the system's own check where it has one, a plain request to check whether code and specs still agree, and "resolve what you found".
No run noticed the drift while making the edit. Of the built-in checks, Spec Kit's converge found it in 6 of 6 runs. OpenSpec's verify checks only an active change, and with everything archived it had nothing to check. Asked plainly, all 24 runs found it.
Told to resolve it, plan mode asked us which side was right in every run, the others in about half. Left to decide, every run sided with the spec, whichever side had moved.
We wrote a 444-word command for OpenSpec, modelled on Spec Kit's converge. It is designed to always surface drift but never fix it: it compares the code with the living specs, names the commit that moved, says for each finding what a person has to decide, and stops. It found the drift in 6 of 6 runs, at about $1.25 for 18–19 requirements.
When a person drives each change and holds the context, plan mode plus an independent review is enough and much cheaper, at $38 for all tasks in the trial together against $57–73 for the others.
This is one codebase and one feature, and AI agents were used to score the runs together with our tests. We haven't yet compared these methods over a series of features where the spec really accumulated. We'll share more experiences on that in the future.
Blog
A coding agent can turn a ticket into a green pull request in minutes. We split the merge decision into narrow checks, let fixed rules decide, set the strictness by the blast radius of a change, and bring in a person only where the agent gets stuck.
We ran three coding agents through four gateways to see what a gateway controls, what it measures correctly and where it misleads you, from budgets that overshoot to telemetry that leaks prompts and data residency set hop by hop.
We compared a stronger model, a different agent, answers from the author and a read-only investigation before any code. On 19 fresh tickets, the investigation went ahead on 9 of 10 wanted changes and held back 7 of 9 requests an author had already declined.
With the team behind the first AI for code.
Backed by frontier coding agent research with ETH Zurich.
Cookie settings
By clicking 'Accept all cookies', you agree to storing cookies on your device to enhance site navigation, analyze site usage and assist in our marketing efforts as outlined in our cookie policy.