Research · By Pavel Ivanov · Last updated: 30 October 2026
Inside a software factory
A software factory turns tickets into verified pull requests with nobody at the keyboard. Run with the lights off, it is a dark factory. Here is what it is built from and how a ticket moves through it.
- components
- 24
- layers
- 4
- loops
- 8
- options compared
- 309
Key findings
- 01Cost
Most parts are free and open source. The running cost is model tokens.
- 02Platforms
Devin, Factory, Copilot, Codex cloud and Cursor run in the vendor's cloud. Few self-host.
- 03The gap
No product judges a patch against its ticket. That gate stays custom.
- 04Churn
TensorZero, LLM Guard and Lost Pixel were archived in 2026. Check activity first.
One agent on one job, on repeat: gather context, act, check the result.
The walls around that loop: a sandbox, its tools, memory, and gates that define done.
Many harnessed loops behind a queue, drained through a review gate into production.
The people who own it: they watch it, measure it, and decide what it may do alone.
00 · How it works
Follow a ticket
Pick a case and press Play. Each case is one loop.
01 · What it is built from
24 parts in four layers
Each layer sits inside the next. Select a part to see what it does, the best open-source and paid picks, and the catch.
The outer loop
The people who own the factory: they see it, measure it, set its rules and decide how much it may do alone.
The factory
Many harnessed loops at once: fed by a queue, drained through a review gate into production, with signals flowing back.
The harness
The walls around one loop: the sandbox and its tools, memory between runs, and the gates that define done.
The loop
One agent on one job, on repeat: gather context, act, check the result.
Sandbox and tools
Memory
Gates
02 · The agents
Ten agents, one job each
Each agent is one model with one job, its own instructions and only the access that job needs. Here is every role a factory can have, in the order a ticket meets them. A first factory needs only the coder; the stage says when each of the others joins.
| Agent | What it does | Where | When it runs |
|---|---|---|---|
| Triage agentStage 2 | Reads every new ticket and alert, drops duplicates and tries to reproduce the bug. Then it decides: take it, ask, plan first, or leave it to a person, and picks the model and the budget.Never changes code. | The factoryTriage, risk and autonomy | Every new ticket2Question loop3Plan loop |
| Refinement agentStage 2 | Turns a ticket into one a run can finish: the Done when list, the scope, the size and how much it could break.Never guesses what done means: when the ticket does not say, it asks. | The factoryWork intake | Every ticket triage takes2Question loop |
| PlannerStage 3 | Splits a ticket too big for one run into small, ordered tasks, each its own pull request.Nothing starts until a person approves the plan. | The factoryPlanning and decomposition | Big tickets only3Plan loop |
| Coderalso: implementerStage 1 | The worker. In its own sandbox it reads the code, edits it, runs the tests and reports what it did. Several run at once, one per ticket.Never decides that the work is done, never merges, never holds a real key. | The loopCoding agents | Every attempt1Attempt loop4Review loop |
| Criticalso: AI reviewerStage 2 | A second model, never the one that wrote the code, judges the patch against the Done when list and quotes its evidence. It also points out deleted tests and broken repository rules.Never edits the code: a no sends the patch back to the coder.Measure the critic before you trust it: give it patches you know are wrong and count how many it sends back. A capable model can approve nearly all of them. | The harnessAI review | After the checks pass1Attempt loop5Merge policy |
| QA agentStage 2 | Opens the change in a browser, works through the Done when list item by item, and compares screenshots before and after.Never edits the code: what it finds goes back to the coder. | The harnessUI and browser checks | Changes a person can see1Attempt loop |
| Security reviewerStage 2 | Reads the diff for security problems: secrets, injection, risky new packages, changes to auth.Never approves on its own: what it finds goes to a person. | The harnessAgent security | Every pull request5Merge policy |
| Incident agentStage 3 | Turns an error after a deploy into a ticket, with the stack trace, the pull request that most likely caused it, and a regression test to write.Never fixes production itself: the fix is a new ticket. | The factoryDeploy and signals | Every new production error6Production loop |
| Learning agentStage 3 | Spots the same failure, question or review comment coming back, and proposes a rule for AGENTS.md.A rule lands only after a person approves it. | The harnessMemory and learning | On a schedule7Learning loop |
| Docs agentStage 3 | Updates the docs and the changelog for what a pull request changed.Never touches code: its edits go in the same pull request, to the same reviewers. | The factoryPublishing and human review | Every pull request |
- Checks
- Tests, types, lint and scans give the same answer every time and cannot be talked round. They run before the critic.
- Merge policy
- Rules, not judgment: a change's risk and the factory's record decide what merges alone.
- People
- They answer questions, approve plans and rules, review pull requests and own the factory.
03 · Eight loops
A factory is a set of loops
A ticket's path is a straight line. The loops around it are what make a factory work and get better: each starts with something going wrong or changing, and feeds back into the path. Fast loops sit inside, slow ones outside. One rule applies to every loop: give it only as much autonomy as you can verify cheaply and reliably.
The closer to the middle, the faster a loop turns. Select a ring.
Comprehension debt. When nobody reads the code, the gap between what exists and what people understand grows, with the tests green the whole time. A loop may run with the lights off only when its check is cheap, runs often, cannot be faked, answers at once and does not drift. Keep people in where a wrong answer is expensive or its blast radius is large.
04 · What it can do
What it can take today, and where it stops
A factory does well on work that is small, clear and checked by tests. It stops where a person has to decide, and each of those places is one of the loops above.
Goes straight through
- Bug fixes with a clear reproductionThe agent writes a failing test first, then the fix, and the test proves it.
- Errors from productionA stack trace becomes a ticket with a regression test to write.
- Small UI changesScreenshots before and after, and an accessibility check, judge the result.
- Dependency updatesWhen the existing tests cover the code they touch.
- Clean-ups across a repositoryLint, type and deprecation fixes, split into small pull requests.
- Missing testsTests for code that works today, checked by mutation testing.
Stops for a person
- Unclear ticketsNo Done when list, so the agent asks instead of guessing.2Question loop
- Big changesAgents drift after a few dozen steps, so a person approves a plan of small tickets.3Plan loop
- Risky codeAuth, payments, data and migrations: a person merges, whatever the checks say.5Merge policy
- Judgment callsProduct, naming and design choices come back as review comments.4Review loop
- Code without testsGreen checks prove nothing there, so a person reads the change.
No share is measured yet. To get one, replay 50 merged pull requests as tickets, three runs each, and count how many pass the team's own tests and review. The tasks and their tests often decide more of the result than the agent does.
05 · Where to start
A first factory, in three stages
A draft for the team to argue with, built from the picks in the options. Open source first, so at the start the only bill is model tokens. Each stage works on its own before the next one begins.
- 1
First run
One ticket at a time, end to end. A person reviews and merges every pull request.
10 partsPart Start with Why Work intake GitHub Issues, with a Done when template Work arrives where the team already is, with a clear end. Models Claude Sonnet 5.5 A strong default for coding agents, at $2 / $10 per million tokens. Coding agents Claude Code in API-key mode Headless, with JSON output. opencode if you want any model. Code context AGENTS.md and ast-grep Free, and every agent reads the same rules. Sandboxes Docker with gVisor The easiest step up from plain Docker. Firecracker later, for snapshots. Environments and caching A Dev Container image per repository, with package caches No run waits on installs. Egress and secrets Smokescreen allow list The sandbox reaches only what it needs and never holds a real key. LLM gateway Bifrost A hard budget per ticket: the run stops when it is spent. LiteLLM is the other choice. Verification The repository's own tests, plus Semgrep, Gitleaks and OSV-Scanner Checks the agent cannot skip. Publishing and human review A GitHub App that opens the pull requests Its own identity, kept off the ruleset bypass list. - 2
Many runs, still reviewed
Tickets in parallel, a second model reviewing each patch, and every run traced and priced.
9 partsPart Start with Why Orchestration River on Postgres A light queue with retries. Temporal only once runs wait days. Triage, risk and autonomy Rules in the factory, routing in the gateway Size and risk decide the model and the budget. AI review A second model judging the patch against the ticket No product does this yet. PR-Agent is the baseline to beat. UI and browser checks Playwright and axe-core in the sandbox A UI change is judged by its screenshots. Service dependencies and test data Testcontainers Real databases beside the sandbox, with no network. Test doubles WireMock Outside APIs faked, offline and without rate limits. Agent security Scorecard and deps.dev on every new package Detectors miss attacks, so network and key limits come first. Observability Langfuse with OpenTelemetry Every run traced, every bill explained. Control plane One screen over the queue, with a stop switch River's own UI is enough at first. - 3
Runs on its own
Big tickets planned, lessons kept, and low-risk work merging without a person once evals show it can.
5 partsPart Start with Why Planning and decomposition Spec Kit plans and git-spice stacks A big ticket lands as small stacked pull requests. Memory and learning Rules in AGENTS.md, added only after a person approves them Lessons carry over without turning into noise. Evals Merged pull requests replayed with Harbor Three runs per task: one run is noise. Governance Rulesets and CODEOWNERS Who may merge, and an audit trail of what did. Deploy and signals Feature flags with OpenFeature and Unleash Merged work ships behind a flag, and production errors become tickets.
06 · What it costs
What a ticket costs to run
Most of the bill is model tokens, so the cost follows from a few numbers. The defaults are guesses for a small ticket: replace them with your own traces after the first week.
07 · Options and prices
309 options compared
For every part, the open-source and paid options we checked, with prices from each vendor's own page on 27 October 2026. Every name links to that page; "unverified" marks what the page did not confirm.
The loop
Models12 optionsOpen sourceQwen3-Coder-Next, gpt-oss-120b on vLLMPaidClaude Sonnet 5.5 ($2 / $10 per million tokens)
Claude Sonnet 5.5 ($2 in, $10 out per million tokens) is a strong default for coding agents. For a self-hosted judge or reviewer, Qwen3-Coder-Next and gpt-oss-120b are Apache-2.0 and fit one GPU node; newer Qwen releases moved to a custom license, so check the exact model. GLM-5.3, Kimi K3 and DeepSeek V4.1 need several GPUs and are realistic only hosted.
- Watch out
- Newer Qwen releases use a custom license
| Option | Type | What it is | Pricing (Oct 2026, per million tokens in / out) | Notes |
|---|---|---|---|---|
| Anthropic API | Paid API | Claude models, 5-minute and 1-hour caching | Sonnet 5.5 $2 / $10; Opus 5.5 $4 / $20; Haiku 5.5 $0.10 / $0.50 (up to 100k-token prompts); Fable 5.1 $10 / $50 | Strong default for coding agents |
| OpenAI API | Paid API | GPT-6 family, Codex models, Batch at 50% off | gpt-6.1-sol $2 / $10; gpt-6-astra $10 / $50; gpt-6-luna $0.10 / $0.50; higher above 272k tokens | Strong second provider |
| Google Gemini API | Paid API, free tier | Gemini models, Batch at 50% off | Gemini 4 Argon $2 / $10 until 2026-11-30, then doubles; 3.8 Flash $0.75 / $3.75 until 2026-12-31, then doubles | Promo prices expire; track the end dates |
| Qwen3-Coder-Next | Open weights (Apache-2.0) | Qwen's coding model, 80B | Free weights; Bedrock $0.15 / $1.20 | Best open model for a single GPU node |
| gpt-oss-120b | Open weights (Apache-2.0) | OpenAI's open reasoning model, 120B and 20B | Free weights | Second open candidate for a single node |
| DeepSeek V4.1 Flash | Open weights (MIT) | 763B multimodal model | Free weights; hosted price unverified | Cleanest license; too big for one node |
| GLM-5.3 | Open weights, custom license | 753B, strong on long agentic tasks | Free weights | Needs several GPUs; use it hosted |
| Kimi K3 | Open weights, custom license | 2.8T-parameter flagship | Together $3 / $15 | Needs several GPUs; use it hosted |
| vLLM and SGLang | Open source (Apache-2.0) | GPU inference servers with OpenAI-style APIs | Free | vLLM is the default; SGLang if prefix caching matters |
| llama.cpp and Ollama | Open source (MIT) | Local and CPU inference | Free | Laptops only; Hugging Face TGI is archived |
| Together AI and Fireworks | Hosted open models | Serverless per-token and dedicated GPUs | Together GLM-5.3-Flash $0.15 / $0.50; GPUs $5.49 to $8.99 per hour | Overflow capacity for open models; Groq and Cerebras prices not published |
| AWS Bedrock and OpenRouter | Marketplaces | Many models behind one account | Bedrock DeepSeek v3.2 $0.62 / $1.85; OpenRouter list price plus 5.5% on credits | Only for enterprise billing or a wide catalog |
Coding agents11 optionsOpen sourceopencode, Codex CLI, Cline CLIPaidClaude Code (API tokens)
The agent must run headless, print machine-readable output, and let a gateway meter its model calls. Claude Code and Codex CLI are the strongest, both with documented JSON output. opencode is the best "any model" agent, and Cline CLI is a second. Agents build model API paths differently (some expect the base URL to end in /v1), so send each one through the gateway in a test before a paid run: a mismatch fails every run without spending anything, which looks like a cheap run rather than a broken one. Before switching agents, compare them on the same model over three or more runs per task: the tasks and their tests often decide more of the result than the agent does.
- Watch out
- Use API-key mode so a gateway can meter it
| Option | Type | Headless mode and models | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| Claude Code | Proprietary | claude -p with JSON or stream-json, --max-budget-usd; Claude models only | Subscriptions from $20/month; API Sonnet 5.5 $2 in / $10 out per million tokens | Documented JSON output; route API keys through a gateway to meter it |
| Codex CLI | Open source (Apache-2.0) | codex exec --json JSONL events, --output-schema; OpenAI models | ChatGPT plans from $8/month, or API per token | JSONL events; API-key mode lets a gateway meter it |
| opencode | Open source (MIT) | opencode run --format json, warm serve mode; any provider | Free; you pay the model | Best generic agent: one JSON format for many providers. At its step limit it sends a prefilled assistant message, which Sonnet 5.5 rejects, so the run errors instead of stopping |
| Cline CLI | Open source (Apache-2.0) | --json events and a final run_result line; any model | Free; you pay the model | Second generic option |
| Gemini CLI | Open source (Apache-2.0) | JSON and stream-json output; Gemini models | Unverified | Its README points to a newer CLI for the latest models |
| Qwen Code | Open source (Apache-2.0) | Terminal agent with SDKs; Qwen, OpenAI, Anthropic, Gemini APIs | Free; you pay the model | Fits a local Qwen setup; headless flags unverified |
| Goose | Open source (Apache-2.0, Linux Foundation) | CLI, desktop, API; 15+ providers, 70+ MCP extensions | Free; you pay the model | Check its machine-readable output first |
| mini-SWE-agent | Open source (MIT) | 100-line bash-only agent, over 74% on SWE-bench Verified; any model | Free; you pay the model | Baseline to compare any agent loop against |
| OpenHands | Open source (MIT) | Agent server with a REST API; can run Claude Code, Codex, opencode | Free to self-host | A full platform; a reference more than a building block |
| GitHub Copilot CLI | Proprietary | Copilot agent in the terminal; output format unverified | Pro $10, Pro+ $39, Max $100/month in AI credits | Needs a seat per user; awkward in a factory |
| Cursor CLI | Proprietary | Headless CLI for scripts; output formats unverified | Pricing page looked inconsistent; unverified | Vendor-locked; low priority |
Not covered in depth: Amp and Kilo Code; Aider calls itself a pair-programming tool and fits a factory poorly.
Code context12 optionsOpen sourceAGENTS.md, ast-grep, Serena, tree-sitterPaidSourcegraph (from $16,000/year), Augment
Sandboxes should stay offline, so local tools win: AGENTS.md as the base, ast-grep and optionally Serena as MCP servers in the image, and a tree-sitter repo map in the prompt (Aider's idea). Hosted context engines need egress and upload code; buy one only if it beats local search in your own evals.
- Watch out
- Hosted engines need egress and upload code
| Option | Type | What it does for a factory | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| AGENTS.md | Open format (Linux Foundation) | Repo file with build, test and convention guidance; read by 20+ agents | Free | The baseline; a reviewer can enforce it |
| ast-grep and ast-grep-mcp | Open source (MIT) | Syntax-aware search and rewrite; MCP wrapper is experimental | Free | Strong: local binary |
| Serena | Open source (GPL-3.0) | Symbol-level search and edits over 40+ languages via LSP, as MCP | Free | Big repos; bake language servers into the image |
| tree-sitter | Open source (MIT) | Parser library under most code tools | Free | Build a repo map or symbol index |
| Aider repo map | Open source | Ranks symbols into a compact map, about 1,000 tokens | Free | Copy the idea into the agent's prompt |
| Zoekt | Open source (Apache-2.0) | Fast code search across many repos | Free | Only for cross-repo search |
| Sourcegraph | Hosted or self-hosted | Code Search and Deep Search with an MCP server | Enterprise from $16,000/year | Large multi-repo estates only |
| Augment Context Engine | Hosted | Semantic code index over MCP | Standard $20/month; Business $100/month | Uploads code; needs egress |
| Context7 | Open source client, hosted backend | Current library docs over MCP | Free 1,000 calls/month; Pro $10/seat/month | Allow-list one host, or vendor docs into the repo |
DeepWiki MCP is free for public repos only, and Greptile overlaps an AI reviewer.
Production context over MCP
| Option | Type | What it does for a factory | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| Grafana MCP | Open source (Apache-2.0) | Agents query dashboards, logs, metrics and alerts | Free | Has a true read-only flag |
| Sentry MCP | Open source (MIT); hosted or self-hosted | Agents search errors, issues and traces | Free with a Sentry plan | Includes write tools; scope it |
| Datadog MCP | Hosted | Agents read logs, metrics and monitors | Unverified | Can write, within the user's permissions |
The harness
Sandboxes15 optionsOpen sourcegVisor, Firecracker, E2B infraPaidE2B ($150/month plus usage), Vercel Sandbox, Runloop
The agent needs a disposable machine with the repo, its tools and no way out. gVisor is the easiest upgrade over plain Docker. Firecracker microVMs add true snapshot and fork, with E2B's open-source infrastructure as the reference design. Hosted sandboxes are quick to adopt but not self-hosted, and Daytona's open repo was archived in June 2026.
- Watch out
- Daytona's open repo was archived in June 2026
| Option | Type | Isolation and snapshots | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| gVisor | Open source (Apache-2.0) | User-space kernel in Go; OCI runtime for Docker; checkpoint and restore | Free | Easiest first step: a drop-in Docker runtime; benchmark package installs |
| Firecracker | Open source (Apache-2.0) | microVMs with memory and disk snapshots; used by E2B and Vercel | Free | Restore a warmed repo instead of reinstalling; you write the VM management |
| Kata Containers | Open source (Apache-2.0) | Each container in a light VM; works with containerd | Free | Strongest isolation with a container workflow; needs KVM |
| E2B | Open source infra (Apache-2.0) and hosted | Firecracker with pause and snapshot; self-host on one KVM host | Pro $150/month plus $0.000014 per vCPU-second | Best self-hostable reference design |
| kubernetes-sigs/agent-sandbox | Open source (Apache-2.0) | Sandbox CRDs and warm pools on gVisor or Kata; snapshots on the roadmap | Free | Warm pools of sandboxes; only on Kubernetes |
| microsandbox | Open source (Apache-2.0), beta | libkrun microVMs with copy-on-write fork | Free | Watch; it warns of breaking changes |
| Sysbox | Open source (Apache-2.0) | User-namespace runtime; Docker inside containers | Free, community support only | Weaker boundary; only for Docker-in-Docker |
| Vercel Sandbox | Hosted | Firecracker microVMs, snapshots, 64 GB disk | Pro $0.128/CPU-hour; snapshots $0.08/GB-month | Cleanest hosted snapshot model |
| Runloop | Hosted | Devboxes, prebuilt blueprints, snapshots, suspend and resume | Pro $250/month plus $0.108/CPU-hour | Blueprints fit per-repo setup; VPC only on Enterprise |
| Modal Sandboxes | Hosted | Programmable sandboxes | Team $250/month; sandboxes $0.00003942/core-second | Costly for long idle agent runs |
| Daytona | Hosted; open repo archived June 2026 | Dedicated kernel, stateful snapshots | $0.0504/vCPU-hour, $200 free credits | Self-hosting is now a dead end |
| Cloudflare Containers | Hosted | Containers behind Workers; up to 4 vCPU, 12 GiB | Workers Paid $5/month plus usage | Too small for monorepo installs |
Also priced: Blaxel (resume in about 25 ms, snapshots $0.20/GB-month), Northflank (bring your own cloud), Fly Machines (Firecracker). Morph Cloud and CodeSandbox prices could not be verified.
The fleet under the sandboxes
| Option | Type | What it does for a factory | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| Kubernetes with Karpenter | Open source (Apache-2.0) | Adds and removes nodes as sandboxes queue, spot included | Free; cloud nodes extra | You run the cluster |
| HashiCorp Nomad | Source-available (BUSL 1.1) | Schedules containers and VMs without Kubernetes | Community edition free | Now IBM; the license bars competing products |
| Spot and preemptible VMs | Cloud service | Cheap interruptible machines for sandbox hosts | Up to 91% off on Google Cloud, up to 90% on AWS | Can be reclaimed with little notice |
Environments and caching9 optionsOpen sourceuv and bun caches, Dev Containers, ZotPaidDepot, Namespace
The biggest speed and disk win is to stop installing per ticket. Build one base image per repo, keyed by the lockfile hash; mount shared package caches; and store each ticket as a copy-on-write diff instead of a full image, which fills disks fast. All of this is free tooling. The paid services speed up CI, not sandboxes. Generated code, such as ORM clients and protobuf stubs, belongs in the base image or the first check; otherwise the tests fail before the agent has changed anything.
- Watch out
- A full image per ticket fills disks fast
| Option | Type | What it does for a factory | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| uv cache and Bun cache | Open source | Shared package caches that link files instead of copying | Free | Highest payoff for least effort; keep the cache on the same filesystem |
| Dev Containers spec | Open spec (MIT) | devcontainer.json per repo, with prebuilt images | Free | Standard way to describe each repo's environment |
| mise | Open source (MIT) | Pins node, python and bun versions with a lockfile | Free | Simple toolchain pinning inside the base image |
| Zot | Open source (Apache-2.0) | Light self-hosted OCI registry | Free | Holds per-repo base images |
| Nydus and stargz-snapshotter | Open source (Apache-2.0) | Lazy image pulls and chunk dedup on containerd | Free | Pays off once many tickets share a big base image |
| Devbox, Nix and Flox | Open source | Reproducible, cacheable environments | Free | Strongest reproducibility; steep for npm, bun and uv repos |
| Depot | Hosted | Container builds, runners, sandboxes with a persistent cache | Developer $20/month; builds $0.04/minute | Only if image builds become the bottleneck |
| Namespace | Hosted | Runners, builds, devboxes with snapshots | Team $100/month; devboxes from $0.004/minute | Its snapshot model is worth studying |
| Blacksmith and BuildJet | Hosted | Faster GitHub Actions runners and caches | From $0.004/minute | CI only; skip for sandboxes |
Egress and secrets11 optionsOpen sourceSmokescreen, Infisical Agent Vault, OpenBaoPaidDoppler, Infisical cloud
Agents should reach only an allow list and never hold a real key. Stripe's Smokescreen (Go, MIT) is a solid allow-list proxy; add host firewall rules so the sandbox can reach nothing but the proxy. Infisical Agent Vault and OneCLI are new open-source tools built on the same idea: the agent holds a placeholder, and the real key is injected outside.
- Watch out
- Agent Vault is still a preview
| Option | Type | What it does for a factory | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| Stripe Smokescreen | Open source (MIT) | Go CONNECT proxy; host allow lists per role; blocks private IPs | Free | Embed or fork; it does not inject credentials |
| Infisical Agent Vault | Open source (MIT), preview | Built for agents: swaps placeholder credentials for real ones on outbound calls | Self-host free; cloud 5,000 requests/month free, then $2 per 1,000 | Placeholder-key design; a preview, API may change |
| OneCLI | Open source core (Apache-2.0), enterprise license | Built for agents: Rust gateway injects real credentials | Core free; enterprise price not published | Study the design; check the license split |
| agentgateway | Open source (Apache-2.0, Linux Foundation) | Built for agents: LLM and MCP traffic with auth and CEL policies | Free | Candidate for MCP tools later |
| OpenBao | Open source (MPL-2.0) | Vault fork in Go: dynamic secrets, leases, revocation | Free | When a secrets file is no longer enough |
| SPIFFE/SPIRE | Open source (Apache-2.0, CNCF) | Workload identity for mTLS and JWT | Free | Heavy; a per-run token is simpler |
| Squid and Envoy | Open source | General forward proxies with ACLs | Free | Work, but heavier than Smokescreen |
| Cilium | Open source (Apache-2.0, CNCF) | FQDN egress policies for Kubernetes | Free | Only on Kubernetes |
| Doppler | Hosted | Secrets store | Free 3 users, then $8/user/month; "AI agents ride free" | Any of these can hold a gateway's keys |
| Infisical Secrets | Open source and hosted | Secrets store | Free 5 identities; Pro $20/identity/month | Self-hostable |
| AWS Secrets Manager | Hosted | Secrets store | $0.40/secret/month plus $0.05 per 10,000 calls | Only on AWS |
Agent security12 optionsOpen sourceLlamaFirewall; Scorecard and deps.dev for new packagesPaidSocket for packages; Lakera Guard for prompts
Agent security has two jobs: stop hostile text (issue bodies, READMEs, web pages, MCP output) from steering the agent, and stop the agent from installing invented or malicious packages. Meta's LlamaFirewall, NVIDIA NeMo Guardrails, Snyk Agent Scan, OpenSSF Scorecard and deps.dev are the active open-source pieces; Socket and Sonatype Repository Firewall cover packages, and Lakera Guard screens prompts at runtime. The catch: no detector stops injection reliably, so design matters more. The model that acts should never read untrusted text directly (the CaMeL and dual-LLM patterns), and the sandbox should limit network and credentials. The field churns: LLM Guard is archived, and Lakera, Protect AI, Invariant and Phylum were all acquired.
- Watch out
- Detectors miss attacks; limit network and keys first
| Option | Type | What it does for a factory | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| LlamaFirewall with Prompt Guard 2 | Open source (MIT; model weights under Llama terms, unverified) | Injection classifier, goal-hijack check and code scanner | Free | Active; no tagged release; needs a model download |
| NeMo Guardrails | Open source (Apache-2.0) | Input, output and tool rails with jailbreak and injection checks | Free | Active; chat-oriented, needs tuning for agents |
| Lakera Guard | Hosted; now part of Check Point | Screens prompts and documents for injection at runtime | Free tier; paid prices unverified | Claims under 50 ms; text leaves your network |
| Snyk Agent Scan | Open source CLI (Apache-2.0); needs a Snyk token | Scans MCP servers and agent skills for poisoning | Free with a Snyk account | Formerly mcp-scan; active |
| Invariant Guardrails | Open source (Apache-2.0) | Rules that block risky sequences of tool calls | Free | Invariant is now part of Snyk; no commits since January 2026 |
| Guardrails AI | Open source (Apache-2.0); Pro hosted | Validates and structures model output | Free; Pro price unverified | Checks output, not an injection defense |
| CaMeL | Research code (Apache-2.0) | The planning model never reads untrusted data; data flow is enforced | Free | A design to copy, not a maintained library |
| Socket | Hosted; CLI open source (MIT) | Flags malicious, typosquatted and risky new packages | Free; Team $25, Business $50 per seat/month | Catches invented package names before install |
| Sonatype Repository Firewall | Hosted | A proxy that blocks malicious components on download | Pro from $4,800/year | Covers npm, Maven, PyPI and NuGet |
| JFrog Curation | Hosted or self-managed | Blocks risky packages, models and MCP servers at the proxy | Enterprise add-on; price unverified | Starts at 200 developers |
| deps.dev API | Free Google API | Package age, advisories and provenance for vetting | Free; limits unverified | Metadata only; blocks nothing by itself |
| OpenSSF Scorecard | Open source (Apache-2.0) | Scores a dependency's security habits from 0 to 10 | Free | A good score does not prove a package is safe |
LLM Guard was archived in July 2026 (Protect AI is now part of Palo Alto Networks), and Phylum's package firewall now lives inside Veracode (pricing unverified).
LLM gateway13 optionsOpen sourceBifrost (Go), LiteLLM (Python)PaidPortkey (budgets on Enterprise), Vercel AI Gateway
A gateway keeps provider keys out of the sandbox, prices every call and stops a run when its budget is spent. Only two open-source gateways confirm hard per-key budgets: Bifrost (Go, answers 402) and LiteLLM (Python). Hosted gateways are simpler but put a third party in every call. TensorZero was archived in June 2026.
- Watch out
- TensorZero was archived in June 2026
| Option | Type | Hard budget per key | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| Bifrost | Open source (Apache-2.0), Go | Yes: virtual keys return 402, reset periods 1 minute to 1 year | Self-host free; hosted price not published | Closest to a hard-budget gateway; behaviour on unpriced models unverified |
| LiteLLM proxy | Open source (MIT core, enterprise folder), Python | Yes: max_budget, optional fail-closed check | Self-host free; Enterprise not published | Most widely used; team budgets answer 422 and alerts are enterprise-only |
| Envoy AI Gateway | Open source (Apache-2.0) | Token rate limits and "budget controls"; USD hard cap unverified | Free | Brings Envoy and Kubernetes; heavy |
| Portkey | Open source gateway (MIT, TypeScript) and cloud | Enterprise tier only | Free 10,000 logs/month; Production $49/month | Budgets are paid-only |
| Helicone | Open source (Apache-2.0) and cloud | Unverified | Free 10,000 requests; Pro $79/month | Logging first; its page notes an ownership change |
| Kong AI Gateway | Open core, mostly enterprise | Token and spend quotas; open-source scope unverified | Not published | Heavy unless you already run Kong |
| Vercel AI Gateway | Hosted | Budgets per team, project and key | No token markup; BYOK free | Hosted |
| Cloudflare AI Gateway | Hosted | None mentioned | Core free; 5% fee on unified billing | Hosted |
| OpenRouter | Hosted | Listed as a plan feature; per-key cap unverified | List price plus 5.5% on credits | Hosted |
| TensorZero | Open source (Apache-2.0), Rust | Not checked | Free | Archived 2026-06-12; do not adopt |
Personal data and retention
| Option | Type | What it does for a factory | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| Microsoft Presidio | Open source (MIT) | Finds and masks personal data before a prompt leaves | Free | No guarantee it finds everything |
| Anthropic zero data retention | Contract term | No retention of inputs and outputs on eligible APIs and Claude Code | Through sales | Approved per organization |
| OpenAI zero data retention | Contract term | Zero retention for eligible API data | Through approval | Stored objects such as threads are excluded; default logs kept 30 days |
Service dependencies and test data12 optionsOpen sourceTestcontainers or Compose sidecars; Greenmask for dataPaidNeon branching; Tonic.ai for masked data
Integration tests need real services (Postgres, Redis, queues) with realistic data, beside a sandbox that has no internet. Testcontainers and plain Docker Compose sidecars are the simplest open-source answer; DBLab Engine and Xata add fast copy-on-write Postgres clones, and Greenmask masks real dumps offline. Neon and Supabase branching and Tonic.ai are the strongest paid choices. The catch: most branching and masking products are cloud services, so an offline sandbox needs its images, dumps or clones prepared ahead. Give each sandbox its own service containers on an internal network with no route out, so a test database never becomes a way to the internet.
- Watch out
- Cloud branching needs network; Neosync is archived
| Option | Type | What it does for a factory | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| Testcontainers | Open source (MIT for Go and Node, Apache-2.0 for Python) | Starts throwaway Postgres, Redis or Kafka containers from test code | Free | Needs a Docker daemon the tests can reach; pre-pull images |
| Testcontainers Cloud | Hosted (Docker) | Runs those containers in the cloud instead of locally | Unverified | Needs network, so not for an offline sandbox |
| Docker Compose sidecars | Open source; self-hosted | Postgres, Redis and queues declared beside the sandbox | Free | Offline once images are cached; no data cloning |
| DBLab Engine | Open source (Apache-2.0) | Thin Postgres clones in seconds, any database size | Free; hosted price unverified | Needs ZFS or LVM on the host |
| Neon | Open source engine (Apache-2.0) and hosted; owned by Databricks | A copy-on-write Postgres branch per ticket | Free plan; Launch $0.106/CU-hour; Scale $0.222/CU-hour; $1.50 per extra branch-month | Needs network access to Neon |
| Supabase branching | Hosted (platform Apache-2.0) | A branch per PR with its own Postgres, auth and storage | Pro from $25/month; branches $0.01344/hour; not on Free | Branches start empty and need seed files |
| Xata | Open source (Apache-2.0) and hosted | Copy-on-write Postgres branches that scale to zero | Free self-hosted; cloud from $0.012/hour plus $0.28/GB-month | Self-hosting needs Kubernetes |
| PlanetScale | Hosted | Managed Postgres and Vitess with branching | Postgres from $5/month per cluster; branch pricing unverified | Prices clusters, not branches |
| Postgres template databases | Built into Postgres | Copies a seeded database fast with CREATE DATABASE ... TEMPLATE | Free | The template must have no open connections |
| Greenmask | Open source (Apache-2.0) | Dumps Postgres or MySQL with masking, subsetting and synthetic data | Free | Single binary; works offline |
| Tonic.ai | Hosted; self-hosted on Enterprise | Masks production data and generates synthetic test data | Fabricate free or $29/month; Structural on quote | The database product is quote-only |
| Synthesized | Hosted or in your environment | Masked, subset and generated test data across databases | On request | Sales-led; focused on enterprise and SAP |
Neosync was archived on 30 August 2025 after an acquisition; do not start new use.
Test doubles9 optionsOpen sourceWireMock, Microcks, HoverflyPaidWireMock Cloud, Speedscale
Agents need fakes of outside services to test integrations without a network: WireMock for HTTP, Microcks when a repo has OpenAPI, AsyncAPI or gRPC specs. Build full twins only where a recorded stub is not enough. One software factory takes this furthest: its "digital twin universe" clones Okta, Jira, Slack and Google Docs, so agents run thousands of scenarios an hour with no rate limits.
- Watch out
- Testcontainers needs Docker in the sandbox
| Option | Type | What it does for a factory | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| WireMock | Open source (Apache-2.0); cloud paid | HTTP stubs with record and playback, faults, state | Free; cloud free at 1,000 calls/month | Best general HTTP double; offline |
| Microcks | Open source (Apache-2.0, CNCF) | Mocks and contract tests from OpenAPI, AsyncAPI, gRPC, GraphQL | Free self-hosted | Spec-driven mocks |
| Testcontainers for Go | Open source (MIT) | Real dependencies (Postgres, Redis) as containers from tests | Free | Needs Docker inside the sandbox |
| Hoverfly | Open source (Apache-2.0), Go | Captures traffic and replays it | Free | Light, embeds well |
| MockServer | Open source (Apache-2.0) | HTTP, gRPC, WebSocket mocks with failure injection | Free | Overlaps WireMock |
| Prism | Open source (Apache-2.0) | Mock server and validating proxy from OpenAPI | Free | Good when OpenAPI specs exist |
| Mockoon | Open source (MIT); Pro paid | Mock APIs with a headless CLI | CLI free; Pro unpublished | Easy in CI; weaker record and replay |
| LocalStack | Paid; open repo archived 2026-03-23 | Local AWS emulator | Hobby free (non-commercial); Base $39/month | Only for AWS; commercial use is paid |
| Speedscale proxymock | Free local tool; paid cloud | Records real traffic, replays as mocks | proxymock free; tiers unpublished | Cloud tiers do not fit offline sandboxes |
Memory and learning20 optionsOpen sourceAGENTS.md; Mem0, Graphiti or Cognee; pgvectorPaidCodeRabbit learnings, Qodo rules
A factory's memory has two jobs: give every fresh sandbox the project's rules, and keep the lessons of earlier runs and reviews. Because the sandbox is thrown away, plain files in the repo (AGENTS.md, CLAUDE.md, rules folders) are the only memory every headless agent reliably reads; per-user cloud memory and machine-local memories do not travel with it. Mem0, Graphiti and Cognee are the strongest open-source memory layers, and CodeRabbit and Qodo learn review rules in a form a person can see and approve. The catch: memory that agents write without review drifts and is a prompt-injection path, so learned rules should land as reviewed pull requests or approved entries. Hosted fine-tuning is fading: OpenAI stops new fine-tuning jobs on 6 January 2027.
- Watch out
- Unreviewed memory turns comments into lasting rules
| Option | Type | What it does for a factory | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| AGENTS.md | Open format (Linux Foundation); a repo file | Build, test and style rules every agent reads | Free | Survives throwaway sandboxes; changes go through PR review |
| Claude Code memory | Proprietary; CLAUDE.md and rules in the repo, auto memory in local files | Repo instructions every session; auto memory notes what it learned | Part of Claude Code | CLAUDE.md travels with the repo; auto memory stays on one machine |
| Cursor rules | Proprietary; rules are repo files | Project, user and team rules; also reads AGENTS.md | Cursor plans; pricing page unverified | Cursor removed its Memories feature in version 2.1; rules remain |
| GitHub Copilot Memory | Proprietary; hosted | The cloud agent reuses repo facts it learned | Pro $10, Pro+ $39, Max $100 per month | Entries expire after 28 days unused; owners can review them |
| Devin Knowledge | Proprietary; hosted | Org-wide tips recalled by trigger; suggests new entries | Pro $20, Max $200, Teams $80 plus $40/user per month | Moving to Skills in Plugins; stored in the vendor's cloud |
| Mem0 | Open source (Apache-2.0) and hosted | Stores and recalls facts across runs through an API | Free 50,000 adds and 10,000 retrievals per month; paid tier on request | Very active; needs a vector store; graph memory only hosted and paid |
| Letta | Open source (Apache-2.0) and hosted | Stateful agents with editable memory blocks | Free; Pro $20/month; API $20/month plus $0.10 per active agent | Code moved to letta-code; check which repo is maintained |
| Zep | Hosted | Time-aware knowledge graph memory as a service | Free 10,000 credits/month; Flex $125/month; Flex Plus $375/month | Self-hosting means Graphiti; credits billed by input size |
| Graphiti | Open source (Apache-2.0) | Builds a time-aware knowledge graph from past events | Free | Active; needs Neo4j, FalkorDB or Neptune |
| Cognee | Open source (Apache-2.0) and hosted | Graph and vector memory built from documents and code | Cloud free for 1M tokens; Standard $1 per 1M tokens | Active; can run on Postgres with pgvector |
| LangMem | Open source (MIT) | Extracts memories and refines agent prompts over time | Free | No releases yet; tied to LangGraph's store |
| Anthropic memory tool | Claude API feature; you host the storage | Claude reads and writes files in a memories folder | Token pricing only | You own storage and path checks; stored text can carry injections |
| Supermemory | Open source (MIT) and hosted | Memory and search API with connectors | Free with $5 credits; Pro $19; Max $100; Scale $399 per month | Young (v0.0.x); self-hosting listed under Scale |
| CodeRabbit learnings | Hosted | Learns review preferences from PR comments | Essentials $24, Team $48, Advanced $72 per user/month (annual) | Learnings can be edited and exported as CSV |
| Greptile | Hosted; self-host on Enterprise | Learns from comments, replies, reactions and commits | Free 50 credits/month; Pro $1 per credit | Learned rules shown on a dashboard; export unverified |
| Qodo rules | Hosted; on-prem on Enterprise | Mines PR history into rules its review enforces | Pro Team $30/month (up to 30 users) plus credits | A person approves each suggested rule before it applies |
| pgvector and Qdrant | Open source (PostgreSQL License, Apache-2.0); Qdrant Cloud | Search past tickets and runs by similarity | Free; Qdrant Cloud free tier, then usage-based (unverified) | pgvector is simplest where Postgres already runs |
| OpenAI fine-tuning | Hosted | Trains a hosted model on accepted changes | Training rates unverified | No new jobs after 6 January 2027; closed to new users since May 2026 |
| Together AI fine-tuning | Hosted | LoRA or full fine-tuning of open models | $0.34 to $40 per 1M training tokens by model size | Code leaves your network; hosting the result is billed separately |
| Unsloth and Axolotl | Open source (Apache-2.0; Unsloth Studio is AGPL-3.0) | Fine-tune open models on your own GPUs | Free; GPU cost extra | Needs hundreds of good examples first |
Also checked: Windsurf and Augment keep team knowledge as rule files in the repo, like Cursor; MemOS needs Neo4j and Qdrant; Memobase remembers users rather than repositories.
Verification19 optionsOpen sourceSemgrep, Gitleaks, OSV-Scanner; fast-check, Hypothesis; StrykerPaidSemgrep Teams ($30/contributor/month), SonarQube Cloud
"Done" must come from checks the agent cannot skip. Next to a repo's own tests, typecheck and lint, add a fixed offline gate: Semgrep, Gitleaks, and OSV-Scanner or Trivy. Ask agents for property-based tests (fast-check, Hypothesis), and run mutation testing (Stryker, mutmut) on changed files to catch hollow tests. Formal methods (TLA+, Dafny, Lean) only fit narrow modules with a written spec; Verus and Kani are Rust-only.
- Watch out
- CodeQL on private repos needs Advanced Security
| Option | Type | What it does for a factory | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| Semgrep | Open source engine (LGPL-2.1), paid rules | Pattern-based static analysis CLI; runs offline with local rules | Free up to 10 contributors; Teams from $30/contributor/month | High: can also encode a repo's own rules as checks |
| Gitleaks | Open source (MIT), Go | Secret scanner over the diff or history | Free | High: feature complete; successor is Betterleaks |
| OSV-Scanner | Open source (Apache-2.0) | Dependency vulnerabilities with an offline database | Free | High: bun.lock and uv.lock support unverified |
| Trivy | Open source (Apache-2.0) | Vulnerabilities, secrets, IaC, SBOM in one binary | Free | High |
| TruffleHog | Open source (AGPL-3.0) | 800+ detectors that test if a secret is live | Free | Needs egress to verify; Gitleaks is simpler |
| fast-check and Hypothesis | Open source | Property-based tests for TypeScript and Python | Free | High: test libraries the agent can use |
| Stryker and mutmut | Open source (Apache-2.0, BSD-3) | Mutation testing: do the new tests fail on bugs? | Free | Medium: slow, run on changed files only |
| Dagger | Open source (Apache-2.0) | Programmable container pipelines with a Go SDK | Engine free | High for reproducible check environments |
| TLA+, Dafny, Lean 4 | Open source | Model checking and proofs | Free | Narrow, spec-able modules only, such as a run state machine |
| SonarQube | Open core and cloud | Quality gate and static analysis server | Cloud free to 50k lines; Team from $34/month | Heavy server for per-ticket gates |
| CodeQL | Free on public repos; private needs GitHub Advanced Security | Deep semantic analysis | Advanced Security price unverified | License blocks private repos without GHAS |
| Snyk Code | Hosted | Static analysis and dependency scanning | Free 100 tests/month; Team from $25/month | Code leaves your network; free limits are small |
| Holdout scenarios | A practice | Plain-English acceptance scenarios the agent never sees, judged by a separate model | Free; model tokens per run | Run each 3 times and require 2 of 3 to pass |
CI services (GitHub Actions at $0.006/minute for Linux, Buildkite from $30/user/month) stay a second opinion after the PR; the factory should decide "done" before it.
Flaky tests and running only the affected tests
| Option | Type | What it does for a factory | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| Trunk Flaky Tests | Hosted | Detects and quarantines flaky tests; comments on PRs | Free up to 5 committers; Team price unverified | Quarantine is in the free tier |
| BuildPulse | Hosted | Detects flaky tests from CI results | Startup $99/month; Team $249/month | Priced by test volume |
| Datadog Test Optimization | Hosted | Flaky test management inside Datadog | $20 per committer/month (annual) | Best if Datadog is already used |
| Nx affected and Turborepo | Open source (MIT) | Run only the tasks whose inputs changed | Free | JavaScript monorepos |
| pytest-testmon | Open source (MIT) | Reruns only the Python tests that touch changed code | Free | Needs its database kept between runs |
| CloudBees Smart Tests | Hosted | A model picks the relevant tests for each change | Unverified | Formerly Launchable |
UI and browser checks14 optionsOpen sourcePlaywright and axe-core inside the sandboxPaidChromatic or Argos for visual review
A reviewer can only judge a UI change by seeing it: a headless browser loads the changed app, runs end-to-end tests, takes screenshots and compares them with the base. Playwright with its MCP server, plus axe-core, is the strongest open-source set, and it runs fully offline in a sandbox once the browsers are in the image. Chromatic, Argos and Percy are the paid visual-review services, and Browserbase is the leading hosted browser. The catch: review dashboards, hosted browsers and AI browser agents all need network access, Lost Pixel was archived in April 2026, and BackstopJS is looking for a maintainer.
- Watch out
- Hosted browsers and AI agents need network
| Option | Type | What it does for a factory | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| Playwright | Open source (Apache-2.0) | End-to-end tests and screenshots in Chromium, Firefox and WebKit | Free | Offline once the browsers are baked into the image |
| Playwright MCP | Open source (Apache-2.0) | Lets a model drive a browser through accessibility snapshots | Free | Offline apart from the install; the model is extra |
| Cypress | Open source runner (MIT); Cypress Cloud hosted | End-to-end tests with video and screenshots | Runner free; Cloud Team $799/year, Business $3,199/year | Runner works offline; the dashboard needs network |
| Chromatic | Hosted | Visual diffs and review of Storybook stories per PR | Free 5,000 snapshots; Starter $179/month; Pro $399/month | Cloud only |
| Percy | Hosted (BrowserStack) | Cross-browser visual diffs per build | Unverified (pricing page did not load) | Cloud only |
| Argos | Open-source core and hosted | Visual diffs for Playwright and Cypress screenshots | Hobby free (5,000 screenshots); Pro $100/month | Has a CLI and an API agents can use |
| Applitools | Hosted | AI visual validation across browsers | Starter $667/month (annual) | Costly; free only as a trial |
| BackstopJS | Open source (MIT) | Screenshot comparison across screen sizes, offline | Free | Looking for a new maintainer |
| axe-core and Pa11y | Open source (MPL-2.0, LGPL-3.0) | Accessibility checks inside a test or against a URL | Free | Offline; cheap to add to every UI ticket |
| Lighthouse CI | Open source (Apache-2.0) | Accessibility, performance and SEO audits per commit | Free | Offline |
| Browser Use | Open source (MIT) and hosted | An AI agent that clicks through the app to check it | Library free; cloud browsers $0.02/hour | Needs a model; offline only with a local one |
| Stagehand | Open source (MIT) | AI-steered Playwright actions | Free; model cost extra | Needs a model API |
| Browserbase | Hosted | Cloud browsers with session replay | Free 1 hour; Developer $20/month; Startup $99/month | Not reachable from an offline sandbox |
| Steel | Open source (Apache-2.0) and hosted | A browser API with a Docker image for agents | Free plus $0.10 per browser-hour; Scale $250/month | The self-hosted image runs offline |
Lost Pixel was archived on 22 April 2026, and Browserless (from $25/month) self-hosts only on Enterprise.
AI review10 optionsOpen sourceQodo PR-Agent, Claude Code ActionPaidCodeRabbit ($24/user/month), Greptile
A second model should judge each patch against its ticket, and flag deleted tests or broken repo rules with quotes. No product does exactly that: commercial reviewers comment on PRs for humans. Self-hosted PR-Agent and the Claude Code actions work as second opinions, or as a baseline to measure a custom judge against. Measure the judge itself, too: re-check what it passed against tests it never saw. A judge that reads the ticket still passes some wrong patches, so its pass is evidence, not proof.
- Watch out
- No product judges a patch against a ticket
| Option | Type | What it does | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| Qodo PR-Agent | Open source (MIT, community-owned) | review, improve, ask on a PR; any model through LiteLLM | Free; you pay the model | High: self-hosted reviewer; needs a PR URL, not a bare diff |
| Claude Code Action | Open source (MIT) | Claude reviews or implements on PRs and issues | Free; you pay the model | High: a judge on your own runners |
| Claude Code Security Review | Open source (MIT) | Security-focused diff review with false-positive filtering | Free; you pay the model | Reuse its prompts; its README warns it is not hardened against prompt injection |
| Codex CLI | Open source (Apache-2.0) | Headless review of a local diff | Plans from $8/month, or API | Cross-model second opinion |
| CodeRabbit | Hosted | Agentic PR, CLI and IDE review, security analysis | Essentials $24/user/month (annual); Team $48 | Has a CLI; no ticket-aware judging |
| Greptile | Hosted, closed source | Review with whole-codebase context | Free 50 credits/month; Pro $1/credit | Self-host on Enterprise only |
| Graphite AI reviews | Hosted | Stacked PRs plus AI review | Starter $20; Team $40/user/month | Now part of Cursor; bound to the GitHub workflow |
| GitHub Copilot code review | Hosted | PR review, local review in IDEs | From Pro $10/month | No headless API for arbitrary diffs |
| Cursor Bugbot | Hosted | PR bug review | Usage-based; unverified | Not self-hostable |
| Sourcery | Hosted | AI review and security scans | Pro $12 to $15/developer/month | Seat pricing; self-host on Enterprise |
The factory
Work intake10 optionsOpen sourceGitHub Spec Kit templates; GitHub Issues APIPaidLinear (free; Basic $10/user/month)
A factory needs tickets from a tracker it can read, each with a clear "Done when". GitHub Issues is the default. Linear is the best source for custom agents: you can assign or @mention an agent and it sends a webhook, so assigning an issue can start a run. GitHub does that only for its own Copilot agent. No tool checks ticket clarity for you; GitHub Spec Kit's templates are a good base for one.
- Watch out
- GitHub assigns issues natively only to Copilot
| Option | Type | What it does for a factory | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| GitHub Issues | Paid platform, free tier | Issues and PRs over REST and webhooks; assign to Copilot natively | Free; Team $4/user/month (first 12 months); Enterprise from $21/user/month | The default source; native assignment only for Copilot |
| Linear | Paid, free tier | Assign or @mention an agent; agent API, webhooks; AI triage on Business | Free (2 teams, 250 issues, agents included); Basic $10; Business $16/user/month | Best source for custom agents: assign an issue, get a webhook |
| Shortcut | Paid, free tier | Tracker with API and webhooks | Free up to 10 users; Team $8.50; Business $12/user/month | API and webhooks; no agent hooks |
| Jira with Rovo | Paid, free tier | REST and webhooks; Rovo agents inside Jira | Free up to 10 users; paid tiers unverified (the page did not load) | Common in enterprises; needs its own importer |
| GitLab with Duo Agent Platform | Open core and paid | Issues and MRs over API and webhooks; Duo agents | Free (5 users); Premium $29/user/month | Needs a GitLab importer and a GitLab publisher |
| GitHub Spec Kit | Open source (MIT) | Spec, plan and task templates for agents | Free | Good base for a ticket clarity check |
| Kiro specs | Paid, free tier | Requirements, design and task files in EARS style | Free 50 credits; Pro $20/month for 1,000 credits | Reference for acceptance-criteria wording only |
Automatic ticket sources
| Option | Type | What it does for a factory | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| Renovate | Open source (AGPL-3.0); Mend's hosted app | Opens dependency update PRs for many ecosystems | Free; Mend Enterprise on request | The broadest choice |
| Dependabot | GitHub feature (core MIT) | Version and security update PRs | Included with GitHub | GitHub only; fewer options than Renovate |
| Scheduled sweeps | A practice | A nightly or weekly job opens one small ticket per anti-pattern or deprecation | Free | Small, readable PRs; a safe first use of a factory |
Triage, risk and autonomy12 optionsOpen sourceLiteLLM or Bifrost to route; gitStream or Kodiak to mergePaidLinear Triage Intelligence; Cursor approval agents
Triage decides which tickets the factory takes, which model and budget each gets, and which resulting pull requests may merge without a person: the step that makes a factory truly dark. LiteLLM and Bifrost route between models with budgets and fallbacks, OpenRouter's auto router adds no fee beyond its credit fee, and Linear's Triage Intelligence sorts incoming issues. For merging, GitHub auto-merge with rulesets, Mergify, Kodiak and LinearB's gitStream apply rule-based gates, and Cursor's approval agents add a risk score before approving low-risk changes. One published playbook waits for a 90% scenario pass rate over the last 20 PRs, under 5% false positives and under 10% human overrides, then has a person audit the first 50 auto-merges. Enable auto-merge for the exact commit that passed (GitHub's API takes the expected head commit), so a later push cannot ride on earlier checks. The catch: routers judge the prompt, not the ticket's risk; autonomy levels exist only as published frameworks; and every risk policy must be tuned against the team's own history.
- Watch out
- Routers see the prompt, not the ticket's risk
| Option | Type | What it does for a factory | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| LiteLLM | Open source with an enterprise tier | Budgets, virtual keys, fallbacks and routing by complexity | Free; Enterprise unverified | The auto router is in beta |
| Bifrost | Open source (Apache-2.0) with enterprise add-ons | Failover, rule-based routing and budget keys, in Go | Free; Enterprise on request | Rules, not learned routing |
| OpenRouter Auto Router | Hosted | Sorts prompts into about 30 task types and picks a model | No router fee; 5.5% fee on credits | Your own keys free up to $25,000/month list price |
| Not Diamond | Hosted; private on Enterprise | Learns which model suits each prompt | $0.05 per million routed tokens (waitlist) | Adds 100 to 150 ms per request |
| RouteLLM | Open source (Apache-2.0) | Routes between a strong and a cheap model | Free | No commits since August 2024 |
| Linear Triage Intelligence | Hosted | Suggests team, assignee, labels and duplicates; can apply them | Business $16/user/month | Needs Business or Enterprise |
| GitHub Agentic Workflows | Open source (MIT) on GitHub Actions | Agent workflows that label and triage new issues, with budget caps | Free; AI credits $0.01 each | Technical preview |
| GitHub auto-merge and rulesets | Platform feature | Merges once required reviews and checks pass | Included; Team $4/user/month | The simplest gate; no risk score |
| LinearB gitStream | Open source rules (Apache-2.0), hosted app | YAML rules that label risk and auto-approve safe PRs | Free with a monthly limit | The most explicit risk rules |
| Mergify | Hosted | Rule-based auto-merge, merge queue and batching | Free up to 5 contributors; Max $21/seat/month | Bots are not billed |
| Kodiak | Open source (AGPL-3.0) and hosted app | Auto-merges labelled PRs after checks and approvals | Free self-hosted; hosted price unverified | Label-triggered; no risk scoring |
| Cursor approval agents | Hosted | Scores PR risk, routes reviewers, approves low-risk changes | Cursor plans; pricing page unverified | Cursor says it does not replace full review |
Martian now presents itself as a research lab, with no router product or price on its site.
Planning and decomposition13 optionsOpen sourceSpec Kit for plans; git-spice for stacksPaidFactory Spec Mode or Kiro; Graphite for stacks
Planning turns one large ticket into a reviewed plan, ordered tasks and a series of small pull requests, with a person approving the plan before any code is written. GitHub Spec Kit and BMAD Method are the open-source choices for specs and plans, and git-spice handles stacked pull requests; Kiro specs, Factory's Spec Mode and Graphite are the paid ones. Agent frameworks (Claude Agent SDK, LangGraph, CrewAI) can split work across agents but bring no plan approval or stacking. The catch: every tool expects a person to approve in an interactive session, so a headless factory must build its own approval gate (a plan comment or pull request) and its own logic for landing a stack.
- Watch out
- Approval gates are interactive; headless ones need custom code
| Option | Type | What it does for a factory | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| GitHub Spec Kit | Open source (MIT) | Spec, plan and task files that agents follow | Free | Very active; approval is manual |
| Kiro specs | Hosted (pricing) | Requirements, design and task files reviewed before work | Free 50 credits; Pro $20; Pro+ $40; Pro Max $100; Power $200 per month | Credit-based; spec tasks use credits |
| BMAD Method | Open source (MIT) | Agent roles that produce a PRD, an architecture and stories | Free | Active; a heavy process |
| Taskmaster | Source-available (MIT with Commons Clause) | Turns a PRD into dependency-ordered tasks via MCP or CLI | Free; model cost extra | Quiet since April 2026 |
| Tessl | Hosted; self-hosted on Enterprise | An agent platform with specs, plugins and evals | Free 1,000 credits/month; Team $100/month | Moved from specs toward a general agent platform |
| Factory Spec Mode | Hosted; on-premise on Enterprise | Read-only planning; a person approves before code | Pro $20; Teams $60/month plus $40/seat | The closest built-in approval gate |
| Google Jules | Hosted | Writes a plan and waits for approval before editing | Free 15 tasks/day; more in Google AI Pro and Ultra | Paid tiers for personal Google accounts only |
| GitHub Copilot cloud agent | Hosted | Plans and implements in the background; opens a draft PR | Pro $10; Pro+ $39; Max $100 per month | No mandatory plan approval |
| Graphite | Hosted; now part of Cursor | Stacked PRs with review and a merge queue | Starter $20, Team $40 per user/month | Still sold on its own |
| git-spice | Open source (GPL-3.0) | Manages stacked branches and opens linked PRs | Free | Active; GitHub, GitLab, Bitbucket, Gitea, Forgejo |
| Claude Agent SDK | Open source library (MIT) | Claude Code as a library, with subagents and hooks | Free; model tokens extra | No plan approval built in |
| LangGraph | Open source (MIT) | Stateful agent graphs that pause for human approval | Free; LangSmith hosting extra | Good fit for approval pauses |
| CrewAI | Open source (MIT) and hosted | Role-based crews of agents and event-driven flows | Free; hosted price unverified | More open-ended than graph control |
Also checked: Sapling and ghstack for stacks, the OpenAI Agents SDK, and Devin (its plan approval is not documented).
Orchestration12 optionsOpen sourceRiver (Go, Postgres); TemporalPaidTemporal Cloud ($50 per million actions)
Runs need a queue, retries, timeouts, and a way to wait for a human answer. On one machine a CLI is enough. Across machines, River (a Go job queue on Postgres) is the light choice. Temporal is the strongest when a run must wait days for an answer, but it is a server plus a database to operate. Retry only infrastructure errors: a run that failed its checks, asked a question or hit its budget would spend again for the same result.
- Watch out
- Temporal is heavy to self-host
| Option | Type | What it does for a factory | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| River | Open source (MPL-2.0); River Pro paid | Go job queue on Postgres: retries, scheduling, unique and periodic jobs, web UI | Free; Pro $125/month (up to 20 developers) adds workflows and global concurrency limits | Best light fit: Go-native, one Postgres |
| Temporal | Open source (MIT) and Temporal Cloud | Durable workflows, timers, signals for "waiting on a human" | Self-host free; Cloud $50 per million actions after $150 free credits | Strongest durability and Go SDK; heavy to self-host |
| DBOS Transact | Open source library, paid Conductor | Durable execution as a library on Postgres; Go SDK | Library free; Pro $99/month (1M checkpoints) | Low-ops Temporal alternative; Go SDK is new |
| Restate | Source-available (BSL 1.1, Apache-2.0 after 4 years) | Single-binary durable execution server; Go SDK | Self-host free; Cloud prices unverified | Easy on one machine; BSL only bars reselling it as a service |
| Hatchet | Open source (MIT) and cloud | Task queue with concurrency and rate limits, cron | Free 1M runs/month, then $30 per million runs | Good concurrency controls; Go SDK unverified |
| Inngest | Open source server, Go SDK (Apache-2.0) | Event-driven durable functions, batching, rate limits | Free 50,000 executions/month; Pro from $99/month | Flow control leans on their cloud |
| Argo Workflows | Open source (Apache-2.0, CNCF) | Kubernetes DAG workflows, cron, retries | Free | Only if runs live on Kubernetes |
| Asynq | Open source (MIT) | Go task queue on Redis | Free | Still v0.x and adds Redis; River is the better pick |
| Trigger.dev | Open source (Apache-2.0) and cloud | Long-running tasks with checkpoints | Free with $5 credits; Pro $50/month plus compute | TypeScript only; no Go SDK |
Changes across many repositories
| Option | Type | What it does for a factory | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| multi-gitter | Open source (Apache-2.0) | Runs a script across repos and opens PRs | Free | Active; works with most forges |
| turbolift | Open source (Apache-2.0) | Clones repos, applies a change and raises PRs | Free | GitHub only; a preview |
| Sourcegraph Batch Changes | Proprietary | Opens and tracks PRs across many repos | Enterprise from $16,000/year | Public source archived in 2024 |
Publishing and human review8 optionsOpen sourceGitHub App with go-github; GitHub merge queuePaidMergify (bots not billed), Graphite
A done run becomes a pull request, and people answer questions and review. A GitHub App gives per-repo permissions and webhooks for replies. GitHub's own merge queue is the first choice; Mergify is a cheap upgrade that does not bill bots as seats. Slack is the natural place for an agent's questions.
- Watch out
- Keep the factory off the ruleset bypass list
| Option | Type | What it does for a factory | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| go-github with a GitHub App | Open source (BSD-3) | Go client: open PRs, comment, set statuses | Free | Direct fit for a Go publisher |
| GitHub merge queue | Platform feature | Queues PRs, drops them on CI failure | Included in GitHub plans | First choice; no extra infrastructure |
| Mergify | Hosted | Merge queue with batching and priorities | Free up to 5 contributors; Max $21/seat/month; bots not billed | Cheap upgrade |
| Graphite | Hosted | Stacked PRs, merge queue, AI review | Starter $20; Team $40/user/month (queue from Team) | Now part of Cursor; for big tickets that land as stacks |
| Aviator | Hosted | Parallel merge queue, stacked PRs | Free 3 repos; Team $20/dev/month | Only at scale; self-host on Enterprise |
| Trunk Merge Queue | Hosted | Merge queue plus flaky-test quarantine | Free up to 5 committers | Flaky-test handling helps agent PRs |
| Slack API | Platform | Buttons and threads for questions and answers | Unverified | Natural channel for an agent's questions |
| GitLab merge trains | Paid tiers | GitLab's merge queue | Premium $29/user/month | Only with a GitLab publisher |
Deploy and signals12 optionsOpen sourceOpenFeature with Unleash; Argo Rollouts or FlaggerPaidSentry with Seer ($40/contributor/month), LaunchDarkly
Closing the loop means production signals become new tickets. A thin adapter from Sentry, Datadog or PagerDuty alerts to tickets does that. For the rest, stay vendor-neutral: OpenFeature with Unleash or GrowthBook for flags, Argo Rollouts or Flagger for canaries. Sentry's Seer and Datadog's Bits already open fix PRs from production errors on their own.
- Watch out
- LocalStack's open repo was archived
| Option | Type | What it does for a factory | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| Sentry with Seer | Hosted | Errors plus an agent that finds root causes and opens fix PRs | Free 5,000 errors/month; Team $29/month; Seer $40 per active contributor/month | Best error-to-ticket source |
| Datadog Bits Code | Hosted | Turns production errors into validated fix PRs | Not published | Only if you already pay for Datadog |
| PagerDuty | Hosted | Alerting and incidents; SRE agent | Free up to 5 users; Professional $25/user/month | Alert source for an alert-to-ticket adapter |
| OpenFeature | Open standard (Apache-2.0, CNCF) | Vendor-neutral feature flag API and SDKs | Free | Keeps the flag vendor a config choice |
| Unleash | Open source (AGPL-3.0); Enterprise paid | Self-hosted flags with gradual rollout | Free; Enterprise unpublished | Ship agent changes dark behind flags |
| GrowthBook | Open source and cloud | Flags, A/B tests, safe rollouts | Free 3 users; Pro $40/seat/month | If you also want experiments |
| Argo Rollouts and Flagger | Open source (Apache-2.0) | Canary and blue-green with automatic rollback on metrics | Free | Pick one, if deploys run on Kubernetes |
| Okteto | Self-hosted free tier, paid | Preview environment per PR on your own Kubernetes | Free up to 5 users | Previews stay in your network |
| Vercel | Hosted | Preview per branch with a checks API | Hobby free; Pro $20/month | Web front ends |
Also priced: LaunchDarkly (Foundation $10 per service connection/month), Signadot (Business $250/month), incident.io (Team $19/user/month). Uffizzi's site failed with an SSL error; avoid it.
Release notes and versioning
| Option | Type | What it does for a factory | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| release-please | Open source (Apache-2.0) | Keeps a release PR with the version bump and changelog | Free | Needs Conventional Commits |
| Changesets | Open source (MIT) | Change files per PR, then versions and changelogs | Free | JavaScript monorepos |
| semantic-release | Open source (MIT) | Versions, writes notes and publishes from commits | Free | Fully automatic; needs strict commit messages |
The outer loop
Control plane13 optionsOpen sourceOpenHands for runs; River, Temporal or Hatchet for the queuePaidRetool or Appsmith over the factory's own API
A control plane gives people one screen for the queue, live runs, spend, questions, approvals and a stop button. No product does all of it for a headless factory, so most setups pair an agent console with a job dashboard. OpenHands and Coder are the strongest self-hosted agent consoles; the dashboards of River, Temporal and Hatchet give the queue view and cancel controls for free; and Retool or Appsmith can wrap a factory's own API into approval and kill-switch screens. The catch: vendor dashboards (Devin, Factory) control only their own agents, desktop tools (Conductor, Sculptor) are built for one person's machine, and Vibe Kanban's company shut down in April 2026.
- Watch out
- Vendor dashboards control only their own agents
| Option | Type | What it does for a factory | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| OpenHands | Open source (MIT) and hosted | A browser console for agent runs; the cloud adds budgets, roles and usage | Free self-hosted; Teams $500/month for 10 users | Very active; bring your own model keys |
| Coder Agents | Open source core (AGPL-3.0) and paid tiers | Starts agents in provisioned workspaces by chat or API | Community free; Premium price unverified | Unlimited agents only on the top tier |
| Temporal UI | Open source (MIT); Temporal Cloud hosted | Workflow history, signals and cancel for long runs | Free self-hosted; Cloud $50 per million actions | Heavy to learn and operate |
| River UI | Open source (MPL-2.0) | A web view of a Postgres job queue: retry, cancel, inspect | Free; River Pro $125/month optional | Go and Postgres only |
| Hatchet | Open source (MIT) and hosted | A task queue with a dashboard, retries and cancellation | Free 1M runs/month, then $30 per million | Still before 1.0 |
| Retool | Hosted; self-host on Enterprise | Approval, queue and kill-switch screens over your own API | Free for 5 users; Team from EUR 9 per builder/month | You build and maintain the console |
| Appsmith | Open source (Apache-2.0) and hosted | The same kind of screens, open source | Free; Business $15/user/month | Same build effort as Retool |
| Backstage | Open source (Apache-2.0) | A developer portal whose plugins can host a run dashboard | Free | High setup effort; no agent features built in |
| Tembo | Hosted | Ticket-to-PR orchestration with a task dashboard | Pro $60/month; Max $200/month; VMs from $0.1066/hour | Vendor-hosted |
| Devin | Hosted | A dashboard for Devin's cloud sessions and usage | Pro $20; Max $200; Teams $80 plus $40/user per month | Controls only Devin's agents |
| Factory | Hosted; on-premise on Enterprise | Droids with usage tracking, an audit log and admin controls | Pro $20; Teams $60/month plus $40/seat | Controls only Factory's agents |
| Conductor | Proprietary Mac app with cloud workspaces | Runs parallel Claude, Codex and Cursor agents | Free; Pro $50/month; Teams $60/user/month | Built for a person at a Mac |
| Sculptor | Open source desktop app (MIT) | Parallel agents in containers with a review screen | Unverified | Built for a person at a desktop |
Vibe Kanban's company shut down in April 2026, and the open-source board is now community-maintained. Also checked: Inngest (from $99/month), ToolJet (AGPL-3.0, from $23 per builder/month) and Claude Squad (AGPL-3.0, free).
Observability9 optionsOpen sourceLangfuse with OpenTelemetryPaidLangfuse Cloud ($29/month), Braintrust, LangSmith
Every run should leave a trace with its cost. Self-hosted Langfuse (MIT) behind an OpenTelemetry collector is the best open match. Claude Code and Codex both export OpenTelemetry, and a gateway can emit the GenAI semantic conventions, so one collector sees every run.
- Watch out
- Hosted tools send prompts and code out
| Option | Type | What it does | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| Langfuse | Open source (MIT) and cloud | Traces, cost dashboards, evals; OTLP over HTTP | Self-host free; cloud Core $29/month | Best self-hosted match |
| Arize Phoenix | Source-available (Elastic 2.0); Arize AX paid | Tracing and evals; lists Claude Code and Codex as supported | Phoenix free; AX Pro $50/month | Fine for internal use |
| Claude Code and Codex OpenTelemetry | Built into the agents | Metrics, logs, events and traces over OTLP; prompt content off by default | Free | Turn on in the sandbox and point at a collector |
| OpenTelemetry GenAI conventions | Open standard (Apache-2.0) | Common names for model, tokens and cost | Free | Emit from the gateway |
| Laminar | Open source (Apache-2.0) and cloud | Tracing and a debugger for long agent runs | Free 1 GB; Starter $30/month | Interesting debugger; Claude Code support unstated |
| LangSmith | Hosted | Tracing and evals | Free 5,000 traces; Plus $39/seat/month | Self-host on Enterprise only |
| Braintrust | Hosted | Evals and logging | Free; Pro $249/month | Eval-first, hosted |
| Datadog LLM Observability | Hosted | LLM and agent monitoring | Unverified | Only if you already use Datadog |
| W&B Weave | Open source (Apache-2.0) and cloud | Tracing and evals | Free 1 GB/month; Pro from $60/month | Python-centric |
Helicone overlaps an LLM gateway, and OpenLIT and OpenLLMetry instrument Python SDKs; neither adds much to a gateway that already emits OpenTelemetry.
Evals16 optionsOpen sourceHarbor, SWE-bench Pro, promptfoo, Inspect AIPaidBraintrust ($249/month), LangSmith
The most useful metric is replaying your own merged PRs: the commit before the merge as the base, the problem as the ticket, and the merged diff to compare with. No hosted eval platform does that; they store scores. Many merged PRs carry no test that proves them, so a replayed task needs tests of its own. Calibrate each one: the tests must fail on the base commit and pass on the merged diff, or the task measures nothing. Name the interface those tests call (functions, routes, files) in the ticket, or a correct patch fails on a name. One run per task is noise, since agents vary between runs: compare pass rates over three or more runs per task, and count runs that produced nothing as failures. Package the tasks in the Harbor format, and use SWE-bench Pro and Terminal-Bench as outside checks. promptfoo fits regression tests for judge prompts.
- Watch out
- One run per task is noise; run each several times
| Option | Type | What it does | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| Harbor | Open source (Apache-2.0) | Agent eval framework from the Terminal-Bench team; custom tasks; runs on Daytona, Modal and others | Free | High: a format for private replay tasks |
| SWE-bench Pro | Open benchmark (MIT) | 642 tasks across 11 repos, in Harbor format | Free to run | Harder outside reference |
| SWE-bench | Open benchmark | Issue-to-patch tasks with a Docker harness (Verified, Multimodal, Multilingual) | Free | Its task shape is the template for replaying your own PRs |
| Terminal-Bench | Open benchmark | Terminal tasks; version 4.0 | Free | General agent skill, not your repos |
| Multi-SWE-bench | Open source (Apache-2.0) | 1,632 tasks in Java, TypeScript, Go, Rust, C and C++ | Free | Includes TypeScript and Go |
| SWE-bench-Live | Open source | SWE-bench harness on a refreshed dataset | Free | Avoids memorized tasks |
| promptfoo | Open source (MIT, now owned by OpenAI) | Declarative prompt and model evals in CI | Free | High: regression tests for judge prompts |
| Inspect AI | Open source (MIT, UK AISI) | Eval framework with 200+ evals and agent scoring | Free | Score the judge on labelled diffs |
| DeepEval | Open source (Apache-2.0) | Pytest-style LLM metrics | Free; Confident AI from $200/month | Python only |
| Braintrust | Hosted | Experiments, datasets, scorers over time | Free 10,000 scores; Pro $249/month | Dashboards; data leaves your network |
| LangSmith | Hosted | Tracing and evals | Free 5,000 traces; Plus $39/seat/month | No Go SDK |
Delivery metrics: does the team ship more?
| Option | Type | What it does for a factory | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| DORA metrics | Research framework | Five measures of delivery speed and stability | Free | Now includes the rework rate |
| Apache DevLake | Open source (Apache-2.0) | Pulls DevOps data into DORA dashboards | Free | Apache incubating; active |
| Swarmia | Hosted | Delivery metrics, DORA and AI impact | Free up to 9 developers; Standard $45/developer/month | Public prices |
| LinearB | Hosted | Delivery metrics and workflow automation | Essentials $29/contributor/month (50 minimum) | Annual only |
| DX and Jellyfish | Hosted | Developer experience, AI impact and investment tracking | On quote | Survey and finance focus |
Governance13 optionsOpen sourceRulesets and CODEOWNERS, OPA, gitsignPaidGitHub Enterprise (attestations on private repos)
The control that matters most is free: GitHub rulesets plus CODEOWNERS, with the factory's GitHub App off the bypass list, so an agent can never merge its own PR. Keep an audit log of every run. Reach for OPA only when rules outgrow code, and for attestations only if the factory ships build artifacts. Whether agent commits carry an AI label is a policy choice.
- Watch out
- Decide whether commits carry an AI label
| Option | Type | What it does for a factory | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| GitHub rulesets and CODEOWNERS | Platform feature | Required checks and code-owner reviews on target branches | Private repos need Team $4/user/month or Enterprise | High: "the agent cannot self-merge" |
| Open Policy Agent | Open source (Apache-2.0, CNCF) | Policy engine embeddable in Go, with decision logs | Free | When rules grow past allow lists |
| Conftest | Open source | Policy assertions on config files and diffs | Free | e.g. block workflow or lockfile changes without a label |
| Assisted-by trailer | Convention (Linux kernel) | Labels AI-assisted commits; the human keeps Signed-off-by | Free | A policy decision: label AI commits or not |
| Agent Trace | Open spec (CC BY 4.0) | Line-level human and AI attribution per commit | Free | Still RFC v0.1.0; watch |
| Sigstore gitsign | Open source | Keyless commit signing with OIDC | Free | Verifiable factory identity; GitHub does not show it as Verified |
| GitHub artifact attestations | Platform feature | Signed build provenance, SLSA Build L2 | Private repos need Enterprise | Only if the factory ships build artifacts |
| in-toto and SLSA | Open specs | Attestation format and build levels | Free | Could wrap "checks and review passed at this commit" |
| Kyverno | Open source (Apache-2.0, CNCF) | Kubernetes admission policy | Free | Only with Kubernetes sandboxes |
Licenses of the dependencies an agent adds
| Option | Type | What it does for a factory | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| OSS Review Toolkit | Open source (Apache-2.0) | Lists dependencies and checks licenses against a policy | Free | A policy gate; heavier setup |
| ScanCode Toolkit | Open source | Finds licenses and copyrights in code and packages | Free | A scanner, not a policy engine |
| Trivy license scanning | Open source (Apache-2.0) | Flags forbidden licenses in packages and images | Free | A simple CI gate |
| FOSSA | Hosted | License and vulnerability scans with policies | Free for 5 projects; Business $20/project/month | Priced per project |
Instead of building
All-in-one platforms12 options
Devin, Factory, the GitHub Copilot agent, Codex cloud and Cursor run in the vendor's cloud with their own sandbox, budgets and review: your code goes there, and your own checks and budgets do not apply. The self-hosted exceptions are OpenHands and Tembo. Factory.ai is the closest commercial match to a full factory.
| Option | Where it runs | What it automates | Pricing (Oct 2026) | Notes |
|---|---|---|---|---|
| OpenHands | Self-hosted (MIT core) | Agent canvas; automations from Slack, GitHub, Linear; bring your own keys | Free self-host; Teams $500/month for 10 users | The only open-source, self-hosted platform |
| Tembo | Cloud VMs; self-host offered | Runs Claude Code, Codex, Cursor with GitHub, Slack, Linear | Pro $60/month (5 users); VMs $0.11 to $2.95/hour | Orchestrates existing agents; self-host terms unverified |
| Factory (Droids) | Cloud; on-prem on Enterprise | Droids in desktop, CLI, SDK; cloud and local background agents | Pro $20/month; Teams $60/month plus $40/seat | Closest commercial match to a full factory |
| Claude Code GitHub Actions | Your GitHub runners | @claude on an issue or PR turns it into a PR | No platform fee; Actions minutes plus tokens | Cheapest start; no per-run budget beyond --max-turns |
| Claude Code on the web | Anthropic cloud | Cloud sessions, PR auto-fix, scheduled runs | In Pro $20, Max from $100, Team plans | Needs a claude.ai login; your own gateway and budgets do not apply |
| GitHub Copilot cloud agent | GitHub cloud | Issue to PR in an ephemeral environment; can hand off to third-party agents | Pro $10, Pro+ $39, Max $100/month | GitHub-only; replaces rather than plugs in |
| OpenAI Codex cloud | OpenAI cloud | Codex on web, CLI, IDE | Plans from $8/month, or API | Replaces only the agent layer |
| Devin | Cognition cloud; VPC on Enterprise | Cloud agents with Slack, Teams, Linear, Jira, GitHub | Pro $20; Teams $80/month plus $40/user (page looked unusual; unverified) | Full replacement in the vendor's cloud |
| Cursor cloud agents | Cursor cloud | Cloud agents and automations | From $30/month (page looked unusual; unverified) | Full replacement in the vendor's cloud |
| Google Jules | Google cloud | Plan approval, then changes on GitHub | Free 15 tasks/day; more on Pro and Ultra | Individual @gmail.com accounts only |
| Augment Code | Cloud | Cosmos agent, CLI, context engine | Standard $20/month plus a 40% fee on model usage | Agent layer only |
| Dark Factory CLI | Self-hosted CLI with Docker sandboxes | GitHub issues to merged PRs: implement, review, scenario tests, a dashboard | Free; source-available (Elastic License 2.0) | Bring your own Anthropic key; no hosted service |
Sweep is now a JetBrains assistant, no longer an issue-to-PR bot.