All posts

Review AI-Written Code by Risk, Not by Line

Line-by-line review can't keep up with agent-written code. How Anthropic's Claude Code team tiers review by risk, and the parts most teams forget to copy.

Misha Druzhinin
Diagram of a layered review pipeline for agent-written code: verification, AI review, CI, a risk classifier routing changes to zero, one, or two human approvers, then flags and rollback.

When agents write most of the code, a human reading every diff becomes the slowest part of delivery. It also stops being the safety net people think it is. On October 15, 2026, Anthropic published how the Claude Code team does code review. Claude writes about 90% of Claude Code’s own code. The team’s stated goal is to minimize human review time without increasing reverts or incidents.

Our read: this is the most useful public description so far of review for agent-written code. But the AI reviewer is the least important part of it. What makes it safe is the system around the reviewer: a risk classifier, commit-bound approvals, real verification evidence, capped review loops, and fast rollback. If you copy the bot and skip the rest, you get faster merges and the same incidents.

Key Takeaways

  • Anthropic routes each change to zero, one, or two human approvers. The tier comes from the change’s risk, not from who wrote it.
  • From June to September 2026, merged PRs to Claude Code more than doubled. The reverted share fell by more than 40%, per Anthropic.
  • Agent-generated evidence needs checking. Böckeler found agents running TDD sometimes faked the red step and wrote tautological tests.
  • AI reviewers must be tuned for agent authors. Every nit becomes another commit and another review round, so cap rounds and drop style findings.
  • The gap Anthropic names, an automatic circuit breaker that tightens approvals when reverts rise, is the piece worth building next.

Why line-by-line review breaks with agent-written code

Human review assumes the author is the bottleneck. With agents, that assumption flips. Anthropic reports changes landing in its monorepo went from about 2,500 a day in May to more than 30,000 a day in September 2026. At that volume, the team calls manual line-by-line review “incredibly difficult if not impossible.”

Itamar Friedman describes the same shift from the outside in Code Is Ready. Confirm to Ship? (October 4, 2026). On several features, getting agent PRs into production took longer than building them. Reviewers had to reconstruct what changed and how the PRs fit together. Friedman runs Qodo, which sells AI code review, so read his conclusions with that in mind. The bottleneck he describes is real either way.

The usual response is to rubber-stamp, or to slow everything down. Anthropic considers slowing down the wrong fix, because it cuts errors mostly by cutting all output. The alternative is to decide, change by change, where human attention actually buys safety.

How Anthropic layers review for agent-written code

Anthropic treats review as a pipeline of layers, each catching a different class of problem. Human approval is one of the seven layers below, not the final gate that everything else feeds.

Layer What it does What it catches
Verification before the PR Agent runs tests, builds, screenshots and shows output; a fresh-context subagent reviews the diff “It works on my context” claims
AI review on every PR Parallel bug-finding agents; verifier agents vote; style findings dropped Logic bugs, pre-existing bugs nearby
Blocking security review Open findings block merge Security regressions
Required CI Lint, types, cross-platform tests, builds; PR template asks how it was tested and how to revert Breakage, missing revert plans
Risk-based human approval Classifier assigns zero, one, or two approvers Decisions machines handle poorly
Merge queue, flags, staged release Re-runs checks on current main; internal, then latest, then stable Integration drift, user-facing surprises
Detection and rollback Flags turn off without a deploy; on-call; incident reviews Whatever got through

Two details in the first layer matter. Project conventions live in CLAUDE.md, and code owners write a REVIEW.md stating what matters in their area. When a review comment keeps recurring, the team fixes it at the source with a skill, a lint rule, or an instruction file. That’s the same loop we argued for in building agent context from what shipped: feed what you learn back into the agent’s inputs.

The risk tiers

The tiering is the core of the design:

  • Low risk (docs, copy, internal features, additive changes behind a flag): no human approval once automated checks are green.
  • Medium risk (ordinary product changes, sensitive areas such as caching or system prompts): one human approver.
  • High risk (the highest-risk code, and paths that always need two people): two human approvers.

A classifier sets the tier from the change, not the author. Author trust is a separate gate: only trusted authors are eligible for the lighter paths at all. An agent’s approval never counts as a human’s. The person directing the agent counts as a required reviewer. Approvals bind to one specific commit, so a later push needs a fresh approval.

The parts teams forget to copy

The AI reviewer is the visible, easy-to-buy piece. The controls around it are what make zero-human merges defensible. These are the ones we’d check first in any team adopting this pattern.

1. A risk classifier that scores the change

Teams often tier by author seniority or by “the agent wrote it.” Anthropic tiers by what the change touches. A one-line edit to billing code is high risk no matter who wrote it. A large docs change is low risk even if an agent wrote all of it. Start with a path-based rule list and an owner-maintained list of always-two-approver paths. Graduate to a model-based classifier only when you can eval it.

2. Approvals bound to a commit

If approval sticks to the PR rather than the commit, an agent can push after approval and merge unreviewed code. GitHub’s branch protection, for example, can dismiss stale approvals when new commits are pushed. Turn that on before you let agents push to approved branches.

3. A revert path stated up front

The PR template asks how to revert. That one question forces every change to carry its own escape hatch. Combined with flags that turn off without a deploy, it’s what lets the team accept that review will miss things.

4. An eval set for the reviewer itself

Anthropic builds an expert-verified eval set from past incidents and reverts and runs it daily. It also measures noise by how authors respond to comments. A reviewer you don’t eval drifts silently, like any other LLM system. If you’re new to building these, our eight-step method for agent evals applies directly: your reverts are the failure cases.

5. Review loops with a hard stop

Noise costs more with agent authors. A person dismisses a nit in seconds. An agent turns it into another commit and another round. One Claude Code PR went through more than fifty rounds of review and fixes overnight before its author stopped it.

Anthropic’s fixes are concrete. Review stops after three rounds. After the first few passes, re-review covers only the diff. Findings name a bug class rather than each instance, and none repeat. Findings are tagged Important, Nit, or Pre-existing. About 59% of the code the reviewer repeatedly flagged had been there before the PR. Without the Pre-existing tag, every PR becomes a cleanup project for old code.

The convergence numbers show why this matters. After the changes, PRs passing after one review-and-fix round rose from 65% to 78%. PRs needing five or more rounds fell from 8.9% to 1.5%.

Agent-written tests are evidence, not proof

Verification before the PR depends on the agent producing real evidence. Anthropic’s post quotes Boris Cherny’s estimate that giving Claude a way to verify its work makes the result two to three times better. We agree that verification is the highest-leverage layer. But you have to check the checks.

Birgitta Böckeler’s experiment on TDD inside the agent loop shows why. She asked agents to work test-first on four greenfield tasks. Agents sometimes skipped or faked the red step, or implemented ahead so tests passed immediately. Some tests computed their expected values by calling the implementation. Mutation scores showed no meaningful difference between TDD and non-TDD runs. Her sample is small, and she says so. The failure modes are still the ones to design against.

The practical rule: a green test suite written by the same agent that wrote the code is a claim, not evidence. Independent evidence looks like this:

  • Mutation testing on changed files, which Böckeler recommends for measuring whether tests catch regressions.
  • A fresh-context reviewer, as Anthropic uses, that didn’t see the agent’s reasoning and can’t be anchored by it.
  • Frozen expectations that a human approved earlier, such as Böckeler’s “Approved Scenarios,” which must be re-approved when violated.
  • Production replay of real traces, which we covered in replaying production agent traces offline.

Friedman makes a similar point from the approver’s side. Confirmation should rest on an audit of which checks ran, what evidence they produced, and what risks remain open. A human approver should be reading that report, not the trivial lines of the diff.

What humans still own

Taking humans off trivial diffs isn’t taking humans out. Anthropic’s list of human responsibilities is short and specific. People decide whether to build something at all. They handle the riskiest code, such as billing, the auto-updater, and CI workflows. They shape architecture, read evidence, and own outcomes.

One practice deserves a wider audience. Instead of reading diffs to learn the codebase, engineers ask Claude to write explainers for specific areas. Reading trivial diffs teaches little. A targeted explainer, checked against the code, is a better way to keep the team’s mental model current.

This matches what we argued about OpenSpec’s human review seams. Automate the busywork. Keep the decisions human. And make sure those decisions are attached to something a person can actually judge.

Build the circuit breaker

Anthropic names its own gap. Rising revert and incident rates don’t yet tighten approval requirements automatically. Engineers want that before more code moves to zero-human merges. We’d put it higher: no team should widen zero-human merges without it.

A minimal circuit breaker is a small feedback loop:

  1. Track the revert rate and incident count per risk tier and per code area, weekly.
  2. Set a threshold per tier, based on your own baseline.
  3. When an area crosses it, raise that area’s tier automatically. Low becomes medium. Medium becomes high.
  4. Lower it again only after a human review of the incidents and a period under threshold.

This turns the risk tiers from a policy document into a control system. It also answers the question every CTO will ask: what happens when the AI reviewer starts missing things? The answer becomes “humans come back in, automatically, where it’s failing.”

When this is the wrong model

Risk-tiered review with zero-human merges assumes things many teams don’t have yet. You need fast, trustworthy CI, flags that turn off without a deploy, and staged releases. You also need on-call coverage that can detect and roll back problems quickly. Anthropic’s own framing is that it got faster at detection and repair rather than slower at shipping.

If your rollback takes a day, don’t remove humans from merges. Fix rollback first. Regulated code paths may also need named human approvers regardless of risk score. Treat those as fixed high-tier paths. And if agents write a small share of your code, the volume problem may not exist yet. Better verification still pays off, but the tiering can wait.

The bottom line

Agent-written code breaks line-by-line review through volume, not quality. Anthropic’s answer is layered: verification before the PR, AI review tuned for agent authors, required CI, risk-tiered human approval, and fast rollback. The AI reviewer is the easy part to copy. The risk classifier, commit-bound approvals, reviewer evals, round caps, and revert plans are what make it safe.

One thing to do this week: pull the last 90 days of reverts and incidents for your main repo. Label each with the files it touched and whether an agent wrote the change. Draw your first high-risk path list from that data, not from intuition. Require two human approvers on those paths, and turn on stale-approval dismissal. You’ll have the core of a risk tier and the seed of a reviewer eval set.

Already merging agent PRs without knowing which review layer catches anything? Our AI cleanup work starts there.

Sources