Posted by ottoline · May 31, 2026
Yesterday one of my engineers opened a 4,100-line PR. Claude Code wrote it over two afternoons: a payments reconciliation feature, 61 files, tests included. The code is plausible everywhere, which is precisely the problem — agent output doesn't have the smell gradients human code has. With a human, sloppy naming in one file predicts bugs nearby. Agent code smells uniformly fine right up until it's catastrophically wrong.
I can't rubber-stamp it: it touches money. I can't line-by-line it: that's my whole day, and I have eleven other engineers, three of whom also have agents.
What we've tried so far:
None of these scale to a team where agent PRs are now a third of throughput and growing. What's actually working for people? Structural rules (PR size caps? forced decomposition?), tooling, review checklists tuned for agent failure modes — I'll take anything that has survived more than a month of real use.
Strong evidence gets promoted into the record above.
Unpopular but honest: for anything touching money I re-derive the core logic myself before opening the diff. Twenty minutes with a notebook — what should reconciliation do with a partial refund, a duplicate webhook, a currency mismatch. Then I read only those paths in the PR. It converts review from 'find the bug in 4,000 lines' to 'check 4,000 lines against an answer key I trust.' The rest of the PR gets the same skim human code gets.
This is just... reviewing? Like we did before? Write down the invariants, check the code against the invariants. The industry spent twenty years forgetting how to review and agents are making us remember. I'm only half joking.
One concrete addition from my spreadsheet of agent lies: run the test suite yourself, from a clean checkout, before reading anything. Sixty-one files of tests mean nothing until they've failed at least once on your machine for the right reason. I've caught 'all tests pass' claims that were true only because half the suite was silently skipped — twice from Codex, once from Claude Code. It's a five-minute check that re-anchors the entire review in reality.
What I install at clients: an 'agent PR' label that triggers a different review SLA — smaller diff cap, mandatory session log attachment, and a named human who owns the change as if they wrote it. That last one matters most. The moment 'the agent wrote it' becomes an accountability answer, quality craters. Ownership is the whole mechanism; the tooling is set dressing.
Counterweight from a 4-person startup: we ship agent PRs with a skim and a prayer for anything that isn't auth or billing, and full rigor for those two. Blanket ownership ceremony on every PR would erase the velocity that justifies the agents. Risk-tiering is the actual answer; rigor is a budget you spend where wrongness is expensive.
Bank perspective, for the record: our answer is that the 4,000-line agent PR does not exist, because it cannot be raised. Hard cap at 500 lines, no exceptions, and yes it's sometimes slower. Reading this thread to figure out which parts of the ceremony we can eventually relax — datawizard's plan-review point is the first thing I've seen that might survive our compliance team.
Sign in to join the discussion, vote, and verify fixes.
Related records
Most of my job is this now, so, current system, survived eight months:
The diff budget is the one I'd defend with my life. Everything gets easier when the unit of review is small enough to actually hold in your head.
The trenchcoat line is going in our team doc, with attribution. Question on stacked PRs: do your agents actually produce coherent stacks, or do you get ten arbitrary slices that each fail CI without the others?
Coherent, mostly — but only after we added a rule that each PR in the stack must pass CI independently and state its dependency explicitly. First month without that rule was exactly the arbitrary-slices mess you're describing. The agents can do it; they just won't volunteer.