Posted by clara.builds · Jul 7, 2026
We spent six months measuring agent impact properly at a 12-person product team, because "it feels faster" was not going to survive our CFO. Sharing methodology and the surprises, since most ROI threads here are vibes with a dollar sign.
What we measured: cycle time per ticket (estimate-normalized), review iterations per PR, rework rate (changes to agent-authored code within 30 days), model spend per merged PR, and — the one we almost didn't track — time-to-first-PR on unfamiliar code areas.
Headline numbers, agent-assisted vs not, same team, similar tickets:
The surprise: the strongest predictor of net-positive ROI wasn't task type, model, or engineer seniority. It was whether the engineer wrote an explicit spec before starting the agent. Spec-first agent tickets: rework rate indistinguishable from human baseline, full cycle-time win kept. Prompt-and-pray tickets: all of the rework regression concentrated there.
We now require a three-line spec (goal, constraints, done-when) in the ticket before agent work starts. Rework has been dropping since. The uncomfortable summary: agents are ROI-positive roughly in proportion to how well you'd have briefed a contractor — which means the ROI was always partly a measurement of our own specification discipline, and agents just made the bill visible.
Happy to share the measurement scripts. What's everyone else's rework rate? Nobody publishes that number and I suspect I know why.
Strong evidence gets promoted into the record above.
Methodological applause with one caveat: same-team before/after has a learning-curve confound — your engineers got better at directing agents over the six months, so late-period numbers blend tool quality with operator skill. The spec-first finding is probably robust (it's a within-period comparison) but the -38% headline likely overstates what a fresh team gets in month one. Still the best-instrumented writeup this site has. The rework-rate taboo needed breaking.
Fair and acknowledged — month-one cycle time was more like -15%, and I should've shown the curve, not the endpoint. Amending the post with the monthly breakdown.
Freelancer translation of your spec finding: I bill fixed-price, so rework comes straight out of my margin, and I learned the contractor-briefing lesson in my own invoices within a month. My version of your three-line spec is written for the client to approve before the agent starts — it fixes the agent's aim and scope-creep disputes in one artifact. The spec was never for the agent. It's for whoever pays for the rework.
The last paragraph deserves a plaque: 'agents made the bill visible.' Half my consulting is teams discovering their velocity problem was always a specification problem, now with a per-token receipt. One warning from the field — teams that adopt your three-line spec as ritual (fill the boxes, start the agent) recover none of the benefit. The spec works when someone can fail it. Yours works because a reviewer can bounce a ticket with a vague done-when. Copy the enforcement, not the template.
Twenty years ago we called this a requirements document and the industry decided it was waterfall and burned it. Genuinely delighted to watch it return with a cost model attached. Some of us kept the templates.
Sign in to join the discussion, vote, and verify fixes.
Related records
Our rework number, since you asked and since somebody has to go second: 26% at month two, 17% now, measured the same way. What moved it wasn't specs (we had those) but banning agent PRs that touch more than one concern — our rework clustered in PRs that did a feature and an opportunistic refactor. Split them and the refactor rework stopped contaminating the feature. Publish the scripts; this thread could become the site's de facto benchmark protocol.
Scripts going up this week. And the single-concern finding matches something in our residuals I couldn't explain — spec-first PRs that still reworked badly were disproportionately 'and also' PRs. You may have just named my mystery variable.