Posted by benchmaster · Jul 26, 2026
The upstream thread has 600+ comments and approximately zero numbers in it, so you can't tell an actual regression from "this is just what it costs". Posting ours in case it's useful as a baseline.
40 completed tasks, mid-size TS monorepo, all on 0.146.0:
median ~210K tokens
p90 ~890K
worst 2.4MMedian is boring. The tail is where all the money went, and for us the tail is basically one thing: search loops. Greps for something, doesn't find it, widens, reads a pile of files that turn out to be irrelevant, and now all of those are in context for the rest of the task.
The 2.4M run was looking for a symbol we renamed six months ago. It read 90 files hunting for something that doesn't exist.
What helped:
The rate-limit-cost-jumped issue is a separate claim and I can't confirm it either way, we changed too much in the same window to say anything useful.
What are other people's medians? Want to know if 210K is normal or if we're doing something dumb.
Strong evidence gets promoted into the record above.
"stale map is worse than no map" — this. we had a directory layout section in AGENTS.md from about eighteen months ago. reorganised the repo twice since. agent trusted it completely every single time. deleting the section was better than leaving it there.
how are you measuring this? I can't find a clean per-task total anywhere
wrapper script that diffs usage before/after. it's not exact, close enough for comparing tasks against each other though.
Sign in to join the discussion, vote, and verify fixes.
Related pages
180K median here, python service repo. so you're normal. our p90 is worse than yours, ~1.3M, same cause though.
our version of it is that our internal naming doesn't match the domain language at all. ticket says "invoice", we called it something else in 2021, agent searches for invoice, off it goes.