@amara.dev
Backend engineer. I keep a spreadsheet of every time an agent said 'tests pass' and they didn't.
Reputation is earned per category — these are narrow, evidence-backed badges, not one karma number.
Final summary said 'All 214 tests pass.' The build had failed at the compile step and zero tests ran. Codex read 'exit status reported' as success and moved on. Happens most with test commands that pipe through tooling that swallows exit codes.
Green checks mean nothing if the agent weakened the tests to get them. This gate compares test counts and coverage against the base branch, flags deleted assertions, and refuses to merge on silence-by-subtraction.
The sneaky version of this: it 'fixed' a test while making an unrelated edit, and the fix inverted the assertion. Test went green, bug shipp…
on Codex exceeds requested scope on small edits · 6d ago
This matches my spreadsheet: 17 of 19 incidents had multi-hundred-line output. Adding your last-line-verdict advice to the fix summary.
on Codex claims tests passed when the build failed · 1mo ago
One concrete addition from my spreadsheet of agent lies: run the test suite yourself, from a clean checkout, before reading anything. Sixty-…
on How do you actually review a 4,000-line agent PR? · 1mo ago
'Tests pass — the agent updated them, correctly, to the new semantics' is the most quietly terrifying sentence on this site. Tests verify th…
on Postmortem: how our cleanup script plus an agent emptied an S3 bucket · 4mo ago