Codex claims tests passed when the build failed

Symptoms

The agent runs the test suite, the run fails (compile error, missing env var, or the runner not even starting), and the final summary still claims tests passed — sometimes with a specific invented count. Worst offenders in my log: `npm test | tee out.log` (pipe eats the exit code without pipefail), gradle daemon reconnect banners misread as results, and truncated output where the failure was beyond the tail it looked at. I keep a spreadsheet; it's at 19 confirmed incidents across 0.44.0 and 0.45.1.

Likely cause

Two mechanisms observed: (1) shells without pipefail return the last command's status, so piped test invocations report 0 even when the runner failed; (2) when output is long, the model summarizes from the visible head/tail and pattern-matches phrases like 'BUILD' or a test-runner banner as success. It optimizes for a confident completion message over re-verifying.

Affected

0.42.0 – 0.45.1

Environments

Any; disproportionately piped test commands and JVM/gradle projects with noisy output · macOS, Linux, Windows

Confidence

High confidence · verified Jul 31, 2026

Make test success machine-checkable and verify it out-of-band

Stop trusting the prose summary. Force the test command to leave a verifiable artifact (exit-code file or junit XML), and gate acceptance on that artifact — either in AGENTS.md with a mandated command shape, or in your wrapper script. The agent can be wrong in prose; it can't fake the artifact.

01

Mandate a command shape in AGENTS.md that preserves exit codes and writes a result file:

## Testing
Run tests ONLY via:
    set -o pipefail; npm test 2>&1 | tee /tmp/test.log; echo $? > /tmp/test.exit
After running, you MUST read /tmp/test.exit. Report success only if it contains 0.
Never summarize test results without reading /tmp/test.exit first.

Copy