A CI gate that actually catches agent-written regressions

Green checks mean nothing if the agent weakened the tests to get them. This gate compares test counts and coverage against the base branch, flags deleted assertions, and refuses to merge on silence-by-subtraction.

Expected result

Agent-authored PRs get an extra required check that fails when test count drops, coverage falls beyond a small tolerance, or assertions were deleted — with a comment explaining exactly what regressed.

Difficulty

Intermediate

Estimated time

~50 min

Version notes

Snippets assume vitest + istanbul JSON formats; jest's summary JSON has the same numTotalTests field. pytest: use --json-report and jq '.summary.total'.

1. 01GitHub Actions (or equivalent) with required status checks
2. 02A test runner with JSON output (jest, vitest, pytest all qualify)
3. 03Coverage already collected on main so there's a baseline
4. 04A convention for marking agent PRs (label or branch prefix)

01

Mark agent PRs so the gate knows when to run

Branch prefix is the low-friction option — your worktree/branch discipline from other recipes already provides it. The gate triggers on agent/*, codex/*, claude/* branches.

# .github/workflows/agent-gate.yml (trigger excerpt)
on:
  pull_request:
    branches: [main]
jobs:
  agent-gate:
    if: startsWith(github.head_ref, 'agent/') || startsWith(github.head_ref, 'codex/') || startsWith(github.head_ref, 'claude/')
    runs-on: ubuntu-latest

Copy

02

Compare test counts against the base branch

The cheapest strong signal: total tests must not decrease. Run the suite on both refs with JSON output and diff the counts. Fewer tests than main is an automatic fail unless a human overrides.

git fetch origin main
git checkout origin/main -- .
npx vitest run --reporter=json --outputFile=/tmp/base.json || true
git checkout HEAD -- .
npx vitest run --reporter=json --outputFile=/tmp/head.json || true
base=$(jq '.numTotalTests' /tmp/base.json)
head=$(jq '.numTotalTests' /tmp/head.json)
echo "tests: base=$base head=$head"
if [ "$head" -lt "$base" ]; then
  echo '::error::Test count decreased on an agent branch.'
  exit 1
fi

Copy

! Common failures

Gate passes but a regression ships anyway

Count and coverage are necessary, not sufficient — agents can write assertion-free tests that execute lines. Add one honesty probe: a mutation-testing spot check (stryker --mutate on changed files) weekly rather than per-PR keeps cost sane.

Base-branch test run is slow, doubling CI time

Cache the base results: main's counts and coverage only change when main moves. Store /tmp/base.json keyed on the merge-base SHA and most PRs skip the base run entirely.

Agents rename test files and the assertion diff misses them

Use git diff -M to follow renames, and treat a renamed test file with heavy deletions as its own warning. Renames are a known laundering pattern for test weakening.