8 repliesDiscussion2h ago
Index
Feed
Posted by rowanp · Jul 18, 2026
For the last quarter, every page at our streaming company has gone to two responders: the on-call human, and a read-only Claude Code session that starts triaging immediately. Time to share real numbers, because "agents will do on-call" discourse is high on vision and low on incident counts.
The setup, deliberately boring: page fires → automation starts an agent session with the alert payload → agent has read-only access to Sentry (via MCP), logs, dashboards, deploy history, and the runbook repo. It cannot restart, rollback, scale, or touch prod state in any way. Its deliverable is a triage brief in the incident channel: probable component, correlated recent deploys, similar past incidents, relevant runbook section, and a proposed first action for the human to take.
A quarter's numbers (114 pages):
Incident A: the agent's brief confidently attributed a latency spike to a deploy that was, in fact, a symptom-driven rollback responding to the spike — causality inverted. The 3am human anchored on it, chased the rollback for 25 minutes. The fix wasn't making the agent smarter; it was reformatting the brief: correlations now render as a timeline with explicit "correlation, not cause" framing, and the probable-cause line carries the agent's stated confidence. Anchoring is a format problem before it's a model problem.
Incident B: during a partial Sentry outage, the agent's MCP queries hung, its brief arrived 20 minutes late into an active incident channel, stale and contradicting established findings. Now: briefs have a 5-minute deadline or they don't post, and late briefs go to a thread, never the main channel. Stale confident text in a live incident channel is negative information.
Where the human stays load-bearing, one quarter in: deciding what NOT to look at. The agent's brief is thorough; incident response is triage under time pressure, and thoroughness is the enemy. The senior-responder skill of ignoring nine plausible threads to pull the right one — the agent doesn't have it, and its briefs, read naively, actively work against it by making all nine threads equally articulate.
Would I keep it? Yes, without hesitation — the deploy-correlation section alone. Would I give it write access to anything? Also no, without hesitation. Ask me again in a year.
Questions welcome, especially from anyone running the write-access version I'm too cowardly for.
Related records
Strong evidence gets promoted into the record above.
Someone has to ask the ugly question: read-only access during incidents is still access. Your agent reads logs during your moments — the moments most likely to have customer data sprayed into error messages, stack traces, and debug output that normal-times log hygiene never sees. What's the story for the agent session transcripts? Because you've built a system that concentrates your most sensitive operational data into the context window of a third-party model, on a schedule determined by your outages.
Fair, and it got argued before launch: transcripts land in the same access-controlled store as incident recordings, log access goes through our existing PII-scrubbing proxy (which, you're right, was built for normal-times traffic — we red-teamed it against incident-shaped logs and patched two gaps, stack-trace locals being the embarrassing one), and the vendor agreement covers the rest to our security team's satisfaction. Not claiming it's solved — claiming it was decided, with eyes open, which is the bar.
'Decided, with eyes open, which is the bar' — I'm going to use this framing in our HIPAA review, where the answer will be no, but it'll be a decided no instead of a vibes no. The stack-trace-locals gap is a genuinely useful specific; our scrubber has the same hole, I just checked. Thread paying rent already.
Small-team translation: we can't afford the automation harness, but the core finding — deploy-correlation is the high-value low-risk slice — ports down. Our version is a manual habit: first responder pastes the alert into a read-only agent session with the deploy log and asks only 'what shipped near this.' Two minutes, no infrastructure, and it's been right often enough that the habit stuck. Sometimes the 20% version is the whole trade.
The 'thoroughness is the enemy' paragraph deserves its own thread. Data engineering on-call has the same failure: agent triage of a pipeline incident enumerates every plausible upstream cause with equal fluency, and junior responders investigate them in list order. Ordering by the agent's actual confidence (which you have to explicitly ask for) instead of narrative convenience changed how our juniors used the briefs more than any accuracy improvement. The model knows more than its prose ordering reveals — make it show its uncertainty structure.
Sign in to join the discussion, vote, and verify fixes.
This matches our pilot almost beat-for-beat, including the anchoring incident — ours inverted a cache-warmup effect into a cause at 4am, same 25-minute detour, and I'd bet money the mechanism is identical: tired humans don't evaluate briefs, they adopt them. Your format fix is right and I'd push further: our briefs now lead with the two-line 'what I'd check first and why,' with everything else collapsed behind a click. Adoption risk scales with brief length, so put the parts you're confident in where the tired brain lands, and make the speculation cost a deliberate action to read.
'Tired humans don't evaluate briefs, they adopt them' is the sentence I needed for the internal writeup — that's the entire risk model of agent-assisted on-call in nine words. Trying the collapse format next sprint.