The 15-minute red build triage playbook

const metadata = ; Main is red. Slack is buzzing. Someone typed "looking" and went back to their standup. This playbook is what the on-call engineer actually runs, in order, before anyone hits re-run. Minute 0–2: Confirm it's real - Check whether the failure reproduces on retry (flake vs code) - Identify the first failing step: ignore downstream red herrings - Note the merge commit and the last green commit If it passes on re-run with no changes, quarantine the flaky test and move on. Don't burn the queue. Minute 2–7: Read the signal Pull these four inputs into one view: 1. Failing step name and exit code 2. Last 50 lines of relevant log (not the full dump) 3. Diff since last green 4. Recent infra changes (runner image, deps, secrets rotation) Most failures cluster into: code regression, dependency drift, infra/config, or flake. Minute 7–12: Route ownership | Failure type | Route to | | | | | Code regression | Last merger or codeowner | | Dependency drift | Platform + affected team | | Infra/config | Platform on-call | | Flake | Test owner + quarantine | Post one Slack message with: failing step, log excerpt, diff link, and suggested owner. Not "CI is broken." Minute 12–15: Unblock or escalate - If fix is obvious and small → pair on patch, re-run targeted job - If unclear → mark queue blocked, assign owner with deadline - If infra → rollback runner/config change first, investigate second When to invoke an agent Use agentic triage when log volume exceeds human scan speed or the failure spans multiple systems (build → test → deploy). The agent should return: root cause hypothesis, evidence links, and suggested next command, not a summary of the logs you already have. Exit criteria Main goes green when: - The failing job passes on a intentional fix (not blind re-run) - Flaky tests are quarantined with an owner assigned - A one-line postmortem is posted if downtime exceeded 30 minutes