Your first CI reliability agent: setup checklist

const metadata = ; Agents that investigate CI failures need the same discipline as agents that deploy to production. This checklist is what platform teams run before flipping the switch. 1. Scope the blast radius Start with read-only investigation on a single pipeline: - [ ] One repo or monorepo package - [ ] Merge queue or main branch only - [ ] Failures on merge-blocking jobs, not every optional check Expand scope only after MTTR improves on the pilot. 2. Wire the signal sources The agent needs structured access to: | Source | Why | | | | | CI logs (step-level) | Primary failure context | | Git history / diff | Correlate failure with change | | Previous run outcomes | Detect flake vs regression | | Run metadata | Runner, image, duration, cache keys | Avoid dumping entire raw logs into the model. Pre-filter to failing step ± context. 3. Set guardrails Non-negotiables before production: - [ ] No auto-merge until review period completes - [ ] Secrets redacted before any model sees logs - [ ] Human approval for suggested patches in week one - [ ] Rate limits on agent invocations per hour - [ ] Audit trail: prompt, tools called, output stored 4. Define success metrics Track these from day one: - Time to first hypothesis (target: under 2 minutes) - Time to human action (target: under 15 minutes) - Re-run rate (should drop, not rise) - False positive rate on root-cause suggestions If re-run rate goes up, the agent is noise, tune before expanding. 5. Choose your model boundary Teams running on open-weight models typically deploy: - Inference inside the VPC (no build logs leave the network) - Smaller models for triage, larger for patch generation - Fallback to human when confidence is low Closed API models can work for low-sensitivity repos, but CI logs are rarely low-sensitivity. 6. Rollout sequence 1. Shadow mode: agent triages, humans ignore output, compare quality 2. Advisory mode: agent posts to Slack/PR, humans act 3. Assisted mode: agent opens draft PR, human merges 4. Autonomous fixes: only for well-scoped, low-risk failures (optional, later) Most teams stall because they skip shadow mode. Don't. Done when You're ready for advisory mode when shadow-mode hypotheses match human diagnosis >70% of the time on a two-week sample of real failures.