CloudOpsBench benchmark

Overview

CloudOpsBench is a 452-scenario Kubernetes root-cause analysis benchmark from Wang et al (arXiv:2603.00468v1, February 2026). Each fault case is a frozen JSON repository served through mocked kubectl-style tool calls, so evaluations are reproducible and do not require a live cluster. OpenSRE evaluates this corpus through a benchmark framework that adds cost tracking, integrity checks (pre-registration, per-stratum reporting, negative results, conflict-of-interest disclosure), per-LLM dispatch with version pinning, and self-contained Markdown and HTML reports. The goal is to publish an opensre+LLM column against the paper’s LLM-alone baselines on the same scenarios.

Paper baseline                 opensre+LLM (this benchmark)
─────────────                  ──────────────────────────
DeepSeek-V3.2  0.73   A@1  →   target 0.78+
GPT-5          0.67   A@1  →   target 0.78+
GPT-4o         0.49   A@1  →   target 0.65+
Claude-4-Sonnet 0.50  A@1  →   target 0.65+

Prerequisites

CloudOpsBench does not require live infrastructure. Frozen snapshots are the environment.

# 1. Python 3.12+ (project standard)

# 2. Benchmark-dedicated LLM API keys (keep separate from production keys)
export ANTHROPIC_API_KEY=...        # Claude-4-Sonnet via Anthropic
export OPENAI_API_KEY=...           # GPT-5, GPT-4o
export DEEPSEEK_API_KEY=...         # DeepSeek-V3.2

# 3. Download the corpus (one-time; a few hundred MB)
make download-cloudopsbench-hf

You do not need AWS credentials, an EKS cluster, kind/minikube, Bedrock, a GPU, Grafana, Datadog, or Prometheus.

Quick start

List adapters

uv run python -m tests.benchmarks._framework.cli list

Validate a config

uv run python -m tests.benchmarks._framework.cli validate \
    tests/benchmarks/cloudopsbench/configs/cloudopsbench_smoke.yml

Validation fails on anti-patterns such as runs_per_case < 3, a missing pre_registration_path, oversized grids, or a system-path output_dir.

Development run

--dev skips integrity gates so you can smoke-test the wiring without a committed pre-registration file. The run ID is prefixed with dev- so development results are not treated as production results.

uv run python -m tests.benchmarks._framework.cli run \
    tests/benchmarks/cloudopsbench/configs/cloudopsbench_smoke.yml --dev

Production run

A production run requires:

A pre-registration YAML at pre_registration_path that lists per-model expected deltas and is committed to git before the run starts
seed: set in the config
Adapter declaration of data_contamination_checked = True
At least one validity metric declared by the adapter

uv run python -m tests.benchmarks._framework.cli run \
    tests/benchmarks/cloudopsbench/configs/cloudopsbench_v1.yml

On completion, the run directory contains report.json, report.md, report.html (self-contained), and cases/*.json (per-cell artifacts).

Re-render an existing report

uv run python -m tests.benchmarks._framework.cli report \
    .bench-results/example/<run-dir>/

Config reference

Example production config shape (see tests/benchmarks/cloudopsbench/configs/cloudopsbench_v1.yml for the full file):

benchmark: cloudopsbench

modes:
  - opensre+llm          # opensre wrapping the LLM
  # - llm_alone          # paper baselines; rerun only if you do not trust them

llms:
  - claude-4-sonnet
  - deepseek-v3.2
  - gpt-5
  - gpt-4o

model_versions:          # pinned provider snapshots
  claude-4-sonnet: claude-sonnet-4-5-20250929
  deepseek-v3.2:   deepseek-chat-v3.2
  gpt-5:           gpt-5-2025-08-07
  gpt-4o:          gpt-4o-2024-11-20

runs_per_case: 3
workers: 4               # serial across LLMs, parallel within
cost_budget_usd: 500
seed: 42

filters:                 # optional case subsetting
  systems: [boutique]
  difficulty: [hard, medium]

output_dir: .bench-results/cloudopsbench_v1/
report_formats: [json, markdown, html]
pre_registration_path: tests/benchmarks/cloudopsbench/configs/preregistrations/cloudopsbench_v1.yml

Environment variable overrides (CI)

Variable	Purpose
`OPENSRE_BENCH_WORKERS`	Override `workers:`
`OPENSRE_BENCH_COST_BUDGET_USD`	Override `cost_budget_usd:`

Integrity guarantees

The framework enforces 11 honest-results mechanisms in code. There is no bypass short of editing the framework itself.

Pre-flight (before any case runs)

IntegrityGuard.pre_flight raises IntegrityViolation if any of the following hold:

M1 — Pre-registration: pre_registration_path is unset, missing, or empty. Expected deltas must be committed before results are known.
M3 — Validity metrics: the adapter declares no validity metric.
M6 — Seeded selection: seed: is None.
M7 — Contamination check: the adapter has not declared data_contamination_checked = True.

Violations are reported together so they can be fixed in one pass.

Report validation (before the report is emitted)

IntegrityGuard.report_validation refuses to publish a report if:

M3 — Not every adapter-declared metric appears in the report
M4 — Per-stratum breakdown is missing or contains only all
M5 — Raw per-case artifacts directory is missing
M9 — negative_results is empty
M10 — coi_disclosure is empty
M1 — Pre-registration path is not carried into the report

Operational mechanisms (not code-enforced)

M8 — External replication of at least one cell by a third party before a public claim
M11 — Blinded LLM-as-judge calibration (BDIL Phase B; tracked separately)

Cost tracking

The framework registers a usage hook on the LLM clients in core/llm/transports/sdk/llm_clients.py. Each successful call records (model, tokens_in, tokens_out) in a CostTracker. When the next call would exceed cost_budget_usd, the runner raises CostBudgetExceeded and stops with a partial-completion report. Per-cell tokens_in / tokens_out / cost_usd may currently be 0 (aggregate cost in report.json is correct; per-cell capture is a follow-up).

Metrics

The paper’s 13 deterministic metrics plus three framework validity metrics:

Family	Metric	Source
Outcome	`a1`, `a3`, `tcr`, `exact`, `in_order`, `any_order`	Paper § 4.2.1
Process — alignment	`rel`, `cov`	Paper § 4.2.2
Process — efficiency	`steps`, `mtti`	Paper § 4.2.2
Process — robustness	`iac`, `rar`, `ztdr`	Paper § 4.2.2
Validity	`citation_grounding_rate`, `entity_existence_rate`, `kubectl_actionability_rate`	Framework

All 16 metrics are deterministic (string or set comparison). Evaluation does not use an LLM as judge.

Existing production entry points

make test-cloudopsbench and opensre tests cloudopsbench use tests/benchmarks/cloudopsbench/run_suite.py (legacy imperative CLI). The framework runner is the YAML-config surface. Both share the same adapter, scoring code, and replay backend.

Reference

Paper: Wang et al, Cloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud Systems, arXiv:2603.00468v1, 28 February 2026 — GitHub
Hugging Face dataset: tracer-cloud/cloud-ops-bench-dataset
Framework source: tests/benchmarks/_framework/
Adapter source: tests/benchmarks/cloudopsbench/

Deploy and APIs

Community

Overview

Prerequisites

Quick start

List adapters

Validate a config

Development run

Production run

Re-render an existing report

Config reference

Environment variable overrides (CI)

Integrity guarantees

Pre-flight (before any case runs)

Report validation (before the report is emitted)

Operational mechanisms (not code-enforced)

Cost tracking

Metrics

Existing production entry points

Reference

​Overview

​Prerequisites

​Quick start

​List adapters

​Validate a config

​Development run

​Production run

​Re-render an existing report

​Config reference

​Environment variable overrides (CI)

​Integrity guarantees

​Pre-flight (before any case runs)

​Report validation (before the report is emitted)

​Operational mechanisms (not code-enforced)

​Cost tracking

​Metrics

​Existing production entry points

​Reference

Overview

Prerequisites

Quick start

List adapters

Validate a config

Development run

Production run

Re-render an existing report

Config reference

Environment variable overrides (CI)

Integrity guarantees

Pre-flight (before any case runs)

Report validation (before the report is emitted)

Operational mechanisms (not code-enforced)

Cost tracking

Metrics

Existing production entry points

Reference