Method
Rootline runs the live system once to freeze snapshots, then replays the same snapshots to every model. Only free open-weight models and free-tier APIs are evaluated, so results describe what budget-constrained teams can do today, not the frontier.
Testbed and faults
The OpenTelemetry Demo runs on Docker Compose, pinned to a release tag and image digests. Ten application faults come from the demo's feature flags, four network faults from Toxiproxy and tc netem, and one healthy control means a model cannot assume every snapshot is broken.
| Fault | Where | Category |
|---|---|---|
| paymentFailure | payment | error |
| paymentUnreachable | checkout → payment | dependency unavailable |
| productCatalogFailure | product-catalog | error |
| productCatalogLockContention | product-catalog | lock contention |
| emailMemoryLeak | memory leak | |
| recommendationCacheFailure | recommendation | memory leak (cache growth) |
| kafkaQueueProblems | fraud-detection | queue lag |
| intlShippingSlowdown | shipping | latency (non-US only) |
| imageSlowLoad | frontend-proxy | latency |
| adHighCpu | ad | CPU saturation |
| toxiLatency | checkout → payment | network latency (300 ± 60 ms) |
| netemLoss | frontend → product-catalog | network loss (5%) |
| toxiReset | checkout → payment | network reset (20% of connections) |
| toxiBandwidth | frontend → product-catalog | network throughput (64 KB/s) |
| healthy | – | none (control) |
Incident protocol
20 simulated users; a 5-minute baseline; the fault for 5 minutes; a snapshot of the window from 5 minutes before to 5 minutes after onset; then the fault is cleared, the system cools down for 3 minutes, and recovery is checked (error ratio within 1.5× and p99 latency within 1.2× of baseline, for up to 10 minutes). Ground-truth labels are stored apart from the snapshots and never shown to a model.
Evidence conditions
Every prompt is at most 6,000 tokens, so conditions differ in signal, not size: metrics (per-service rates, error ratios and latency quantiles in 30-second buckets as change from baseline), traces (the 20 slowest and 20 errored traces as critical paths), logs (Drain3 templates ranked by rate change, two examples each), and combined (a third of the budget each). Every line has a stable ID so cited evidence can be checked. Perturbation P1 replaces service names with consistent anonymous ones.
Metrics
- Top-1 and top-3 accuracy on the (service, category) pair
- Layer attribution: network vs application
- Evidence validity: cited IDs that exist and belong to the root-cause service
- Healthy false-alarm rate; calibration (Brier score, 10-bin ECE)
- Cascade accuracy per 1,000 tokens and escalation rate
- Remediation success within 120 s and harmful-action rate, from live trials
Statistics
McNemar's exact test for paired comparisons on the same snapshots, Fisher's exact test for unpaired ones, 10,000-resample bootstrap confidence intervals with a fixed seed, and Holm correction across the research questions. Every result is reported, including hypotheses that fail.
Limitations
- This is a pilot: two replicas per incident type give wide confidence intervals.
- One testbed; real production systems are larger and noisier.
- Network faults are emulated; the hardware arm compares them with real Wi-Fi faults.
- Models may have seen the demo's public code; P1 measures reliance on names.