Rootline v0.1 pilot

Method

Rootline runs the live system once to freeze snapshots, then replays the same snapshots to every model. Only free open-weight models and free-tier APIs are evaluated, so results describe what budget-constrained teams can do today, not the frontier.

Testbed and faults

The OpenTelemetry Demo runs on Docker Compose, pinned to a release tag and image digests. Ten application faults come from the demo's feature flags, four network faults from Toxiproxy and tc netem, and one healthy control means a model cannot assume every snapshot is broken.

FaultWhereCategory
paymentFailurepaymenterror
paymentUnreachablecheckout → paymentdependency unavailable
productCatalogFailureproduct-catalogerror
productCatalogLockContentionproduct-cataloglock contention
emailMemoryLeakemailmemory leak
recommendationCacheFailurerecommendationmemory leak (cache growth)
kafkaQueueProblemsfraud-detectionqueue lag
intlShippingSlowdownshippinglatency (non-US only)
imageSlowLoadfrontend-proxylatency
adHighCpuadCPU saturation
toxiLatencycheckout → paymentnetwork latency (300 ± 60 ms)
netemLossfrontend → product-catalognetwork loss (5%)
toxiResetcheckout → paymentnetwork reset (20% of connections)
toxiBandwidthfrontend → product-catalognetwork throughput (64 KB/s)
healthy–none (control)

Incident protocol

20 simulated users; a 5-minute baseline; the fault for 5 minutes; a snapshot of the window from 5 minutes before to 5 minutes after onset; then the fault is cleared, the system cools down for 3 minutes, and recovery is checked (error ratio within 1.5× and p99 latency within 1.2× of baseline, for up to 10 minutes). Ground-truth labels are stored apart from the snapshots and never shown to a model.

Evidence conditions

Every prompt is at most 6,000 tokens, so conditions differ in signal, not size: metrics (per-service rates, error ratios and latency quantiles in 30-second buckets as change from baseline), traces (the 20 slowest and 20 errored traces as critical paths), logs (Drain3 templates ranked by rate change, two examples each), and combined (a third of the budget each). Every line has a stable ID so cited evidence can be checked. Perturbation P1 replaces service names with consistent anonymous ones.

Metrics

Statistics

McNemar's exact test for paired comparisons on the same snapshots, Fisher's exact test for unpaired ones, 10,000-resample bootstrap confidence intervals with a fixed seed, and Holm correction across the research questions. Every result is reported, including hypotheses that fail.

Limitations