Rootline v0.1 pilot

● Open benchmark · pilot (n=42 snapshots)

When the shop breaks, can an AI find why?

Rootline breaks a real microservice shop on purpose, freezes what its engineers would see, and asks free, open AI models for the root cause. It measures which evidence they need, and whether they can tell a broken network from a broken service.

  • 15incident types
  • 42frozen snapshots
  • 3open-weight models
  • 2Arduino UNO Q boards

Requests from checkout to payment are lost on the way. Payment itself is fine. Drag to turn. Hover a service.

01 / The problem

Everyone wants AI on call. Nobody checks what it looked at.

When a web shop fails, the error shows up far from where it started. A checkout page times out because payment is slow, because a link is dropping packets, because a radio is weak. Engineers trace back through metrics, traces and logs until they find the one thing that broke.

Language models are now pitched for that job. But most evaluations grade a free-text answer with another model, and say little about which evidence the answer rests on, or whether the model can tell a network problem from an application one.

Rootline grades by evidence: every line a model sees has an ID, and an answer only counts if it cites lines that exist.

prompt.py · the exact instructionsevery model, every run
# excerpt
You are the on-call engineer for a microservice web shop. Below is telemetry from a 10-minute window; "fault onset" is the midpoint. The window may contain a fault, or the system may be healthy.

Find the root cause using only the evidence below.

Layers: application (the service itself is broken), network (the link between services is degraded), none (healthy).

Rules: give up to 3 hypotheses ranked best first; cite only IDs that appear below; confidence is your probability that the hypothesis is right.

02 / The method

Break it, freeze it, replay it, score it.

  1. i.

    Break

    A Go controller injects one of 15 incident types into the OpenTelemetry Demo, a shop of about 20 services under steady load.

    flagd · Toxiproxy · tc netem · SSH to the boards
  2. ii.

    Freeze

    Ten minutes of spans, metrics and logs around the fault are written to Parquet. The answer is stored apart.

    OTLP · Apache Arrow · Parquet
  3. iii.

    Replay

    The same frozen snapshot goes to every model, packed into the same 6,000 tokens, under four evidence conditions.

    Polars · Drain3 · tiktoken
  4. iv.

    Score

    Answers follow a strict JSON contract and are scored for service, category, layer and cited evidence.

    Pydantic · exact tests · bootstrap CIs

One incident · drag the timeline

T0 + 2.0 minFaultin snapshot

At T0 the fault is injected: a feature flag, a Toxiproxy toxic, a tc netem rule, or a command on an Arduino board.

The first incident of a night adds a 10-minute warm-up. One incident takes 13 to 23 minutes, depending on how fast health recovers; the 30 pilot snapshots run unattended in one night.

15 incident types · filter, open a row

Application faults use the demo's own feature flags. Network faults degrade one link between two services. The healthy control checks for false alarms.

The 15 incident types of the pilot
#IncidentRoot causeCategory
01paymenterror
02checkout → paymentdependency unavailable
03product-catalogerror
04product-cataloglock contention
05emailmemory leak
06recommendationmemory leak
07fraud-detectionqueue lag
08shippinglatency
09frontend-proxylatency
10adCPU saturation
11checkout → paymentnetwork latency
12frontend → product-catalognetwork loss
13checkout → paymentnetwork reset
network layer · how it is injected

Toxiproxy reset_peer toxic: 20% of connections are cut.

14frontend → product-catalognetwork throughput
15–none

03 / Two questions

What evidence it needs, and where it thinks the fault is.

RQ1

Which signals does a model need to find the cause?

Why
Teams pay to keep metrics, traces and logs. If one signal carries most of the answer, budget-constrained teams can keep less.
Measure
Top-1 accuracy of the strongest model with combined evidence against its best single signal, on the same snapshots.
Test
McNemar exact test; threshold +10 points.
RQ2

Can a model tell a broken link from a broken service?

Why
A slow network and a slow service look alike from the caller's side. Restarting a healthy service does nothing for a degraded link.
Measure
Layer accuracy (application, network or none) on network faults against application faults.
Test
Fisher exact test; threshold 15 points lower on network faults.

RQ1 · what a model receives

Pick a condition. The token budget never changes, only what fills it.

metrics · 2,000
traces · 2,000
logs · 2,000
06,000 tokens (cl100k_base)

A third of each

Metrics, traces and logs at a third of the budget each. RQ1 asks whether this beats the best single signal.

Every line carries a stable ID (M · T · L-0042), and answers must cite the IDs they rely on, so cited evidence can be checked.

Four more questions (name reliance, cheap-first cascades, whether fixes work, classical baselines) are written and pre-stated, and held for v0.2.

04 / Results

What the models found.

ModelTierTop-1 (95% CI)Top-3LayerEvidence validityHealthy false alarms
gpt-oss-120b@groqC9.5% (2.4%–19.0%)19.0%66.7%16.9%100.0%
gemma3-4b-q4@ollamaA4.8% (0.0%–11.9%)9.5%47.6%14.8%100.0%
qwen3-8b-q4@ollamaB2.4% (0.0%–7.1%)2.4%31.0%17.0%100.0%
RQ1

Combined signals beat the best single signal by >= 10 pp top-1

Effect
-0.024
p (Holm)
1.000
Result
not supported
RQ2

Layer accuracy is >= 15 pts lower on network faults than application faults

Effect
0.300
p (Holm)
0.165
Result
threshold met, not significant

Every negative result is reported too. Replay any snapshot on the incidents pages.

05 / Hardware

Two small boards, one real radio link.

Emulated faults are tidy. Real Wi-Fi is not. The hardware arm checks whether models read a genuinely degraded link the way they read an emulated one.

Arduino UNO Q · 68.6 × 53.3 mm · drawing, positions approximate12345A
Fig. · Board ATap a numbered part
In Rootline

edge-relay, a small Go program run by systemd, carries every checkout → payment request over the board's real Wi-Fi. Faults: transmit power dropped to 1 dBm, or the radio switched off for 30 s.

Board A, the relay

  • Nameaffan1
  • JobCarries checkout → payment over Wi-Fi
  • FaultsWeak signal, 30-second Wi-Fi outage
Arduino UNO Q · 68.6 × 53.3 mm · drawing, positions approximate12345B
Fig. · Board BTap a numbered part
In Rootline

Sends 30 Mbit/s of UDP with iperf3 to the PC for 5 minutes, crowding the 2.4 GHz channel Board A depends on. Only ever to a private address on the home network.

Board B, the crowd

  • Nameaffan2
  • JobCrowds the 2.4 GHz channel on cue
  • FaultsChannel contention

How it is wired · switch a fault on

PC · WSL · Dockercheckoutpaymentsnapshotteriperf3 srvWi-Fi router2.4 GHz · ch 6Board A · affan1edge-relay :50051tx power 1 dBmBoard B · affan2iperf3 idlecheckout → payment

Weak signal Board A's transmit power drops from about 20 dBm to 1 dBm. Its uplink gets lossy; the board puts power back by itself after 8 minutes at the latest.

All three hardware faults run unattended over SSH, and each board schedules its own undo, so a lost connection never leaves a fault in place. Hardware details

06 / Limitations

What this pilot cannot tell you.

Read every result with these in mind. The full list is in the repository's README.

  1. A small pilot

    Two replicas of each incident type: 30 snapshots. Confidence intervals are wide, and a result that misses significance may only mean the sample is too small.

  2. Two questions, not six

    Name reliance, cascades, whether fixes work and classical baselines are held for v0.2.

  3. One testbed, one PC

    The OpenTelemetry Demo 3.1.0 on Docker Compose under 20 simulated users. Production systems are larger and less tidy.

  4. Changed resource limits

    A few services, the database among them, got more memory so the testbed itself stayed up. Snapshots hit by a testbed crash were moved aside and run again.

  5. Faults a model may have seen

    The application faults are the demo's public feature flags. Fault names never reach a model, service names do.

  6. Free models only

    Open-weight models and free tiers, one run per snapshot and condition at temperature 0. Hosted models can change without notice.