Rootline v0.1 pilot

← All incidents

hwWifiOutage-r3

What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)

Service graph during the window

Evidence and answers

4785 prompt tokens

metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean

M-0001svc=accounting metric=dotnet.thread_pool.thread.count.rate base=0 delta/30s=[na,+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%]
M-0002svc=accounting metric=process.thread.count.rate base=0 delta/30s=[na,+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%]
M-0003svc=checkout metric=error_ratio base=0 delta/30s=[na,+0%,na,new:0.00971,na,new:0.00787,na,+0%,na,+0%]
M-0004svc=checkout metric=go.goroutine.count.rate base=0 delta/30s=[na,+0%,na,new:0.133,na,+0%,na,new:0.0667,na,+0%]
M-0005svc=checkout metric=otel.sdk.span.live base=0 delta/30s=[na,new:2,na,+0%,na,+0%,na,+0%,na,+0%]
M-0006svc=frontend metric=error_ratio base=0 delta/30s=[na,new:0.000747,na,+0%,na,new:0.000929,na,+0%,na,+0%]
M-0007svc=frontend-proxy metric=error_ratio base=0 delta/30s=[na,new:0.000239,na,new:0.000241,na,new:0.000488,na,+0%,na,+0%]
M-0008svc=kafka metric=jvm.gc.duration.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0009svc=kafka metric=jvm.thread.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0010svc=load-generator metric=error_ratio base=0 delta/30s=[na,new:0.0024,na,new:0.00203,na,new:0.00189,na,+0%,na,+0%]
M-0011svc=load-generator metric=http.client.duration.p50 base=7.78 delta/30s=[na,+128382%,na,+128382%,na,+1%,na,-5%,na,+3%]
M-0012svc=quote metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:1]
M-0013svc=unknown metric=system.disk.pending_operations base=0 delta/30s=[na,new:2,na,+0%,na,+0%,na,+0%,na,+0%]
M-0014svc=checkout metric=rpc.client.call.duration.p50 base=0.0178 delta/30s=[na,+56238%,na,+64%,na,+20%,na,+13%,na,+32%]
M-0015svc=checkout metric=rpc.client.call.duration.p95 base=0.0452 delta/30s=[na,+22010%,na,+22010%,na,+71%,na,+154%,na,+58%]
M-0016svc=checkout metric=rpc.client.call.duration.p99 base=0.053 delta/30s=[na,+18752%,na,+18752%,na,+80%,na,+320%,na,+40%]
M-0017svc=checkout metric=traces.span.metrics.duration.p50 base=127.5 delta/30s=[na,+135%,na,+11665%,na,-22%,na,-49%,na,+57%]
M-0018svc=frontend metric=traces.span.metrics.duration.p50 base=141.7 delta/30s=[na,+10488%,na,+10488%,na,-8%,na,-6%,na,+41%]
M-0019svc=checkout metric=rpc.server.call.duration.p50 base=0.124 delta/30s=[na,+7981%,na,+90%,na,-29%,na,-46%,na,+153%]
M-0020svc=load-generator metric=http.client.duration.p95 base=313.0 delta/30s=[na,+3095%,na,+3095%,na,-10%,na,-34%,na,+124%]
M-0021svc=checkout metric=traces.span.metrics.duration.p95 base=674.0 delta/30s=[na,+87%,na,+2126%,na,+54%,na,-63%,na,+28%]
M-0022svc=checkout metric=traces.span.metrics.duration.p99 base=806.8 delta/30s=[na,+70%,na,+1759%,na,+65%,na,-54%,na,+20%]
M-0023svc=product-catalog metric=otel.sdk.processor.log.queue.size base=1.4 delta/30s=[+1757%,na,+1757%,na,+1543%,na,+1186%,na,-100%,na]
M-0024svc=frontend-proxy metric=traces.span.metrics.duration.p50 base=832.1 delta/30s=[na,+1703%,na,+1703%,na,+146%,na,-99%,na,+147%]
M-0025svc=frontend metric=traces.span.metrics.duration.p95 base=956.0 delta/30s=[na,+1469%,na,+1469%,na,+19%,na,-37%,na,-10%]
M-0026svc=checkout metric=rpc.server.call.duration.p95 base=0.677 delta/30s=[na,+1377%,na,+1377%,na,+70%,na,-22%,na,+39%]
M-0027svc=frontend metric=http.server.request.duration.p99 base=0.692 delta/30s=[na,+1344%,na,+1344%,na,+16%,na,-17%,na,+34%]
M-0028svc=frontend metric=traces.span.metrics.duration.p99 base=1.12e+03 delta/30s=[na,+1240%,na,+1240%,na,+20%,na,-32%,na,-13%]
M-0029svc=load-generator metric=http.client.duration.p99 base=843.5 delta/30s=[na,+1086%,na,+1086%,na,+63%,na,-50%,na,+11%]
M-0030svc=checkout metric=rpc.server.call.duration.p99 base=0.895 delta/30s=[na,+1017%,na,+1017%,na,+149%,na,-21%,na,+10%]
M-0031svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,+900%,-100%,-100%,-100%,-100%,-100%,-100%,-100%,-100%]
M-0032svc=frontend-web metric=traces.span.metrics.duration.p95 base=800.0 delta/30s=[na,-22%,na,-37%,na,+40%,na,+728%,na,-20%]
M-0033svc=load-generator metric=otel.sdk.span.live base=1 delta/30s=[na,+300%,na,+100%,na,+100%,na,+300%,na,+700%]
M-0034svc=frontend metric=http.server.request.duration.p95 base=0.269 delta/30s=[na,+56%,na,+616%,na,-26%,na,-16%,na,+100%]
M-0035svc=email metric=traces.span.metrics.duration.p99 base=7.04 delta/30s=[na,+40%,na,+525%,na,+8%,na,-16%,na,+10%]
M-0036svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0.00667 delta/30s=[na,-100%,na,+400%,na,-100%,na,-100%,na,-100%]
M-0037svc=accounting metric=dns.lookup.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,+400%,na,-100%,na,-100%,na,-100%]
876 lower-ranked lines omitted

traces: slowest and errored traces after fault onset, critical path (! = error span)

T-0001trace=ae166f59 dur=21021ms ERROR path: load-generator:user_checkout_multi 15057ms! > frontend-proxy:POST 15000ms! > frontend:POST 15000ms! > checkout:oteldemo.CheckoutService/PlaceOrder 19209ms!
T-0002trace=f741774b dur=45352ms ok path: accounting:order-consumed 45352ms
T-0003trace=c68cb53f dur=16921ms ERROR path: load-generator:user_checkout_multi 15057ms! > frontend-proxy:POST 15000ms! > frontend:POST 15001ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 8ms
T-0004trace=8fae15d8 dur=22510ms ok path: accounting:order-consumed 22510ms
T-0005trace=d4719125 dur=20ms ERROR path: load-generator:user_checkout_single 20ms! > frontend-proxy:POST 6ms! > frontend:POST /api/checkout 6ms! > checkout:oteldemo.CheckoutService/PlaceOrder 4ms!
T-0006trace=d92ebe1f dur=13586ms ok path: accounting:order-consumed 13586ms
T-0007trace=818fe841 dur=13444ms ok path: accounting:order-consumed 13444ms
T-0008trace=1215590a dur=12901ms ok path: accounting:order-consumed 12901ms
T-0009trace=aac12890 dur=12029ms ok path: accounting:order-consumed 12029ms
T-0010trace=749a89f2 dur=11626ms ok path: accounting:order-consumed 11626ms
T-0011trace=59ad4182 dur=9896ms ok path: accounting:order-consumed 9896ms
T-0012trace=60ae02db dur=9587ms ok path: load-generator:user_checkout_multi 7934ms > frontend-proxy:POST 7867ms > frontend:POST /api/checkout 7868ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 7ms
T-0013trace=136af213 dur=9515ms ok path: load-generator:browser_change_currency 9515ms
T-0014trace=9d09f6ee dur=9476ms ok path: accounting:order-consumed 9476ms
T-0015trace=1a8f3466 dur=9180ms ok path: accounting:order-consumed 9180ms
T-0016trace=b7b5f625 dur=8061ms ok path: accounting:order-consumed 8061ms
T-0017trace=46c6abb4 dur=7564ms ok path: accounting:order-consumed 7564ms
T-0018trace=3458e33f dur=7505ms ok path: accounting:order-consumed 7505ms
T-0019trace=5558e6e3 dur=7486ms ok path: load-generator:browser_add_to_cart 7486ms
T-0020trace=e319262a dur=7455ms ok path: frontend-web:documentLoad 7455ms
T-0021trace=43949eea dur=7062ms ok path: accounting:order-consumed 7062ms
T-0022trace=b65aada8 dur=6844ms ok path: accounting:order-consumed 6844ms
T-0023trace=c399d9be dur=6812ms ok path: accounting:order-consumed 6812ms

logs: log templates per service, rate before vs after fault onset, with examples

L-0001svc=load-generator sev=INFO base=11.2/min fault=18.0/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0002svc=load-generator sev=INFO base=8.8/min fault=14.0/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user 2ede0c24-dbfd-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user 2ede0c24-dbfd-11f1-8306-d6b650b5399b']
L-0003svc=load-generator sev=INFO base=59.2/min fault=83.6/min tpl="User <UUID> adding <NUM> of product <*> to cart" ex=['User 2ede0c24-dbfd-11f1-8306-d6b650b5399b adding 3 of product 66VCHSJNUP to cart', 'User 2ede0c24-dbfd-11f1-8306-d6b650b5399b adding 3 of product 66VCHSJNUP to cart']
L-0004svc=frontend sev=ERROR base=0.0/min fault=0.4/min tpl="Checkout failed to place order" ex=['Checkout failed to place order', 'Checkout failed to place order']
L-0005svc=load-generator sev=INFO base=48.8/min fault=37.2/min tpl="User viewing cart" ex=['User viewing cart', 'User viewing cart']
L-0006svc=cart sev=INFORMATION base=44.0/min fault=56.0/min tpl="AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}" ex=['AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}', 'AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}']
L-0007svc=checkout sev=INFO base=11.2/min fault=13.8/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0008svc=checkout sev=INFO base=0.0/min fault=0.2/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0009svc=accounting sev=INFORMATION base=11.2/min fault=13.4/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0010svc=cart sev=INFORMATION base=11.2/min fault=13.4/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0011svc=checkout sev=INFO base=11.2/min fault=13.4/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 171.496µs', 'Successful to write message. offset: 0, duration: 142.125µs']
L-0012svc=checkout sev=INFO base=11.2/min fault=13.4/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0013svc=checkout sev=INFO base=11.2/min fault=13.4/min tpl="order placed" ex=['order placed', 'order placed']
L-0014svc=checkout sev=INFO base=11.2/min fault=13.4/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0015svc=checkout sev=INFO base=11.2/min fault=13.4/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0016svc=email sev=INFO base=11.2/min fault=13.4/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0017svc=fraud-detection sev=INFO base=11.2/min fault=13.4/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: 2ee59bd6-dbfd-11f1-9d0e-3288f2b1222e, and updated total count to: 22400', 'Consumed record with orderId: 412b4826-dbfd-11f1-9d0e-3288f2b1222e, and updated total count to: 22401']
L-0018svc=frontend sev=INFO base=11.2/min fault=13.4/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0019svc=payment sev=INFO base=11.2/min fault=13.4/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0020svc=payment sev=INFO base=11.2/min fault=13.4/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0021svc=shipping sev=INFO base=11.2/min fault=13.4/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0022svc=frontend-proxy sev=UNSET base=241.2/min fault=269.8/min tpl="<*> <*> <*> HTTP/<NUM>" <NUM> <*> <*> - "-" <NUM> <NUM> <NUM> <*> "-" <*> <*> <*> <*> <*> <*> <NUM>.<NUM>:<NUM> <NUM>.<NUM>:<NUM> - -" ex=['[2026-11-09T05:19:47.690Z] "GET /api/data/ HTTP/1.1" 308 - via_upstream - "-" 0 9 1 0 "-" "python-requests/2.34.2" "400aa86e-1098-93ac-a577…', '[2026-11-09T05:19:47.693Z] "GET /api/data HTTP/1.1" 200 - via_upstream - "-" 0 217 2 2 "-" "python-requests/2.34.2" "4c041c1b-07bb-9d16-ae1…']
L-0023svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 73053 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 73170 breach: LogSegment(baseOff…']
L-0024svc=kafka sev=INFO base=1.0/min fault=0.8/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=72985, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=73105, siz…']
L-0025svc=shipping sev=INFO base=1.0/min fault=0.8/min tpl="" ex=['', '']
36 lower-ranked lines omitted
Loading runs in your browser…