Rootline v0.1 pilot

← All incidents

toxiReset-r2

What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)

Service graph during the window

Evidence and answers

4616 prompt tokens

metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean

M-0001svc=ad metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:1,na,new:1]
M-0002svc=frontend-proxy metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:0.000227,na,+0%]
M-0003svc=load-generator metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:0.00229,na,+0%]
M-0004svc=payment metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,new:0.0667,na,+0%,na,+0%]
M-0005svc=payment metric=traces.span.metrics.duration.p50 base=1.11 delta/30s=[na,-1%,na,-10%,na,+1351653%,na,+10%,na,-1%]
M-0006svc=payment metric=traces.span.metrics.duration.p95 base=7.92 delta/30s=[na,-52%,na,-76%,na,+189294%,na,-38%,na,-63%]
M-0007svc=payment metric=traces.span.metrics.duration.p99 base=12.8 delta/30s=[na,+222%,na,-85%,na,+117234%,na,-55%,na,-70%]
M-0008svc=recommendation metric=process.runtime.cpython.cpu.utilization base=0 delta/30s=[new:13.1,na,new:13.8,na,new:13.4,na,+0%,na,+0%,na]
M-0009svc=unknown metric=system.disk.pending_operations base=0 delta/30s=[na,+0%,na,+0%,na,new:33.0,na,+0%,na,+0%]
M-0010svc=unknown metric=system.processes.count.rate base=0.02 delta/30s=[na,+567%,na,+900%,na,+1567%,na,+900%,na,+567%]
M-0011svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,-100%,-100%,-100%,-100%,-100%,-100%,+900%,-100%,-100%]
M-0012svc=frontend-web metric=traces.span.metrics.duration.p99 base=668.4 delta/30s=[na,+112%,na,+504%,na,+518%,na,+1%,na,+18%]
M-0013svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,-100%,na,+400%,na,-100%,na]
M-0014svc=accounting metric=dns.lookup.duration.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,-100%,na,+400%,na,-100%,na]
M-0015svc=cart metric=dotnet.thread_pool.thread.count.rate base=0.00667 delta/30s=[+400%,na,-100%,na,-100%,na,-100%,na,-100%,na]
M-0016svc=cart metric=process.thread.count.rate base=0.00667 delta/30s=[+400%,na,-100%,na,-100%,na,-100%,na,-100%,na]
M-0017svc=checkout metric=otel.sdk.processor.log.queue.size base=1.2 delta/30s=[na,-100%,na,-100%,na,-100%,na,+400%,na,-100%]
M-0018svc=fraud-detection metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,-100%,na,+400%,na,-100%,na]
M-0019svc=product-catalog metric=go.goroutine.count.rate base=0.0133 delta/30s=[na,-100%,na,-100%,na,-100%,na,+400%,na,-100%]
M-0020svc=frontend-proxy metric=traces.span.metrics.duration.p50 base=424.4 delta/30s=[na,-98%,na,-99%,na,+384%,na,-99%,na,+382%]
M-0021svc=shipping metric=http.server.request.duration.p50 base=0.0366 delta/30s=[na,-46%,na,+255%,na,+296%,na,+2%,na,+122%]
M-0022svc=recommendation metric=otel.sdk.processor.log.queue.size base=0.8 delta/30s=[-100%,na,+25%,na,-100%,na,-100%,na,+275%,na]
M-0023svc=load-generator metric=otel.sdk.processor.log.queue.size base=2.8 delta/30s=[na,+43%,na,+257%,na,+43%,na,+257%,na,-29%]
M-0024svc=frontend metric=http.server.request.duration.p95 base=0.17 delta/30s=[+17%,na,+193%,na,+241%,na,-42%,na,-15%,na]
M-0025svc=checkout metric=http.client.request.duration.p50 base=0.00524 delta/30s=[na,-13%,na,+234%,na,-5%,na,-12%,na,-20%]
M-0026svc=quote metric=traces.span.metrics.duration.p50 base=7.2 delta/30s=[na,-28%,na,+224%,na,+150%,na,+2%,na,+11%]
M-0027svc=shipping metric=traces.span.metrics.duration.p95 base=574.7 delta/30s=[na,+74%,na,+72%,na,+140%,na,-13%,na,+206%]
M-0028svc=checkout metric=http.client.request.duration.p95 base=0.382 delta/30s=[na,+155%,na,+201%,na,-42%,na,-58%,na,+96%]
M-0029svc=checkout metric=rpc.server.call.duration.p50 base=0.0592 delta/30s=[na,+101%,na,+196%,na,+69%,na,-16%,na,-16%]
M-0030svc=shipping metric=http.server.request.duration.p95 base=0.7 delta/30s=[na,+59%,na,+150%,na,+145%,na,-19%,na,+190%]
M-0031svc=accounting metric=traces.span.metrics.duration.p50 base=3.46e+03 delta/30s=[na,-47%,na,+34%,na,+189%,na,+30%,na,+45%]
M-0032svc=load-generator metric=http.client.duration.p95 base=221.4 delta/30s=[na,+137%,na,+179%,na,+10%,na,-41%,na,+103%]
M-0033svc=frontend-web metric=traces.span.metrics.duration.p95 base=414.7 delta/30s=[na,-4%,na,+170%,na,-4%,na,-4%,na,+18%]
M-0034svc=flagd metric=error_ratio base=0.000184 delta/30s=[na,-100%,na,-100%,na,+170%,na,-100%,na,-100%]
M-0035svc=quote metric=traces.span.metrics.duration.p95 base=246.6 delta/30s=[na,+66%,na,+135%,na,+165%,na,+1%,na,+127%]
M-0036svc=load-generator metric=process.runtime.cpython.gc_count base=252.2 delta/30s=[na,+163%,na,-97%,na,-96%,na,-94%,na,-36%]
M-0037svc=otelcol-contrib metric=otelcol_exporter_in_flight_requests base=0.4 delta/30s=[na,+150%,na,+150%,na,-100%,na,-100%,na,-100%]
876 lower-ranked lines omitted

traces: slowest and errored traces after fault onset, critical path (! = error span)

T-0001trace=53662886 dur=3ms ERROR path: load-generator:user_browse_product 3ms! > frontend-proxy:GET 0ms!
T-0002trace=a8254545 dur=25021ms ok path: accounting:order-consumed 25021ms
T-0003trace=b5668533 dur=16643ms ok path: accounting:order-consumed 16643ms
T-0004trace=81fcb890 dur=15364ms ok path: accounting:order-consumed 15364ms
T-0005trace=06996227 dur=13515ms ok path: accounting:order-consumed 13515ms
T-0006trace=dfd4475b dur=12265ms ok path: accounting:order-consumed 12265ms
T-0007trace=d1b9e77b dur=12147ms ok path: accounting:order-consumed 12147ms
T-0008trace=49819e8f dur=11207ms ok path: accounting:order-consumed 11207ms
T-0009trace=e9720eb4 dur=10688ms ok path: accounting:order-consumed 10688ms
T-0010trace=162aa997 dur=10446ms ok path: accounting:order-consumed 10446ms
T-0011trace=046b2f2b dur=10308ms ok path: accounting:order-consumed 10308ms
T-0012trace=cc8c1eab dur=10060ms ok path: accounting:order-consumed 10060ms
T-0013trace=9ff41b96 dur=9581ms ok path: accounting:order-consumed 9581ms
T-0014trace=c798a48e dur=8989ms ok path: accounting:order-consumed 8989ms
T-0015trace=2b913a52 dur=8672ms ok path: accounting:order-consumed 8672ms
T-0016trace=15445141 dur=7395ms ok path: accounting:order-consumed 7395ms
T-0017trace=29fdb821 dur=6479ms ok path: frontend-proxy:POST 6479ms > flagd:flagd.evaluation.v1.Service/EventStream 6478ms
T-0018trace=2ca41a67 dur=6260ms ok path: accounting:order-consumed 6260ms
T-0019trace=39077555 dur=6214ms ok path: accounting:order-consumed 6214ms
T-0020trace=dfc7a739 dur=6051ms ok path: frontend-proxy:POST 6051ms > flagd:flagd.evaluation.v1.Service/EventStream 6051ms
T-0021trace=ad25d381 dur=5946ms ok path: accounting:order-consumed 5946ms

logs: log templates per service, rate before vs after fault onset, with examples

L-0001svc=load-generator sev=INFO base=14.4/min fault=8.0/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user 28f3a6ac-dbe9-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user 28f3a6ac-dbe9-11f1-8306-d6b650b5399b']
L-0002svc=shipping sev=INFO base=1.6/min fault=0.8/min tpl="" ex=['', '']
L-0003svc=load-generator sev=INFO base=76.0/min fault=59.2/min tpl="User <UUID> adding <NUM> of product <*> to cart" ex=['User 24dcc850-dbe9-11f1-8306-d6b650b5399b adding 4 of product L9ECAV7KIM to cart', 'User 24dcc850-dbe9-11f1-8306-d6b650b5399b adding 4 of product L9ECAV7KIM to cart']
L-0004svc=load-generator sev=INFO base=13.6/min fault=17.6/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0005svc=load-generator sev=INFO base=25.6/min fault=31.6/min tpl="Product added to cart successfully" ex=['Product added to cart successfully', 'Product added to cart successfully']
L-0006svc=accounting sev=INFORMATION base=13.4/min fault=10.8/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0007svc=cart sev=INFORMATION base=13.4/min fault=10.8/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0008svc=checkout sev=INFO base=13.4/min fault=10.8/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 203.619µs', 'Successful to write message. offset: 0, duration: 257.451µs']
L-0009svc=checkout sev=INFO base=13.4/min fault=10.8/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0010svc=checkout sev=INFO base=13.4/min fault=10.8/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0011svc=checkout sev=INFO base=13.4/min fault=10.8/min tpl="order placed" ex=['order placed', 'order placed']
L-0012svc=checkout sev=INFO base=13.4/min fault=10.8/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0013svc=checkout sev=INFO base=13.4/min fault=10.8/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0014svc=email sev=INFO base=13.4/min fault=10.8/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0015svc=fraud-detection sev=INFO base=13.4/min fault=10.8/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: 28ff35a0-dbe9-11f1-82f3-d279f8fa1f6d, and updated total count to: 20653', 'Consumed record with orderId: 2b0559c8-dbe9-11f1-82f3-d279f8fa1f6d, and updated total count to: 20654']
L-0016svc=frontend sev=INFO base=13.4/min fault=10.8/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0017svc=payment sev=INFO base=13.4/min fault=10.8/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0018svc=payment sev=INFO base=13.4/min fault=10.8/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0019svc=shipping sev=INFO base=13.4/min fault=10.8/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0020svc=checkout sev=INFO base=0.2/min fault=0.0/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0021svc=load-generator sev=INFO base=35.6/min fault=29.6/min tpl="Currency changed to CHF" ex=['Currency changed to CHF', 'Currency changed to CHF']
L-0022svc=currency sev=INFO base=122.0/min fault=103.0/min tpl="conversion successful" ex=['conversion successful', 'conversion successful']
L-0023svc=ad sev=INFO base=2.6/min fault=3.2/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0024svc=ad sev=INFO base=46.4/min fault=52.4/min tpl="Targeted ad request received for <*>" ex=['Targeted ad request received for [binoculars]', 'Targeted ad request received for [binoculars]']
L-0025svc=cart sev=INFORMATION base=50.8/min fault=45.2/min tpl="AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}" ex=['AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}', 'AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}']
L-0026svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=55716, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=55866, siz…']
L-0027svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 55932 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 56049 breach: LogSegment(baseOff…']
L-0028svc=kafka sev=INFO base=7.6/min fault=8.4/min tpl="Deleted <*> index <*>" ex=['Deleted offset index /tmp/kafka-logs/__cluster_metadata-0/00000000000000055716.index.deleted.', 'Deleted offset index /tmp/kafka-logs/__cluster_metadata-0/00000000000000055746.index.deleted.']
32 lower-ranked lines omitted
Loading runs in your browser…