Rootline v0.1 pilot

← All incidents

hwContention-r3

What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)

Service graph during the window

Evidence and answers

4573 prompt tokens

metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean

M-0001svc=accounting metric=dotnet.thread_pool.thread.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0002svc=accounting metric=process.thread.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0003svc=ad metric=jvm.gc.duration.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0004svc=checkout metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[+0%,na,new:1,na,+0%,na,+0%,na,new:5,na]
M-0005svc=checkout metric=otel.sdk.span.live base=0 delta/30s=[+0%,na,new:2,na,+0%,na,+0%,na,+0%,na]
M-0006svc=fraud-detection metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:1,na,+0%,na]
M-0007svc=fraud-detection metric=otel.sdk.processor.span.queue.size base=0 delta/30s=[+0%,na,new:2,na,+0%,na,new:2,na,+0%,na]
M-0008svc=kafka metric=jvm.class.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0009svc=quote metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[+0%,na,new:1,na,new:1,na,+0%,na,new:1,na]
M-0010svc=checkout metric=rpc.client.call.duration.p50 base=0.022 delta/30s=[+99%,na,+697%,na,+1038%,na,+317%,na,+981%,na]
M-0011svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,-100%,-100%,-100%,+900%,-100%,-100%,-100%,-100%,-100%]
M-0012svc=load-generator metric=process.runtime.cpython.gc_count base=93.8 delta/30s=[+679%,na,+74%,na,-83%,na,-86%,na,-89%,na]
M-0013svc=otelcol-contrib metric=otelcol_exporter_queue_size base=1.8 delta/30s=[+11%,na,+678%,na,+233%,na,+11%,na,+67%,na]
M-0014svc=checkout metric=rpc.server.call.duration.p50 base=0.104 delta/30s=[+623%,na,+241%,na,+313%,na,+69%,na,+282%,na]
M-0015svc=quote metric=otel.sdk.processor.span.queue.size base=1.8 delta/30s=[-100%,na,+67%,na,+567%,na,+67%,na,+67%,na]
M-0016svc=checkout metric=traces.span.metrics.duration.p50 base=100.0 delta/30s=[+500%,na,+267%,na,+280%,na,+100%,na,+333%,na]
M-0017svc=load-generator metric=http.client.duration.p95 base=321.2 delta/30s=[+492%,na,+182%,na,+316%,na,+40%,na,+100%,na]
M-0018svc=frontend metric=http.server.request.duration.p95 base=0.286 delta/30s=[+214%,na,+485%,na,+66%,na,+44%,na,+83%,na]
M-0019svc=frontend-web metric=traces.span.metrics.duration.p99 base=2.47e+03 delta/30s=[+254%,na,+464%,na,+458%,na,-45%,na,+67%,na]
M-0020svc=frontend-proxy metric=traces.span.metrics.duration.p50 base=414.5 delta/30s=[+397%,na,+410%,na,-99%,na,+396%,na,-98%,na]
M-0021svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,-100%,na,+400%,na,-100%,na]
M-0022svc=accounting metric=dns.lookup.duration.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,-100%,na,+400%,na,-100%,na]
M-0023svc=fraud-detection metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,+400%,na,-100%,na,-100%,na]
M-0024svc=kafka metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,-100%,na,+400%,na,-100%]
M-0025svc=otelcol-contrib metric=otelcol_exporter_in_flight_requests base=0.2 delta/30s=[-100%,na,-100%,na,+400%,na,-100%,na,-100%,na]
M-0026svc=product-catalog metric=go.goroutine.count.rate base=0.0133 delta/30s=[-100%,na,-100%,na,+400%,na,-100%,na,+400%,na]
M-0027svc=recommendation metric=otel.sdk.processor.span.queue.size base=4.8 delta/30s=[na,+400%,na,+275%,na,+400%,na,+275%,na,+88%]
M-0028svc=checkout metric=rpc.client.call.duration.p95 base=0.849 delta/30s=[+371%,na,+239%,na,+312%,na,-62%,na,-20%,na]
M-0029svc=checkout metric=rpc.client.call.duration.p99 base=1.1 delta/30s=[+337%,na,+317%,na,+328%,na,-58%,na,-33%,na]
M-0030svc=quote metric=traces.span.metrics.duration.p50 base=5.92 delta/30s=[+294%,na,-3%,na,+238%,na,+69%,na,+61%,na]
M-0031svc=product-catalog metric=otel.sdk.processor.log.queue.size base=8.6 delta/30s=[-19%,na,-100%,na,+284%,na,-100%,na,-53%,na]
M-0032svc=frontend metric=traces.span.metrics.duration.p50 base=164.2 delta/30s=[+265%,na,+134%,na,+133%,na,+22%,na,+164%,na]
M-0033svc=frontend-web metric=traces.span.metrics.duration.p95 base=1.37e+03 delta/30s=[-43%,na,+261%,na,+243%,na,-11%,na,+36%,na]
M-0034svc=load-generator metric=traces.span.metrics.duration.p95 base=4.85e+03 delta/30s=[+147%,na,+142%,na,+209%,na,+0%,na,+0%,na]
M-0035svc=shipping metric=http.server.request.duration.p50 base=0.0573 delta/30s=[+206%,na,+104%,na,-34%,na,-29%,na,+53%,na]
M-0036svc=accounting metric=traces.span.metrics.duration.p50 base=4.92e+03 delta/30s=[+205%,na,-29%,na,-37%,na,-35%,na,-6%,na]
M-0037svc=load-generator metric=traces.span.metrics.duration.p99 base=4.97e+03 delta/30s=[+190%,na,+189%,na,+202%,na,+0%,na,+0%,na]
876 lower-ranked lines omitted

traces: slowest and errored traces after fault onset, critical path (! = error span)

T-0001trace=528ae9ca dur=22312ms ok path: accounting:order-consumed 22312ms
T-0002trace=c93bc44f dur=19397ms ok path: accounting:order-consumed 19397ms
T-0003trace=06fe6ff5 dur=16395ms ok path: load-generator:browser_change_currency 16395ms
T-0004trace=d806f2a7 dur=14225ms ok path: accounting:order-consumed 14225ms
T-0005trace=4f5d4a78 dur=14088ms ok path: frontend-web:documentLoad 14088ms
T-0006trace=7cbc21c1 dur=13918ms ok path: load-generator:browser_change_currency 13918ms
T-0007trace=3a149062 dur=12895ms ok path: accounting:order-consumed 12895ms
T-0008trace=ba393ec4 dur=12478ms ok path: accounting:order-consumed 12478ms
T-0009trace=88874654 dur=12443ms ok path: accounting:order-consumed 12443ms
T-0010trace=6fbb7b59 dur=11866ms ok path: frontend-web:documentLoad 11866ms
T-0011trace=fd328a1a dur=9989ms ok path: accounting:order-consumed 9989ms
T-0012trace=b1a285cd dur=9382ms ok path: accounting:order-consumed 9382ms
T-0013trace=d12dfb92 dur=9264ms ok path: accounting:order-consumed 9264ms
T-0014trace=317029dc dur=8525ms ok path: accounting:order-consumed 8525ms
T-0015trace=bd99350d dur=7806ms ok path: accounting:order-consumed 7806ms
T-0016trace=c0b2fe79 dur=7177ms ok path: load-generator:browser_change_currency 7177ms
T-0017trace=a11f1db4 dur=6958ms ok path: accounting:order-consumed 6958ms
T-0018trace=7436da8d dur=6615ms ok path: accounting:order-consumed 6615ms
T-0019trace=034d81e0 dur=6464ms ok path: load-generator:browser_change_currency 6464ms
T-0020trace=b6fe9d30 dur=6444ms ok path: load-generator:browser_add_to_cart 6444ms

logs: log templates per service, rate before vs after fault onset, with examples

L-0001svc=shipping sev=INFO base=0.2/min fault=1.4/min tpl="" ex=['', '']
L-0002svc=load-generator sev=INFO base=11.2/min fault=18.8/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0003svc=load-generator sev=INFO base=9.6/min fault=14.0/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user 93b79932-dbfb-11f1-8306-d6b650b5399b', 'Checkout completed for user 93b79932-dbfb-11f1-8306-d6b650b5399b']
L-0004svc=checkout sev=INFO base=9.8/min fault=13.0/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0005svc=accounting sev=INFORMATION base=9.8/min fault=12.8/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0006svc=cart sev=INFORMATION base=9.8/min fault=12.8/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0007svc=checkout sev=INFO base=9.8/min fault=12.8/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 169.206µs', 'Successful to write message. offset: 0, duration: 180.807µs']
L-0008svc=checkout sev=INFO base=9.8/min fault=12.8/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0009svc=checkout sev=INFO base=9.8/min fault=12.8/min tpl="order placed" ex=['order placed', 'order placed']
L-0010svc=checkout sev=INFO base=9.8/min fault=12.8/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0011svc=checkout sev=INFO base=9.8/min fault=12.8/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0012svc=email sev=INFO base=9.8/min fault=12.8/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0013svc=fraud-detection sev=INFO base=9.8/min fault=12.8/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: 916c9ff7-dbfb-11f1-9d0e-3288f2b1222e, and updated total count to: 22266', 'Consumed record with orderId: 92b81cd9-dbfb-11f1-9d0e-3288f2b1222e, and updated total count to: 22267']
L-0014svc=frontend sev=INFO base=9.8/min fault=12.8/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0015svc=payment sev=INFO base=9.8/min fault=12.8/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0016svc=payment sev=INFO base=9.8/min fault=12.8/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0017svc=shipping sev=INFO base=9.8/min fault=12.8/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0018svc=load-generator sev=INFO base=42.0/min fault=34.4/min tpl="User viewing cart" ex=['User viewing cart', 'User viewing cart']
L-0019svc=checkout sev=INFO base=0.0/min fault=0.2/min tpl="Closed connection to broker kafka:<NUM>" ex=['Closed connection to broker kafka:9092']
L-0020svc=checkout sev=INFO base=0.0/min fault=0.2/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0021svc=checkout sev=INFO base=0.0/min fault=0.2/min tpl="client/seedbrokers close seed broker #-<NUM> at kafka:<NUM> due to socket error: broken pipe" ex=['client/seedbrokers close seed broker #-1 at kafka:9092 due to socket error: broken pipe']
L-0022svc=load-generator sev=INFO base=30.4/min fault=25.2/min tpl="Product added to cart successfully" ex=['Product added to cart successfully', 'Product added to cart successfully']
L-0023svc=frontend-proxy sev=UNSET base=1755.0/min fault=1524.6/min tpl="<*> <*> <*> HTTP/<NUM>" <NUM> <*> <*> - "-" <NUM> <NUM> <NUM> <NUM> "-" "Mozilla/<NUM> (X11; Linux x86_64) AppleWebKit/<NUM> (KHTML, like G…" ex=['[2026-11-09T05:08:13.540Z] "GET /icons/Chevron.svg HTTP/1.1" 200 - via_upstream - "-" 0 793 2 2 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleW…', '[2026-11-09T05:08:13.540Z] "GET /images/opentelemetry-demo-logo.png HTTP/1.1" 200 - via_upstream - "-" 0 100140 0 0 "-" "Mozilla/5.0 (X11; …']
L-0024svc=load-generator sev=INFO base=10.0/min fault=11.6/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user 9163b512-dbfb-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user 9163b512-dbfb-11f1-8306-d6b650b5399b']
L-0025svc=kafka sev=INFO base=0.4/min fault=0.6/min tpl="[QuorumController id=<NUM>] Periodic task <*> generated <NUM> records in <NUM> microseconds." ex=['[QuorumController id=1] Periodic task expireDelegationTokens generated 0 records in 321 microseconds.', '[QuorumController id=1] Periodic task electPreferred generated 0 records in 16 microseconds.']
L-0026svc=ad sev=INFO base=3.4/min fault=4.0/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0027svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=71425, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=71545, siz…']
35 lower-ranked lines omitted
Loading runs in your browser…