Rootline v0.1 pilot

← All incidents

toxiBandwidth-r1

What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)

Service graph during the window

Evidence and answers

5341 prompt tokens

metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean

M-0001svc=accounting metric=dotnet.thread_pool.thread.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0002svc=accounting metric=process.thread.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0003svc=cart metric=feature_flag.evaluation_active_count base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:1,na,+0%]
M-0004svc=cart metric=http.server.active_requests base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:1,na,+0%]
M-0005svc=payment metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,new:0.0435,na,+0%,na,+0%]
M-0006svc=payment metric=traces.span.metrics.duration.p50 base=1.48 delta/30s=[na,-32%,na,-32%,na,+1014489%,na,-1%,na,-23%]
M-0007svc=payment metric=traces.span.metrics.duration.p95 base=4.04 delta/30s=[na,-53%,na,-53%,na,+371187%,na,+75%,na,+296%]
M-0008svc=payment metric=traces.span.metrics.duration.p99 base=4.65 delta/30s=[na,-57%,na,-57%,na,+322619%,na,+68%,na,+829%]
M-0009svc=product-catalog metric=go.goroutine.count.rate base=0 delta/30s=[na,+0%,na,new:0.467,na,+0%,na,+0%,na,+0%]
M-0010svc=product-catalog metric=otel.sdk.span.live base=0 delta/30s=[na,+0%,na,new:4,na,+0%,na,+0%,na,+0%]
M-0011svc=recommendation metric=error_ratio base=0 delta/30s=[na,+0%,na,new:0.00365,na,+0%,na,+0%,na,+0%]
M-0012svc=recommendation metric=traces.span.metrics.duration.p50 base=3.26 delta/30s=[na,+0%,na,+459415%,na,-4%,na,+2%,na,+0%]
M-0013svc=recommendation metric=traces.span.metrics.duration.p95 base=5.66 delta/30s=[na,-1%,na,+264927%,na,-4%,na,+4%,na,+0%]
M-0014svc=recommendation metric=traces.span.metrics.duration.p99 base=11.7 delta/30s=[na,-47%,na,+128558%,na,-40%,na,-36%,na,-41%]
M-0015svc=shipping metric=http.server.active_requests base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:2,na,+0%]
M-0016svc=payment metric=v8js.gc.duration.p99 base=0.0187 delta/30s=[na,+30%,na,-48%,na,+1206%,na,+2497%,na,+32%]
M-0017svc=payment metric=v8js.gc.duration.p95 base=0.0174 delta/30s=[na,+22%,na,-48%,na,+1166%,na,+2345%,na,+35%]
M-0018svc=ad metric=otel.sdk.processor.log.queue.size base=0.2 delta/30s=[na,-100%,na,+1400%,na,-100%,na,-100%,na,-100%]
M-0019svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,-100%,-100%,-100%,-100%,-100%,-100%,+900%,-100%,-100%]
M-0020svc=payment metric=nodejs.eventloop.delay.max base=0.0396 delta/30s=[na,-49%,na,-63%,na,+243%,na,+591%,na,-34%]
M-0021svc=payment metric=nodejs.eventloop.delay.stddev base=0.000577 delta/30s=[na,-59%,na,-76%,na,+285%,na,+507%,na,-40%]
M-0022svc=load-generator metric=otel.sdk.span.live base=1.4 delta/30s=[-29%,na,-29%,na,+471%,na,-100%,na,+43%,na]
M-0023svc=payment metric=v8js.gc.duration.p50 base=0.00667 delta/30s=[na,-6%,na,-50%,na,+463%,na,+185%,na,+50%]
M-0024svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0.00667 delta/30s=[na,-100%,na,+400%,na,-100%,na,-100%,na,-100%]
M-0025svc=accounting metric=dns.lookup.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,+400%,na,-100%,na,-100%,na,-100%]
M-0026svc=cart metric=dotnet.thread_pool.thread.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,-100%,na,+400%,na,-100%]
M-0027svc=cart metric=process.thread.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,-100%,na,+400%,na,-100%]
M-0028svc=checkout metric=go.goroutine.count.rate base=0.0133 delta/30s=[na,+400%,na,+400%,na,-100%,na,+400%,na,-100%]
M-0029svc=fraud-detection metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,+400%,na,-100%,na,-100%]
M-0030svc=accounting metric=db.client.connection.npgsql.create_time.p99 base=0.00995 delta/30s=[na,na,na,+398%,na,na,na,na,na,na]
M-0031svc=accounting metric=db.client.connection.npgsql.create_time.p95 base=0.00975 delta/30s=[na,na,na,+392%,na,na,na,na,na,na]
M-0032svc=frontend-web metric=traces.span.metrics.duration.p99 base=1.96e+03 delta/30s=[na,+122%,na,+353%,na,-10%,na,-1%,na,+93%]
M-0033svc=checkout metric=http.client.request.duration.p50 base=0.394 delta/30s=[na,-94%,na,-90%,na,-94%,na,+344%,na,-89%]
M-0034svc=accounting metric=db.client.connection.npgsql.create_time.p50 base=0.0075 delta/30s=[na,na,na,+300%,na,na,na,na,na,na]
M-0035svc=load-generator metric=otel.sdk.processor.log.queue.size base=2 delta/30s=[+300%,na,+200%,na,+200%,na,-100%,na,+100%,na]
M-0036svc=frontend-web metric=traces.span.metrics.duration.p95 base=1.26e+03 delta/30s=[na,+54%,na,+290%,na,+1%,na,+34%,na,-51%]
M-0037svc=unknown metric=redis.commands base=8.3 delta/30s=[+261%,+20%,+213%,-40%,-4%,+45%,-28%,+129%,-16%,-16%]
876 lower-ranked lines omitted

traces: slowest and errored traces after fault onset, critical path (! = error span)

T-0001trace=9108d7f1 dur=2540ms ERROR path: load-generator:user_checkout_single 2540ms > frontend-proxy:POST 2484ms > frontend:POST /api/checkout 2483ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 454ms | errors off path: checkout:POST
T-0002trace=190a3aa3 dur=15287ms ok path: accounting:order-consumed 15287ms
T-0003trace=ad88eb15 dur=609ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 609ms!
T-0004trace=b86f38b6 dur=14952ms ok path: accounting:order-consumed 14952ms
T-0005trace=43d8e2e8 dur=600ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 600ms!
T-0006trace=e106153e dur=13858ms ok path: accounting:order-consumed 13858ms
T-0007trace=1b8bd27d dur=371ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 371ms!
T-0008trace=cae42898 dur=13328ms ok path: accounting:order-consumed 13328ms
T-0009trace=dd070625 dur=350ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 350ms!
T-0010trace=10788673 dur=13070ms ok path: accounting:order-consumed 13070ms
T-0011trace=714bfac2 dur=317ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 317ms!
T-0012trace=33770201 dur=11809ms ok path: accounting:order-consumed 11809ms
T-0013trace=f8e5e00f dur=202ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 202ms!
T-0014trace=30c282f0 dur=10164ms ok path: accounting:order-consumed 10164ms
T-0015trace=712a2566 dur=199ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 199ms!
T-0016trace=b231d2e1 dur=9595ms ok path: accounting:order-consumed 9595ms
T-0017trace=f2dc3b02 dur=191ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 191ms!
T-0018trace=230688e0 dur=9236ms ok path: accounting:order-consumed 9236ms
T-0019trace=e32b1b24 dur=67ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 67ms!
T-0020trace=977b3b7d dur=8201ms ok path: accounting:order-consumed 8201ms
T-0021trace=ef863671 dur=55ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 55ms!
T-0022trace=d51fca83 dur=8036ms ok path: accounting:order-consumed 8036ms
T-0023trace=7542c658 dur=32ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 32ms!
T-0024trace=2c3835fc dur=7973ms ok path: accounting:order-consumed 7973ms
T-0025trace=a1ac5f9e dur=17ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 17ms!
T-0026trace=70aa1eea dur=7784ms ok path: accounting:order-consumed 7784ms
T-0027trace=8e62420b dur=10ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 10ms!
T-0028trace=b581e6ba dur=7460ms ok path: accounting:order-consumed 7460ms
T-0029trace=49709559 dur=10ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 10ms!
T-0030trace=d4d16673 dur=6615ms ok path: load-generator:browser_add_to_cart 6615ms
T-0031trace=5bdf304d dur=9ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 9ms!
T-0032trace=07412ed4 dur=6545ms ok path: frontend-proxy:POST 6542ms > flagd:flagd.evaluation.v1.Service/EventStream 6533ms
T-0033trace=234b486c dur=9ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 9ms!
T-0034trace=f8fc5742 dur=6083ms ok path: frontend-proxy:POST 6080ms > flagd:flagd.evaluation.v1.Service/EventStream 6065ms
T-0035trace=b3f6bcfe dur=8ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 8ms!
T-0036trace=b5df68cc dur=5886ms ok path: accounting:order-consumed 5886ms
T-0037trace=11840b1a dur=5ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 5ms!
T-0038trace=2c00d229 dur=5804ms ok path: accounting:order-consumed 5804ms
T-0039trace=dadbef60 dur=5ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 5ms!
T-0040trace=10e98f03 dur=5792ms ok path: frontend-proxy:POST 5789ms > flagd:flagd.evaluation.v1.Service/EventStream 5784ms

logs: log templates per service, rate before vs after fault onset, with examples

L-0001svc=load-generator sev=INFO base=13.2/min fault=18.4/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user fdf30d36-dbdd-11f1-8306-d6b650b5399b', 'Checkout completed for user fdf30d36-dbdd-11f1-8306-d6b650b5399b']
L-0002svc=load-generator sev=INFO base=34.4/min fault=25.2/min tpl="Currency changed to CHF" ex=['Currency changed to CHF', 'Currency changed to CHF']
L-0003svc=load-generator sev=INFO base=45.2/min fault=34.8/min tpl="User viewing cart" ex=['User viewing cart', 'User viewing cart']
L-0004svc=load-generator sev=INFO base=10.4/min fault=13.2/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0005svc=load-generator sev=INFO base=14.8/min fault=12.0/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user 012d27d4-dbde-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user 012d27d4-dbde-11f1-8306-d6b650b5399b']
L-0006svc=checkout sev=INFO base=0.2/min fault=0.0/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0007svc=ad sev=INFO base=3.2/min fault=2.6/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0008svc=shipping sev=INFO base=2.0/min fault=1.6/min tpl="" ex=['', '']
L-0009svc=currency sev=INFO base=121.4/min fault=108.4/min tpl="conversion successful" ex=['conversion successful', 'conversion successful']
L-0010svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=46114, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=46234, siz…']
L-0011svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 46299 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 46416 breach: LogSegment(baseOff…']
L-0012svc=shipping sev=INFO base=27.4/min fault=30.4/min tpl="Requesting quote" ex=['Requesting quote', 'Requesting quote']
L-0013svc=quote sev=INFO base=27.4/min fault=30.2/min tpl="Calculated quote" ex=['Calculated quote', 'Calculated quote']
L-0014svc=kafka sev=INFO base=7.6/min fault=8.4/min tpl="Deleted <*> index <*>" ex=['Deleted offset index /tmp/kafka-logs/__cluster_metadata-0/00000000000000046114.index.deleted.', 'Deleted offset index /tmp/kafka-logs/__cluster_metadata-0/00000000000000046144.index.deleted.']
L-0015svc=shipping sev=INFO base=27.4/min fault=30.0/min tpl="Sending Quote" ex=['Sending Quote', 'Sending Quote']
L-0016svc=currency sev=INFO base=30.4/min fault=27.8/min tpl="GetSupportedCurrencies successful" ex=['GetSupportedCurrencies successful', 'GetSupportedCurrencies successful']
L-0017svc=load-generator sev=INFO base=27.2/min fault=29.6/min tpl="Product added to cart successfully" ex=['Product added to cart successfully', 'Product added to cart successfully']
L-0018svc=load-generator sev=INFO base=126.4/min fault=137.2/min tpl="User browsing product: <*>" ex=['User browsing product: HQTGWGPNH4', 'User browsing product: HQTGWGPNH4']
L-0019svc=kafka sev=INFO base=3.8/min fault=4.2/min tpl="Deleted log /tmp/kafka-logs/__cluster_metadata-<NUM>/<HEX>.log.deleted." ex=['Deleted log /tmp/kafka-logs/__cluster_metadata-0/00000000000000046114.log.deleted.', 'Deleted log /tmp/kafka-logs/__cluster_metadata-0/00000000000000046144.log.deleted.']
L-0020svc=kafka sev=INFO base=3.8/min fault=4.2/min tpl="Deleted producer state snapshot /tmp/kafka-logs/__cluster_metadata-<NUM>/<HEX>.snapshot.deleted" ex=['Deleted producer state snapshot /tmp/kafka-logs/__cluster_metadata-0/00000000000000046114.snapshot.deleted', 'Deleted producer state snapshot /tmp/kafka-logs/__cluster_metadata-0/00000000000000046144.snapshot.deleted']
L-0021svc=checkout sev=INFO base=13.8/min fault=15.0/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0022svc=accounting sev=INFORMATION base=14.0/min fault=15.2/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0023svc=cart sev=INFORMATION base=14.0/min fault=15.2/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0024svc=checkout sev=INFO base=14.0/min fault=15.2/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 170.925µs', 'Successful to write message. offset: 0, duration: 194.035µs']
L-0025svc=checkout sev=INFO base=14.0/min fault=15.2/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0026svc=checkout sev=INFO base=14.0/min fault=15.2/min tpl="order placed" ex=['order placed', 'order placed']
L-0027svc=checkout sev=INFO base=14.0/min fault=15.2/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0028svc=checkout sev=INFO base=14.0/min fault=15.2/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0029svc=email sev=INFO base=14.0/min fault=15.2/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
32 lower-ranked lines omitted
Loading runs in your browser…