Rootline v0.1 pilot

← All incidents

toxiLatency-r1

What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)

Service graph during the window

Evidence and answers

4581 prompt tokens

metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean

M-0001svc=ad metric=jvm.gc.duration.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%]
M-0002svc=checkout metric=go.goroutine.count.rate base=0 delta/30s=[na,new:0.0667,na,+0%,na,+0%,na,+0%,na,+0%]
M-0003svc=fraud-detection metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:1,na,+0%]
M-0004svc=kafka metric=jvm.gc.duration.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%]
M-0005svc=recommendation metric=process.runtime.cpython.cpu.utilization base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:5.36]
M-0006svc=recommendation metric=process.thread.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0007svc=shipping metric=http.server.active_requests base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:1]
M-0008svc=unknown metric=system.disk.pending_operations base=0 delta/30s=[na,+0%,na,+0%,na,new:3,na,+0%,na,new:1]
M-0009svc=checkout metric=rpc.client.call.duration.p50 base=0.00326 delta/30s=[na,+9283%,na,+11415%,na,+11415%,na,+11415%,na,+10989%]
M-0010svc=cart metric=dotnet.thread_pool.thread.count.rate base=0.00667 delta/30s=[na,+400%,na,-100%,na,-100%,na,-100%,na,+900%]
M-0011svc=cart metric=process.thread.count.rate base=0.00667 delta/30s=[na,+400%,na,-100%,na,-100%,na,-100%,na,+900%]
M-0012svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,+900%,-100%,-100%,-100%,-100%,-100%,-100%,-100%,-100%]
M-0013svc=checkout metric=rpc.client.call.duration.p95 base=0.0529 delta/30s=[na,+808%,na,+821%,na,+821%,na,+821%,na,+818%]
M-0014svc=load-generator metric=process.runtime.cpython.gc_count base=69.8 delta/30s=[+265%,na,+496%,na,+434%,na,+692%,na,-87%,na]
M-0015svc=checkout metric=rpc.client.call.duration.p99 base=0.0658 delta/30s=[na,+654%,na,+656%,na,+656%,na,+656%,na,+656%]
M-0016svc=fraud-detection metric=otel.sdk.processor.span.queue.size base=0.8 delta/30s=[na,+650%,na,-100%,na,-100%,na,+400%,na,+150%]
M-0017svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,-100%,na,-100%,na,+400%]
M-0018svc=accounting metric=dns.lookup.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,-100%,na,-100%,na,+400%]
M-0019svc=ad metric=jvm.thread.count.rate base=0.00667 delta/30s=[na,+400%,na,+400%,na,-100%,na,-100%,na,-100%]
M-0020svc=fraud-detection metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,+400%,na,-100%,na,-100%]
M-0021svc=kafka metric=jvm.thread.count.rate base=0.00667 delta/30s=[na,+400%,na,-100%,na,+400%,na,+400%,na,-100%]
M-0022svc=otelcol-contrib metric=otelcol_exporter_in_flight_requests base=0.2 delta/30s=[na,-100%,na,-100%,na,-100%,na,-100%,na,+400%]
M-0023svc=frontend-web metric=traces.span.metrics.duration.p95 base=443.3 delta/30s=[na,-12%,na,-12%,na,+385%,na,-10%,na,+148%]
M-0024svc=checkout metric=rpc.server.call.duration.p50 base=0.105 delta/30s=[na,+233%,na,+337%,na,+297%,na,+290%,na,+337%]
M-0025svc=checkout metric=traces.span.metrics.duration.p99 base=1.08e+03 delta/30s=[na,+326%,na,+27%,na,-27%,na,+25%,na,+27%]
M-0026svc=checkout metric=traces.span.metrics.duration.p50 base=103.1 delta/30s=[na,+240%,na,+269%,na,+249%,na,+320%,na,+269%]
M-0027svc=recommendation metric=error_ratio base=0.000202 delta/30s=[na,-100%,na,+131%,na,+72%,na,+52%,na,+317%]
M-0028svc=frontend metric=http.server.request.duration.p99 base=0.986 delta/30s=[na,+97%,na,+81%,na,-33%,na,+33%,na,+278%]
M-0029svc=frontend metric=http.server.request.duration.p95 base=0.23 delta/30s=[na,+273%,na,+96%,na,+101%,na,+108%,na,+275%]
M-0030svc=frontend-web metric=traces.span.metrics.duration.p99 base=1.21e+03 delta/30s=[na,-67%,na,-51%,na,+265%,na,+6%,na,+10%]
M-0031svc=frontend metric=traces.span.metrics.duration.p99 base=1.34e+03 delta/30s=[na,+242%,na,+2%,na,-41%,na,+0%,na,+2%]
M-0032svc=checkout metric=traces.span.metrics.duration.p95 base=896.0 delta/30s=[na,+224%,na,+36%,na,-17%,na,+23%,na,+36%]
M-0033svc=frontend metric=traces.span.metrics.duration.p50 base=151.7 delta/30s=[na,+98%,na,+164%,na,+164%,na,+208%,na,+216%]
M-0034svc=load-generator metric=http.client.duration.p95 base=398.1 delta/30s=[+14%,na,+214%,na,+21%,na,+17%,na,+77%,na]
M-0035svc=quote metric=traces.span.metrics.duration.p50 base=10.8 delta/30s=[na,-47%,na,-26%,na,+178%,na,-63%,na,-26%]
M-0036svc=payment metric=v8js.gc.duration.p95 base=0.0358 delta/30s=[na,-35%,na,+172%,na,-73%,na,-41%,na,-74%]
M-0037svc=frontend metric=traces.span.metrics.duration.p95 base=1.1e+03 delta/30s=[na,+165%,na,+9%,na,-31%,na,+2%,na,+9%]
852 lower-ranked lines omitted

traces: slowest and errored traces after fault onset, critical path (! = error span)

T-0001trace=e7e415ad dur=34302ms ok path: accounting:order-consumed 34302ms
T-0002trace=dafbc655 dur=20848ms ok path: accounting:order-consumed 20848ms
T-0003trace=644937ec dur=15437ms ok path: accounting:order-consumed 15437ms
T-0004trace=6b622384 dur=15097ms ok path: accounting:order-consumed 15097ms
T-0005trace=7eba7c7c dur=14267ms ok path: accounting:order-consumed 14267ms
T-0006trace=469daec4 dur=13258ms ok path: accounting:order-consumed 13258ms
T-0007trace=a208fda7 dur=12936ms ok path: accounting:order-consumed 12936ms
T-0008trace=01946580 dur=12907ms ok path: accounting:order-consumed 12907ms
T-0009trace=a32a0e30 dur=11847ms ok path: accounting:order-consumed 11847ms
T-0010trace=3d14c2c6 dur=10904ms ok path: accounting:order-consumed 10904ms
T-0011trace=8c8362a7 dur=10349ms ok path: accounting:order-consumed 10349ms
T-0012trace=d12854a0 dur=8318ms ok path: accounting:order-consumed 8318ms
T-0013trace=a3066a57 dur=7554ms ok path: accounting:order-consumed 7554ms
T-0014trace=c4dc2c5a dur=7490ms ok path: accounting:order-consumed 7490ms
T-0015trace=c0f968c9 dur=5944ms ok path: accounting:order-consumed 5944ms
T-0016trace=797d1ae8 dur=5786ms ok path: accounting:order-consumed 5786ms
T-0017trace=d7b297ab dur=5566ms ok path: accounting:order-consumed 5566ms
T-0018trace=c95936b7 dur=5521ms ok path: frontend-proxy:POST 5520ms > flagd:flagd.evaluation.v1.Service/EventStream 5521ms
T-0019trace=a34d13e0 dur=5467ms ok path: frontend-proxy:POST 5467ms > flagd:flagd.evaluation.v1.Service/EventStream 5466ms
T-0020trace=c0dd4767 dur=5343ms ok path: accounting:order-consumed 5343ms

logs: log templates per service, rate before vs after fault onset, with examples

L-0001svc=shipping sev=INFO base=0.8/min fault=0.4/min tpl="" ex=['', '']
L-0002svc=load-generator sev=INFO base=34.0/min fault=43.6/min tpl="User viewing cart" ex=['User viewing cart', 'User viewing cart']
L-0003svc=load-generator sev=INFO base=18.0/min fault=14.4/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0004svc=checkout sev=INFO base=0.2/min fault=0.0/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0005svc=load-generator sev=INFO base=13.6/min fault=11.2/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user fee5101a-dbea-11f1-8306-d6b650b5399b', 'Checkout completed for user fee5101a-dbea-11f1-8306-d6b650b5399b']
L-0006svc=load-generator sev=INFO base=84.4/min fault=70.8/min tpl="User getting <*> for <*> <*>" ex=['User getting recommendations for product: L9ECAV7KIM', 'User getting recommendations for product: L9ECAV7KIM']
L-0007svc=accounting sev=INFORMATION base=13.0/min fault=10.8/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0008svc=cart sev=INFORMATION base=13.0/min fault=10.8/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0009svc=checkout sev=INFO base=13.0/min fault=10.8/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 171.268µs', 'Successful to write message. offset: 0, duration: 212.2µs']
L-0010svc=checkout sev=INFO base=13.0/min fault=10.8/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0011svc=checkout sev=INFO base=13.0/min fault=10.8/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0012svc=checkout sev=INFO base=13.0/min fault=10.8/min tpl="order placed" ex=['order placed', 'order placed']
L-0013svc=checkout sev=INFO base=13.0/min fault=10.8/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0014svc=checkout sev=INFO base=13.0/min fault=10.8/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0015svc=email sev=INFO base=13.0/min fault=10.8/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0016svc=fraud-detection sev=INFO base=13.0/min fault=10.8/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: fee80af7-dbea-11f1-82f3-d279f8fa1f6d, and updated total count to: 20823', 'Consumed record with orderId: ffbe70c3-dbea-11f1-82f3-d279f8fa1f6d, and updated total count to: 20824']
L-0017svc=frontend sev=INFO base=13.0/min fault=10.8/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0018svc=payment sev=INFO base=13.0/min fault=10.8/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0019svc=payment sev=INFO base=13.0/min fault=10.8/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0020svc=shipping sev=INFO base=13.0/min fault=10.8/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0021svc=load-generator sev=INFO base=12.4/min fault=10.4/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user ffb8545c-dbea-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user ffb8545c-dbea-11f1-8306-d6b650b5399b']
L-0022svc=ad sev=INFO base=3.2/min fault=2.6/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0023svc=kafka sev=INFO base=0.4/min fault=0.6/min tpl="[QuorumController id=<NUM>] Periodic task <*> generated <NUM> records in <NUM> microseconds." ex=['[QuorumController id=1] Periodic task expireDelegationTokens generated 0 records in 25 microseconds.', '[QuorumController id=1] Periodic task electPreferred generated 0 records in 15 microseconds.']
L-0024svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=57276, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=57426, siz…']
L-0025svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 57492 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 57609 breach: LogSegment(baseOff…']
L-0026svc=kafka sev=INFO base=7.6/min fault=8.4/min tpl="Deleted <*> index <*>" ex=['Deleted offset index /tmp/kafka-logs/__cluster_metadata-0/00000000000000057276.index.deleted.', 'Deleted offset index /tmp/kafka-logs/__cluster_metadata-0/00000000000000057306.index.deleted.']
L-0027svc=kafka sev=INFO base=3.8/min fault=4.2/min tpl="Deleted log /tmp/kafka-logs/__cluster_metadata-<NUM>/<HEX>.log.deleted." ex=['Deleted log /tmp/kafka-logs/__cluster_metadata-0/00000000000000057276.log.deleted.', 'Deleted log /tmp/kafka-logs/__cluster_metadata-0/00000000000000057306.log.deleted.']
33 lower-ranked lines omitted
Loading runs in your browser…