Rootline v0.1 pilot

← All incidents

hwWeakSignal-r2

What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)

Service graph during the window

Evidence and answers

4586 prompt tokens

metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean

M-0001svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%]
M-0002svc=accounting metric=dns.lookup.duration.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%]
M-0003svc=ad metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0108]
M-0004svc=ad metric=rpc.client.duration.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0005svc=ad metric=rpc.client.request.size.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0006svc=ad metric=rpc.client.response.size.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0007svc=ad metric=traces.span.metrics.duration.p50 base=1 delta/30s=[na,-0%,na,-0%,na,-0%,na,+2%,na,+1493924%]
M-0008svc=ad metric=traces.span.metrics.duration.p95 base=1.91 delta/30s=[na,-0%,na,-0%,na,-0%,na,+2%,na,+786228%]
M-0009svc=ad metric=traces.span.metrics.duration.p99 base=2.18 delta/30s=[na,-9%,na,-9%,na,-9%,na,+44%,na,+687973%]
M-0010svc=cart metric=dotnet.thread_pool.thread.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%]
M-0011svc=checkout metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[na,+0%,na,new:5,na,+0%,na,+0%,na,+0%]
M-0012svc=flagd metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.000969]
M-0013svc=fraud-detection metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0345]
M-0014svc=fraud-detection metric=jvm.thread.count.rate base=0 delta/30s=[na,new:0.0333,na,new:0.0333,na,+0%,na,+0%,na,new:0.0667]
M-0015svc=fraud-detection metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[na,new:1,na,+0%,na,new:1,na,+0%,na,+0%]
M-0016svc=fraud-detection metric=otel.sdk.processor.span.queue.size base=0 delta/30s=[na,new:2,na,+0%,na,new:2,na,+0%,na,+0%]
M-0017svc=fraud-detection metric=rpc.client.duration.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0018svc=fraud-detection metric=rpc.client.request.size.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0019svc=fraud-detection metric=rpc.client.response.size.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0020svc=quote metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[na,new:1,na,+0%,na,+0%,na,new:2,na,new:1]
M-0021svc=fraud-detection metric=traces.span.metrics.duration.p50 base=50.6 delta/30s=[na,-17%,na,-21%,na,-37%,na,-41%,na,+29568%]
M-0022svc=fraud-detection metric=traces.span.metrics.duration.p95 base=90.5 delta/30s=[na,+4%,na,+3%,na,-0%,na,-1%,na,+16473%]
M-0023svc=fraud-detection metric=traces.span.metrics.duration.p99 base=98.1 delta/30s=[na,+1%,na,+1%,na,-0%,na,-0%,na,+15190%]
M-0024svc=recommendation metric=otel.sdk.processor.span.queue.size base=0.6 delta/30s=[+3900%,na,+4400%,na,+1400%,na,+3900%,na,+4400%,na]
M-0025svc=unknown metric=system.disk.pending_operations base=0.2 delta/30s=[na,-100%,na,+400%,na,+400%,na,+1900%,na,+400%]
M-0026svc=cart metric=demo.cart.add_item.latency.p99 base=0.00495 delta/30s=[na,+0%,na,+0%,na,+0%,na,+713%,na,+1102%]
M-0027svc=cart metric=traces.span.metrics.duration.p99 base=6.62 delta/30s=[na,-12%,na,-11%,na,+23%,na,+538%,na,+950%]
M-0028svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,-100%,-100%,-100%,-100%,-100%,-100%,+900%,-100%,-100%]
M-0029svc=cart metric=http.server.request.duration.p99 base=0.00675 delta/30s=[na,-27%,na,-27%,na,+42%,na,+496%,na,+781%]
M-0030svc=recommendation metric=process.runtime.cpython.gc_count base=58.0 delta/30s=[+97%,na,-17%,na,+238%,na,+703%,na,+640%,na]
M-0031svc=checkout metric=http.client.request.duration.p95 base=0.31 delta/30s=[na,-39%,na,+449%,na,+158%,na,-48%,na,+195%]
M-0032svc=ad metric=jvm.thread.count.rate base=0.0133 delta/30s=[na,-100%,na,-100%,na,-100%,na,-100%,na,+400%]
M-0033svc=cart metric=process.thread.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,+400%,na,-100%,na,-100%]
M-0034svc=checkout metric=go.goroutine.count.rate base=0.0133 delta/30s=[na,-100%,na,-100%,na,-100%,na,-100%,na,+400%]
M-0035svc=fraud-detection metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,+400%,na,-100%,na,-100%,na,-100%]
M-0036svc=otelcol-contrib metric=otelcol_exporter_in_flight_requests base=0.4 delta/30s=[na,+150%,na,+150%,na,-100%,na,+150%,na,+400%]
M-0037svc=product-catalog metric=go.goroutine.count.rate base=0.0133 delta/30s=[na,-100%,na,+400%,na,-100%,na,+400%,na,-100%]
864 lower-ranked lines omitted

traces: slowest and errored traces after fault onset, critical path (! = error span)

T-0001trace=7ca62c93 dur=23258ms ok path: accounting:order-consumed 23258ms
T-0002trace=2467b90e dur=13411ms ok path: accounting:order-consumed 13411ms
T-0003trace=abc57c96 dur=11897ms ok path: accounting:order-consumed 11897ms
T-0004trace=64e14e9a dur=11335ms ok path: accounting:order-consumed 11335ms
T-0005trace=15f8896b dur=10190ms ok path: accounting:order-consumed 10190ms
T-0006trace=c98fc0e0 dur=9268ms ok path: accounting:order-consumed 9268ms
T-0007trace=8e8512a7 dur=8935ms ok path: accounting:order-consumed 8935ms
T-0008trace=aabee271 dur=8871ms ok path: accounting:order-consumed 8871ms
T-0009trace=f9d9983c dur=8647ms ok path: accounting:order-consumed 8647ms
T-0010trace=3ee4e7fd dur=8021ms ok path: accounting:order-consumed 8021ms
T-0011trace=12ddd45f dur=7955ms ok path: accounting:order-consumed 7955ms
T-0012trace=621921cc dur=7939ms ok path: accounting:order-consumed 7939ms
T-0013trace=db0adcf3 dur=7434ms ok path: accounting:order-consumed 7434ms
T-0014trace=fd328343 dur=7345ms ok path: accounting:order-consumed 7345ms
T-0015trace=3bcfbe37 dur=7263ms ok path: accounting:order-consumed 7263ms
T-0016trace=ac6c3a8d dur=7127ms ok path: accounting:order-consumed 7127ms
T-0017trace=3d7b1be3 dur=6922ms ok path: accounting:order-consumed 6922ms
T-0018trace=2af22191 dur=6912ms ok path: accounting:order-consumed 6912ms
T-0019trace=7bc66671 dur=6574ms ok path: accounting:order-consumed 6574ms
T-0020trace=6a4644d4 dur=6133ms ok path: frontend-proxy:POST 6132ms > flagd:flagd.evaluation.v1.Service/EventStream 6132ms

logs: log templates per service, rate before vs after fault onset, with examples

L-0001svc=load-generator sev=INFO base=8.4/min fault=11.2/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user 70498e52-dc04-11f1-8306-d6b650b5399b', 'Checkout completed for user 70498e52-dc04-11f1-8306-d6b650b5399b']
L-0002svc=shipping sev=INFO base=2.2/min fault=1.6/min tpl="" ex=['', '']
L-0003svc=checkout sev=INFO base=0.0/min fault=0.2/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0004svc=kafka sev=INFO base=0.6/min fault=0.4/min tpl="[QuorumController id=<NUM>] Periodic task <*> generated <NUM> records in <NUM> microseconds." ex=['[QuorumController id=1] Periodic task electPreferred generated 0 records in 16 microseconds.', '[QuorumController id=1] Periodic task electUnclean generated 0 records in 9 microseconds.']
L-0005svc=load-generator sev=INFO base=16.4/min fault=14.4/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user 79d1d236-dc04-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user 79d1d236-dc04-11f1-8306-d6b650b5399b']
L-0006svc=load-generator sev=INFO base=88.0/min fault=78.0/min tpl="User getting <*> for <*> <*>" ex=['User getting ads for category: books', 'User getting ads for category: books']
L-0007svc=ad sev=INFO base=2.8/min fault=2.4/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0008svc=load-generator sev=INFO base=13.2/min fault=14.8/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0009svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 79293 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 79410 breach: LogSegment(baseOff…']
L-0010svc=kafka sev=INFO base=1.0/min fault=0.8/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=79225, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=79345, siz…']
L-0011svc=load-generator sev=INFO base=139.2/min fault=127.2/min tpl="User browsing product: <*>" ex=['User browsing product: LS4PSXUNUM', 'User browsing product: LS4PSXUNUM']
L-0012svc=kafka sev=INFO base=8.4/min fault=7.6/min tpl="Deleted <*> index <*>" ex=['Deleted offset index /tmp/kafka-logs/__cluster_metadata-0/00000000000000079135.index.deleted.', 'Deleted offset index /tmp/kafka-logs/__cluster_metadata-0/00000000000000079165.index.deleted.']
L-0013svc=kafka sev=INFO base=4.2/min fault=3.8/min tpl="Deleted log /tmp/kafka-logs/__cluster_metadata-<NUM>/<HEX>.log.deleted." ex=['Deleted log /tmp/kafka-logs/__cluster_metadata-0/00000000000000079135.log.deleted.', 'Deleted log /tmp/kafka-logs/__cluster_metadata-0/00000000000000079165.log.deleted.']
L-0014svc=kafka sev=INFO base=4.2/min fault=3.8/min tpl="Deleted producer state snapshot /tmp/kafka-logs/__cluster_metadata-<NUM>/<HEX>.snapshot.deleted" ex=['Deleted producer state snapshot /tmp/kafka-logs/__cluster_metadata-0/00000000000000079135.snapshot.deleted', 'Deleted producer state snapshot /tmp/kafka-logs/__cluster_metadata-0/00000000000000079165.snapshot.deleted']
L-0015svc=frontend-proxy sev=UNSET base=271.6/min fault=252.2/min tpl="<*> <*> <*> HTTP/<NUM>" <NUM> <*> <*> - "-" <NUM> <NUM> <NUM> <*> "-" <*> <*> <*> <*> <*> <*> <NUM>.<NUM>:<NUM> <NUM>.<NUM>:<NUM> - -" ex=['[2026-11-09T06:11:44.452Z] "GET /api/data/?contextKeys=books HTTP/1.1" 308 - via_upstream - "-" 0 27 1 0 "-" "python-requests/2.34.2" "289c…', '[2026-11-09T06:11:44.455Z] "GET /api/data?contextKeys=books HTTP/1.1" 200 - via_upstream - "-" 0 211 2 2 "-" "python-requests/2.34.2" "b5a4…']
L-0016svc=kafka sev=INFO base=2.2/min fault=2.0/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 79332 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 79371 breach: LogSegment(baseOff…']
L-0017svc=load-generator sev=INFO base=82.8/min fault=77.6/min tpl="User <UUID> adding <NUM> of product <*> to cart" ex=['User 70498e52-dc04-11f1-8306-d6b650b5399b adding 5 of product 6E92ZMYYFZ to cart', 'User 70498e52-dc04-11f1-8306-d6b650b5399b adding 5 of product 6E92ZMYYFZ to cart']
L-0018svc=ad sev=INFO base=49.4/min fault=46.6/min tpl="Targeted ad request received for <*>" ex=['Targeted ad request received for [books]', 'Targeted ad request received for [accessories]']
L-0019svc=kafka sev=INFO base=3.2/min fault=3.0/min tpl="Deleted snapshot files for snapshot OffsetAndEpoch[offset=<NUM>, epoch=<NUM>]." ex=['Deleted snapshot files for snapshot OffsetAndEpoch[offset=79137, epoch=1].', 'Deleted snapshot files for snapshot OffsetAndEpoch[offset=79176, epoch=1].']
41 lower-ranked lines omitted
Loading runs in your browser…