Rootline v0.1 pilot

← All incidents

hwWeakSignal-r4

What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)

Service graph during the window

Evidence and answers

4596 prompt tokens

metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean

M-0001svc=accounting metric=dotnet.thread_pool.thread.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%]
M-0002svc=accounting metric=process.thread.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%]
M-0003svc=ad metric=error_ratio base=0 delta/30s=[na,+0%,na,new:0.0128,na,+0%,na,+0%,na,+0%]
M-0004svc=ad metric=jvm.thread.count.rate base=0 delta/30s=[na,+0%,na,new:0.0667,na,+0%,na,+0%,na,+0%]
M-0005svc=ad metric=rpc.client.duration.count.rate base=0 delta/30s=[na,+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%]
M-0006svc=ad metric=rpc.client.request.size.count.rate base=0 delta/30s=[na,+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%]
M-0007svc=ad metric=rpc.client.response.size.count.rate base=0 delta/30s=[na,+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%]
M-0008svc=ad metric=traces.span.metrics.duration.p50 base=1 delta/30s=[na,-0%,na,+1494041%,na,-0%,na,-0%,na,-0%]
M-0009svc=ad metric=traces.span.metrics.duration.p95 base=1.91 delta/30s=[na,-0%,na,+786290%,na,-0%,na,-0%,na,-0%]
M-0010svc=ad metric=traces.span.metrics.duration.p99 base=2.18 delta/30s=[na,-9%,na,+689238%,na,-9%,na,-9%,na,-9%]
M-0011svc=flagd metric=error_ratio base=0 delta/30s=[na,+0%,na,new:0.000898,na,+0%,na,+0%,na,+0%]
M-0012svc=fraud-detection metric=error_ratio base=0 delta/30s=[na,+0%,na,new:0.0435,na,+0%,na,+0%,na,+0%]
M-0013svc=fraud-detection metric=rpc.client.duration.count.rate base=0 delta/30s=[na,+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%]
M-0014svc=fraud-detection metric=rpc.client.request.size.count.rate base=0 delta/30s=[na,+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%]
M-0015svc=fraud-detection metric=rpc.client.response.size.count.rate base=0 delta/30s=[na,+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%]
M-0016svc=fraud-detection metric=traces.span.metrics.duration.p50 base=47.3 delta/30s=[na,+6%,na,+31583%,na,-1%,na,-23%,na,-23%]
M-0017svc=fraud-detection metric=traces.span.metrics.duration.p95 base=94.9 delta/30s=[na,+0%,na,+15703%,na,-0%,na,-3%,na,-2%]
M-0018svc=fraud-detection metric=traces.span.metrics.duration.p99 base=114.6 delta/30s=[na,-14%,na,+12989%,na,-14%,na,-14%,na,-14%]
M-0019svc=payment metric=error_ratio base=0.00673 delta/30s=[na,-28%,na,+43%,na,-27%,na,+4389%,na,-26%]
M-0020svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,+900%,-100%,-100%,-100%,-100%,-100%,-100%,-100%,-100%]
M-0021svc=product-catalog metric=otel.sdk.processor.log.queue.size base=1 delta/30s=[+500%,na,+0%,na,+800%,na,+0%,na,+500%,na]
M-0022svc=email metric=traces.span.metrics.duration.p99 base=7.36 delta/30s=[na,-19%,na,-19%,na,-20%,na,+492%,na,-22%]
M-0023svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0.00667 delta/30s=[na,-100%,na,+400%,na,-100%,na,-100%,na,-100%]
M-0024svc=accounting metric=dns.lookup.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,+400%,na,-100%,na,-100%,na,-100%]
M-0025svc=cart metric=dotnet.thread_pool.thread.count.rate base=0.00667 delta/30s=[na,+400%,na,-100%,na,-100%,na,+400%,na,-100%]
M-0026svc=cart metric=process.thread.count.rate base=0.00667 delta/30s=[na,+400%,na,-100%,na,-100%,na,+400%,na,-100%]
M-0027svc=checkout metric=go.goroutine.count.rate base=0.0133 delta/30s=[na,-100%,na,+400%,na,-100%,na,+400%,na,-100%]
M-0028svc=fraud-detection metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[na,+400%,na,-100%,na,-100%,na,-100%,na,+400%]
M-0029svc=kafka metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,-100%,na,-100%,na,+400%,na]
M-0030svc=kafka metric=jvm.thread.count.rate base=0.00667 delta/30s=[+400%,na,-100%,na,-100%,na,-100%,na,-100%,na]
M-0031svc=product-catalog metric=go.goroutine.count.rate base=0.0133 delta/30s=[+400%,na,-100%,na,+400%,na,-100%,na,-100%,na]
M-0032svc=cart metric=traces.span.metrics.duration.p95 base=6.42 delta/30s=[na,-28%,na,+305%,na,-27%,na,-31%,na,-27%]
M-0033svc=recommendation metric=otel.sdk.processor.log.queue.size base=1 delta/30s=[+100%,na,+300%,na,+0%,na,+0%,na,-100%,na]
M-0034svc=load-generator metric=process.runtime.cpython.gc_count base=74.2 delta/30s=[na,-93%,na,+269%,na,-69%,na,-73%,na,-78%]
M-0035svc=checkout metric=rpc.client.call.duration.p99 base=0.0629 delta/30s=[na,+13%,na,+17%,na,+261%,na,-23%,na,-23%]
M-0036svc=frontend-web metric=traces.span.metrics.duration.p99 base=1.16e+03 delta/30s=[na,+26%,na,-21%,na,+250%,na,+236%,na,-66%]
M-0037svc=fraud-detection metric=jvm.thread.count.rate base=0.02 delta/30s=[na,+67%,na,+233%,na,-100%,na,-100%,na,-100%]
864 lower-ranked lines omitted

traces: slowest and errored traces after fault onset, critical path (! = error span)

T-0001trace=ee90eb40 dur=20678ms ok path: accounting:order-consumed 20678ms
T-0002trace=9d280109 dur=14505ms ok path: accounting:order-consumed 14505ms
T-0003trace=29cb9fc8 dur=12525ms ok path: accounting:order-consumed 12525ms
T-0004trace=afbc7c55 dur=11821ms ok path: accounting:order-consumed 11821ms
T-0005trace=1840d2cf dur=11453ms ok path: accounting:order-consumed 11453ms
T-0006trace=b47ab4e0 dur=10471ms ok path: accounting:order-consumed 10471ms
T-0007trace=c83dde68 dur=10080ms ok path: accounting:order-consumed 10080ms
T-0008trace=7a064e2f dur=9820ms ok path: accounting:order-consumed 9820ms
T-0009trace=bfb30fe6 dur=9042ms ok path: accounting:order-consumed 9042ms
T-0010trace=529f5af3 dur=8175ms ok path: accounting:order-consumed 8175ms
T-0011trace=72decc2b dur=7858ms ok path: accounting:order-consumed 7858ms
T-0012trace=dc2ace74 dur=7402ms ok path: accounting:order-consumed 7402ms
T-0013trace=96f1d91c dur=6959ms ok path: accounting:order-consumed 6959ms
T-0014trace=c7c8d258 dur=6874ms ok path: accounting:order-consumed 6874ms
T-0015trace=2e7ae851 dur=6279ms ok path: accounting:order-consumed 6279ms
T-0016trace=eb2c397b dur=6100ms ok path: accounting:order-consumed 6100ms
T-0017trace=78cc7768 dur=5737ms ok path: accounting:order-consumed 5737ms
T-0018trace=b0e69ef3 dur=5722ms ok path: accounting:order-consumed 5722ms
T-0019trace=6cc42752 dur=5702ms ok path: accounting:order-consumed 5702ms
T-0020trace=f9d765db dur=5473ms ok path: frontend-proxy:POST 5472ms > flagd:flagd.evaluation.v1.Service/EventStream 5472ms

logs: log templates per service, rate before vs after fault onset, with examples

L-0001svc=ad sev=INFO base=2.0/min fault=3.0/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0002svc=shipping sev=INFO base=1.2/min fault=1.8/min tpl="" ex=['', '']
L-0003svc=load-generator sev=INFO base=38.0/min fault=46.8/min tpl="User viewing cart" ex=['User viewing cart', 'User viewing cart']
L-0004svc=checkout sev=INFO base=0.2/min fault=0.0/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0005svc=load-generator sev=INFO base=14.4/min fault=12.0/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0006svc=ad sev=INFO base=48.6/min fault=42.8/min tpl="Targeted ad request received for <*>" ex=['Targeted ad request received for [assembly]', 'Targeted ad request received for [travel]']
L-0007svc=load-generator sev=INFO base=30.4/min fault=27.2/min tpl="Product added to cart successfully" ex=['Product added to cart successfully', 'Product added to cart successfully']
L-0008svc=kafka sev=INFO base=1.0/min fault=0.8/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=75955, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=76105, siz…']
L-0009svc=kafka sev=INFO base=1.0/min fault=0.8/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 76173 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 76290 breach: LogSegment(baseOff…']
L-0010svc=load-generator sev=INFO base=79.6/min fault=87.2/min tpl="User getting <*> for <*> <*>" ex=['User getting ads for category: assembly', 'User getting ads for category: assembly']
L-0011svc=kafka sev=INFO base=8.4/min fault=7.6/min tpl="Deleted <*> index <*>" ex=['Deleted offset index /tmp/kafka-logs/__cluster_metadata-0/00000000000000075895.index.deleted.', 'Deleted time index /tmp/kafka-logs/__cluster_metadata-0/00000000000000075895.timeindex.deleted.']
L-0012svc=kafka sev=INFO base=4.2/min fault=3.8/min tpl="Deleted log /tmp/kafka-logs/__cluster_metadata-<NUM>/<HEX>.log.deleted." ex=['Deleted log /tmp/kafka-logs/__cluster_metadata-0/00000000000000075895.log.deleted.', 'Deleted log /tmp/kafka-logs/__cluster_metadata-0/00000000000000075925.log.deleted.']
L-0013svc=kafka sev=INFO base=4.2/min fault=3.8/min tpl="Deleted producer state snapshot /tmp/kafka-logs/__cluster_metadata-<NUM>/<HEX>.snapshot.deleted" ex=['Deleted producer state snapshot /tmp/kafka-logs/__cluster_metadata-0/00000000000000075895.snapshot.deleted', 'Deleted producer state snapshot /tmp/kafka-logs/__cluster_metadata-0/00000000000000075925.snapshot.deleted']
L-0014svc=quote sev=INFO base=30.2/min fault=28.0/min tpl="Calculated quote" ex=['Calculated quote', 'Calculated quote']
L-0015svc=shipping sev=INFO base=30.2/min fault=28.0/min tpl="Requesting quote" ex=['Requesting quote', 'Requesting quote']
L-0016svc=shipping sev=INFO base=30.2/min fault=28.0/min tpl="Sending Quote" ex=['Sending Quote', 'Sending Quote']
L-0017svc=load-generator sev=INFO base=31.2/min fault=33.6/min tpl="Currency changed to CHF" ex=['Currency changed to CHF', 'Currency changed to CHF']
L-0018svc=load-generator sev=INFO base=14.0/min fault=13.2/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user acbd0f70-dc00-11f1-8306-d6b650b5399b', 'Checkout completed for user acbd0f70-dc00-11f1-8306-d6b650b5399b']
L-0019svc=kafka sev=INFO base=3.2/min fault=3.0/min tpl="Deleted snapshot files for snapshot OffsetAndEpoch[offset=<NUM>, epoch=<NUM>]." ex=['Deleted snapshot files for snapshot OffsetAndEpoch[offset=75900, epoch=1].', 'Deleted snapshot files for snapshot OffsetAndEpoch[offset=75939, epoch=1].']
L-0020svc=kafka sev=INFO base=3.2/min fault=3.0/min tpl="[RaftLog partition=__cluster_metadata-<NUM>, nodeId=<NUM>] Marking snapshot OffsetAndEpoch[offset=<NUM>, epoch=<NUM>] for deletion because …" ex=['[RaftLog partition=__cluster_metadata-0, nodeId=1] Marking snapshot OffsetAndEpoch[offset=76017, epoch=1] for deletion because its timestam…', '[RaftLog partition=__cluster_metadata-0, nodeId=1] Marking snapshot OffsetAndEpoch[offset=76056, epoch=1] for deletion because its timestam…']
L-0021svc=kafka sev=INFO base=3.2/min fault=3.0/min tpl="[SnapshotEmitter id=<NUM>] Successfully wrote snapshot <HEX>-<NUM>" ex=['[SnapshotEmitter id=1] Successfully wrote snapshot 00000000000000076212-0000000001', '[SnapshotEmitter id=1] Successfully wrote snapshot 00000000000000076251-0000000001']
L-0022svc=kafka sev=INFO base=3.2/min fault=3.0/min tpl="[SnapshotGenerator id=<NUM>] Creating new KRaft snapshot file snapshot <HEX>-<NUM> because we have replayed at least <NUM> bytes." ex=['[SnapshotGenerator id=1] Creating new KRaft snapshot file snapshot 00000000000000076212-0000000001 because we have replayed at least 2800 b…', '[SnapshotGenerator id=1] Creating new KRaft snapshot file snapshot 00000000000000076251-0000000001 because we have replayed at least 2800 b…']
L-0023svc=kafka sev=INFO base=3.2/min fault=3.0/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Incremented log start offset to <NUM> due to snapshot generated" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Incremented log start offset to 76056 due to snapshot generated', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Incremented log start offset to 76095 due to snapshot generated']
37 lower-ranked lines omitted
Loading runs in your browser…