Rootline v0.1 pilot

← All incidents

hwWeakSignal-r3

What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)

Service graph during the window

Evidence and answers

4525 prompt tokens

metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean

M-0001svc=accounting metric=dotnet.thread_pool.thread.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%]
M-0002svc=accounting metric=process.thread.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,+0%,na,new:0.0333,na,+0%]
M-0003svc=ad metric=jvm.gc.duration.count.rate base=0 delta/30s=[na,+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%]
M-0004svc=checkout metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[na,+0%,na,new:6,na,+0%,na,+0%,na,+0%]
M-0005svc=kafka metric=jvm.gc.duration.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%]
M-0006svc=otelcol-contrib metric=otelcol_exporter_in_flight_requests base=0 delta/30s=[na,+0%,na,new:1,na,+0%,na,+0%,na,+0%]
M-0007svc=product-catalog metric=go.goroutine.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0667]
M-0008svc=payment metric=error_ratio base=0.00691 delta/30s=[na,+43%,na,-100%,na,+44%,na,-26%,na,+4234%]
M-0009svc=cart metric=demo.cart.add_item.latency.p99 base=0.00608 delta/30s=[na,+451%,na,-19%,na,-19%,na,-19%,na,+912%]
M-0010svc=recommendation metric=otel.sdk.processor.log.queue.size base=0.6 delta/30s=[na,+400%,na,+400%,na,-100%,na,-100%,na,+900%]
M-0011svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,-100%,-100%,+900%,-100%,-100%,-100%,-100%,-100%,-100%]
M-0012svc=cart metric=http.server.request.duration.p99 base=0.00622 delta/30s=[na,+438%,na,-20%,na,-20%,na,-20%,na,+888%]
M-0013svc=cart metric=traces.span.metrics.duration.p99 base=6.06 delta/30s=[na,+511%,na,-5%,na,-34%,na,-35%,na,+362%]
M-0014svc=otelcol-contrib metric=otelcol_exporter_queue_size base=2.4 delta/30s=[na,+442%,na,+483%,na,-17%,na,-100%,na,+108%]
M-0015svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,-100%,na,+400%,na,-100%]
M-0016svc=accounting metric=dns.lookup.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,-100%,na,+400%,na,-100%]
M-0017svc=fraud-detection metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,+400%,na,-100%,na,-100%,na,-100%]
M-0018svc=quote metric=otel.sdk.processor.log.queue.size base=0.4 delta/30s=[na,-100%,na,-100%,na,-100%,na,-100%,na,+400%]
M-0019svc=frontend metric=http.server.request.duration.p95 base=0.118 delta/30s=[na,-27%,na,-24%,na,-39%,na,+78%,na,+386%]
M-0020svc=load-generator metric=http.client.duration.p95 base=192.3 delta/30s=[na,-50%,na,-1%,na,-32%,na,+54%,na,+251%]
M-0021svc=unknown metric=system.processes.count.rate base=0.107 delta/30s=[na,+88%,na,+244%,na,+244%,na,+244%,na,+88%]
M-0022svc=checkout metric=rpc.server.call.duration.p50 base=0.0511 delta/30s=[na,-20%,na,+10%,na,-18%,na,+47%,na,+243%]
M-0023svc=checkout metric=traces.span.metrics.duration.p50 base=52.4 delta/30s=[na,-25%,na,-14%,na,-24%,na,+43%,na,+234%]
M-0024svc=ad metric=otel.sdk.processor.log.queue.size base=0.6 delta/30s=[na,-100%,na,-100%,na,-100%,na,+233%,na,-100%]
M-0025svc=load-generator metric=process.runtime.cpython.gc_count base=234.8 delta/30s=[na,-64%,na,-56%,na,+6%,na,+125%,na,+205%]
M-0026svc=accounting metric=dotnet.gc.last_collection.heap.fragmentation.size base=2.89e+05 delta/30s=[na,+167%,na,+167%,na,+167%,na,+167%,na,+5%]
M-0027svc=frontend-proxy metric=traces.span.metrics.duration.p50 base=824.5 delta/30s=[na,-99%,na,+154%,na,-99%,na,-99%,na,-99%]
M-0028svc=cart metric=dotnet.thread_pool.thread.count.rate base=0.0133 delta/30s=[na,-100%,na,-100%,na,+150%,na,-100%,na,+150%]
M-0029svc=cart metric=process.thread.count.rate base=0.0133 delta/30s=[na,-100%,na,-100%,na,+150%,na,-100%,na,+150%]
M-0030svc=checkout metric=go.goroutine.count.rate base=0.0267 delta/30s=[na,-100%,na,-100%,na,-100%,na,-100%,na,+150%]
M-0031svc=fraud-detection metric=otel.sdk.processor.span.queue.size base=1.6 delta/30s=[na,-100%,na,-100%,na,+150%,na,-100%,na,-100%]
M-0032svc=load-generator metric=otel.sdk.span.live base=1.6 delta/30s=[na,+25%,na,-38%,na,-100%,na,+150%,na,-100%]
M-0033svc=recommendation metric=process.runtime.cpython.gc_count base=129.4 delta/30s=[na,-92%,na,+65%,na,-44%,na,-93%,na,+148%]
M-0034svc=checkout metric=traces.span.metrics.duration.p99 base=773.6 delta/30s=[na,+71%,na,+148%,na,-50%,na,-50%,na,+145%]
M-0035svc=checkout metric=traces.span.metrics.duration.p95 base=668.0 delta/30s=[na,+53%,na,+137%,na,-49%,na,-51%,na,+123%]
M-0036svc=frontend metric=traces.span.metrics.duration.p99 base=2.04e+03 delta/30s=[na,+119%,na,-5%,na,-52%,na,-4%,na,-7%]
M-0037svc=accounting metric=request_rate base=0.947 delta/30s=[na,+111%,na,+37%,na,+27%,na,+62%,na,+80%]
876 lower-ranked lines omitted

traces: slowest and errored traces after fault onset, critical path (! = error span)

T-0001trace=d2b61cc7 dur=15466ms ok path: accounting:order-consumed 15466ms
T-0002trace=b603a6b2 dur=15111ms ok path: accounting:order-consumed 15111ms
T-0003trace=8b8168fa dur=14888ms ok path: accounting:order-consumed 14888ms
T-0004trace=1b00aad7 dur=14818ms ok path: accounting:order-consumed 14818ms
T-0005trace=1beca288 dur=13176ms ok path: accounting:order-consumed 13176ms
T-0006trace=06ad0635 dur=8908ms ok path: accounting:order-consumed 8908ms
T-0007trace=b4f5ee71 dur=8644ms ok path: accounting:order-consumed 8644ms
T-0008trace=6d608c24 dur=8515ms ok path: accounting:order-consumed 8515ms
T-0009trace=9a0a376d dur=8407ms ok path: accounting:order-consumed 8407ms
T-0010trace=125c5199 dur=8381ms ok path: accounting:order-consumed 8381ms
T-0011trace=b06bfca1 dur=7802ms ok path: accounting:order-consumed 7802ms
T-0012trace=a9e45dd8 dur=7469ms ok path: accounting:order-consumed 7469ms
T-0013trace=1a1e5452 dur=7347ms ok path: accounting:order-consumed 7347ms
T-0014trace=f3805218 dur=7165ms ok path: accounting:order-consumed 7165ms
T-0015trace=d7e11f44 dur=6749ms ok path: accounting:order-consumed 6749ms
T-0016trace=d61cd541 dur=6635ms ok path: accounting:order-consumed 6635ms
T-0017trace=803ad78d dur=6536ms ok path: frontend-proxy:POST 6536ms > flagd:flagd.evaluation.v1.Service/EventStream 6536ms
T-0018trace=eafe88c4 dur=6199ms ok path: accounting:order-consumed 6199ms
T-0019trace=628a10a6 dur=6051ms ok path: accounting:order-consumed 6051ms
T-0020trace=02678a6c dur=5965ms ok path: accounting:order-consumed 5965ms

logs: log templates per service, rate before vs after fault onset, with examples

L-0001svc=shipping sev=INFO base=1.0/min fault=2.8/min tpl="" ex=['', '']
L-0002svc=load-generator sev=INFO base=11.2/min fault=17.2/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user 3a335470-dc08-11f1-8306-d6b650b5399b', 'Checkout completed for user 3a335470-dc08-11f1-8306-d6b650b5399b']
L-0003svc=checkout sev=INFO base=0.0/min fault=0.2/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0004svc=accounting sev=INFORMATION base=12.4/min fault=14.6/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0005svc=cart sev=INFORMATION base=12.4/min fault=14.6/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0006svc=checkout sev=INFO base=12.4/min fault=14.6/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 180.147µs', 'Successful to write message. offset: 0, duration: 162.087µs']
L-0007svc=checkout sev=INFO base=12.4/min fault=14.6/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0008svc=checkout sev=INFO base=12.4/min fault=14.6/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0009svc=checkout sev=INFO base=12.4/min fault=14.6/min tpl="order placed" ex=['order placed', 'order placed']
L-0010svc=checkout sev=INFO base=12.4/min fault=14.6/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0011svc=checkout sev=INFO base=12.4/min fault=14.6/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0012svc=email sev=INFO base=12.4/min fault=14.6/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0013svc=fraud-detection sev=INFO base=12.4/min fault=14.6/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: 3652ab61-dc08-11f1-9d0e-3288f2b1222e, and updated total count to: 23451', 'Consumed record with orderId: 36dfcb23-dc08-11f1-9d0e-3288f2b1222e, and updated total count to: 23452']
L-0014svc=frontend sev=INFO base=12.4/min fault=14.6/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0015svc=payment sev=INFO base=12.4/min fault=14.6/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0016svc=payment sev=INFO base=12.4/min fault=14.6/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0017svc=shipping sev=INFO base=12.4/min fault=14.6/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0018svc=load-generator sev=INFO base=13.6/min fault=12.0/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user 364d2728-dc08-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user 364d2728-dc08-11f1-8306-d6b650b5399b']
L-0019svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=82345, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=82465, siz…']
L-0020svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 82530 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 82647 breach: LogSegment(baseOff…']
L-0021svc=load-generator sev=INFO base=44.8/min fault=40.8/min tpl="User viewing cart" ex=['User viewing cart', 'User viewing cart']
L-0022svc=currency sev=INFO base=107.8/min fault=117.0/min tpl="conversion successful" ex=['conversion successful', 'conversion successful']
L-0023svc=quote sev=INFO base=27.4/min fault=29.8/min tpl="Calculated quote" ex=['Calculated quote', 'Calculated quote']
L-0024svc=shipping sev=INFO base=27.4/min fault=29.8/min tpl="Requesting quote" ex=['Requesting quote', 'Requesting quote']
L-0025svc=shipping sev=INFO base=27.4/min fault=29.8/min tpl="Sending Quote" ex=['Sending Quote', 'Sending Quote']
L-0026svc=load-generator sev=INFO base=128.4/min fault=139.2/min tpl="User browsing product: <*>" ex=['User browsing product: 1YMWWN1N4O', 'User browsing product: 1YMWWN1N4O']
L-0027svc=load-generator sev=INFO base=28.4/min fault=30.4/min tpl="Currency changed to CHF" ex=['Currency changed to CHF', 'Currency changed to CHF']
L-0028svc=kafka sev=INFO base=2.2/min fault=2.0/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=82405, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=82435, siz…']
32 lower-ranked lines omitted
Loading runs in your browser…