Rootline v0.1 pilot

← All incidents

netemLoss-r2

What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)

Service graph during the window

Evidence and answers

4391 prompt tokens

metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean

M-0001svc=accounting metric=dotnet.thread_pool.thread.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333,na]
M-0002svc=ad metric=jvm.class.count.rate base=0 delta/30s=[na,+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%]
M-0003svc=ad metric=jvm.gc.duration.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%]
M-0004svc=fraud-detection metric=jvm.class.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0005svc=otelcol-contrib metric=otelcol_exporter_in_flight_requests base=0 delta/30s=[na,+0%,na,+0%,na,new:1,na,+0%,na,+0%]
M-0006svc=payment metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,new:0.0256,na,+0%,na,+0%]
M-0007svc=payment metric=traces.span.metrics.duration.p50 base=1.2 delta/30s=[na,-10%,na,-2%,na,+1247860%,na,+33%,na,+16%]
M-0008svc=payment metric=traces.span.metrics.duration.p95 base=11.6 delta/30s=[na,-60%,na,-25%,na,+129099%,na,-38%,na,-66%]
M-0009svc=recommendation metric=process.runtime.cpython.cpu.utilization base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:12.8,na,+0%]
M-0010svc=payment metric=traces.span.metrics.duration.p99 base=20.6 delta/30s=[na,-72%,na,-53%,na,+72850%,na,-62%,na,-63%]
M-0011svc=unknown metric=system.disk.pending_operations base=0.6 delta/30s=[na,-100%,na,-100%,na,-100%,na,+2567%,na,-100%]
M-0012svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,-100%,-100%,-100%,-100%,+900%,-100%,-100%,-100%,-100%]
M-0013svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,-100%,na,+400%,na,-100%,na]
M-0014svc=accounting metric=dns.lookup.duration.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,-100%,na,+400%,na,-100%,na]
M-0015svc=accounting metric=process.thread.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,-100%,na,-100%,na,+400%,na]
M-0016svc=cart metric=dotnet.thread_pool.thread.count.rate base=0.00667 delta/30s=[+400%,na,-100%,na,-100%,na,-100%,na,-100%,na]
M-0017svc=cart metric=process.thread.count.rate base=0.00667 delta/30s=[+400%,na,-100%,na,+400%,na,-100%,na,-100%,na]
M-0018svc=fraud-detection metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[+400%,na,-100%,na,-100%,na,-100%,na,+400%,na]
M-0019svc=fraud-detection metric=otel.sdk.processor.log.queue.size base=0.2 delta/30s=[-100%,na,-100%,na,+400%,na,-100%,na,-100%,na]
M-0020svc=kafka metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,-100%,na,-100%,na,+400%]
M-0021svc=product-catalog metric=go.goroutine.count.rate base=0.0133 delta/30s=[na,-100%,na,-100%,na,+400%,na,-100%,na,-100%]
M-0022svc=frontend-proxy metric=traces.span.metrics.duration.p50 base=432.4 delta/30s=[na,-99%,na,-99%,na,-98%,na,-99%,na,+375%]
M-0023svc=otelcol-contrib metric=otelcol_exporter_queue_size base=2 delta/30s=[na,-50%,na,+100%,na,+300%,na,+50%,na,+100%]
M-0024svc=checkout metric=go.goroutine.count.rate base=0.0533 delta/30s=[na,-100%,na,+25%,na,+275%,na,-100%,na,+25%]
M-0025svc=ad metric=jvm.cpu.recent_utilization base=0.000124 delta/30s=[na,-10%,na,+260%,na,-4%,na,-10%,na,+7%]
M-0026svc=fraud-detection metric=otel.sdk.processor.span.queue.size base=1.2 delta/30s=[+233%,na,+233%,na,+67%,na,+67%,na,-100%,na]
M-0027svc=load-generator metric=otel.sdk.span.live base=1.4 delta/30s=[na,-29%,na,+43%,na,+43%,na,+43%,na,+186%]
M-0028svc=flagd metric=traces.span.metrics.duration.p50 base=5.85e+03 delta/30s=[na,-39%,na,-40%,na,+157%,na,+157%,na,-40%]
M-0029svc=ad metric=jvm.thread.count.rate base=0.0133 delta/30s=[na,-100%,na,-100%,na,+150%,na,+150%,na,-100%]
M-0030svc=fraud-detection metric=jvm.thread.count.rate base=0.0133 delta/30s=[+150%,na,+150%,na,-100%,na,-100%,na,-100%,na]
M-0031svc=shipping metric=http.server.request.duration.p95 base=0.801 delta/30s=[na,-58%,na,-17%,na,+6%,na,+8%,na,+145%]
M-0032svc=frontend metric=v8js.gc.duration.p99 base=0.00992 delta/30s=[+0%,na,+135%,na,+143%,na,-0%,na,+135%,na]
M-0033svc=flagd metric=error_ratio base=0.000192 delta/30s=[na,-100%,na,-100%,na,+132%,na,-100%,na,-100%]
M-0034svc=checkout metric=http.client.request.duration.p95 base=0.643 delta/30s=[na,-63%,na,-67%,na,-63%,na,-16%,na,+125%]
M-0035svc=frontend metric=v8js.gc.duration.p95 base=0.00959 delta/30s=[+1%,na,+75%,na,+114%,na,-0%,na,+75%,na]
M-0036svc=flagd metric=traces.span.metrics.duration.p95 base=7.45e+03 delta/30s=[na,-34%,na,-35%,na,+101%,na,+101%,na,-35%]
M-0037svc=checkout metric=rpc.client.call.duration.p99 base=0.0348 delta/30s=[na,-34%,na,-31%,na,+33%,na,-32%,na,+100%]
876 lower-ranked lines omitted

traces: slowest and errored traces after fault onset, critical path (! = error span)

T-0001trace=37f4f31b dur=16074ms ok path: accounting:order-consumed 16074ms
T-0002trace=264e2a28 dur=13492ms ok path: accounting:order-consumed 13492ms
T-0003trace=d1fc1fc2 dur=12256ms ok path: accounting:order-consumed 12256ms
T-0004trace=2762ced0 dur=12221ms ok path: accounting:order-consumed 12221ms
T-0005trace=364d74da dur=11069ms ok path: accounting:order-consumed 11069ms
T-0006trace=bbb96230 dur=9548ms ok path: accounting:order-consumed 9548ms
T-0007trace=37948761 dur=9273ms ok path: accounting:order-consumed 9273ms
T-0008trace=5aee3cc3 dur=9242ms ok path: accounting:order-consumed 9242ms
T-0009trace=c0c0cb26 dur=8949ms ok path: accounting:order-consumed 8949ms
T-0010trace=d4429b1a dur=8800ms ok path: accounting:order-consumed 8800ms
T-0011trace=ba17178c dur=8483ms ok path: accounting:order-consumed 8483ms
T-0012trace=2144ef68 dur=7234ms ok path: accounting:order-consumed 7234ms
T-0013trace=d1bf4c13 dur=7230ms ok path: accounting:order-consumed 7230ms
T-0014trace=f0190a5a dur=7079ms ok path: accounting:order-consumed 7079ms
T-0015trace=f47b2d36 dur=6229ms ok path: accounting:order-consumed 6229ms
T-0016trace=b6edcf2e dur=6195ms ok path: accounting:order-consumed 6195ms
T-0017trace=96322e0e dur=5997ms ok path: accounting:order-consumed 5997ms
T-0018trace=35669112 dur=5923ms ok path: accounting:order-consumed 5923ms
T-0019trace=f79eb131 dur=5864ms ok path: accounting:order-consumed 5864ms
T-0020trace=2e098251 dur=5689ms ok path: accounting:order-consumed 5689ms

logs: log templates per service, rate before vs after fault onset, with examples

L-0001svc=load-generator sev=INFO base=12.8/min fault=18.4/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user 78e7a054-dbf4-11f1-8306-d6b650b5399b', 'Checkout completed for user 78e7a054-dbf4-11f1-8306-d6b650b5399b']
L-0002svc=load-generator sev=INFO base=11.2/min fault=14.0/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0003svc=shipping sev=INFO base=1.8/min fault=2.4/min tpl="" ex=['', '']
L-0004svc=checkout sev=INFO base=0.2/min fault=0.0/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0005svc=ad sev=INFO base=3.2/min fault=3.8/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0006svc=load-generator sev=INFO base=44.0/min fault=38.4/min tpl="User viewing cart" ex=['User viewing cart', 'User viewing cart']
L-0007svc=load-generator sev=INFO base=15.6/min fault=13.6/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user 76b2776e-dbf4-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user 76b2776e-dbf4-11f1-8306-d6b650b5399b']
L-0008svc=accounting sev=INFORMATION base=14.2/min fault=16.0/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0009svc=cart sev=INFORMATION base=14.2/min fault=16.0/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0010svc=checkout sev=INFO base=14.2/min fault=16.0/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 163.577µs', 'Successful to write message. offset: 0, duration: 118.046µs']
L-0011svc=checkout sev=INFO base=14.2/min fault=16.0/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0012svc=checkout sev=INFO base=14.2/min fault=16.0/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0013svc=checkout sev=INFO base=14.2/min fault=16.0/min tpl="order placed" ex=['order placed', 'order placed']
L-0014svc=checkout sev=INFO base=14.2/min fault=16.0/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0015svc=checkout sev=INFO base=14.2/min fault=16.0/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0016svc=email sev=INFO base=14.2/min fault=16.0/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0017svc=fraud-detection sev=INFO base=14.2/min fault=16.0/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: 76b75e8e-dbf4-11f1-82f3-d279f8fa1f6d, and updated total count to: 21623', 'Consumed record with orderId: 78e9e66e-dbf4-11f1-82f3-d279f8fa1f6d, and updated total count to: 21624']
L-0018svc=frontend sev=INFO base=14.2/min fault=16.0/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0019svc=payment sev=INFO base=14.2/min fault=16.0/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0020svc=payment sev=INFO base=14.2/min fault=16.0/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0021svc=shipping sev=INFO base=14.2/min fault=16.0/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0022svc=quote sev=INFO base=28.8/min fault=31.6/min tpl="Calculated quote" ex=['Calculated quote', 'Calculated quote']
L-0023svc=shipping sev=INFO base=28.8/min fault=31.6/min tpl="Requesting quote" ex=['Requesting quote', 'Requesting quote']
L-0024svc=shipping sev=INFO base=28.8/min fault=31.6/min tpl="Sending Quote" ex=['Sending Quote', 'Sending Quote']
L-0025svc=load-generator sev=INFO base=81.2/min fault=76.0/min tpl="User getting <*> for <*> <*>" ex=['User getting ads for category: telescopes', 'User getting ads for category: telescopes']
L-0026svc=kafka sev=INFO base=2.0/min fault=2.2/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=65429, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=65519, siz…']
L-0027svc=kafka sev=INFO base=2.0/min fault=2.2/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 65604 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 65682 breach: LogSegment(baseOff…']
33 lower-ranked lines omitted
Loading runs in your browser…