Rootline v0.1 pilot

← All incidents

adHighCpu-r1

What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)

Service graph during the window

Evidence and answers

5291 prompt tokens

metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean

M-0001svc=accounting metric=process.thread.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333,na]
M-0002svc=ad metric=error_ratio base=0 delta/30s=[+0%,na,new:0.0105,na,+0%,na,+0%,na,+0%,na]
M-0003svc=ad metric=jvm.class.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0004svc=ad metric=jvm.cpu.recent_utilization base=0.00014 delta/30s=[+13063%,na,+117713%,na,+117387%,na,+117575%,na,+117375%,na]
M-0005svc=ad metric=jvm.thread.count.rate base=0 delta/30s=[new:0.2,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0006svc=ad metric=rpc.client.duration.count.rate base=0 delta/30s=[new:0.0333,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0007svc=ad metric=rpc.client.request.size.count.rate base=0 delta/30s=[new:0.0333,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0008svc=ad metric=rpc.client.response.size.count.rate base=0 delta/30s=[new:0.0333,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0009svc=ad metric=traces.span.metrics.duration.p50 base=1.01 delta/30s=[+14724%,na,+1482251%,na,+6%,na,+1%,na,-1%,na]
M-0010svc=ad metric=traces.span.metrics.duration.p95 base=1.92 delta/30s=[+10042%,na,+780085%,na,+48%,na,+1%,na,-1%,na]
M-0011svc=ad metric=traces.span.metrics.duration.p99 base=2.56 delta/30s=[+11366%,na,+584923%,na,+100%,na,+100%,na,-23%,na]
M-0012svc=fraud-detection metric=error_ratio base=0 delta/30s=[+0%,na,new:0.0345,na,+0%,na,+0%,na,+0%,na]
M-0013svc=fraud-detection metric=jvm.thread.count.rate base=0 delta/30s=[new:0.0333,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0014svc=fraud-detection metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0015svc=fraud-detection metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0016svc=fraud-detection metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0017svc=fraud-detection metric=traces.span.metrics.duration.p50 base=47.5 delta/30s=[+58%,na,+31455%,na,+26%,na,+31%,na,-12%,na]
M-0018svc=fraud-detection metric=traces.span.metrics.duration.p95 base=94.5 delta/30s=[+3%,na,+15767%,na,+2%,na,+2%,na,-1%,na]
M-0019svc=fraud-detection metric=traces.span.metrics.duration.p99 base=98.9 delta/30s=[+1%,na,+15066%,na,+0%,na,+0%,na,-0%,na]
M-0020svc=ad metric=rpc.server.duration.p99 base=4.95 delta/30s=[+7476%,na,+0%,na,+57%,na,+0%,na,+0%,na]
M-0021svc=ad metric=jvm.cpu.time base=101.8 delta/30s=[+27%,na,+262%,na,+498%,na,+734%,na,+970%,na]
M-0022svc=payment metric=nodejs.eventloop.delay.stddev base=0.000809 delta/30s=[-1%,na,-75%,na,+961%,na,+9%,na,-63%,na]
M-0023svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,-100%,-100%,-100%,+900%,-100%,-100%,-100%,-100%,-100%]
M-0024svc=payment metric=nodejs.eventloop.delay.max base=0.0616 delta/30s=[+7%,na,-76%,na,+663%,na,-12%,na,-61%,na]
M-0025svc=payment metric=nodejs.eventloop.utilization base=0.00615 delta/30s=[-19%,na,-32%,na,+569%,na,+88%,na,-27%,na]
M-0026svc=payment metric=v8js.gc.duration.p50 base=0.008 delta/30s=[-53%,na,-61%,na,+525%,na,+369%,na,+9%,na]
M-0027svc=payment metric=v8js.gc.duration.p99 base=0.0443 delta/30s=[+11%,na,-78%,na,+451%,na,+67%,na,-45%,na]
M-0028svc=payment metric=v8js.gc.duration.p95 base=0.0413 delta/30s=[+12%,na,-79%,na,+432%,na,+69%,na,-45%,na]
M-0029svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,-100%,na,-100%,na,+400%,na]
M-0030svc=accounting metric=dns.lookup.duration.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,-100%,na,-100%,na,+400%,na]
M-0031svc=cart metric=dotnet.thread_pool.thread.count.rate base=0.00667 delta/30s=[-100%,na,+400%,na,-100%,na,-100%,na,-100%,na]
M-0032svc=cart metric=process.thread.count.rate base=0.00667 delta/30s=[-100%,na,+400%,na,-100%,na,-100%,na,-100%,na]
M-0033svc=fraud-detection metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[+400%,na,-100%,na,-100%,na,-100%,na,+400%,na]
M-0034svc=fraud-detection metric=kafka.consumer.heartbeat_response_time_max base=1 delta/30s=[+0%,na,+400%,na,+0%,na,+0%,na,+200%,na]
M-0035svc=accounting metric=db.client.connection.npgsql.create_time.p99 base=0.00995 delta/30s=[na,na,na,na,na,na,na,na,+398%,na]
M-0036svc=accounting metric=db.client.connection.npgsql.create_time.p95 base=0.00975 delta/30s=[na,na,na,na,na,na,na,na,+392%,na]
M-0037svc=checkout metric=rpc.client.call.duration.p50 base=0.0644 delta/30s=[-3%,na,-22%,na,+55%,na,+366%,na,-29%,na]
858 lower-ranked lines omitted

traces: slowest and errored traces after fault onset, critical path (! = error span)

T-0001trace=dfef3838 dur=900ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 900ms!
T-0002trace=541c7956 dur=18341ms ok path: accounting:order-consumed 18341ms
T-0003trace=722daef5 dur=900ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 900ms!
T-0004trace=d360d841 dur=13594ms ok path: accounting:order-consumed 13594ms
T-0005trace=a54b4cc4 dur=746ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 746ms!
T-0006trace=36ca4275 dur=12939ms ok path: accounting:order-consumed 12939ms
T-0007trace=40b60f95 dur=742ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 742ms!
T-0008trace=911ce7ba dur=11346ms ok path: accounting:order-consumed 11346ms
T-0009trace=c0af2824 dur=740ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 740ms!
T-0010trace=36b30bdb dur=10918ms ok path: accounting:order-consumed 10918ms
T-0011trace=897c4fc3 dur=717ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 717ms!
T-0012trace=9e867e0f dur=9640ms ok path: accounting:order-consumed 9640ms
T-0013trace=945934ce dur=294ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 294ms!
T-0014trace=9fa567e8 dur=8887ms ok path: accounting:order-consumed 8887ms
T-0015trace=881c85ce dur=293ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 293ms!
T-0016trace=5c3304e1 dur=8495ms ok path: accounting:order-consumed 8495ms
T-0017trace=eec1d239 dur=292ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 292ms!
T-0018trace=10c08e0f dur=8249ms ok path: accounting:order-consumed 8249ms
T-0019trace=8455a622 dur=289ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 289ms!
T-0020trace=f41701cb dur=7881ms ok path: accounting:order-consumed 7881ms
T-0021trace=f53e8bbc dur=288ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 288ms!
T-0022trace=12d7bcd2 dur=7797ms ok path: accounting:order-consumed 7797ms
T-0023trace=4d1ba781 dur=288ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 288ms!
T-0024trace=c4e31d7e dur=7774ms ok path: accounting:order-consumed 7774ms
T-0025trace=0c6f2dcd dur=284ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 284ms!
T-0026trace=1101a7aa dur=7751ms ok path: accounting:order-consumed 7751ms
T-0027trace=31a1872f dur=280ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 280ms!
T-0028trace=2cd38635 dur=7685ms ok path: accounting:order-consumed 7685ms
T-0029trace=3fd82ec6 dur=213ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 213ms!
T-0030trace=2df824cb dur=7580ms ok path: accounting:order-consumed 7580ms
T-0031trace=0bbb8c3a dur=122ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 122ms!
T-0032trace=92ced336 dur=7479ms ok path: accounting:order-consumed 7479ms
T-0033trace=44edbd0d dur=120ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 120ms!
T-0034trace=0c8d3e23 dur=6977ms ok path: accounting:order-consumed 6977ms
T-0035trace=4b382920 dur=98ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 98ms!
T-0036trace=04018bd2 dur=6668ms ok path: accounting:order-consumed 6668ms
T-0037trace=61be125d dur=97ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 97ms!
T-0038trace=d75edc40 dur=6631ms ok path: accounting:order-consumed 6631ms
T-0039trace=c8aed2ba dur=54ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 54ms!
T-0040trace=3c89ce21 dur=6447ms ok path: accounting:order-consumed 6447ms

logs: log templates per service, rate before vs after fault onset, with examples

L-0001svc=ad sev=INFO base=0.0/min fault=46.8/min tpl="High CPU-Load problempattern enabled" ex=['High CPU-Load problempattern enabled', 'High CPU-Load problempattern enabled']
L-0002svc=shipping sev=INFO base=0.2/min fault=1.2/min tpl="" ex=['', '']
L-0003svc=load-generator sev=INFO base=8.8/min fault=14.0/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user 242e87c0-dbdc-11f1-8306-d6b650b5399b', 'Checkout completed for user 242e87c0-dbdc-11f1-8306-d6b650b5399b']
L-0004svc=checkout sev=WARN base=0.4/min fault=0.0/min tpl="failed to send order confirmation: failed POST to email service: Post "http://email:<NUM>/send_order_confirmation": context deadline exceed…" ex=['failed to send order confirmation: failed POST to email service: Post "http://email:6060/send_order_confirmation": context deadline exceeded', 'failed to send order confirmation: failed POST to email service: Post "http://email:6060/send_order_confirmation": context deadline exceeded']
L-0005svc=load-generator sev=INFO base=40.0/min fault=48.8/min tpl="User viewing cart" ex=['User viewing cart', 'User viewing cart']
L-0006svc=ad sev=INFO base=2.4/min fault=1.8/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0007svc=checkout sev=INFO base=0.0/min fault=0.2/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0008svc=checkout sev=INFO base=10.4/min fault=12.4/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0009svc=load-generator sev=INFO base=12.8/min fault=10.8/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user 2485cef4-dbdc-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user 2485cef4-dbdc-11f1-8306-d6b650b5399b']
L-0010svc=load-generator sev=INFO base=26.0/min fault=30.0/min tpl="Currency changed to CHF" ex=['Currency changed to CHF', 'Currency changed to CHF']
L-0011svc=accounting sev=INFORMATION base=10.8/min fault=12.4/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0012svc=cart sev=INFORMATION base=10.8/min fault=12.4/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0013svc=checkout sev=INFO base=10.8/min fault=12.4/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 271.444µs', 'Successful to write message. offset: 0, duration: 172.809µs']
L-0014svc=checkout sev=INFO base=10.8/min fault=12.4/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0015svc=checkout sev=INFO base=10.8/min fault=12.4/min tpl="order placed" ex=['order placed', 'order placed']
L-0016svc=checkout sev=INFO base=10.8/min fault=12.4/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0017svc=checkout sev=INFO base=10.8/min fault=12.4/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0018svc=email sev=INFO base=10.8/min fault=12.4/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0019svc=fraud-detection sev=INFO base=10.8/min fault=12.4/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: 24322fac-dbdc-11f1-82f3-d279f8fa1f6d, and updated total count to: 19451', 'Consumed record with orderId: 24c4132c-dbdc-11f1-82f3-d279f8fa1f6d, and updated total count to: 19452']
L-0020svc=frontend sev=INFO base=10.8/min fault=12.4/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0021svc=payment sev=INFO base=10.8/min fault=12.4/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0022svc=payment sev=INFO base=10.8/min fault=12.4/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0023svc=shipping sev=INFO base=10.8/min fault=12.4/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0024svc=kafka sev=INFO base=2.0/min fault=2.4/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=44464, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=44494, siz…']
L-0025svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 44622 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 44739 breach: LogSegment(baseOff…']
L-0026svc=kafka sev=INFO base=1.0/min fault=0.8/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=44554, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=44674, siz…']
L-0027svc=load-generator sev=INFO base=84.0/min fault=75.6/min tpl="User getting <*> for <*> <*>" ex=['User getting ads for category: books', 'User getting ads for category: books']
35 lower-ranked lines omitted
Loading runs in your browser…