Rootline v0.1 pilot

← All incidents

adHighCpu-r2

What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)

Service graph during the window

Evidence and answers

4601 prompt tokens

metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean

M-0001svc=ad metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.00917]
M-0002svc=ad metric=jvm.cpu.recent_utilization base=0.000131 delta/30s=[na,+73157%,na,+126011%,na,+126128%,na,+126213%,na,+126385%]
M-0003svc=ad metric=jvm.gc.duration.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%]
M-0004svc=ad metric=rpc.client.duration.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0005svc=ad metric=rpc.client.request.size.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0006svc=ad metric=rpc.client.response.size.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0007svc=ad metric=traces.span.metrics.duration.p50 base=1.01 delta/30s=[na,+594%,na,-1%,na,-1%,na,-1%,na,+1487915%]
M-0008svc=ad metric=traces.span.metrics.duration.p95 base=1.92 delta/30s=[na,+312%,na,-1%,na,-1%,na,-1%,na,+783066%]
M-0009svc=ad metric=traces.span.metrics.duration.p99 base=2.38 delta/30s=[na,+1314%,na,-17%,na,-17%,na,-17%,na,+631213%]
M-0010svc=fraud-detection metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0476]
M-0011svc=fraud-detection metric=rpc.client.duration.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0012svc=fraud-detection metric=rpc.client.request.size.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0013svc=fraud-detection metric=rpc.client.response.size.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0014svc=kafka metric=jvm.gc.duration.count.rate base=0 delta/30s=[+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%,na]
M-0015svc=kafka metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[+0%,na,+0%,na,new:2,na,+0%,na,+0%,na]
M-0016svc=recommendation metric=process.runtime.cpython.cpu.utilization base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:13.2,na]
M-0017svc=shipping metric=http.server.active_requests base=0 delta/30s=[na,+0%,na,+0%,na,new:2,na,+0%,na,+0%]
M-0018svc=fraud-detection metric=traces.span.metrics.duration.p50 base=46.6 delta/30s=[na,-14%,na,-19%,na,-1%,na,+34%,na,+32100%]
M-0019svc=fraud-detection metric=traces.span.metrics.duration.p95 base=94.5 delta/30s=[na,-1%,na,-2%,na,-0%,na,+2%,na,+15780%]
M-0020svc=fraud-detection metric=traces.span.metrics.duration.p99 base=98.9 delta/30s=[na,-0%,na,-0%,na,-0%,na,+0%,na,+15068%]
M-0021svc=ad metric=jvm.thread.count.rate base=0.00667 delta/30s=[na,+1900%,na,-100%,na,-100%,na,-100%,na,+900%]
M-0022svc=flagd metric=error_ratio base=9.24e-05 delta/30s=[na,-100%,na,-100%,na,-100%,na,-100%,na,+985%]
M-0023svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,-100%,-100%,-100%,-100%,-100%,-100%,-100%,-100%,+900%]
M-0024svc=ad metric=otel.sdk.processor.span.queue.size base=1.8 delta/30s=[na,-100%,na,+344%,na,+622%,na,-100%,na,+122%]
M-0025svc=kafka metric=kafka.purgatory.size base=9.6 delta/30s=[+233%,na,+171%,na,-58%,na,+483%,na,+67%,na]
M-0026svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,-100%,na,-100%,na,+400%]
M-0027svc=accounting metric=dns.lookup.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,-100%,na,-100%,na,+400%]
M-0028svc=ad metric=otel.sdk.processor.log.queue.size base=0.4 delta/30s=[na,-100%,na,+400%,na,+400%,na,-100%,na,-100%]
M-0029svc=checkout metric=go.goroutine.count.rate base=0.0133 delta/30s=[na,-100%,na,+400%,na,-100%,na,-100%,na,-100%]
M-0030svc=fraud-detection metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,-100%,na,+400%,na,-100%]
M-0031svc=fraud-detection metric=jvm.thread.count.rate base=0.02 delta/30s=[na,+233%,na,-100%,na,-100%,na,+67%,na,+400%]
M-0032svc=fraud-detection metric=otel.sdk.processor.log.queue.size base=0.2 delta/30s=[na,-100%,na,+400%,na,-100%,na,-100%,na,-100%]
M-0033svc=fraud-detection metric=otel.sdk.processor.span.queue.size base=0.8 delta/30s=[na,-100%,na,+150%,na,+150%,na,-100%,na,+400%]
M-0034svc=load-generator metric=process.runtime.cpython.gc_count base=78.4 delta/30s=[-80%,na,+108%,na,+311%,na,+146%,na,+318%,na]
M-0035svc=load-generator metric=otel.sdk.processor.log.queue.size base=2.4 delta/30s=[-17%,na,-17%,na,-17%,na,-17%,na,+317%,na]
M-0036svc=ad metric=rpc.server.duration.p99 base=4.95 delta/30s=[na,+305%,na,+0%,na,+0%,na,+0%,na,-0%]
M-0037svc=fraud-detection metric=kafka.consumer.commit_latency_max base=1 delta/30s=[na,+100%,na,+300%,na,+300%,na,+100%,na,+0%]
858 lower-ranked lines omitted

traces: slowest and errored traces after fault onset, critical path (! = error span)

T-0001trace=5e0a039a dur=32583ms ok path: accounting:order-consumed 32583ms
T-0002trace=17b31362 dur=25477ms ok path: accounting:order-consumed 25477ms
T-0003trace=b63c1442 dur=13488ms ok path: accounting:order-consumed 13488ms
T-0004trace=20074897 dur=12769ms ok path: accounting:order-consumed 12769ms
T-0005trace=d0431b3f dur=11739ms ok path: accounting:order-consumed 11739ms
T-0006trace=4f2a20f7 dur=11548ms ok path: accounting:order-consumed 11548ms
T-0007trace=9ba8e290 dur=9680ms ok path: accounting:order-consumed 9680ms
T-0008trace=d35b5eb7 dur=9573ms ok path: accounting:order-consumed 9573ms
T-0009trace=b797ff03 dur=9298ms ok path: accounting:order-consumed 9298ms
T-0010trace=b1b8191d dur=9293ms ok path: accounting:order-consumed 9293ms
T-0011trace=e5c76484 dur=8876ms ok path: accounting:order-consumed 8876ms
T-0012trace=54b6986a dur=8355ms ok path: accounting:order-consumed 8355ms
T-0013trace=68bcc888 dur=8070ms ok path: frontend-web:GET 692ms > frontend-proxy:GET 675ms > frontend:GET /api/shipping 674ms
T-0014trace=5e1b0f50 dur=7991ms ok path: accounting:order-consumed 7991ms
T-0015trace=5528d903 dur=7846ms ok path: accounting:order-consumed 7846ms
T-0016trace=84b7e59d dur=7256ms ok path: accounting:order-consumed 7256ms
T-0017trace=a9bf5cbd dur=6966ms ok path: accounting:order-consumed 6966ms
T-0018trace=74edbc67 dur=6721ms ok path: accounting:order-consumed 6721ms
T-0019trace=c92fcc95 dur=6366ms ok path: accounting:order-consumed 6366ms
T-0020trace=ce641b5c dur=6353ms ok path: accounting:order-consumed 6353ms

logs: log templates per service, rate before vs after fault onset, with examples

L-0001svc=ad sev=INFO base=0.0/min fault=54.0/min tpl="High CPU-Load problempattern enabled" ex=['High CPU-Load problempattern enabled', 'High CPU-Load problempattern enabled']
L-0002svc=ad sev=INFO base=1.8/min fault=3.8/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0003svc=load-generator sev=INFO base=14.8/min fault=11.2/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user c60310c0-dbf0-11f1-8306-d6b650b5399b', 'Checkout completed for user c60310c0-dbf0-11f1-8306-d6b650b5399b']
L-0004svc=load-generator sev=INFO base=11.2/min fault=14.8/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0005svc=accounting sev=INFORMATION base=15.0/min fault=11.6/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0006svc=cart sev=INFORMATION base=15.0/min fault=11.6/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0007svc=checkout sev=INFO base=15.0/min fault=11.6/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 171.549µs', 'Successful to write message. offset: 0, duration: 159.628µs']
L-0008svc=checkout sev=INFO base=15.0/min fault=11.6/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0009svc=checkout sev=INFO base=15.0/min fault=11.6/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0010svc=checkout sev=INFO base=15.0/min fault=11.6/min tpl="order placed" ex=['order placed', 'order placed']
L-0011svc=checkout sev=INFO base=15.0/min fault=11.6/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0012svc=checkout sev=INFO base=15.0/min fault=11.6/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0013svc=email sev=INFO base=15.0/min fault=11.6/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0014svc=fraud-detection sev=INFO base=15.0/min fault=11.6/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: bdfd13af-dbf0-11f1-82f3-d279f8fa1f6d, and updated total count to: 21350', 'Consumed record with orderId: c605adb3-dbf0-11f1-82f3-d279f8fa1f6d, and updated total count to: 21351']
L-0015svc=frontend sev=INFO base=15.0/min fault=11.6/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0016svc=payment sev=INFO base=15.0/min fault=11.6/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0017svc=payment sev=INFO base=15.0/min fault=11.6/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0018svc=shipping sev=INFO base=15.0/min fault=11.6/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0019svc=load-generator sev=INFO base=15.2/min fault=12.0/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user bdf3f318-dbf0-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user bdf3f318-dbf0-11f1-8306-d6b650b5399b']
L-0020svc=load-generator sev=INFO base=43.6/min fault=36.0/min tpl="User viewing cart" ex=['User viewing cart', 'User viewing cart']
L-0021svc=checkout sev=INFO base=0.0/min fault=0.2/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0022svc=load-generator sev=INFO base=91.2/min fault=78.0/min tpl="User <UUID> adding <NUM> of product <*> to cart" ex=['User bdf3f318-dbf0-11f1-8306-d6b650b5399b adding 3 of product 6E92ZMYYFZ to cart', 'User bdf3f318-dbf0-11f1-8306-d6b650b5399b adding 3 of product 6E92ZMYYFZ to cart']
L-0023svc=currency sev=INFO base=122.0/min fault=105.2/min tpl="conversion successful" ex=['conversion successful', 'conversion successful']
L-0024svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=62226, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=62346, siz…']
L-0025svc=cart sev=INFORMATION base=60.4/min fault=54.4/min tpl="AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}" ex=['AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}', 'AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}']
L-0026svc=load-generator sev=INFO base=77.6/min fault=85.6/min tpl="User getting <*> for <*> <*>" ex=['User getting ads for category: assembly', 'User getting ads for category: assembly']
L-0027svc=quote sev=INFO base=29.8/min fault=27.0/min tpl="Calculated quote" ex=['Calculated quote', 'Calculated quote']
L-0028svc=shipping sev=INFO base=29.8/min fault=27.0/min tpl="Requesting quote" ex=['Requesting quote', 'Requesting quote']
L-0029svc=shipping sev=INFO base=29.8/min fault=27.0/min tpl="Sending Quote" ex=['Sending Quote', 'Sending Quote']
L-0030svc=ad sev=INFO base=45.8/min fault=50.2/min tpl="Targeted ad request received for <*>" ex=['Targeted ad request received for [assembly]', 'Targeted ad request received for [binoculars]']
31 lower-ranked lines omitted
Loading runs in your browser…