Rootline v0.1 pilot

← All incidents

emailMemoryLeak-r1

What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)

Service graph during the window

Evidence and answers

5282 prompt tokens

metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean

M-0001svc=accounting metric=dotnet.thread_pool.thread.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333,na]
M-0002svc=accounting metric=process.thread.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333,na]
M-0003svc=ad metric=error_ratio base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.0108,na]
M-0004svc=ad metric=jvm.thread.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,new:0.0667,na]
M-0005svc=ad metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333,na]
M-0006svc=ad metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333,na]
M-0007svc=ad metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333,na]
M-0008svc=ad metric=traces.span.metrics.duration.p50 base=1.01 delta/30s=[+2%,na,+0%,na,-1%,na,-1%,na,+1478962%,na]
M-0009svc=ad metric=traces.span.metrics.duration.p95 base=1.93 delta/30s=[+2%,na,+0%,na,-1%,na,-1%,na,+778354%,na]
M-0010svc=ad metric=traces.span.metrics.duration.p99 base=2.67 delta/30s=[+30%,na,+80%,na,-26%,na,-26%,na,+562119%,na]
M-0011svc=fraud-detection metric=error_ratio base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.0323,na]
M-0012svc=fraud-detection metric=jvm.thread.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,new:0.1,na,new:0.0667,na]
M-0013svc=fraud-detection metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0014svc=fraud-detection metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0015svc=fraud-detection metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0016svc=kafka metric=jvm.thread.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,new:0.0333,na]
M-0017svc=product-catalog metric=otel.sdk.span.live base=0 delta/30s=[new:1,na,+0%,na,new:1,na,+0%,na,new:1,na]
M-0018svc=shipping metric=http.server.active_requests base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:1,na]
M-0019svc=fraud-detection metric=traces.span.metrics.duration.p50 base=46.6 delta/30s=[-36%,na,+7%,na,+0%,na,+34%,na,+32081%,na]
M-0020svc=fraud-detection metric=traces.span.metrics.duration.p95 base=93.6 delta/30s=[+76%,na,+2%,na,+1%,na,+3%,na,+15934%,na]
M-0021svc=unknown metric=system.disk.pending_operations base=1.4 delta/30s=[-100%,na,+1400%,na,-29%,na,-100%,na,+15757%,na]
M-0022svc=fraud-detection metric=traces.span.metrics.duration.p99 base=98.7 delta/30s=[+96%,na,+0%,na,+0%,na,+1%,na,+15096%,na]
M-0023svc=checkout metric=otel.sdk.span.live base=0.4 delta/30s=[-100%,na,-100%,na,+900%,na,-100%,na,-100%,na]
M-0024svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[+900%,-100%,-100%,-100%,-100%,-100%,-100%,-100%,-100%,-100%]
M-0025svc=payment metric=v8js.gc.duration.p50 base=0.00758 delta/30s=[-51%,na,-45%,na,+724%,na,+230%,na,+131%,na]
M-0026svc=otelcol-contrib metric=otelcol_exporter_queue_size base=1.2 delta/30s=[+233%,na,+150%,na,-100%,na,+567%,na,+67%,na]
M-0027svc=quote metric=otel.sdk.processor.span.queue.size base=1.8 delta/30s=[na,+567%,na,-100%,na,+67%,na,+67%,na,+233%]
M-0028svc=checkout metric=go.goroutine.count.rate base=0.0333 delta/30s=[+100%,na,-100%,na,+500%,na,-100%,na,-100%,na]
M-0029svc=product-catalog metric=otel.sdk.processor.log.queue.size base=4.2 delta/30s=[-29%,na,-52%,na,+43%,na,-29%,na,+495%,na]
M-0030svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,+400%,na,-100%,na,-100%,na]
M-0031svc=accounting metric=dns.lookup.duration.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,+400%,na,-100%,na,-100%,na]
M-0032svc=cart metric=dotnet.thread_pool.thread.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,+400%,na,-100%,na,-100%,na]
M-0033svc=cart metric=process.thread.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,+400%,na,-100%,na,+400%,na]
M-0034svc=fraud-detection metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[-100%,na,+400%,na,-100%,na,-100%,na,-100%,na]
M-0035svc=kafka metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,-100%,na,-100%,na,+400%,na]
M-0036svc=quote metric=otel.sdk.processor.log.queue.size base=0.2 delta/30s=[na,+400%,na,-100%,na,+400%,na,-100%,na,-100%]
M-0037svc=kafka metric=kafka.purgatory.size base=7.4 delta/30s=[+76%,na,-73%,na,+62%,na,+386%,na,+116%,na]
858 lower-ranked lines omitted

traces: slowest and errored traces after fault onset, critical path (! = error span)

T-0001trace=79dc366f dur=864ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 864ms!
T-0002trace=6d8cb3c0 dur=16441ms ok path: accounting:order-consumed 16441ms
T-0003trace=28858a7b dur=805ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 805ms!
T-0004trace=2d9459d9 dur=11444ms ok path: accounting:order-consumed 11444ms
T-0005trace=8b4af789 dur=788ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 788ms!
T-0006trace=a75a6387 dur=11235ms ok path: accounting:order-consumed 11235ms
T-0007trace=b226f383 dur=786ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 786ms!
T-0008trace=45446275 dur=10273ms ok path: accounting:order-consumed 10273ms
T-0009trace=79b8884f dur=693ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 693ms!
T-0010trace=9d5b7c2b dur=9135ms ok path: accounting:order-consumed 9135ms
T-0011trace=a22a263d dur=535ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 535ms!
T-0012trace=f734c340 dur=8469ms ok path: accounting:order-consumed 8469ms
T-0013trace=53fa30e0 dur=532ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 532ms!
T-0014trace=d2684aba dur=7990ms ok path: accounting:order-consumed 7990ms
T-0015trace=3a765e6b dur=532ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 532ms!
T-0016trace=2cbb06d3 dur=7906ms ok path: accounting:order-consumed 7906ms
T-0017trace=236a7409 dur=532ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 532ms!
T-0018trace=b1496daa dur=7111ms ok path: accounting:order-consumed 7111ms
T-0019trace=7dfc7438 dur=532ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 532ms!
T-0020trace=2ac0c5b5 dur=6687ms ok path: accounting:order-consumed 6687ms
T-0021trace=586c496f dur=518ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 518ms!
T-0022trace=525c4e4b dur=6441ms ok path: accounting:order-consumed 6441ms
T-0023trace=aaf06d25 dur=428ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 428ms!
T-0024trace=04b5162f dur=6239ms ok path: accounting:order-consumed 6239ms
T-0025trace=d6ae9b6a dur=410ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 410ms!
T-0026trace=ad5e2437 dur=6174ms ok path: accounting:order-consumed 6174ms
T-0027trace=0ee17919 dur=387ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 387ms!
T-0028trace=98a716c3 dur=5999ms ok path: accounting:order-consumed 5999ms
T-0029trace=3cc03dd6 dur=368ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 368ms!
T-0030trace=304c179d dur=5982ms ok path: accounting:order-consumed 5982ms
T-0031trace=b2372c19 dur=162ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 162ms!
T-0032trace=503a295c dur=5975ms ok path: accounting:order-consumed 5975ms
T-0033trace=a82bdaae dur=140ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 140ms!
T-0034trace=36cca430 dur=5937ms ok path: accounting:order-consumed 5937ms
T-0035trace=3f82eadc dur=120ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 120ms!
T-0036trace=c0a4ff6e dur=5780ms ok path: accounting:order-consumed 5780ms
T-0037trace=a7ac79cd dur=113ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 113ms!
T-0038trace=298d5012 dur=5742ms ok path: frontend-proxy:POST 5573ms > flagd:flagd.evaluation.v1.Service/EventStream 5614ms
T-0039trace=8284bb53 dur=103ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 103ms!
T-0040trace=04810d7a dur=5568ms ok path: accounting:order-consumed 5568ms

logs: log templates per service, rate before vs after fault onset, with examples

L-0001svc=shipping sev=INFO base=1.4/min fault=2.4/min tpl="" ex=['', '']
L-0002svc=ad sev=INFO base=3.4/min fault=2.4/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0003svc=load-generator sev=INFO base=12.8/min fault=15.6/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user 52071952-dbda-11f1-8306-d6b650b5399b', 'Checkout completed for user 52071952-dbda-11f1-8306-d6b650b5399b']
L-0004svc=checkout sev=INFO base=0.2/min fault=0.0/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0005svc=checkout sev=WARN base=0.2/min fault=0.0/min tpl="failed to send order confirmation: failed POST to email service: Post "http://email:<NUM>/send_order_confirmation": context deadline exceed…" ex=['failed to send order confirmation: failed POST to email service: Post "http://email:6060/send_order_confirmation": context deadline exceeded']
L-0006svc=load-generator sev=INFO base=73.2/min fault=88.0/min tpl="User <UUID> adding <NUM> of product <*> to cart" ex=['User 457e51dc-dbda-11f1-8306-d6b650b5399b adding 2 of product 6E92ZMYYFZ to cart', 'User 457e51dc-dbda-11f1-8306-d6b650b5399b adding 2 of product 6E92ZMYYFZ to cart']
L-0007svc=checkout sev=INFO base=12.6/min fault=15.2/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0008svc=cart sev=INFORMATION base=51.2/min fault=60.4/min tpl="AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}" ex=['AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}', 'AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}']
L-0009svc=ad sev=INFO base=40.2/min fault=47.4/min tpl="Targeted ad request received for <*>" ex=['Targeted ad request received for [binoculars]', 'Targeted ad request received for [binoculars]']
L-0010svc=accounting sev=INFORMATION base=12.8/min fault=15.2/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0011svc=cart sev=INFORMATION base=12.8/min fault=15.2/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0012svc=checkout sev=INFO base=12.8/min fault=15.2/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 186.74µs', 'Successful to write message. offset: 0, duration: 171.03µs']
L-0013svc=checkout sev=INFO base=12.8/min fault=15.2/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0014svc=checkout sev=INFO base=12.8/min fault=15.2/min tpl="order placed" ex=['order placed', 'order placed']
L-0015svc=checkout sev=INFO base=12.8/min fault=15.2/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0016svc=checkout sev=INFO base=12.8/min fault=15.2/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0017svc=email sev=INFO base=12.8/min fault=15.2/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0018svc=fraud-detection sev=INFO base=12.8/min fault=15.2/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: 45b7a9a0-dbda-11f1-82f3-d279f8fa1f6d, and updated total count to: 19266', 'Consumed record with orderId: 500f5ee7-dbda-11f1-82f3-d279f8fa1f6d, and updated total count to: 19267']
L-0019svc=frontend sev=INFO base=12.8/min fault=15.2/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0020svc=payment sev=INFO base=12.8/min fault=15.2/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0021svc=payment sev=INFO base=12.8/min fault=15.2/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0022svc=shipping sev=INFO base=12.8/min fault=15.2/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0023svc=shipping sev=ERROR base=0.2/min fault=0.4/min tpl="failed to resolve integer flag" ex=['failed to resolve integer flag', 'failed to resolve integer flag']
L-0024svc=quote sev=INFO base=27.4/min fault=31.6/min tpl="Calculated quote" ex=['Calculated quote', 'Calculated quote']
L-0025svc=shipping sev=INFO base=27.4/min fault=31.6/min tpl="Requesting quote" ex=['Requesting quote', 'Requesting quote']
L-0026svc=shipping sev=INFO base=27.4/min fault=31.6/min tpl="Sending Quote" ex=['Sending Quote', 'Sending Quote']
L-0027svc=load-generator sev=INFO base=40.0/min fault=34.8/min tpl="User viewing cart" ex=['User viewing cart', 'User viewing cart']
L-0028svc=load-generator sev=INFO base=12.8/min fault=14.8/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user 457e51dc-dbda-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user 457e51dc-dbda-11f1-8306-d6b650b5399b']
L-0029svc=kafka sev=INFO base=0.4/min fault=0.6/min tpl="[QuorumController id=<NUM>] Periodic task <*> generated <NUM> records in <NUM> microseconds." ex=['[QuorumController id=1] Periodic task expireDelegationTokens generated 0 records in 396 microseconds.', '[QuorumController id=1] Periodic task electPreferred generated 0 records in 27 microseconds.']
L-0030svc=load-generator sev=INFO base=32.0/min fault=28.0/min tpl="Currency changed to CHF" ex=['Currency changed to CHF', 'Currency changed to CHF']
32 lower-ranked lines omitted
Loading runs in your browser…