Rootline v0.1 pilot

← All incidents

paymentUnreachable-r1

What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)

Service graph during the window

Evidence and answers

5637 prompt tokens

metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean

M-0001svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%]
M-0002svc=accounting metric=dns.lookup.duration.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%]
M-0003svc=accounting metric=http.client.connection.duration.count.rate base=0 delta/30s=[na,+0%,na,new:0.0667,na,+0%,na,+0%,na,+0%]
M-0004svc=ad metric=rpc.client.duration.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0005svc=ad metric=rpc.client.request.size.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0006svc=ad metric=rpc.client.response.size.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0007svc=fraud-detection metric=rpc.client.duration.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0008svc=fraud-detection metric=rpc.client.request.size.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0009svc=fraud-detection metric=rpc.client.response.size.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0010svc=frontend metric=error_ratio base=0 delta/30s=[na,new:0.00662,na,new:0.0183,na,new:0.0164,na,new:0.0066,na,new:0.0146]
M-0011svc=frontend-proxy metric=error_ratio base=0 delta/30s=[na,new:0.00337,na,new:0.00895,na,new:0.00766,na,new:0.00402,na,new:0.00543]
M-0012svc=kafka metric=jvm.gc.duration.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%]
M-0013svc=load-generator metric=error_ratio base=0 delta/30s=[na,new:0.0147,na,new:0.0369,na,new:0.0317,na,new:0.0179,na,new:0.0254]
M-0014svc=load-generator metric=process.open_file_descriptor.count.rate base=0 delta/30s=[+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%,na]
M-0015svc=load-generator metric=process.runtime.cpython.cpu.utilization base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:17.5,na,+0%,na]
M-0016svc=quote metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:1,na,+0%]
M-0017svc=recommendation metric=process.runtime.cpython.cpu.utilization base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:13.2,na,+0%,na]
M-0018svc=shipping metric=http.server.active_requests base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:1,na,+0%]
M-0019svc=frontend metric=http.server.request.duration.p50 base=0.00643 delta/30s=[na,+60005%,na,+57022%,na,+58238%,na,+58238%,na,+54348%]
M-0020svc=checkout metric=error_ratio base=0.00122 delta/30s=[na,+6816%,na,+17546%,na,+18379%,na,+13249%,na,+20400%]
M-0021svc=load-generator metric=http.client.duration.p50 base=32.9 delta/30s=[+13203%,na,+11303%,na,+11091%,na,+11303%,na,+10542%,na]
M-0022svc=checkout metric=go.goroutine.count.rate base=0.0467 delta/30s=[na,+2400%,na,+5043%,na,+4257%,na,+2686%,na,+2757%]
M-0023svc=kafka metric=kafka.purgatory.size base=13.2 delta/30s=[na,+1021%,na,+2839%,na,+4658%,na,+1173%,na,+2983%]
M-0024svc=fraud-detection metric=jvm.thread.count.rate base=0.0133 delta/30s=[na,+150%,na,-100%,na,-100%,na,-100%,na,+900%]
M-0025svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,-100%,-100%,-100%,-100%,-100%,-100%,-100%,-100%,+900%]
M-0026svc=checkout metric=rpc.server.call.duration.p50 base=0.458 delta/30s=[na,+773%,na,+702%,na,+718%,na,+718%,na,+669%]
M-0027svc=frontend metric=traces.span.metrics.duration.p50 base=477.7 delta/30s=[na,+695%,na,+614%,na,+633%,na,+633%,na,+580%]
M-0028svc=ad metric=jvm.thread.count.rate base=0.0133 delta/30s=[na,-100%,na,-100%,na,-100%,na,-100%,na,+650%]
M-0029svc=frontend metric=http.server.request.duration.p95 base=0.855 delta/30s=[na,+601%,na,+469%,na,+470%,na,+470%,na,+467%]
M-0030svc=load-generator metric=otel.sdk.span.live base=0.6 delta/30s=[+67%,na,+233%,na,+233%,na,+67%,na,+567%,na]
M-0031svc=load-generator metric=http.client.duration.p95 base=1.12e+03 delta/30s=[+535%,na,+335%,na,+334%,na,+335%,na,+332%,na]
M-0032svc=unknown metric=system.disk.pending_operations base=3.8 delta/30s=[na,-74%,na,+453%,na,-100%,na,-100%,na,-74%]
M-0033svc=unknown metric=system.processes.count.rate base=0.14 delta/30s=[na,-29%,na,+424%,na,+400%,na,-76%,na,-76%]
M-0034svc=ad metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,-100%,na,-100%,na,+400%]
M-0035svc=fraud-detection metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,+400%,na,-100%,na,-100%]
M-0036svc=checkout metric=traces.span.metrics.duration.p50 base=773.0 delta/30s=[na,+385%,na,+341%,na,+353%,na,+353%,na,+317%]
856 lower-ranked lines omitted

traces: slowest and errored traces after fault onset, critical path (! = error span)

T-0001trace=ef59537e dur=7229ms ERROR path: load-generator:user_checkout_multi 7229ms! > frontend-proxy:POST 7167ms! > frontend:POST /api/checkout 7167ms! > checkout:oteldemo.CheckoutService/PlaceOrder 7155ms!
T-0002trace=3eeca890 dur=4890ms ok path: frontend-proxy:POST 4879ms > flagd:flagd.evaluation.v1.Service/EventStream 4888ms
T-0003trace=946cc41f dur=5221ms ERROR path: load-generator:user_checkout_single 5221ms! > frontend-proxy:POST 4701ms! > frontend:POST /api/checkout 4701ms! > checkout:oteldemo.CheckoutService/PlaceOrder 4698ms!
T-0004trace=a83ed6e8 dur=4884ms ok path: frontend-proxy:POST 4882ms > flagd:flagd.evaluation.v1.Service/EventStream 4879ms
T-0005trace=7db6b6a0 dur=4608ms ERROR path: load-generator:user_checkout_multi 4608ms! > frontend-proxy:POST 3566ms! > frontend:POST /api/checkout 3566ms! > checkout:oteldemo.CheckoutService/PlaceOrder 3563ms!
T-0006trace=98c0e5a7 dur=4848ms ok path: frontend-proxy:POST 4844ms > flagd:flagd.evaluation.v1.Service/EventStream 4844ms
T-0007trace=c52b20a3 dur=4566ms ERROR path: load-generator:user_checkout_multi 4566ms! > frontend-proxy:POST 3176ms! > frontend:POST /api/checkout 3176ms! > checkout:oteldemo.CheckoutService/PlaceOrder 3173ms!
T-0008trace=50f43b41 dur=4355ms ok path: frontend-proxy:POST 4354ms > flagd:flagd.evaluation.v1.Service/EventStream 4346ms
T-0009trace=607f3350 dur=4538ms ERROR path: load-generator:user_checkout_single 4538ms! > frontend-proxy:POST 4010ms! > frontend:POST /api/checkout 4011ms! > checkout:oteldemo.CheckoutService/PlaceOrder 4008ms!
T-0010trace=091cd7c7 dur=4308ms ok path: frontend-proxy:POST 4302ms > flagd:flagd.evaluation.v1.Service/EventStream 4304ms
T-0011trace=27d444b5 dur=4077ms ERROR path: load-generator:user_checkout_multi 4077ms! > frontend-proxy:POST 4012ms! > frontend:POST /api/checkout 4011ms! > checkout:oteldemo.CheckoutService/PlaceOrder 4009ms!
T-0012trace=0ce69618 dur=4296ms ok path: frontend-proxy:POST 4293ms > flagd:flagd.evaluation.v1.Service/EventStream 4293ms
T-0013trace=8184aea1 dur=3854ms ERROR path: load-generator:user_checkout_multi 3854ms! > frontend-proxy:POST 3796ms! > frontend:POST /api/checkout 3795ms! > checkout:oteldemo.CheckoutService/PlaceOrder 3793ms!
T-0014trace=0f067c18 dur=4281ms ok path: frontend-proxy:POST 4277ms > flagd:flagd.evaluation.v1.Service/EventStream 4277ms
T-0015trace=2854f973 dur=3779ms ERROR path: load-generator:user_checkout_single 3779ms! > frontend-proxy:POST 3741ms! > frontend:POST /api/checkout 3740ms! > checkout:oteldemo.CheckoutService/PlaceOrder 3737ms!
T-0016trace=893c484b dur=4197ms ok path: frontend-proxy:POST 4195ms > flagd:flagd.evaluation.v1.Service/EventStream 4196ms
T-0017trace=005b540c dur=3754ms ERROR path: load-generator:user_checkout_single 3754ms! > frontend-proxy:POST 3707ms! > frontend:POST /api/checkout 3706ms! > checkout:oteldemo.CheckoutService/PlaceOrder 3703ms!
T-0018trace=0cc263f7 dur=4142ms ok path: frontend-proxy:POST 3693ms > flagd:flagd.evaluation.v1.Service/EventStream 4118ms
T-0019trace=9eef2fa9 dur=3752ms ERROR path: load-generator:user_checkout_multi 3752ms! > frontend-proxy:POST 3657ms! > frontend:POST /api/checkout 3657ms! > checkout:oteldemo.CheckoutService/PlaceOrder 3655ms!
T-0020trace=de5ed0e6 dur=4076ms ok path: load-generator:browser_add_to_cart 4076ms
T-0021trace=ce1ed9bb dur=3686ms ERROR path: load-generator:user_checkout_single 3686ms! > frontend-proxy:POST 3215ms! > frontend:POST /api/checkout 3215ms! > checkout:oteldemo.CheckoutService/PlaceOrder 3213ms!
T-0022trace=021ae598 dur=4072ms ok path: load-generator:browser_add_to_cart 4072ms
T-0023trace=53e41bd2 dur=3674ms ERROR path: load-generator:user_checkout_multi 3674ms! > frontend-proxy:POST 3197ms! > frontend:POST /api/checkout 3197ms! > checkout:oteldemo.CheckoutService/PlaceOrder 3195ms!
T-0024trace=53bdd6d7 dur=4033ms ok path: frontend-proxy:POST 4026ms > flagd:flagd.evaluation.v1.Service/EventStream 4021ms
T-0025trace=8068525e dur=3620ms ERROR path: load-generator:user_checkout_single 3620ms! > frontend-proxy:POST 3602ms! > frontend:POST /api/checkout 3602ms! > checkout:oteldemo.CheckoutService/PlaceOrder 3598ms!
T-0026trace=1c313345 dur=3982ms ok path: frontend-proxy:POST 3981ms > flagd:flagd.evaluation.v1.Service/EventStream 3972ms
T-0027trace=b8d59588 dur=3536ms ERROR path: load-generator:user_checkout_multi 3536ms! > frontend-proxy:POST 3504ms! > frontend:POST /api/checkout 3504ms! > checkout:oteldemo.CheckoutService/PlaceOrder 3501ms!
T-0028trace=84b314de dur=3971ms ok path: frontend-proxy:POST 3968ms > flagd:flagd.evaluation.v1.Service/EventStream 3970ms
T-0029trace=7d63f90a dur=3520ms ERROR path: load-generator:user_checkout_single 3520ms! > frontend-proxy:POST 3493ms! > frontend:POST /api/checkout 3492ms! > checkout:oteldemo.CheckoutService/PlaceOrder 3490ms!
T-0030trace=cf7ee143 dur=3913ms ok path: frontend-proxy:POST 3904ms > flagd:flagd.evaluation.v1.Service/EventStream 3909ms
T-0031trace=8dd6a00e dur=3427ms ERROR path: load-generator:user_checkout_multi 3427ms! > frontend-proxy:POST 3349ms! > frontend:POST /api/checkout 3348ms! > checkout:oteldemo.CheckoutService/PlaceOrder 3346ms!
9 lower-ranked lines omitted

logs: log templates per service, rate before vs after fault onset, with examples

L-0001svc=accounting sev=INFORMATION base=14.2/min fault=0.0/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0002svc=cart sev=INFORMATION base=14.2/min fault=0.0/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0003svc=checkout sev=INFO base=14.2/min fault=0.0/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 244.246µs', 'Successful to write message. offset: 0, duration: 148.58µs']
L-0004svc=checkout sev=INFO base=14.2/min fault=0.0/min tpl="order placed" ex=['order placed', 'order placed']
L-0005svc=checkout sev=INFO base=14.2/min fault=0.0/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0006svc=checkout sev=INFO base=14.2/min fault=0.0/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0007svc=email sev=INFO base=14.2/min fault=0.0/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0008svc=fraud-detection sev=INFO base=14.2/min fault=0.0/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: 1befce98-dbc7-11f1-82f3-d279f8fa1f6d, and updated total count to: 10300', 'Consumed record with orderId: 1daefac4-dbc7-11f1-82f3-d279f8fa1f6d, and updated total count to: 10301']
L-0009svc=frontend sev=INFO base=14.2/min fault=0.0/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0010svc=payment sev=INFO base=14.2/min fault=0.0/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0011svc=payment sev=INFO base=14.2/min fault=0.0/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0012svc=shipping sev=INFO base=14.2/min fault=0.0/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0013svc=checkout sev=INFO base=14.0/min fault=0.0/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0014svc=frontend sev=ERROR base=0.0/min fault=12.6/min tpl="Checkout failed to place order" ex=['Checkout failed to place order', 'Checkout failed to place order']
L-0015svc=shipping sev=INFO base=0.8/min fault=0.0/min tpl="" ex=['', '']
L-0016svc=load-generator sev=INFO base=7.6/min fault=14.0/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0017svc=load-generator sev=INFO base=34.8/min fault=28.4/min tpl="Product added to cart successfully" ex=['Product added to cart successfully', 'Product added to cart successfully']
L-0018svc=ad sev=INFO base=2.8/min fault=3.6/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0019svc=load-generator sev=INFO base=26.8/min fault=32.4/min tpl="Currency changed to CHF" ex=['Currency changed to CHF', 'Currency changed to CHF']
L-0020svc=checkout sev=INFO base=0.2/min fault=0.0/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0021svc=checkout sev=WARN base=0.2/min fault=0.0/min tpl="failed to send order confirmation: failed POST to email service: Post "http://email:<NUM>/send_order_confirmation": context deadline exceed…" ex=['failed to send order confirmation: failed POST to email service: Post "http://email:6060/send_order_confirmation": context deadline exceeded']
L-0022svc=quote sev=INFO base=31.6/min fault=26.6/min tpl="Calculated quote" ex=['Calculated quote', 'Calculated quote']
L-0023svc=shipping sev=INFO base=31.6/min fault=26.6/min tpl="Requesting quote" ex=['Requesting quote', 'Requesting quote']
L-0024svc=shipping sev=INFO base=31.6/min fault=26.6/min tpl="Sending Quote" ex=['Sending Quote', 'Sending Quote']
L-0025svc=load-generator sev=INFO base=41.6/min fault=35.2/min tpl="User viewing cart" ex=['User viewing cart', 'User viewing cart']
L-0026svc=load-generator sev=INFO base=14.4/min fault=12.4/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user d3022aaa-dbc7-11f1-8306-d6b650b5399b', 'Checkout completed for user d3022aaa-dbc7-11f1-8306-d6b650b5399b']
L-0027svc=kafka sev=INFO base=2.0/min fault=2.4/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 27228 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 27267 breach: LogSegment(baseOff…']
L-0028svc=load-generator sev=INFO base=75.6/min fault=85.6/min tpl="User getting <*> for <*> <*>" ex=['User getting recommendations for product: 9SIQT8TOJO', 'User getting recommendations for product: 9SIQT8TOJO']
L-0029svc=cart sev=INFORMATION base=58.6/min fault=52.0/min tpl="AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}" ex=['AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}', 'AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}']
L-0030svc=checkout sev=INFO base=14.2/min fault=12.6/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
33 lower-ranked lines omitted
Loading runs in your browser…