Rootline v0.1 pilot

← All incidents

paymentUnreachable-r2

What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)

Service graph during the window

Evidence and answers

5710 prompt tokens

metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean

M-0001svc=accounting metric=dotnet.thread_pool.thread.count.rate base=0 delta/30s=[+0%,na,new:0.0667,na,+0%,na,+0%,na,+0%,na]
M-0002svc=accounting metric=http.client.connection.duration.count.rate base=0 delta/30s=[+0%,na,new:0.0667,na,+0%,na,+0%,na,+0%,na]
M-0003svc=accounting metric=process.thread.count.rate base=0 delta/30s=[+0%,na,new:0.0667,na,+0%,na,+0%,na,+0%,na]
M-0004svc=ad metric=error_ratio base=0 delta/30s=[+0%,na,new:0.0164,na,+0%,na,+0%,na,+0%,na]
M-0005svc=ad metric=jvm.gc.duration.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0006svc=ad metric=jvm.thread.count.rate base=0 delta/30s=[new:0.0333,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0007svc=ad metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0008svc=ad metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0009svc=ad metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0010svc=ad metric=traces.span.metrics.duration.p50 base=1.01 delta/30s=[-1%,na,+1492247%,na,+2%,na,+2%,na,-1%,na]
M-0011svc=ad metric=traces.span.metrics.duration.p95 base=1.91 delta/30s=[-1%,na,+785346%,na,+2%,na,+2%,na,-1%,na]
M-0012svc=ad metric=traces.span.metrics.duration.p99 base=2.22 delta/30s=[-11%,na,+674360%,na,+125%,na,+36%,na,-11%,na]
M-0013svc=checkout metric=error_ratio base=0 delta/30s=[new:0.0276,na,new:0.197,na,new:0.218,na,new:0.181,na,new:0.204,na]
M-0014svc=checkout metric=otel.sdk.span.live base=0 delta/30s=[+0%,na,new:2,na,+0%,na,new:1,na,new:3,na]
M-0015svc=fraud-detection metric=error_ratio base=0 delta/30s=[+0%,na,new:1,na,+0%,na,+0%,na,+0%,na]
M-0016svc=fraud-detection metric=jvm.thread.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0017svc=fraud-detection metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0018svc=fraud-detection metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0019svc=fraud-detection metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0020svc=frontend metric=error_ratio base=0 delta/30s=[new:0.00199,na,new:0.0153,na,new:0.0185,na,new:0.0162,na,new:0.0212,na]
M-0021svc=frontend-proxy metric=error_ratio base=0 delta/30s=[new:0.00107,na,new:0.00724,na,new:0.00917,na,new:0.00749,na,new:0.0107,na]
M-0022svc=kafka metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[+0%,na,+0%,na,new:1,na,+0%,na,+0%,na]
M-0023svc=load-generator metric=error_ratio base=0 delta/30s=[new:0.00412,na,new:0.0277,na,new:0.039,na,new:0.0304,na,new:0.0405,na]
M-0024svc=frontend metric=http.server.request.duration.p50 base=0.00665 delta/30s=[+56325%,na,+56325%,na,+53638%,na,+56325%,na,+53503%,na]
M-0025svc=load-generator metric=http.client.duration.p50 base=8.57 delta/30s=[-7%,na,+43648%,na,+42790%,na,+42606%,na,+42190%,na]
M-0026svc=checkout metric=go.goroutine.count.rate base=0.0133 delta/30s=[+2400%,na,+16400%,na,+18900%,na,+15650%,na,+25900%,na]
M-0027svc=fraud-detection metric=traces.span.metrics.duration.p50 base=59.6 delta/30s=[+1%,na,+25072%,na,na,na,na,na,na,na]
M-0028svc=fraud-detection metric=traces.span.metrics.duration.p95 base=104.6 delta/30s=[-8%,na,+14243%,na,na,na,na,na,na,na]
M-0029svc=fraud-detection metric=traces.span.metrics.duration.p99 base=116.9 delta/30s=[-15%,na,+12730%,na,na,na,na,na,na,na]
M-0030svc=checkout metric=traces.span.metrics.duration.p50 base=87.9 delta/30s=[+3882%,na,+3882%,na,+3787%,na,+3760%,na,+3712%,na]
M-0031svc=checkout metric=rpc.server.call.duration.p50 base=0.108 delta/30s=[+3368%,na,+3368%,na,+3304%,na,+3286%,na,+3253%,na]
M-0032svc=kafka metric=kafka.purgatory.size base=26.4 delta/30s=[+59%,na,+627%,na,+1533%,na,+2442%,na,+3351%,na]
M-0033svc=frontend metric=traces.span.metrics.duration.p50 base=101.7 delta/30s=[+3343%,na,+3343%,na,+3261%,na,+3244%,na,+3187%,na]
M-0034svc=checkout metric=http.client.request.duration.p50 base=0.00623 delta/30s=[-4%,na,+1745%,na,+20%,na,+569%,na,+205%,na]
M-0035svc=load-generator metric=http.client.duration.p95 base=275.2 delta/30s=[+111%,na,+1671%,na,+1668%,na,+1668%,na,+1667%,na]
860 lower-ranked lines omitted

traces: slowest and errored traces after fault onset, critical path (! = error span)

T-0001trace=8b76e999 dur=4915ms ERROR path: load-generator:user_checkout_single 4915ms! > frontend-proxy:POST 4898ms! > frontend:POST /api/checkout 4898ms! > checkout:oteldemo.CheckoutService/PlaceOrder 4895ms!
T-0002trace=e1c25317 dur=6145ms ok path: frontend-proxy:POST 6145ms > flagd:flagd.evaluation.v1.Service/EventStream 6144ms
T-0003trace=6e041990 dur=4860ms ERROR path: load-generator:user_checkout_single 4860ms! > frontend-proxy:POST 4845ms! > frontend:POST /api/checkout 4845ms! > checkout:oteldemo.CheckoutService/PlaceOrder 4842ms!
T-0004trace=243b7d58 dur=5444ms ok path: frontend-proxy:POST 5443ms > flagd:flagd.evaluation.v1.Service/EventStream 5443ms
T-0005trace=459b20b0 dur=4322ms ERROR path: load-generator:user_checkout_multi 4322ms! > frontend-proxy:POST 4224ms! > frontend:POST /api/checkout 4222ms! > checkout:oteldemo.CheckoutService/PlaceOrder 4219ms!
T-0006trace=f8cee378 dur=5136ms ok path: load-generator:browser_add_to_cart 5136ms
T-0007trace=254780de dur=4274ms ERROR path: load-generator:user_checkout_single 4274ms! > frontend-proxy:POST 4252ms! > frontend:POST /api/checkout 4252ms! > checkout:oteldemo.CheckoutService/PlaceOrder 4249ms!
T-0008trace=05287ff3 dur=5113ms ok path: load-generator:browser_add_to_cart 5113ms
T-0009trace=e334da59 dur=4071ms ERROR path: load-generator:user_checkout_single 4071ms! > frontend-proxy:POST 3987ms! > frontend:POST /api/checkout 3988ms! > checkout:oteldemo.CheckoutService/PlaceOrder 3983ms!
T-0010trace=25cc1ad4 dur=4815ms ok path: frontend-proxy:POST 4814ms > flagd:flagd.evaluation.v1.Service/EventStream 4814ms
T-0011trace=bda8262f dur=4064ms ERROR path: load-generator:user_checkout_multi 4064ms! > frontend-proxy:POST 4018ms! > frontend:POST /api/checkout 4018ms! > checkout:oteldemo.CheckoutService/PlaceOrder 4014ms!
T-0012trace=2407c91d dur=4812ms ok path: frontend-proxy:POST 4811ms > flagd:flagd.evaluation.v1.Service/EventStream 4811ms
T-0013trace=618b1c48 dur=3953ms ERROR path: load-generator:user_checkout_multi 3953ms! > frontend-proxy:POST 3889ms! > frontend:POST /api/checkout 3890ms! > checkout:oteldemo.CheckoutService/PlaceOrder 3886ms!
T-0014trace=3eb86597 dur=4639ms ok path: frontend-proxy:POST 4638ms > flagd:flagd.evaluation.v1.Service/EventStream 4638ms
T-0015trace=37aac248 dur=3889ms ERROR path: load-generator:user_checkout_multi 3889ms! > frontend-proxy:POST 3794ms! > frontend:POST /api/checkout 3793ms! > checkout:oteldemo.CheckoutService/PlaceOrder 3790ms!
T-0016trace=53019298 dur=4627ms ok path: load-generator:browser_add_to_cart 4627ms
T-0017trace=a3e9124f dur=3862ms ERROR path: load-generator:user_checkout_multi 3862ms! > frontend-proxy:POST 3794ms! > frontend:POST /api/checkout 3793ms! > checkout:oteldemo.CheckoutService/PlaceOrder 3791ms!
T-0018trace=21f2c48a dur=4357ms ok path: frontend-proxy:POST 4356ms > flagd:flagd.evaluation.v1.Service/EventStream 4356ms
T-0019trace=9f7ade8f dur=3802ms ERROR path: load-generator:user_checkout_single 3802ms! > frontend-proxy:POST 3775ms! > frontend:POST /api/checkout 3774ms! > checkout:oteldemo.CheckoutService/PlaceOrder 3772ms!
T-0020trace=34109e50 dur=4306ms ok path: frontend-proxy:POST 4305ms > flagd:flagd.evaluation.v1.Service/EventStream 4305ms
T-0021trace=e3dd7f10 dur=3744ms ERROR path: load-generator:user_checkout_multi 3744ms! > frontend-proxy:POST 3691ms! > frontend:POST /api/checkout 3690ms! > checkout:oteldemo.CheckoutService/PlaceOrder 3688ms!
T-0022trace=b1ca37f7 dur=4229ms ok path: frontend-proxy:POST 4229ms > flagd:flagd.evaluation.v1.Service/EventStream 4228ms
T-0023trace=58a3e8b9 dur=3742ms ERROR path: load-generator:user_checkout_single 3742ms! > frontend-proxy:POST 3718ms! > frontend:POST /api/checkout 3719ms! > checkout:oteldemo.CheckoutService/PlaceOrder 3715ms!
T-0024trace=b04770ae dur=4138ms ok path: frontend-proxy:POST 4138ms > flagd:flagd.evaluation.v1.Service/EventStream 4138ms
T-0025trace=e21ac78b dur=3671ms ERROR path: load-generator:user_checkout_multi 3671ms! > frontend-proxy:POST 3632ms! > frontend:POST /api/checkout 3632ms! > checkout:oteldemo.CheckoutService/PlaceOrder 3629ms!
T-0026trace=22e6e05e dur=4033ms ok path: frontend-proxy:POST 4033ms > flagd:flagd.evaluation.v1.Service/EventStream 4033ms
T-0027trace=2862a20d dur=3649ms ERROR path: load-generator:user_checkout_multi 3649ms! > frontend-proxy:POST 3570ms! > frontend:POST /api/checkout 3569ms! > checkout:oteldemo.CheckoutService/PlaceOrder 3566ms!
T-0028trace=7c1ea563 dur=4021ms ok path: frontend-proxy:POST 4021ms > flagd:flagd.evaluation.v1.Service/EventStream 4021ms
T-0029trace=c9554c6a dur=3609ms ERROR path: load-generator:user_checkout_single 3609ms! > frontend-proxy:POST 3593ms! > frontend:POST /api/checkout 3593ms! > checkout:oteldemo.CheckoutService/PlaceOrder 3590ms!
T-0030trace=9d42f921 dur=4011ms ok path: frontend-proxy:POST 4011ms > flagd:flagd.evaluation.v1.Service/EventStream 4011ms
T-0031trace=f08001cf dur=3559ms ERROR path: load-generator:user_checkout_multi 3559ms! > frontend-proxy:POST 3405ms! > frontend:POST /api/checkout 3400ms! > checkout:oteldemo.CheckoutService/PlaceOrder 3397ms!
T-0032trace=768cd989 dur=4001ms ok path: frontend-proxy:POST 3999ms > flagd:flagd.evaluation.v1.Service/EventStream 4001ms
8 lower-ranked lines omitted

logs: log templates per service, rate before vs after fault onset, with examples

L-0001svc=frontend sev=ERROR base=0.0/min fault=16.2/min tpl="Checkout failed to place order" ex=['Checkout failed to place order', 'Checkout failed to place order']
L-0002svc=accounting sev=INFORMATION base=11.8/min fault=0.0/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0003svc=cart sev=INFORMATION base=11.8/min fault=0.0/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0004svc=checkout sev=INFO base=11.8/min fault=0.0/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 153.218µs', 'Successful to write message. offset: 0, duration: 160.658µs']
L-0005svc=checkout sev=INFO base=11.8/min fault=0.0/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0006svc=checkout sev=INFO base=11.8/min fault=0.0/min tpl="order placed" ex=['order placed', 'order placed']
L-0007svc=checkout sev=INFO base=11.8/min fault=0.0/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0008svc=checkout sev=INFO base=11.8/min fault=0.0/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0009svc=email sev=INFO base=11.8/min fault=0.0/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0010svc=fraud-detection sev=INFO base=11.8/min fault=0.0/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: e7299fc2-dbf1-11f1-82f3-d279f8fa1f6d, and updated total count to: 21450', 'Consumed record with orderId: ed2108f6-dbf1-11f1-82f3-d279f8fa1f6d, and updated total count to: 21451']
L-0011svc=frontend sev=INFO base=11.8/min fault=0.0/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0012svc=payment sev=INFO base=11.8/min fault=0.0/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0013svc=payment sev=INFO base=11.8/min fault=0.0/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0014svc=shipping sev=INFO base=11.8/min fault=0.0/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0015svc=shipping sev=INFO base=0.8/min fault=0.0/min tpl="" ex=['', '']
L-0016svc=ad sev=INFO base=3.8/min fault=1.8/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0017svc=load-generator sev=INFO base=10.0/min fault=17.2/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user 9b46233e-dbf2-11f1-8306-d6b650b5399b', 'Checkout completed for user 9b46233e-dbf2-11f1-8306-d6b650b5399b']
L-0018svc=checkout sev=INFO base=11.8/min fault=16.4/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0019svc=load-generator sev=INFO base=10.8/min fault=13.6/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0020svc=checkout sev=INFO base=0.0/min fault=0.2/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0021svc=load-generator sev=INFO base=76.0/min fault=90.8/min tpl="User <UUID> adding <NUM> of product <*> to cart" ex=['User 99a3eda4-dbf2-11f1-8306-d6b650b5399b adding 4 of product LS4PSXUNUM to cart', 'User 99a3eda4-dbf2-11f1-8306-d6b650b5399b adding 4 of product LS4PSXUNUM to cart']
L-0022svc=quote sev=INFO base=26.6/min fault=30.6/min tpl="Calculated quote" ex=['Calculated quote', 'Calculated quote']
L-0023svc=shipping sev=INFO base=26.6/min fault=30.6/min tpl="Requesting quote" ex=['Requesting quote', 'Requesting quote']
L-0024svc=shipping sev=INFO base=26.6/min fault=30.6/min tpl="Sending Quote" ex=['Sending Quote', 'Sending Quote']
L-0025svc=cart sev=INFORMATION base=52.8/min fault=59.6/min tpl="AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}" ex=['AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}', 'AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}']
L-0026svc=currency sev=INFO base=110.6/min fault=124.2/min tpl="conversion successful" ex=['conversion successful', 'conversion successful']
L-0027svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=63786, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=63906, siz…']
L-0028svc=load-generator sev=INFO base=13.6/min fault=15.2/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user 99a3eda4-dbf2-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user 99a3eda4-dbf2-11f1-8306-d6b650b5399b']
L-0029svc=kafka sev=INFO base=7.6/min fault=8.4/min tpl="Deleted <*> index <*>" ex=['Deleted offset index /tmp/kafka-logs/__cluster_metadata-0/00000000000000063756.index.deleted.', 'Deleted time index /tmp/kafka-logs/__cluster_metadata-0/00000000000000063756.timeindex.deleted.']
32 lower-ranked lines omitted
Loading runs in your browser…