Rootline v0.1 pilot

← All incidents

hwWifiOutage-r1

What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)

Service graph during the window

Evidence and answers

4950 prompt tokens

metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean

M-0001svc=ad metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:0.00952,na,+0%]
M-0002svc=ad metric=jvm.thread.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,new:0.1,na,new:0.1,na,+0%]
M-0003svc=ad metric=rpc.client.duration.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%]
M-0004svc=ad metric=rpc.client.request.size.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%]
M-0005svc=ad metric=rpc.client.response.size.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%]
M-0006svc=ad metric=traces.span.metrics.duration.p50 base=1 delta/30s=[na,-0%,na,-0%,na,-0%,na,+1493802%,na,-0%]
M-0007svc=ad metric=traces.span.metrics.duration.p95 base=1.91 delta/30s=[na,-0%,na,-0%,na,-0%,na,+786164%,na,-0%]
M-0008svc=ad metric=traces.span.metrics.duration.p99 base=2.18 delta/30s=[na,-9%,na,-9%,na,-9%,na,+686713%,na,-9%]
M-0009svc=cart metric=dotnet.timer.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%]
M-0010svc=checkout metric=error_ratio base=0 delta/30s=[na,new:0.0426,na,+0%,na,+0%,na,+0%,na,+0%]
M-0011svc=checkout metric=otel.sdk.span.live base=0 delta/30s=[na,new:4,na,+0%,na,+0%,na,+0%,na,+0%]
M-0012svc=edge-relay metric=relay.connection.duration.count.rate base=0 delta/30s=[+0%,+0%,+0%,+0%,+0%,new:0.0333,+0%,+0%,+0%,+0%]
M-0013svc=flagd metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:0.000926,na,+0%]
M-0014svc=fraud-detection metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:0.0435,na,+0%]
M-0015svc=fraud-detection metric=rpc.client.duration.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%]
M-0016svc=fraud-detection metric=rpc.client.request.size.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%]
M-0017svc=fraud-detection metric=rpc.client.response.size.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%]
M-0018svc=frontend metric=error_ratio base=0 delta/30s=[na,new:0.0039,na,+0%,na,+0%,na,+0%,na,+0%]
M-0019svc=frontend metric=http.server.request.duration.p50 base=0.00613 delta/30s=[na,+163019%,na,+5%,na,-16%,na,-3%,na,-34%]
M-0020svc=frontend-proxy metric=error_ratio base=0 delta/30s=[na,new:0.00166,na,+0%,na,+0%,na,+0%,na,+0%]
M-0021svc=load-generator metric=error_ratio base=0 delta/30s=[na,new:0.00828,na,+0%,na,+0%,na,+0%,na,+0%]
M-0022svc=load-generator metric=http.client.duration.p50 base=8.32 delta/30s=[na,+120131%,na,-1%,na,-3%,na,-8%,na,-9%]
M-0023svc=recommendation metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[new:2,na,+0%,na,new:2,na,new:1,na,na,+0%]
M-0024svc=recommendation metric=process.runtime.cpython.cpu.utilization base=0 delta/30s=[+0%,na,+0%,na,new:6.83,na,+0%,na,na,+0%]
M-0025svc=checkout metric=rpc.client.call.duration.p50 base=0.0178 delta/30s=[na,+56155%,na,+67%,na,+3%,na,-2%,na,+5%]
M-0026svc=fraud-detection metric=traces.span.metrics.duration.p50 base=44.7 delta/30s=[na,+21%,na,-3%,na,-3%,na,+33491%,na,-17%]
M-0027svc=fraud-detection metric=traces.span.metrics.duration.p95 base=96.0 delta/30s=[na,-1%,na,-2%,na,+46%,na,+15529%,na,-4%]
M-0028svc=checkout metric=traces.span.metrics.duration.p50 base=82.3 delta/30s=[na,+15085%,na,-3%,na,-46%,na,+70%,na,-44%]
M-0029svc=checkout metric=rpc.server.call.duration.p50 base=0.0694 delta/30s=[na,+14314%,na,-7%,na,-28%,na,+26%,na,-30%]
M-0030svc=checkout metric=rpc.client.call.duration.p95 base=0.0729 delta/30s=[na,+13614%,na,+11814%,na,-42%,na,-19%,na,-26%]
M-0031svc=fraud-detection metric=traces.span.metrics.duration.p99 base=115.2 delta/30s=[na,-14%,na,-14%,na,+63%,na,+12921%,na,-15%]
M-0032svc=frontend metric=traces.span.metrics.duration.p50 base=112.5 delta/30s=[na,+11011%,na,+167%,na,-56%,na,+16%,na,-22%]
M-0033svc=checkout metric=rpc.client.call.duration.p99 base=0.0946 delta/30s=[na,+10473%,na,+10195%,na,-49%,na,-24%,na,-25%]
M-0034svc=frontend metric=http.server.request.duration.p95 base=0.164 delta/30s=[na,+5995%,na,-41%,na,+32%,na,-5%,na,+4%]
M-0035svc=load-generator metric=http.client.duration.p95 base=218.8 delta/30s=[na,+4470%,na,-41%,na,+13%,na,-30%,na,+69%]
M-0036svc=frontend-proxy metric=traces.span.metrics.duration.p50 base=415.1 delta/30s=[na,+2711%,na,-99%,na,+394%,na,-99%,na,-99%]
M-0037svc=checkout metric=traces.span.metrics.duration.p95 base=626.0 delta/30s=[na,+2296%,na,+1258%,na,-11%,na,+92%,na,-47%]
864 lower-ranked lines omitted

traces: slowest and errored traces after fault onset, critical path (! = error span)

T-0001trace=cd8244be dur=20704ms ERROR path: load-generator:user_checkout_single 15014ms! > frontend-proxy:POST 14997ms! > frontend:POST 14999ms! > checkout:oteldemo.CheckoutService/PlaceOrder 19180ms!
T-0002trace=37abbc52 dur=44570ms ok path: accounting:order-consumed 44570ms
T-0003trace=5069e966 dur=15785ms ERROR path: load-generator:user_checkout_multi 14393ms! > frontend-proxy:POST 14332ms! > frontend:POST /api/checkout 14333ms! > checkout:oteldemo.CheckoutService/PlaceOrder 14327ms!
T-0004trace=0b545817 dur=17153ms ok path: accounting:order-consumed 17153ms
T-0005trace=75dff034 dur=12744ms ERROR path: load-generator:user_checkout_multi 11253ms! > frontend-proxy:POST 11222ms! > frontend:POST /api/checkout 11222ms! > checkout:oteldemo.CheckoutService/PlaceOrder 11218ms!
T-0006trace=4ce6336d dur=17029ms ok path: accounting:order-consumed 17029ms
T-0007trace=b98d4129 dur=2729ms ERROR path: load-generator:user_checkout_single 1289ms! > frontend-proxy:POST 1274ms! > frontend:POST /api/checkout 1273ms! > checkout:oteldemo.CheckoutService/PlaceOrder 1270ms!
T-0008trace=fb821f10 dur=14677ms ok path: accounting:order-consumed 14677ms
T-0009trace=af33f49e dur=14149ms ok path: accounting:order-consumed 14149ms
T-0010trace=182d4623 dur=13275ms ok path: accounting:order-consumed 13275ms
T-0011trace=6ea9a64d dur=10406ms ok path: accounting:order-consumed 10406ms
T-0012trace=60ce9037 dur=10277ms ok path: accounting:order-consumed 10277ms
T-0013trace=963f5c84 dur=9920ms ok path: load-generator:user_checkout_multi 8714ms > frontend-proxy:POST 8657ms > frontend:POST /api/checkout 8657ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 7ms
T-0014trace=c6909514 dur=9676ms ok path: load-generator:user_checkout_single 8614ms > frontend-proxy:POST 8594ms > frontend:POST /api/checkout 8595ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 1ms
T-0015trace=9e770427 dur=9527ms ok path: accounting:order-consumed 9527ms
T-0016trace=54ea4533 dur=9129ms ok path: accounting:order-consumed 9129ms
T-0017trace=2ee506a4 dur=8733ms ok path: load-generator:user_checkout_multi 7263ms > frontend-proxy:POST 7225ms > frontend:POST /api/checkout 7226ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 1ms
T-0018trace=b755e2fd dur=8210ms ok path: accounting:order-consumed 8210ms
T-0019trace=32175e3a dur=7461ms ok path: accounting:order-consumed 7461ms
T-0020trace=a1229749 dur=7378ms ok path: accounting:order-consumed 7378ms
T-0021trace=e6275f9b dur=7229ms ok path: accounting:order-consumed 7229ms
T-0022trace=d9ec0105 dur=5640ms ok path: accounting:order-consumed 5640ms
T-0023trace=37895bb0 dur=5494ms ok path: accounting:order-consumed 5494ms
T-0024trace=4893b8e5 dur=5148ms ok path: accounting:order-consumed 5148ms

logs: log templates per service, rate before vs after fault onset, with examples

L-0001svc=frontend sev=ERROR base=0.0/min fault=0.8/min tpl="Checkout failed to place order" ex=['Checkout failed to place order', 'Checkout failed to place order']
L-0002svc=load-generator sev=INFO base=16.4/min fault=11.6/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user b9d95c62-dc0b-11f1-8306-d6b650b5399b', 'Checkout completed for user b9d95c62-dc0b-11f1-8306-d6b650b5399b']
L-0003svc=load-generator sev=INFO base=47.2/min fault=36.8/min tpl="User viewing cart" ex=['User viewing cart', 'User viewing cart']
L-0004svc=checkout sev=INFO base=0.2/min fault=0.0/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0005svc=accounting sev=INFORMATION base=15.2/min fault=13.0/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0006svc=cart sev=INFORMATION base=15.2/min fault=13.0/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0007svc=checkout sev=INFO base=15.2/min fault=13.0/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 185.077µs', 'Successful to write message. offset: 0, duration: 165.427µs']
L-0008svc=checkout sev=INFO base=15.2/min fault=13.0/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0009svc=checkout sev=INFO base=15.2/min fault=13.0/min tpl="order placed" ex=['order placed', 'order placed']
L-0010svc=checkout sev=INFO base=15.2/min fault=13.0/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0011svc=checkout sev=INFO base=15.2/min fault=13.0/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0012svc=email sev=INFO base=15.2/min fault=13.0/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0013svc=fraud-detection sev=INFO base=15.2/min fault=13.0/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: b64263f8-dc0b-11f1-9d0e-3288f2b1222e, and updated total count to: 23788', 'Consumed record with orderId: cbbcee42-dc0b-11f1-9d0e-3288f2b1222e, and updated total count to: 23789']
L-0014svc=frontend sev=INFO base=15.2/min fault=13.0/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0015svc=payment sev=INFO base=15.2/min fault=13.0/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0016svc=payment sev=INFO base=15.2/min fault=13.0/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0017svc=shipping sev=INFO base=15.2/min fault=13.0/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0018svc=load-generator sev=INFO base=12.0/min fault=14.0/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0019svc=load-generator sev=INFO base=14.0/min fault=16.0/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user b63ab966-dc0b-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user b63ab966-dc0b-11f1-8306-d6b650b5399b']
L-0020svc=load-generator sev=INFO base=73.6/min fault=83.2/min tpl="User getting <*> for <*> <*>" ex=['User getting ads for category: binoculars', 'User getting ads for category: binoculars']
L-0021svc=shipping sev=INFO base=2.6/min fault=2.2/min tpl="" ex=['', '']
L-0022svc=load-generator sev=INFO base=140.4/min fault=125.2/min tpl="User browsing product: <*>" ex=['User browsing product: OLJCESPC7Z', 'User browsing product: OLJCESPC7Z']
L-0023svc=kafka sev=INFO base=1.0/min fault=0.8/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=85434, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=85584, siz…']
L-0024svc=kafka sev=INFO base=1.0/min fault=0.8/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 85650 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 85767 breach: LogSegment(baseOff…']
L-0025svc=load-generator sev=INFO base=28.4/min fault=31.6/min tpl="Currency changed to CHF" ex=['Currency changed to CHF', 'Currency changed to CHF']
L-0026svc=quote sev=INFO base=31.4/min fault=28.6/min tpl="Calculated quote" ex=['Calculated quote', 'Calculated quote']
L-0027svc=shipping sev=INFO base=31.4/min fault=28.6/min tpl="Requesting quote" ex=['Requesting quote', 'Requesting quote']
L-0028svc=shipping sev=INFO base=31.4/min fault=28.6/min tpl="Sending Quote" ex=['Sending Quote', 'Sending Quote']
L-0029svc=checkout sev=INFO base=15.2/min fault=13.8/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0030svc=kafka sev=INFO base=8.4/min fault=7.6/min tpl="Deleted <*> index <*>" ex=['Deleted offset index /tmp/kafka-logs/__cluster_metadata-0/00000000000000085374.index.deleted.', 'Deleted time index /tmp/kafka-logs/__cluster_metadata-0/00000000000000085374.timeindex.deleted.']
31 lower-ranked lines omitted
Loading runs in your browser…