Rootline v0.1 pilot

← All incidents

hwWifiOutage-r4

What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)

Service graph during the window

Evidence and answers

5082 prompt tokens

metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean

M-0001svc=ad metric=error_ratio base=0 delta/30s=[+0%,na,new:0.00862,na,+0%,na,+0%,na,+0%,na]
M-0002svc=ad metric=jvm.gc.duration.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0003svc=ad metric=jvm.thread.count.rate base=0 delta/30s=[+0%,na,new:0.0667,na,+0%,na,+0%,na,+0%,na]
M-0004svc=ad metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0005svc=ad metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0006svc=ad metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0007svc=ad metric=traces.span.metrics.duration.p50 base=1 delta/30s=[+2%,na,+1492791%,na,-0%,na,-0%,na,+2%,na]
M-0008svc=ad metric=traces.span.metrics.duration.p95 base=1.91 delta/30s=[+2%,na,+785632%,na,-0%,na,-0%,na,+2%,na]
M-0009svc=ad metric=traces.span.metrics.duration.p99 base=3.01 delta/30s=[+0%,na,+497908%,na,-34%,na,-34%,na,+68%,na]
M-0010svc=cart metric=dotnet.thread_pool.thread.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0011svc=cart metric=process.thread.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0012svc=checkout metric=error_ratio base=0 delta/30s=[+0%,na,new:0.0893,na,+0%,na,+0%,na,+0%,na]
M-0013svc=checkout metric=otel.sdk.span.live base=0 delta/30s=[new:4,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0014svc=edge-relay metric=relay.connection.duration.count.rate base=0 delta/30s=[+0%,+0%,+0%,+0%,+0%,new:0.0333,+0%,+0%,+0%,+0%]
M-0015svc=flagd metric=error_ratio base=0 delta/30s=[+0%,na,new:0.00101,na,+0%,na,+0%,na,+0%,na]
M-0016svc=fraud-detection metric=error_ratio base=0 delta/30s=[+0%,na,new:0.0909,na,+0%,na,+0%,na,+0%,na]
M-0017svc=fraud-detection metric=jvm.thread.count.rate base=0 delta/30s=[+0%,na,new:0.0667,na,+0%,na,+0%,na,+0%,na]
M-0018svc=fraud-detection metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:1,na,+0%,na]
M-0019svc=fraud-detection metric=otel.sdk.processor.span.queue.size base=0 delta/30s=[+0%,na,new:2,na,+0%,na,new:6,na,+0%,na]
M-0020svc=fraud-detection metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0021svc=fraud-detection metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0022svc=fraud-detection metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0023svc=frontend metric=error_ratio base=0 delta/30s=[+0%,na,new:0.00454,na,+0%,na,+0%,na,+0%,na]
M-0024svc=frontend metric=http.server.request.duration.p50 base=0.00586 delta/30s=[-2%,na,+149328%,na,+6%,na,-25%,na,-29%,na]
M-0025svc=frontend-proxy metric=error_ratio base=0 delta/30s=[+0%,na,new:0.00214,na,+0%,na,+0%,na,+0%,na]
M-0026svc=frontend-proxy metric=traces.span.metrics.duration.p50 base=5.98 delta/30s=[+0%,na,+195006%,na,+35019%,na,+34131%,na,-11%,na]
M-0027svc=load-generator metric=error_ratio base=0 delta/30s=[+0%,na,new:0.0139,na,+0%,na,+0%,na,+0%,na]
M-0028svc=load-generator metric=http.client.duration.p50 base=8.21 delta/30s=[+121687%,na,+121687%,na,-24%,na,-11%,na,-11%,na]
M-0029svc=recommendation metric=process.runtime.cpython.cpu.utilization base=0 delta/30s=[na,new:12.0,na,+0%,na,+0%,na,+0%,na,+0%]
M-0030svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0 delta/30s=[new:0.0333,+0%,+0%,+0%,+0%,+0%,+0%,+0%,+0%,+0%]
M-0031svc=checkout metric=rpc.client.call.duration.p50 base=0.017 delta/30s=[+21%,na,+58727%,na,-12%,na,-7%,na,+3%,na]
M-0032svc=fraud-detection metric=traces.span.metrics.duration.p50 base=48.4 delta/30s=[+24%,na,+30919%,na,-10%,na,-7%,na,-9%,na]
M-0033svc=checkout metric=rpc.client.call.duration.p95 base=0.0453 delta/30s=[+1%,na,+21987%,na,-47%,na,-25%,na,-4%,na]
M-0034svc=checkout metric=rpc.client.call.duration.p99 base=0.0531 delta/30s=[-7%,na,+18748%,na,-53%,na,-12%,na,-8%,na]
M-0035svc=fraud-detection metric=traces.span.metrics.duration.p95 base=94.3 delta/30s=[+2%,na,+15809%,na,-0%,na,+0%,na,-0%,na]
M-0036svc=fraud-detection metric=traces.span.metrics.duration.p99 base=98.9 delta/30s=[+0%,na,+15073%,na,-0%,na,+0%,na,-0%,na]
M-0037svc=checkout metric=traces.span.metrics.duration.p50 base=105.7 delta/30s=[-5%,na,+11724%,na,-57%,na,+4%,na,-41%,na]
861 lower-ranked lines omitted

traces: slowest and errored traces after fault onset, critical path (! = error span)

T-0001trace=185d3598 dur=20820ms ERROR path: load-generator:user_checkout_single 15014ms! > frontend-proxy:POST 14996ms! > frontend:POST 14997ms! > checkout:oteldemo.CheckoutService/PlaceOrder 19188ms!
T-0002trace=150310a2 dur=24331ms ok path: accounting:order-consumed 24331ms
T-0003trace=58bf08cf dur=20427ms ERROR path: load-generator:user_checkout_single 15018ms! > frontend-proxy:POST 14998ms! > frontend:POST 15000ms! > checkout:oteldemo.CheckoutService/PlaceOrder 18795ms!
T-0004trace=41d681d7 dur=20378ms ok path: accounting:order-consumed 20378ms
T-0005trace=e425c158 dur=16727ms ERROR path: load-generator:user_checkout_single 15015ms! > frontend-proxy:POST 15001ms! > frontend:POST 15000ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 2ms
T-0006trace=c07a5979 dur=14062ms ok path: accounting:order-consumed 14062ms
T-0007trace=d76d6de9 dur=14131ms ERROR path: load-generator:user_checkout_multi 12543ms! > frontend-proxy:POST 12518ms! > frontend:POST /api/checkout 12519ms! > checkout:oteldemo.CheckoutService/PlaceOrder 12517ms!
T-0008trace=641d018c dur=13385ms ok path: accounting:order-consumed 13385ms
T-0009trace=31c91619 dur=10545ms ERROR path: load-generator:user_checkout_single 9446ms! > frontend-proxy:POST 9431ms! > frontend:POST /api/checkout 9432ms! > checkout:oteldemo.CheckoutService/PlaceOrder 9429ms!
T-0010trace=d8dea2c6 dur=13154ms ok path: load-generator:user_checkout_multi 11583ms > frontend-proxy:POST 11530ms > frontend:POST /api/checkout 11530ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 8ms
T-0011trace=14c87fd5 dur=2642ms ERROR path: load-generator:user_checkout_multi 1064ms! > frontend-proxy:POST 1033ms! > frontend:POST /api/checkout 1033ms! > checkout:oteldemo.CheckoutService/PlaceOrder 1029ms!
T-0012trace=dc08cc8c dur=12803ms ok path: accounting:order-consumed 12803ms
T-0013trace=86777511 dur=11906ms ok path: accounting:order-consumed 11906ms
T-0014trace=1611936a dur=11311ms ok path: load-generator:user_checkout_single 9877ms > frontend-proxy:POST 9861ms > frontend:POST /api/checkout 9860ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 1ms
T-0015trace=c973f368 dur=11283ms ok path: accounting:order-consumed 11283ms
T-0016trace=4cdd8c05 dur=9970ms ok path: accounting:order-consumed 9970ms
T-0017trace=a7773fd2 dur=8902ms ok path: accounting:order-consumed 8902ms
T-0018trace=f9dca41e dur=8366ms ok path: accounting:order-consumed 8366ms
T-0019trace=2445d258 dur=7154ms ok path: accounting:order-consumed 7154ms
T-0020trace=7f3b119e dur=7114ms ok path: accounting:order-consumed 7114ms
T-0021trace=4c33feac dur=6958ms ok path: accounting:order-consumed 6958ms
T-0022trace=b5f501b2 dur=6952ms ok path: accounting:order-consumed 6952ms
T-0023trace=cdb3cc7c dur=6886ms ok path: accounting:order-consumed 6886ms
T-0024trace=382d0263 dur=6463ms ok path: accounting:order-consumed 6463ms
T-0025trace=b35799ee dur=6139ms ok path: accounting:order-consumed 6139ms
T-0026trace=5bc11074 dur=6138ms ok path: frontend-proxy:POST 6138ms > flagd:flagd.evaluation.v1.Service/EventStream 6137ms

logs: log templates per service, rate before vs after fault onset, with examples

L-0001svc=frontend sev=ERROR base=0.0/min fault=1.0/min tpl="Checkout failed to place order" ex=['Checkout failed to place order', 'Checkout failed to place order']
L-0002svc=shipping sev=INFO base=2.0/min fault=0.8/min tpl="" ex=['', '']
L-0003svc=load-generator sev=INFO base=14.4/min fault=9.6/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user 55d3d6ac-dc06-11f1-8306-d6b650b5399b', 'Checkout completed for user 55d3d6ac-dc06-11f1-8306-d6b650b5399b']
L-0004svc=accounting sev=INFORMATION base=14.2/min fault=9.8/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0005svc=cart sev=INFORMATION base=14.2/min fault=9.8/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0006svc=checkout sev=INFO base=14.2/min fault=9.8/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 194.748µs', 'Successful to write message. offset: 0, duration: 232.14µs']
L-0007svc=checkout sev=INFO base=14.2/min fault=9.8/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0008svc=checkout sev=INFO base=14.2/min fault=9.8/min tpl="order placed" ex=['order placed', 'order placed']
L-0009svc=checkout sev=INFO base=14.2/min fault=9.8/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0010svc=checkout sev=INFO base=14.2/min fault=9.8/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0011svc=email sev=INFO base=14.2/min fault=9.8/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0012svc=fraud-detection sev=INFO base=14.2/min fault=9.8/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: 64a555d4-dc06-11f1-9d0e-3288f2b1222e, and updated total count to: 23296', 'Consumed record with orderId: 66c89fd7-dc06-11f1-9d0e-3288f2b1222e, and updated total count to: 23297']
L-0013svc=frontend sev=INFO base=14.2/min fault=9.8/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0014svc=payment sev=INFO base=14.2/min fault=9.8/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0015svc=payment sev=INFO base=14.2/min fault=9.8/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0016svc=shipping sev=INFO base=14.2/min fault=9.8/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0017svc=checkout sev=INFO base=14.2/min fault=10.8/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0018svc=ad sev=INFO base=39.4/min fault=51.0/min tpl="Targeted ad request received for <*>" ex=['Targeted ad request received for [binoculars]', 'Targeted ad request received for [assembly]']
L-0019svc=load-generator sev=INFO base=25.2/min fault=30.8/min tpl="Product added to cart successfully" ex=['Product added to cart successfully', 'Product added to cart successfully']
L-0020svc=checkout sev=INFO base=0.2/min fault=0.0/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0021svc=load-generator sev=INFO base=36.0/min fault=30.0/min tpl="Currency changed to CHF" ex=['Currency changed to CHF', 'Currency changed to CHF']
L-0022svc=currency sev=INFO base=124.2/min fault=106.8/min tpl="conversion successful" ex=['conversion successful', 'conversion successful']
L-0023svc=load-generator sev=INFO base=14.0/min fault=12.0/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user 59cca0d6-dc06-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user 59cca0d6-dc06-11f1-8306-d6b650b5399b']
L-0024svc=load-generator sev=INFO base=77.6/min fault=69.2/min tpl="User <UUID> adding <NUM> of product <*> to cart" ex=['User 53abd046-dc06-11f1-8306-d6b650b5399b adding 2 of product HQTGWGPNH4 to cart', 'User 53abd046-dc06-11f1-8306-d6b650b5399b adding 2 of product HQTGWGPNH4 to cart']
L-0025svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 80853 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 80970 breach: LogSegment(baseOff…']
L-0026svc=kafka sev=INFO base=1.0/min fault=0.8/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=80785, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=80905, siz…']
L-0027svc=kafka sev=INFO base=8.4/min fault=7.6/min tpl="Deleted <*> index <*>" ex=['Deleted offset index /tmp/kafka-logs/__cluster_metadata-0/00000000000000080695.index.deleted.', 'Deleted time index /tmp/kafka-logs/__cluster_metadata-0/00000000000000080695.timeindex.deleted.']
L-0028svc=load-generator sev=INFO base=73.6/min fault=80.0/min tpl="User getting <*> for <*> <*>" ex=['User getting recommendations for product: 2ZYFJ3GM2N', 'User getting recommendations for product: 2ZYFJ3GM2N']
33 lower-ranked lines omitted
Loading runs in your browser…