Rootline v0.1 pilot

← All incidents

intlShippingSlowdown-r1

What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)

Service graph during the window

Evidence and answers

4683 prompt tokens

metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean

M-0001svc=ad metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0104]
M-0002svc=ad metric=jvm.thread.count.rate base=0 delta/30s=[na,new:0.0667,na,+0%,na,+0%,na,+0%,na,new:0.0667]
M-0003svc=ad metric=rpc.client.duration.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0004svc=ad metric=rpc.client.request.size.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0005svc=ad metric=rpc.client.response.size.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0006svc=ad metric=traces.span.metrics.duration.p50 base=1.01 delta/30s=[na,-1%,na,-1%,na,-1%,na,-1%,na,+1488600%]
M-0007svc=ad metric=traces.span.metrics.duration.p95 base=1.91 delta/30s=[na,-1%,na,-1%,na,-1%,na,-1%,na,+783426%]
M-0008svc=ad metric=traces.span.metrics.duration.p99 base=2.36 delta/30s=[na,-16%,na,-16%,na,-16%,na,-16%,na,+636572%]
M-0009svc=fraud-detection metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.027]
M-0010svc=fraud-detection metric=rpc.client.duration.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0011svc=fraud-detection metric=rpc.client.request.size.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0012svc=fraud-detection metric=rpc.client.response.size.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0013svc=kafka metric=jvm.class.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%]
M-0014svc=kafka metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:2,na,+0%]
M-0015svc=recommendation metric=process.thread.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0016svc=shipping metric=http.server.active_requests base=0 delta/30s=[na,+0%,na,+0%,na,new:2,na,+0%,na,+0%]
M-0017svc=fraud-detection metric=traces.span.metrics.duration.p50 base=60.6 delta/30s=[na,+5%,na,-37%,na,-27%,na,-39%,na,+24654%]
M-0018svc=fraud-detection metric=traces.span.metrics.duration.p95 base=96.0 delta/30s=[na,+0%,na,-3%,na,-2%,na,-2%,na,+15528%]
M-0019svc=fraud-detection metric=traces.span.metrics.duration.p99 base=99.2 delta/30s=[na,+0%,na,-0%,na,-0%,na,-0%,na,+15022%]
M-0020svc=checkout metric=http.client.request.duration.p95 base=0.406 delta/30s=[na,+165%,na,+1481%,na,-42%,na,+1440%,na,+368%]
M-0021svc=load-generator metric=http.client.duration.p95 base=374.2 delta/30s=[na,-20%,na,+1403%,na,-38%,na,+227%,na,+157%]
M-0022svc=checkout metric=traces.span.metrics.duration.p95 base=718.0 delta/30s=[na,+95%,na,+1153%,na,+45%,na,+1084%,na,+701%]
M-0023svc=checkout metric=traces.span.metrics.duration.p99 base=815.6 delta/30s=[na,+131%,na,+1102%,na,+63%,na,+1089%,na,+1022%]
M-0024svc=flagd metric=error_ratio base=8.83e-05 delta/30s=[na,+401%,na,-100%,na,-100%,na,-100%,na,+921%]
M-0025svc=checkout metric=rpc.server.call.duration.p95 base=0.686 delta/30s=[na,+160%,na,+914%,na,+57%,na,+902%,na,+665%]
M-0026svc=ad metric=otel.sdk.processor.log.queue.size base=0.2 delta/30s=[na,-100%,na,-100%,na,+900%,na,-100%,na,-100%]
M-0027svc=fraud-detection metric=jvm.thread.count.rate base=0.00667 delta/30s=[na,+400%,na,-100%,na,-100%,na,-100%,na,+900%]
M-0028svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,-100%,-100%,-100%,-100%,-100%,-100%,-100%,-100%,+900%]
M-0029svc=checkout metric=http.client.request.duration.p99 base=0.743 delta/30s=[na,+198%,na,+880%,na,+160%,na,+875%,na,+788%]
M-0030svc=load-generator metric=http.client.duration.p99 base=729.6 delta/30s=[na,+169%,na,+877%,na,+117%,na,+830%,na,+702%]
M-0031svc=checkout metric=rpc.server.call.duration.p99 base=0.777 delta/30s=[na,+203%,na,+851%,na,+185%,na,+849%,na,+807%]
M-0032svc=shipping metric=traces.span.metrics.duration.p95 base=940.0 delta/30s=[na,+19%,na,+849%,na,-40%,na,+831%,na,+459%]
M-0033svc=checkout metric=otel.sdk.processor.log.queue.size base=1.4 delta/30s=[na,-100%,na,+257%,na,+757%,na,-100%,na,+329%]
M-0034svc=frontend metric=http.server.request.duration.p99 base=0.832 delta/30s=[na,+72%,na,+728%,na,+129%,na,+698%,na,+516%]
M-0035svc=product-catalog metric=otel.sdk.processor.log.queue.size base=1 delta/30s=[na,-100%,na,+600%,na,-100%,na,-100%,na,-100%]
M-0036svc=shipping metric=http.server.request.duration.p95 base=1.08 delta/30s=[na,+17%,na,+542%,na,-44%,na,+538%,na,-15%]
M-0037svc=frontend metric=traces.span.metrics.duration.p95 base=1.47e+03 delta/30s=[na,-3%,na,+508%,na,-30%,na,+496%,na,+275%]
858 lower-ranked lines omitted

traces: slowest and errored traces after fault onset, critical path (! = error span)

T-0001trace=890e23d4 dur=17655ms ok path: accounting:order-consumed 17655ms
T-0002trace=358b18a3 dur=15676ms ok path: accounting:order-consumed 15676ms
T-0003trace=c91fcfd5 dur=14828ms ok path: accounting:order-consumed 14828ms
T-0004trace=dfc72087 dur=13252ms ok path: accounting:order-consumed 13252ms
T-0005trace=c8aa01b7 dur=12036ms ok path: accounting:order-consumed 12036ms
T-0006trace=6df55434 dur=11841ms ok path: accounting:order-consumed 11841ms
T-0007trace=33aff183 dur=11077ms ok path: accounting:order-consumed 11077ms
T-0008trace=925d4be0 dur=10416ms ok path: accounting:order-consumed 10416ms
T-0009trace=0bafcf70 dur=8829ms ok path: accounting:order-consumed 8829ms
T-0010trace=0de34d7c dur=8412ms ok path: accounting:order-consumed 8412ms
T-0011trace=4e4186ca dur=7707ms ok path: accounting:order-consumed 7707ms
T-0012trace=a8ea6719 dur=7655ms ok path: accounting:order-consumed 7655ms
T-0013trace=26cf8a8d dur=7646ms ok path: accounting:order-consumed 7646ms
T-0014trace=682a4fd5 dur=7546ms ok path: load-generator:user_checkout_multi 6027ms > frontend-proxy:POST 6000ms > frontend:POST /api/checkout 6000ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 1ms
T-0015trace=770229aa dur=7327ms ok path: accounting:order-consumed 7327ms
T-0016trace=fbce5ebe dur=7114ms ok path: load-generator:user_checkout_single 5344ms > frontend-proxy:POST 5327ms > frontend:POST /api/checkout 5326ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 2ms
T-0017trace=007543d8 dur=7043ms ok path: load-generator:user_checkout_single 5127ms > frontend-proxy:POST 5103ms > frontend:POST /api/checkout 5104ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 2ms
T-0018trace=ecf4c30a dur=7015ms ok path: load-generator:user_checkout_multi 5133ms > frontend-proxy:POST 5079ms > frontend:POST /api/checkout 5079ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 9ms
T-0019trace=030fb52b dur=7008ms ok path: load-generator:user_checkout_single 5121ms > frontend-proxy:POST 5105ms > frontend:POST /api/checkout 5106ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 1ms
T-0020trace=b5c15552 dur=7000ms ok path: accounting:order-consumed 7000ms

logs: log templates per service, rate before vs after fault onset, with examples

L-0001svc=shipping sev=INFO base=0.0/min fault=1.2/min tpl="Delaying international shipment due to <redacted> feature flag" ex=['Delaying international shipment due to <redacted> feature flag', 'Delaying international shipment due to <redacted> feature flag']
L-0002svc=load-generator sev=INFO base=10.8/min fault=14.4/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user 536cbc40-dbf6-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user 536cbc40-dbf6-11f1-8306-d6b650b5399b']
L-0003svc=load-generator sev=INFO base=73.2/min fault=89.6/min tpl="User <UUID> adding <NUM> of product <*> to cart" ex=['User 529ad5d6-dbf6-11f1-8306-d6b650b5399b adding 2 of product 1YMWWN1N4O to cart', 'User 529ad5d6-dbf6-11f1-8306-d6b650b5399b adding 2 of product 1YMWWN1N4O to cart']
L-0004svc=accounting sev=INFORMATION base=12.8/min fault=15.8/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0005svc=cart sev=INFORMATION base=12.8/min fault=15.8/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0006svc=checkout sev=INFO base=12.8/min fault=15.8/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 160.396µs', 'Successful to write message. offset: 0, duration: 171.757µs']
L-0007svc=checkout sev=INFO base=12.8/min fault=15.8/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0008svc=checkout sev=INFO base=12.8/min fault=15.8/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0009svc=checkout sev=INFO base=12.8/min fault=15.8/min tpl="order placed" ex=['order placed', 'order placed']
L-0010svc=checkout sev=INFO base=12.8/min fault=15.8/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0011svc=checkout sev=INFO base=12.8/min fault=15.8/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0012svc=email sev=INFO base=12.8/min fault=15.8/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0013svc=fraud-detection sev=INFO base=12.8/min fault=15.8/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: 53322e64-dbf6-11f1-82f3-d279f8fa1f6d, and updated total count to: 21807', 'Consumed record with orderId: 5372fbd6-dbf6-11f1-82f3-d279f8fa1f6d, and updated total count to: 21808']
L-0014svc=frontend sev=INFO base=12.8/min fault=15.8/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0015svc=payment sev=INFO base=12.8/min fault=15.8/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0016svc=payment sev=INFO base=12.8/min fault=15.8/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0017svc=shipping sev=INFO base=12.8/min fault=15.8/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0018svc=checkout sev=INFO base=0.0/min fault=0.2/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0019svc=shipping sev=INFO base=1.6/min fault=1.2/min tpl="" ex=['', '']
L-0020svc=load-generator sev=INFO base=14.8/min fault=12.4/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0021svc=load-generator sev=INFO base=14.8/min fault=17.2/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user 532fd406-dbf6-11f1-8306-d6b650b5399b', 'Checkout completed for user 532fd406-dbf6-11f1-8306-d6b650b5399b']
L-0022svc=currency sev=INFO base=99.4/min fault=114.4/min tpl="conversion successful" ex=['conversion successful', 'conversion successful']
L-0023svc=cart sev=INFORMATION base=53.8/min fault=61.0/min tpl="AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}" ex=['AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}', 'AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}']
L-0024svc=load-generator sev=INFO base=34.0/min fault=38.0/min tpl="User viewing cart" ex=['User viewing cart', 'User viewing cart']
L-0025svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=67019, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=67135, siz…']
L-0026svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 67203 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 67320 breach: LogSegment(baseOff…']
L-0027svc=load-generator sev=INFO base=24.4/min fault=26.8/min tpl="Currency changed to CHF" ex=['Currency changed to CHF', 'Currency changed to CHF']
34 lower-ranked lines omitted
Loading runs in your browser…