Rootline v0.1 pilot

← All incidents

healthy-r2

What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)

Service graph during the window

Evidence and answers

5396 prompt tokens

metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean

M-0001svc=ad metric=jvm.gc.duration.count.rate base=0 delta/30s=[na,+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%]
M-0002svc=checkout metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,new:0.00995,na,+0%,na,new:0.0108]
M-0003svc=checkout metric=otel.sdk.span.live base=0 delta/30s=[na,+0%,na,+0%,na,new:2,na,+0%,na,+0%]
M-0004svc=fraud-detection metric=rpc.client.duration.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0005svc=fraud-detection metric=rpc.client.request.size.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0006svc=fraud-detection metric=rpc.client.response.size.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0007svc=kafka metric=jvm.gc.duration.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%]
M-0008svc=kafka metric=jvm.thread.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:0.0333,na,new:0.0333]
M-0009svc=payment metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.037]
M-0010svc=payment metric=traces.span.metrics.duration.p50 base=1.26 delta/30s=[na,-10%,na,-9%,na,+19%,na,+377%,na,+1193452%]
M-0011svc=payment metric=traces.span.metrics.duration.p95 base=8.51 delta/30s=[na,-17%,na,-39%,na,+3189%,na,+378%,na,+176094%]
M-0012svc=payment metric=traces.span.metrics.duration.p99 base=12.9 delta/30s=[na,-39%,na,-55%,na,+2814%,na,+273%,na,+116155%]
M-0013svc=recommendation metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:0.00342,na,+0%]
M-0014svc=recommendation metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:5]
M-0015svc=recommendation metric=traces.span.metrics.duration.p50 base=3.38 delta/30s=[na,+5%,na,-5%,na,-0%,na,+443875%,na,-5%]
M-0016svc=recommendation metric=traces.span.metrics.duration.p95 base=5.88 delta/30s=[na,+4%,na,+11%,na,+7%,na,+255096%,na,-1%]
M-0017svc=recommendation metric=traces.span.metrics.duration.p99 base=8.71 delta/30s=[na,-9%,na,+198%,na,+13%,na,+172027%,na,-9%]
M-0018svc=unknown metric=system.disk.pending_operations base=0.2 delta/30s=[na,+1900%,na,+7400%,na,+2400%,na,+400%,na,-100%]
M-0019svc=checkout metric=go.goroutine.count.rate base=0.0133 delta/30s=[na,-100%,na,-100%,na,+650%,na,-100%,na,+1400%]
M-0020svc=unknown metric=system.processes.count.rate base=0.06 delta/30s=[na,+344%,na,+844%,na,+1233%,na,+67%,na,-44%]
M-0021svc=checkout metric=otel.sdk.processor.log.queue.size base=0.8 delta/30s=[na,-100%,na,-100%,na,+275%,na,-100%,na,+1150%]
M-0022svc=payment metric=nodejs.eventloop.delay.stddev base=0.00062 delta/30s=[-6%,na,-68%,na,-68%,na,+998%,na,+477%,na]
M-0023svc=load-generator metric=otel.sdk.processor.span.queue.size base=4.6 delta/30s=[+922%,na,+596%,na,+878%,na,+400%,na,+596%,na]
M-0024svc=accounting metric=dotnet.thread_pool.thread.count.rate base=0.00667 delta/30s=[-100%,na,+900%,na,-100%,na,-100%,na,-100%,na]
M-0025svc=accounting metric=process.thread.count.rate base=0.00667 delta/30s=[-100%,na,+900%,na,-100%,na,-100%,na,-100%,na]
M-0026svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,-100%,-100%,+900%,-100%,-100%,-100%,-100%,-100%,-100%]
M-0027svc=payment metric=nodejs.eventloop.delay.max base=0.0467 delta/30s=[-9%,na,-66%,na,-60%,na,+781%,na,+462%,na]
M-0028svc=cart metric=traces.span.metrics.duration.p99 base=649.2 delta/30s=[na,+105%,na,-70%,na,+197%,na,-43%,na,+615%]
M-0029svc=email metric=traces.span.metrics.duration.p95 base=592.0 delta/30s=[na,+40%,na,-48%,na,+96%,na,+89%,na,+593%]
M-0030svc=email metric=traces.span.metrics.duration.p99 base=726.4 delta/30s=[na,+33%,na,-47%,na,+86%,na,+85%,na,+564%]
M-0031svc=payment metric=v8js.gc.duration.p99 base=0.0366 delta/30s=[+35%,na,-73%,na,-73%,na,+563%,na,-32%,na]
M-0032svc=payment metric=v8js.gc.duration.p95 base=0.0349 delta/30s=[+34%,na,-74%,na,-72%,na,+509%,na,-30%,na]
M-0033svc=cart metric=traces.span.metrics.duration.p95 base=526.0 delta/30s=[na,+102%,na,-64%,na,+212%,na,-51%,na,+508%]
M-0034svc=checkout metric=rpc.client.call.duration.p99 base=0.844 delta/30s=[na,+166%,na,-72%,na,+185%,na,-44%,na,+451%]
M-0035svc=payment metric=nodejs.eventloop.utilization base=0.00592 delta/30s=[-9%,na,-27%,na,-33%,na,+436%,na,+172%,na]
M-0036svc=checkout metric=rpc.client.call.duration.p95 base=0.62 delta/30s=[na,+98%,na,-71%,na,+225%,na,-41%,na,+424%]
M-0037svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,-100%,na,-100%,na,+400%,na]
864 lower-ranked lines omitted

traces: slowest and errored traces after fault onset, critical path (! = error span)

T-0001trace=e7aed1e7 dur=6252ms ERROR path: load-generator:user_checkout_single 5704ms > frontend-proxy:POST 5380ms > frontend:POST /api/checkout 5379ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 501ms | errors off path: checkout:POST
T-0002trace=9358bc4d dur=20267ms ok path: accounting:order-consumed 20267ms
T-0003trace=ec3fbece dur=5091ms ERROR path: load-generator:user_checkout_multi 3179ms > frontend-proxy:POST 2408ms > frontend:POST /api/checkout 2407ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 511ms | errors off path: checkout:POST
T-0004trace=8bca9b74 dur=18851ms ok path: accounting:order-consumed 18851ms
T-0005trace=116fe355 dur=3323ms ERROR path: load-generator:user_checkout_single 3323ms > frontend-proxy:POST 3264ms > frontend:POST /api/checkout 3262ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 501ms | errors off path: checkout:POST
T-0006trace=37ed16a9 dur=15436ms ok path: accounting:order-consumed 15436ms
T-0007trace=11e42c00 dur=1938ms ERROR path: load-generator:user_checkout_multi 1938ms > frontend-proxy:POST 1849ms > frontend:POST /api/checkout 1848ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 501ms | errors off path: checkout:POST
T-0008trace=6944f5f1 dur=13593ms ok path: accounting:order-consumed 13593ms
T-0009trace=9e25d134 dur=1420ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 1420ms!
T-0010trace=a1164366 dur=11474ms ok path: accounting:order-consumed 11474ms
T-0011trace=a6bdffeb dur=787ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 787ms!
T-0012trace=d5deb53b dur=11355ms ok path: accounting:order-consumed 11355ms
T-0013trace=5401e8f8 dur=769ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 769ms!
T-0014trace=ec8b5f67 dur=11229ms ok path: accounting:order-consumed 11229ms
T-0015trace=a7734eb9 dur=769ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 769ms!
T-0016trace=ea644a45 dur=10891ms ok path: accounting:order-consumed 10891ms
T-0017trace=f1aae1b2 dur=760ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 760ms!
T-0018trace=b77a72c3 dur=9972ms ok path: accounting:order-consumed 9972ms
T-0019trace=35e6cdbe dur=632ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 632ms!
T-0020trace=e6a347db dur=8943ms ok path: accounting:order-consumed 8943ms
T-0021trace=0c468d00 dur=469ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 469ms!
T-0022trace=004bb88c dur=8925ms ok path: accounting:order-consumed 8925ms
T-0023trace=c4b5f413 dur=465ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 465ms!
T-0024trace=18740fda dur=8805ms ok path: accounting:order-consumed 8805ms
T-0025trace=868b008a dur=463ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 463ms!
T-0026trace=0f20ed6f dur=8716ms ok path: accounting:order-consumed 8716ms
T-0027trace=ffbd8363 dur=452ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 452ms!
T-0028trace=b216e5f8 dur=7227ms ok path: load-generator:browser_change_currency 7227ms
T-0029trace=2e153cbf dur=440ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 440ms!
T-0030trace=3dd85ee0 dur=7125ms ok path: accounting:order-consumed 7125ms
T-0031trace=068e3d78 dur=439ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 439ms!
T-0032trace=5f703c0a dur=6932ms ok path: accounting:order-consumed 6932ms
T-0033trace=535bb717 dur=437ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 437ms!
T-0034trace=b468f4b1 dur=6790ms ok path: accounting:order-consumed 6790ms
T-0035trace=dbc696e5 dur=434ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 434ms!
T-0036trace=a7a72a7f dur=6508ms ok path: accounting:order-consumed 6508ms
T-0037trace=d9f8fceb dur=318ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 318ms!
T-0038trace=8920eafe dur=6435ms ok path: accounting:order-consumed 6435ms
T-0039trace=91e3bd31 dur=310ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 310ms!
T-0040trace=e5188123 dur=6387ms ok path: load-generator:browser_add_to_cart 6387ms

logs: log templates per service, rate before vs after fault onset, with examples

L-0001svc=checkout sev=WARN base=0.2/min fault=0.8/min tpl="failed to send order confirmation: failed POST to email service: Post "http://email:<NUM>/send_order_confirmation": context deadline exceed…" ex=['failed to send order confirmation: failed POST to email service: Post "http://email:6060/send_order_confirmation": context deadline exceeded', 'failed to send order confirmation: failed POST to email service: Post "http://email:6060/send_order_confirmation": context deadline exceeded']
L-0002svc=load-generator sev=INFO base=33.2/min fault=44.4/min tpl="User viewing cart" ex=['User viewing cart', 'User viewing cart']
L-0003svc=load-generator sev=INFO base=24.8/min fault=30.4/min tpl="Currency changed to CHF" ex=['Currency changed to CHF', 'Currency changed to CHF']
L-0004svc=ad sev=INFO base=45.8/min fault=37.6/min tpl="Targeted ad request received for <*>" ex=['Targeted ad request received for [binoculars]', 'Targeted ad request received for [binoculars]']
L-0005svc=checkout sev=INFO base=0.0/min fault=0.2/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0006svc=ad sev=INFO base=2.4/min fault=3.0/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0007svc=kafka sev=INFO base=2.0/min fault=2.4/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=39783, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=39813, siz…']
L-0008svc=kafka sev=INFO base=2.0/min fault=2.4/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 39942 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 39981 breach: LogSegment(baseOff…']
L-0009svc=load-generator sev=INFO base=17.2/min fault=15.2/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0010svc=kafka sev=INFO base=1.0/min fault=0.8/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=39843, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=39993, siz…']
L-0011svc=kafka sev=INFO base=1.0/min fault=0.8/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 40059 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 40176 breach: LogSegment(baseOff…']
L-0012svc=load-generator sev=INFO base=32.4/min fault=29.2/min tpl="Product added to cart successfully" ex=['Product added to cart successfully', 'Product added to cart successfully']
L-0013svc=shipping sev=INFO base=1.4/min fault=1.6/min tpl="" ex=['', '']
L-0014svc=currency sev=INFO base=102.8/min fault=111.4/min tpl="conversion successful" ex=['conversion successful', 'conversion successful']
L-0015svc=load-generator sev=INFO base=76.4/min fault=81.2/min tpl="User <UUID> adding <NUM> of product <*> to cart" ex=['User 8fca5974-dbd6-11f1-8306-d6b650b5399b adding 3 of product L9ECAV7KIM to cart', 'User 8fca5974-dbd6-11f1-8306-d6b650b5399b adding 3 of product L9ECAV7KIM to cart']
L-0016svc=load-generator sev=INFO base=12.4/min fault=13.2/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user 9272a6ea-dbd6-11f1-8306-d6b650b5399b', 'Checkout completed for user 9272a6ea-dbd6-11f1-8306-d6b650b5399b']
L-0017svc=load-generator sev=INFO base=13.6/min fault=12.8/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user 91a28910-dbd6-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user 91a28910-dbd6-11f1-8306-d6b650b5399b']
L-0018svc=quote sev=INFO base=29.2/min fault=27.6/min tpl="Calculated quote" ex=['Calculated quote', 'Calculated quote']
L-0019svc=shipping sev=INFO base=29.2/min fault=27.6/min tpl="Requesting quote" ex=['Requesting quote', 'Requesting quote']
L-0020svc=shipping sev=INFO base=29.2/min fault=27.6/min tpl="Sending Quote" ex=['Sending Quote', 'Sending Quote']
L-0021svc=kafka sev=INFO base=3.2/min fault=3.0/min tpl="[SnapshotEmitter id=<NUM>] Successfully wrote snapshot <HEX>-<NUM>" ex=['[SnapshotEmitter id=1] Successfully wrote snapshot 00000000000000040059-0000000001', '[SnapshotEmitter id=1] Successfully wrote snapshot 00000000000000040098-0000000001']
L-0022svc=kafka sev=INFO base=3.2/min fault=3.0/min tpl="[SnapshotGenerator id=<NUM>] Creating new KRaft snapshot file snapshot <HEX>-<NUM> because we have replayed at least <NUM> bytes." ex=['[SnapshotGenerator id=1] Creating new KRaft snapshot file snapshot 00000000000000040059-0000000001 because we have replayed at least 2800 b…', '[SnapshotGenerator id=1] Creating new KRaft snapshot file snapshot 00000000000000040098-0000000001 because we have replayed at least 2800 b…']
39 lower-ranked lines omitted
Loading runs in your browser…