Rootline v0.1 pilot

← All incidents

productCatalogLockContention-r2

What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)

Service graph during the window

Evidence and answers

5814 prompt tokens

metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean

M-0001svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0002svc=accounting metric=dns.lookup.duration.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0003svc=ad metric=error_ratio base=0 delta/30s=[+0%,na,new:0.0588,na,+0%,na,+0%,na,+0%,na]
M-0004svc=ad metric=jvm.gc.duration.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0005svc=ad metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0006svc=ad metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0007svc=ad metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0008svc=ad metric=traces.span.metrics.duration.p50 base=1 delta/30s=[-0%,na,+1492956%,na,+3%,na,-0%,na,-0%,na]
M-0009svc=ad metric=traces.span.metrics.duration.p95 base=1.91 delta/30s=[-0%,na,+785719%,na,+3%,na,-0%,na,-0%,na]
M-0010svc=ad metric=traces.span.metrics.duration.p99 base=2.21 delta/30s=[-10%,na,+679248%,na,+57%,na,-10%,na,-10%,na]
M-0011svc=checkout metric=otel.sdk.span.live base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:3,na,+0%,na]
M-0012svc=fraud-detection metric=error_ratio base=0 delta/30s=[+0%,na,new:0.0769,na,+0%,na,+0%,na,+0%,na]
M-0013svc=fraud-detection metric=jvm.thread.count.rate base=0 delta/30s=[new:0.0333,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0014svc=fraud-detection metric=kafka.consumer.last_heartbeat_seconds_ago base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:2,na]
M-0015svc=fraud-detection metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0016svc=fraud-detection metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0017svc=fraud-detection metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0018svc=frontend-proxy metric=error_ratio base=0 delta/30s=[+0%,na,new:0.0218,na,new:0.0106,na,new:0.0222,na,new:0.0112,na]
M-0019svc=kafka metric=jvm.gc.duration.count.rate base=0 delta/30s=[na,+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%]
M-0020svc=kafka metric=jvm.thread.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%]
M-0021svc=load-generator metric=error_ratio base=0 delta/30s=[+0%,na,new:0.131,na,new:0.0791,na,new:0.138,na,new:0.0671,na]
M-0022svc=product-catalog metric=db.client.operation.duration.p95 base=0.00421 delta/30s=[-4%,na,+237675%,na,+237675%,na,+237675%,na,+237675%,na]
M-0023svc=product-catalog metric=db.client.operation.duration.p99 base=0.00485 delta/30s=[-1%,na,+206042%,na,+206042%,na,+206042%,na,+206042%,na]
M-0024svc=load-generator metric=http.client.duration.p50 base=17.8 delta/30s=[+56042%,na,+56042%,na,+56042%,na,na,+56042%,na,+56042%]
M-0025svc=fraud-detection metric=traces.span.metrics.duration.p50 base=49.5 delta/30s=[+31%,na,+30202%,na,-84%,na,-26%,na,-49%,na]
M-0026svc=fraud-detection metric=traces.span.metrics.duration.p95 base=94.5 delta/30s=[+2%,na,+15772%,na,-5%,na,-5%,na,-4%,na]
M-0027svc=fraud-detection metric=traces.span.metrics.duration.p99 base=98.9 delta/30s=[+0%,na,+15067%,na,-1%,na,-1%,na,-1%,na]
M-0028svc=product-catalog metric=go.goroutine.count.rate base=0.04 delta/30s=[-100%,na,+4733%,na,-100%,na,+5900%,na,-100%,na]
M-0029svc=product-catalog metric=rpc.server.call.duration.p50 base=0.0129 delta/30s=[-15%,na,+3109%,na,+3889%,na,+3804%,na,+4040%,na]
M-0030svc=recommendation metric=otel.sdk.span.live base=0.8 delta/30s=[+3400%,na,+3775%,na,+2275%,na,+3775%,na,+775%,na]
M-0031svc=frontend metric=traces.span.metrics.duration.p50 base=401.3 delta/30s=[-0%,na,+1769%,na,+2599%,na,+3638%,na,+3015%,na]
M-0032svc=product-catalog metric=traces.span.metrics.duration.p95 base=467.9 delta/30s=[+32%,na,+3106%,na,+3106%,na,+3106%,na,+3106%,na]
M-0033svc=product-catalog metric=traces.span.metrics.duration.p50 base=15.1 delta/30s=[-9%,na,+2709%,na,+2749%,na,+2703%,na,+3033%,na]
M-0034svc=flagd metric=error_ratio base=0.00284 delta/30s=[+66%,na,+1686%,na,+1836%,na,+810%,na,+2891%,na]
M-0035svc=cart metric=http.server.request.duration.p50 base=0.0627 delta/30s=[-28%,na,-60%,na,+2691%,na,-0%,na,+1296%,na]
M-0036svc=load-generator metric=otel.sdk.span.live base=1.2 delta/30s=[+1900%,na,+1983%,na,+2400%,na,na,+1483%,na,+2317%]
853 lower-ranked lines omitted

traces: slowest and errored traces after fault onset, critical path (! = error span)

T-0001trace=1fa53c62 dur=34917ms ERROR path: load-generator:user_checkout_multi 34917ms > frontend-proxy:POST 5146ms > frontend:POST /api/checkout 5145ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 501ms | errors off path: checkout:POST, frontend-proxy:GET, load-generator:GET
T-0002trace=5a96369a dur=41240ms ok path: accounting:order-consumed 41240ms
T-0003trace=12371e85 dur=33385ms ERROR path: load-generator:user_checkout_multi 33385ms > frontend-proxy:POST 3193ms > frontend:POST /api/checkout 3189ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 158ms | errors off path: frontend-proxy:GET, load-generator:GET
T-0004trace=cd029adc dur=40229ms ok path: accounting:order-consumed 40229ms
T-0005trace=246284c7 dur=32590ms ERROR path: load-generator:user_checkout_multi 32590ms > frontend-proxy:POST 4830ms > frontend:POST /api/checkout 4826ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 505ms | errors off path: frontend-proxy:GET, load-generator:GET
T-0006trace=7fed87bb dur=34460ms ok path: accounting:order-consumed 34460ms
T-0007trace=ce20aa0a dur=32033ms ERROR path: load-generator:user_checkout_single 30014ms! > frontend-proxy:POST 14999ms! > frontend:POST 14999ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 327ms | errors off path: frontend-proxy:GET, load-generator:GET
T-0008trace=a5dfad69 dur=34261ms ok path: accounting:order-consumed 34261ms
T-0009trace=1d1643a6 dur=31340ms ERROR path: load-generator:user_checkout_single 15525ms! > frontend-proxy:POST 15002ms! > frontend:POST 15003ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 29902ms
T-0010trace=40af99fc dur=34147ms ok path: accounting:order-consumed 34147ms
T-0011trace=5bf1f02f dur=30648ms ERROR path: load-generator:user_checkout_multi 30533ms > frontend-proxy:POST 2617ms > frontend:POST /api/checkout 2612ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 231ms | errors off path: frontend-proxy:GET, load-generator:GET
T-0012trace=c51ec430 dur=33818ms ok path: accounting:order-consumed 33818ms
T-0013trace=94fab531 dur=30135ms ERROR path: load-generator:user_get_recommendations 15004ms! > frontend-proxy:GET 15001ms! > frontend:GET 15001ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 515ms
T-0014trace=d77e50a4 dur=30415ms ok path: frontend-proxy:GET 3023ms > frontend:GET 3023ms > currency:oteldemo.CurrencyService/Convert 0ms
T-0015trace=78101322 dur=30070ms ERROR path: load-generator:user_browse_product 15003ms! > frontend-proxy:GET 15000ms! > frontend:GET 15001ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 30048ms
T-0016trace=851be50a dur=30380ms ok path: frontend-proxy:GET 3071ms > frontend:GET 3071ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 534ms
T-0017trace=81dd8232 dur=29999ms ERROR path: load-generator:user_browse_product 14999ms! > frontend-proxy:GET 14997ms! > frontend:GET 14999ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 29984ms
T-0018trace=5d93f001 dur=30291ms ok path: frontend-proxy:GET 3049ms > frontend:GET 3050ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 527ms
T-0019trace=8c5f7e2c dur=29899ms ERROR path: load-generator:user_get_recommendations 15002ms! > frontend-proxy:GET 14999ms! > frontend:GET 14999ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 119ms
T-0020trace=881b4a48 dur=30039ms ok path: frontend-proxy:GET 2682ms > frontend:GET 2680ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 30015ms
T-0021trace=28e3cbff dur=29634ms ERROR path: load-generator:user_browse_product 15001ms! > frontend-proxy:GET 14999ms! > frontend:GET 14999ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 29625ms
T-0022trace=b39e1294 dur=29912ms ok path: frontend-proxy:GET 3023ms > frontend:GET 3023ms > currency:oteldemo.CurrencyService/Convert 0ms
T-0023trace=ccc6d173 dur=29630ms ERROR path: load-generator:user_browse_product 15003ms! > frontend-proxy:GET 15000ms! > frontend:GET 15000ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 29603ms
T-0024trace=9f327e83 dur=29805ms ok path: frontend-proxy:GET 3045ms > frontend:GET 3045ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 523ms
T-0025trace=df5eac3d dur=29594ms ERROR path: load-generator:user_browse_product 15000ms! > frontend-proxy:GET 14997ms! > frontend:GET 15000ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 29580ms
T-0026trace=88a0ec71 dur=29799ms ok path: frontend-proxy:GET 3062ms > frontend:GET 3062ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 526ms
T-0027trace=82175509 dur=29526ms ERROR path: load-generator:user_browse_product 15005ms! > frontend-proxy:GET 15002ms! > frontend:GET 15004ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 29503ms
T-0028trace=0a3d8cc6 dur=29199ms ok path: frontend-proxy:GET 483ms > frontend:GET 481ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 534ms
T-0029trace=fc4c2643 dur=29474ms ERROR path: load-generator:user_browse_product 15005ms! > frontend-proxy:GET 15003ms! > frontend:GET 15004ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 29450ms
T-0030trace=19025814 dur=29118ms ok path: frontend-proxy:GET 3091ms > frontend:GET 3091ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 509ms
T-0031trace=c27cdb3a dur=29454ms ERROR path: load-generator:user_add_to_cart 15005ms > frontend-proxy:POST 5ms > frontend:POST /api/cart 4ms > cart:POST /oteldemo.CartService/GetCart 0ms | errors off path: frontend-proxy:GET, load-generator:GET
9 lower-ranked lines omitted

logs: log templates per service, rate before vs after fault onset, with examples

L-0001svc=load-generator sev=ERROR base=0.0/min fault=4.8/min tpl="Error in add to cart task: Timeout <NUM> exceeded while waiting for event "response" =========================== logs =====================…" ex=['Error in add to cart task: Timeout 15000ms exceeded while waiting for event "response" =========================== logs ===================…', 'Error in add to cart task: Timeout 15000ms exceeded while waiting for event "response" =========================== logs ===================…']
L-0002svc=product-catalog sev=INFO base=0.0/min fault=1.6/min tpl="lock contention scenario active: holding ACCESS EXCLUSIVE lock on catalog.products" ex=['lock contention scenario active: holding ACCESS EXCLUSIVE lock on catalog.products', 'lock contention scenario active: holding ACCESS EXCLUSIVE lock on catalog.products']
L-0003svc=load-generator sev=INFO base=38.8/min fault=14.4/min tpl="User viewing cart" ex=['User viewing cart', 'User viewing cart']
L-0004svc=ad sev=INFO base=3.6/min fault=0.8/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0005svc=load-generator sev=INFO base=27.2/min fault=11.2/min tpl="Product added to cart successfully" ex=['Product added to cart successfully', 'Product added to cart successfully']
L-0006svc=load-generator sev=INFO base=12.4/min fault=5.2/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user c958ec0e-dbe1-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user c958ec0e-dbe1-11f1-8306-d6b650b5399b']
L-0007svc=ad sev=INFO base=39.4/min fault=17.8/min tpl="Targeted ad request received for <*>" ex=['Targeted ad request received for [travel]', 'Targeted ad request received for [books]']
L-0008svc=quote sev=INFO base=25.6/min fault=11.6/min tpl="Calculated quote" ex=['Calculated quote', 'Calculated quote']
L-0009svc=shipping sev=INFO base=25.6/min fault=11.6/min tpl="Requesting quote" ex=['Requesting quote', 'Requesting quote']
L-0010svc=shipping sev=INFO base=25.6/min fault=11.6/min tpl="Sending Quote" ex=['Sending Quote', 'Sending Quote']
L-0011svc=frontend-proxy sev=UNSET base=1627.0/min fault=779.2/min tpl="<*> <*> <*> HTTP/<NUM>" <NUM> <*> <*> - "-" <NUM> <NUM> <NUM> <*> "-" "Mozilla/<NUM> (X11; Linux x86_64) AppleWebKit/<NUM> (KHTML, like Gec…" ex=['[2026-11-09T02:03:09.980Z] "POST /flagservice/flagd.evaluation.v1.Service/ResolveAll HTTP/1.1" 200 - via_upstream - "-" 136 364 604 604 "-"…', '[2026-11-09T02:03:12.807Z] "GET /cart HTTP/1.1" 200 - via_upstream - "-" 0 2972 4 4 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36…']
L-0012svc=checkout sev=INFO base=12.2/min fault=5.4/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0013svc=product-catalog sev=INFO base=582.0/min fault=287.0/min tpl="Product Found" ex=['Product Found', 'Product Found']
L-0014svc=recommendation sev=INFO base=204.4/min fault=101.2/min tpl="Receive ListRecommendations for product <*> <*> <*> <*> <*>" ex=["Receive ListRecommendations for product ids:['1YMWWN1N4O', 'HQTGWGPNH4', 'LS4PSXUNUM', 'OLJCESPC7Z', '66VCHSJNUP']", "Receive ListRecommendations for product ids:['1YMWWN1N4O', 'HQTGWGPNH4', 'LS4PSXUNUM', 'OLJCESPC7Z', '66VCHSJNUP']"]
L-0015svc=cart sev=INFORMATION base=51.0/min fault=25.0/min tpl="AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}" ex=['AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}', 'AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}']
L-0016svc=frontend sev=INFO base=497.0/min fault=249.8/min tpl="API request completed" ex=['API request completed', 'API request completed']
L-0017svc=currency sev=INFO base=103.4/min fault=51.6/min tpl="conversion successful" ex=['conversion successful', 'conversion successful']
L-0018svc=cart sev=INFORMATION base=165.8/min fault=84.2/min tpl="GetCartAsync called with userId={userId}" ex=['GetCartAsync called with userId={userId}', 'GetCartAsync called with userId={userId}']
L-0019svc=product-catalog sev=INFO base=129.4/min fault=66.6/min tpl="Found <NUM> products from database" ex=['Found 10 products from database', 'Found 10 products from database']
L-0020svc=load-generator sev=INFO base=14.8/min fault=7.2/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0021svc=accounting sev=INFORMATION base=12.2/min fault=6.0/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0022svc=checkout sev=INFO base=12.2/min fault=6.0/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 270.892µs', 'Successful to write message. offset: 0, duration: 275.443µs']
L-0023svc=checkout sev=INFO base=12.2/min fault=6.0/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0024svc=email sev=INFO base=12.2/min fault=6.0/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0025svc=fraud-detection sev=INFO base=12.2/min fault=6.0/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: c9af9b7d-dbe1-11f1-82f3-d279f8fa1f6d, and updated total count to: 20000', 'Consumed record with orderId: ca0421d5-dbe1-11f1-82f3-d279f8fa1f6d, and updated total count to: 20001']
L-0026svc=frontend sev=INFO base=12.2/min fault=6.0/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0027svc=cart sev=INFORMATION base=12.0/min fault=6.0/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
37 lower-ranked lines omitted
Loading runs in your browser…