productCatalogLockContention-r1
What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)
Service graph during the window
Evidence and answers
5773 prompt tokens
metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean
M-0001svc=ad metric=jvm.gc.duration.count.rate base=0 delta/30s=[+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%,na]
M-0002svc=cart metric=dotnet.thread_pool.thread.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0003svc=cart metric=process.thread.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0004svc=checkout metric=error_ratio base=0 delta/30s=[+0%,na,+0%,na,new:0.0625,na,+0%,na,new:0.00971,na]
M-0005svc=checkout metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:1,na,+0%,na]
M-0006svc=checkout metric=otel.sdk.span.live base=0 delta/30s=[+0%,na,new:4,na,+0%,na,new:5,na,+0%,na]
M-0007svc=frontend-proxy metric=error_ratio base=0 delta/30s=[+0%,na,new:0.0195,na,new:0.00948,na,new:0.0197,na,new:0.0111,na]
M-0008svc=load-generator metric=error_ratio base=0 delta/30s=[+0%,na,new:0.127,na,new:0.0742,na,new:0.116,na,new:0.0709,na]
M-0009svc=payment metric=error_ratio base=0 delta/30s=[+0%,na,new:0.0714,na,+0%,na,+0%,na,+0%,na]
M-0010svc=payment metric=traces.span.metrics.duration.p50 base=1.23 delta/30s=[-2%,na,+1219921%,na,+42%,na,-19%,na,+36%,na]
M-0011svc=payment metric=traces.span.metrics.duration.p95 base=7.45 delta/30s=[-1%,na,+201242%,na,-2%,na,-74%,na,+303%,na]
M-0012svc=payment metric=traces.span.metrics.duration.p99 base=12.7 delta/30s=[-38%,na,+118103%,na,-38%,na,-84%,na,+262%,na]
M-0013svc=product-catalog metric=db.client.operation.duration.p95 base=0.00408 delta/30s=[+0%,na,+244949%,na,+244949%,na,+244949%,na,+244949%,na]
M-0014svc=product-catalog metric=db.client.operation.duration.p99 base=0.00482 delta/30s=[-0%,na,+207386%,na,+207386%,na,+207386%,na,+207386%,na]
M-0015svc=recommendation metric=process.runtime.cpython.cpu.utilization base=0 delta/30s=[new:8.87,na,+0%,na,+0%,na,new:13.7,na,+0%,na]
M-0016svc=unknown metric=postgresql.rollbacks base=0 delta/30s=[+0%,new:1,new:2,new:2,new:3,new:4,new:5,new:5,new:6,new:7]
M-0017svc=load-generator metric=http.client.duration.p50 base=23.7 delta/30s=[-13%,na,+42105%,na,+42105%,na,+42105%,na,+42105%,na]
M-0018svc=product-catalog metric=otel.sdk.span.live base=0.8 delta/30s=[+4900%,na,+9150%,na,+5900%,na,+8400%,na,+8400%,na]
M-0019svc=flagd metric=error_ratio base=0.00136 delta/30s=[-100%,na,+2014%,na,+4353%,na,+6288%,na,+8711%,na]
M-0020svc=email metric=traces.span.metrics.duration.p50 base=45.8 delta/30s=[-42%,na,+173%,na,+5356%,na,+409%,na,+1100%,na]
M-0021svc=cart metric=http.server.request.duration.p50 base=0.0663 delta/30s=[-34%,na,+23%,na,+2542%,na,+655%,na,+4931%,na]
M-0022svc=frontend metric=traces.span.metrics.duration.p50 base=312.7 delta/30s=[-4%,na,+4298%,na,+4697%,na,+2299%,na,+4697%,na]
M-0023svc=checkout metric=traces.span.metrics.duration.p50 base=253.3 delta/30s=[-8%,na,+689%,na,+4341%,na,+1282%,na,+1874%,na]
M-0024svc=checkout metric=http.client.request.duration.p50 base=0.0394 delta/30s=[-51%,na,+281%,na,+4340%,na,+851%,na,+4340%,na]
M-0025svc=product-catalog metric=rpc.server.call.duration.p50 base=0.0141 delta/30s=[-28%,na,+3124%,na,+3970%,na,+3898%,na,+4146%,na]
M-0026svc=cart metric=traces.span.metrics.duration.p50 base=76.2 delta/30s=[+425%,na,+15%,na,+2329%,na,+688%,na,+3839%,na]
M-0027svc=product-catalog metric=traces.span.metrics.duration.p95 base=431.7 delta/30s=[-36%,na,+3375%,na,+3375%,na,+3375%,na,+3375%,na]
M-0028svc=checkout metric=rpc.client.call.duration.p50 base=0.0615 delta/30s=[-9%,na,+510%,na,+1831%,na,+916%,na,+3152%,na]
M-0029svc=recommendation metric=otel.sdk.span.live base=1 delta/30s=[+0%,na,+0%,na,+0%,na,+3000%,na,+0%,na]
M-0030svc=product-catalog metric=traces.span.metrics.duration.p50 base=18.6 delta/30s=[-46%,na,+1915%,na,+2769%,na,+2891%,na,+2705%,na]
M-0031svc=checkout metric=go.goroutine.count.rate base=0.0133 delta/30s=[-100%,na,+1400%,na,-100%,na,+2400%,na,-100%,na]
M-0032svc=checkout metric=rpc.server.call.duration.p50 base=0.278 delta/30s=[-23%,na,+800%,na,+2376%,na,+1250%,na,+1250%,na]
M-0033svc=product-catalog metric=rpc.server.call.duration.p95 base=0.42 delta/30s=[-29%,na,+2280%,na,+2280%,na,+2280%,na,+2280%,na]
M-0034svc=product-catalog metric=traces.span.metrics.duration.p99 base=704.3 delta/30s=[-29%,na,+2030%,na,+2030%,na,+2030%,na,+2030%,na]
M-0035svc=product-catalog metric=db.sql.connection.open base=2 delta/30s=[+950%,na,+1800%,na,+1150%,na,+1650%,na,+1650%,na]
M-0036svc=load-generator metric=otel.sdk.span.live base=1.8 delta/30s=[+1456%,na,+1400%,na,+1678%,na,+1344%,na,+1567%,na]
874 lower-ranked lines omitted
traces: slowest and errored traces after fault onset, critical path (! = error span)
T-0001trace=db929688 dur=33043ms ERROR path: load-generator:user_checkout_single 30014ms! > frontend-proxy:POST 15003ms! > frontend:POST 15004ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 502ms | errors off path: frontend-proxy:GET, load-generator:GET
T-0002trace=6fec178a dur=42746ms ok path: accounting:order-consumed 42746ms
T-0003trace=d928765e dur=32635ms ERROR path: load-generator:user_checkout_multi 32635ms > frontend-proxy:POST 1987ms > frontend:POST /api/checkout 1985ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 87ms | errors off path: frontend-proxy:GET, load-generator:GET
T-0004trace=eb8f0bb0 dur=39263ms ok path: accounting:order-consumed 39263ms
T-0005trace=8e7563d5 dur=32579ms ERROR path: load-generator:user_checkout_multi 32242ms > frontend-proxy:POST 1900ms > frontend:POST /api/checkout 1900ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 321ms | errors off path: frontend-proxy:GET, load-generator:GET
T-0006trace=530c7c8b dur=38011ms ok path: accounting:order-consumed 38011ms
T-0007trace=5b2e8b6d dur=32154ms ERROR path: load-generator:user_checkout_single 30014ms! > frontend-proxy:POST 15003ms! > frontend:POST 15003ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 502ms | errors off path: checkout:POST, frontend-proxy:GET, load-generator:GET
T-0008trace=5ed1eeb2 dur=37699ms ok path: accounting:order-consumed 37699ms
T-0009trace=3fb1b96e dur=31420ms ERROR path: load-generator:user_checkout_multi 31420ms > frontend-proxy:POST 2801ms > frontend:POST /api/checkout 2793ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 193ms | errors off path: frontend-proxy:GET, load-generator:GET
T-0010trace=a52ccea0 dur=36872ms ok path: accounting:order-consumed 36872ms
T-0011trace=90140342 dur=31344ms ERROR path: load-generator:user_checkout_multi 31285ms > frontend-proxy:POST 1508ms > frontend:POST /api/checkout 1508ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 95ms | errors off path: frontend-proxy:GET, load-generator:GET
T-0012trace=91276a87 dur=36844ms ok path: accounting:order-consumed 36844ms
T-0013trace=d0e1265f dur=31163ms ERROR path: load-generator:user_checkout_single 29830ms > frontend-proxy:POST 14816ms > frontend:POST /api/checkout 14818ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 502ms | errors off path: checkout:POST, frontend-proxy:GET, load-generator:GET
T-0014trace=611ef62c dur=31737ms ok path: accounting:order-consumed 31737ms
T-0015trace=81d8165e dur=30456ms ERROR path: load-generator:user_checkout_single 15117ms! > frontend-proxy:POST 15004ms! > frontend:POST 15004ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 29204ms
T-0016trace=77745cd2 dur=30489ms ok path: frontend-proxy:GET 3072ms > frontend:GET 3073ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 599ms
T-0017trace=05f9ac0e dur=30236ms ERROR path: load-generator:user_get_recommendations 15001ms! > frontend-proxy:GET 14999ms! > frontend:GET 15000ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 529ms
T-0018trace=88f80b86 dur=30347ms ok path: frontend-proxy:GET 3020ms > frontend:GET 3020ms > currency:oteldemo.CurrencyService/Convert 0ms
T-0019trace=3aa08014 dur=30120ms ERROR path: load-generator:user_browse_product 15002ms! > frontend-proxy:GET 14999ms! > frontend:GET 14998ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 30107ms
T-0020trace=8f00551f dur=30315ms ok path: frontend-proxy:GET 3065ms > frontend:GET 3065ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 517ms
T-0021trace=f5c475eb dur=30070ms ERROR path: load-generator:user_browse_product 15001ms! > frontend-proxy:GET 14998ms! > frontend:GET 14997ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 30051ms
T-0022trace=3c48a33f dur=30247ms ok path: frontend-proxy:GET 3023ms > frontend:GET 3024ms > currency:oteldemo.CurrencyService/Convert 0ms
T-0023trace=c424bebb dur=29965ms ERROR path: load-generator:user_add_to_cart 15011ms > frontend-proxy:POST 4ms > frontend:POST /api/cart 4ms > cart:POST /oteldemo.CartService/GetCart 0ms | errors off path: frontend-proxy:GET, load-generator:GET
T-0024trace=06a1c2c9 dur=30246ms ok path: frontend-proxy:GET 3090ms > frontend:GET 3090ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 512ms
T-0025trace=1e0a02f0 dur=29962ms ERROR path: load-generator:user_add_to_cart 15007ms > frontend-proxy:POST 5ms > frontend:POST /api/cart 4ms > cart:POST /oteldemo.CartService/GetCart 0ms | errors off path: frontend-proxy:GET, load-generator:GET
T-0026trace=a0a45201 dur=30216ms ok path: frontend-proxy:GET 3023ms > frontend:GET 3023ms > currency:oteldemo.CurrencyService/Convert 0ms
T-0027trace=9e533744 dur=29848ms ERROR path: load-generator:user_browse_product 15001ms! > frontend-proxy:GET 14998ms! > frontend:GET 14999ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 29842ms
T-0028trace=e27fc21b dur=30213ms ok path: frontend-proxy:GET 3078ms > frontend:GET 3079ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 30192ms
T-0029trace=56b1bc1a dur=29682ms ERROR path: load-generator:user_browse_product 15001ms! > frontend-proxy:GET 14999ms! > frontend:GET 15000ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 29661ms
T-0030trace=d240ad2d dur=30199ms ok path: frontend-proxy:GET 3062ms > frontend:GET 3061ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 30169ms
10 lower-ranked lines omitted
logs: log templates per service, rate before vs after fault onset, with examples
L-0001svc=load-generator sev=ERROR base=0.0/min fault=5.6/min tpl="Error in add to cart task: Timeout <NUM> exceeded while waiting for event "response" =========================== logs =====================…" ex=['Error in add to cart task: Timeout 15000ms exceeded while waiting for event "response" =========================== logs ===================…', 'Error in add to cart task: Timeout 15000ms exceeded while waiting for event "response" =========================== logs ===================…']
L-0002svc=load-generator sev=INFO base=29.2/min fault=10.0/min tpl="Product added to cart successfully" ex=['Product added to cart successfully', 'Product added to cart successfully']
L-0003svc=product-catalog sev=INFO base=0.0/min fault=1.6/min tpl="lock contention scenario active: holding ACCESS EXCLUSIVE lock on catalog.products" ex=['lock contention scenario active: holding ACCESS EXCLUSIVE lock on catalog.products', 'lock contention scenario active: holding ACCESS EXCLUSIVE lock on catalog.products']
L-0004svc=checkout sev=INFO base=14.0/min fault=4.8/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0005svc=quote sev=INFO base=28.8/min fault=10.6/min tpl="Calculated quote" ex=['Calculated quote', 'Calculated quote']
L-0006svc=shipping sev=INFO base=28.8/min fault=10.6/min tpl="Requesting quote" ex=['Requesting quote', 'Requesting quote']
L-0007svc=shipping sev=INFO base=28.8/min fault=10.6/min tpl="Sending Quote" ex=['Sending Quote', 'Sending Quote']
L-0008svc=load-generator sev=INFO base=13.6/min fault=4.8/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user 4ccf130c-dbcd-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user 4ccf130c-dbcd-11f1-8306-d6b650b5399b']
L-0009svc=ad sev=INFO base=44.4/min fault=17.4/min tpl="Targeted ad request received for <*>" ex=['Targeted ad request received for [assembly]', 'Targeted ad request received for [accessories]']
L-0010svc=cart sev=INFORMATION base=55.8/min fault=22.2/min tpl="AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}" ex=['AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}', 'AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}']
L-0011svc=frontend sev=INFO base=14.0/min fault=5.6/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0012svc=accounting sev=INFORMATION base=14.0/min fault=5.8/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0013svc=cart sev=INFORMATION base=14.0/min fault=5.8/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0014svc=checkout sev=INFO base=14.0/min fault=5.8/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 167.617µs', 'Successful to write message. offset: 0, duration: 166.497µs']
L-0015svc=checkout sev=INFO base=14.0/min fault=5.8/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0016svc=checkout sev=INFO base=14.0/min fault=5.8/min tpl="order placed" ex=['order placed', 'order placed']
L-0017svc=checkout sev=INFO base=14.0/min fault=5.8/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0018svc=checkout sev=INFO base=14.0/min fault=5.8/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0019svc=email sev=INFO base=14.0/min fault=5.8/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0020svc=fraud-detection sev=INFO base=14.0/min fault=5.8/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: 48e31187-dbcd-11f1-82f3-d279f8fa1f6d, and updated total count to: 10808', 'Consumed record with orderId: 48fbf467-dbcd-11f1-82f3-d279f8fa1f6d, and updated total count to: 10809']
L-0021svc=payment sev=INFO base=14.0/min fault=5.8/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0022svc=payment sev=INFO base=14.0/min fault=5.8/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0023svc=shipping sev=INFO base=14.0/min fault=5.8/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0024svc=load-generator sev=INFO base=82.0/min fault=36.8/min tpl="User <UUID> adding <NUM> of product <*> to cart" ex=['User 471cfcc6-dbcd-11f1-8306-d6b650b5399b adding 3 of product 9SIQT8TOJO to cart', 'User 471cfcc6-dbcd-11f1-8306-d6b650b5399b adding 3 of product 9SIQT8TOJO to cart']
L-0025svc=load-generator sev=INFO base=14.4/min fault=6.4/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user 48c15554-dbcd-11f1-8306-d6b650b5399b', 'Checkout completed for user 48c15554-dbcd-11f1-8306-d6b650b5399b']
L-0026svc=product-catalog sev=INFO base=642.4/min fault=310.8/min tpl="Product Found" ex=['Product Found', 'Product Found']
L-0027svc=load-generator sev=INFO base=92.4/min fault=44.4/min tpl="User getting <*> for <*> <*>" ex=['User getting ads for category: assembly', 'User getting ads for category: assembly']
L-0028svc=frontend sev=INFO base=536.8/min fault=267.2/min tpl="API request completed" ex=['API request completed', 'API request completed']
L-0029svc=recommendation sev=INFO base=230.4/min fault=114.4/min tpl="Receive ListRecommendations for product <*> <*> <*> <*> <*>" ex=["Receive ListRecommendations for product ids:['LS4PSXUNUM', '1YMWWN1N4O', 'HQTGWGPNH4', '0PUK6V6EV0', '66VCHSJNUP']", "Receive ListRecommendations for product ids:['LS4PSXUNUM', '1YMWWN1N4O', 'HQTGWGPNH4', '0PUK6V6EV0', '66VCHSJNUP']"]
36 lower-ranked lines omitted
Loading runs in your browser…