paymentFailure-r1
What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)
Service graph during the window
Evidence and answers
5781 prompt tokens
metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean
M-0001svc=ad metric=error_ratio base=0 delta/30s=[+0%,na,new:0.0101,na,+0%,na,+0%,na,+0%,na]
M-0002svc=ad metric=jvm.thread.count.rate base=0 delta/30s=[new:0.0333,na,new:0.0333,na,+0%,na,+0%,na,new:0.0333,na]
M-0003svc=ad metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0004svc=ad metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0005svc=ad metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0006svc=ad metric=traces.span.metrics.duration.p50 base=1.01 delta/30s=[-1%,na,+1488450%,na,+1%,na,+1%,na,-1%,na]
M-0007svc=ad metric=traces.span.metrics.duration.p95 base=1.91 delta/30s=[-1%,na,+783347%,na,+1%,na,+1%,na,-1%,na]
M-0008svc=ad metric=traces.span.metrics.duration.p99 base=2.28 delta/30s=[-13%,na,+658951%,na,+28%,na,+115%,na,-13%,na]
M-0009svc=checkout metric=error_ratio base=0 delta/30s=[new:0.0309,na,new:0.0865,na,new:0.1,na,new:0.0698,na,new:0.0595,na]
M-0010svc=checkout metric=go.goroutine.count.rate base=0 delta/30s=[+0%,na,new:0.0667,na,+0%,na,new:0.0667,na,+0%,na]
M-0011svc=fraud-detection metric=error_ratio base=0 delta/30s=[+0%,na,new:0.0526,na,+0%,na,+0%,na,+0%,na]
M-0012svc=fraud-detection metric=jvm.thread.count.rate base=0 delta/30s=[new:0.0333,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0013svc=fraud-detection metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0014svc=fraud-detection metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0015svc=fraud-detection metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0016svc=frontend metric=error_ratio base=0 delta/30s=[new:0.00147,na,new:0.00448,na,new:0.00419,na,new:0.00327,na,new:0.00244,na]
M-0017svc=frontend-proxy metric=error_ratio base=0 delta/30s=[new:0.000944,na,new:0.00189,na,new:0.00194,na,new:0.00166,na,new:0.000936,na]
M-0018svc=kafka metric=jvm.gc.duration.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0019svc=kafka metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:2,na,+0%,na]
M-0020svc=load-generator metric=error_ratio base=0 delta/30s=[new:0.00609,na,new:0.0186,na,new:0.0176,na,new:0.0123,na,new:0.00969,na]
M-0021svc=otelcol-contrib metric=otelcol_exporter_in_flight_requests base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:1,na,new:1,na]
M-0022svc=quote metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[na,+0%,na,new:2,na,new:1,na,+0%,na,new:4]
M-0023svc=recommendation metric=process.runtime.cpython.cpu.utilization base=0 delta/30s=[new:5.71,na,new:12.4,na,+0%,na,+0%,na,+0%,na]
M-0024svc=fraud-detection metric=traces.span.metrics.duration.p50 base=52.1 delta/30s=[-38%,na,+28709%,na,+8%,na,-4%,na,-36%,na]
M-0025svc=fraud-detection metric=traces.span.metrics.duration.p95 base=106.9 delta/30s=[-19%,na,+13929%,na,-11%,na,-11%,na,-15%,na]
M-0026svc=fraud-detection metric=traces.span.metrics.duration.p99 base=117.4 delta/30s=[-17%,na,+12678%,na,-16%,na,-16%,na,-16%,na]
M-0027svc=payment metric=error_ratio base=0.0105 delta/30s=[+1210%,na,+4900%,na,+4967%,na,+4056%,na,+2433%,na]
M-0028svc=unknown metric=system.disk.pending_operations base=0.2 delta/30s=[-100%,na,-100%,na,-100%,na,+4900%,na,-100%,na]
M-0029svc=frontend metric=http.server.request.duration.p50 base=0.00648 delta/30s=[na,+209%,na,+2602%,na,+257%,na,+672%,na,+672%]
M-0030svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,-100%,+900%,-100%,-100%,-100%,-100%,-100%,-100%,-100%]
M-0031svc=unknown metric=system.processes.count.rate base=0.0867 delta/30s=[-23%,na,+54%,na,+54%,na,+823%,na,+900%,na]
M-0032svc=flagd metric=error_ratio base=0.0001 delta/30s=[-100%,na,+812%,na,-100%,na,-100%,na,-100%,na]
M-0033svc=checkout metric=rpc.client.call.duration.p99 base=0.0656 delta/30s=[+621%,na,-64%,na,-85%,na,-64%,na,-27%,na]
M-0034svc=checkout metric=rpc.client.call.duration.p95 base=0.0518 delta/30s=[+600%,na,-65%,na,-82%,na,-63%,na,-25%,na]
M-0035svc=load-generator metric=http.client.duration.p50 base=8.35 delta/30s=[+155%,na,+349%,na,+163%,na,+499%,na,+349%,na]
M-0036svc=load-generator metric=process.runtime.cpython.cpu.utilization base=3.58 delta/30s=[+437%,na,-100%,na,-100%,na,-100%,na,-100%,na]
856 lower-ranked lines omitted
traces: slowest and errored traces after fault onset, critical path (! = error span)
T-0001trace=43bbbc55 dur=2507ms ERROR path: load-generator:user_checkout_multi 960ms! > frontend-proxy:POST 908ms! > frontend:POST /api/checkout 908ms! > checkout:oteldemo.CheckoutService/PlaceOrder 905ms! > payment:oteldemo.PaymentService/Charge 1ms!
T-0002trace=8e7f9be6 dur=31553ms ok path: accounting:order-consumed 31553ms
T-0003trace=416ffd75 dur=2500ms ERROR path: load-generator:user_checkout_single 876ms! > frontend-proxy:POST 842ms! > frontend:POST /api/checkout 841ms! > checkout:oteldemo.CheckoutService/PlaceOrder 838ms! > payment:oteldemo.PaymentService/Charge 1ms!
T-0004trace=5ae6f4ee dur=19659ms ok path: accounting:order-consumed 19659ms
T-0005trace=72cc29e1 dur=2232ms ERROR path: load-generator:user_checkout_multi 885ms! > frontend-proxy:POST 795ms! > frontend:POST /api/checkout 794ms! > checkout:oteldemo.CheckoutService/PlaceOrder 792ms! > payment:oteldemo.PaymentService/Charge 1ms!
T-0006trace=b3138ca8 dur=18757ms ok path: accounting:order-consumed 18757ms
T-0007trace=1ddd2d01 dur=2202ms ERROR path: load-generator:user_checkout_multi 664ms! > frontend-proxy:POST 621ms! > frontend:POST /api/checkout 620ms! > checkout:oteldemo.CheckoutService/PlaceOrder 618ms! > payment:oteldemo.PaymentService/Charge 0ms!
T-0008trace=e3003757 dur=16935ms ok path: accounting:order-consumed 16935ms
T-0009trace=044f3e6c dur=2200ms ERROR path: load-generator:user_checkout_multi 1274ms! > frontend-proxy:POST 1211ms! > frontend:POST /api/checkout 1210ms! > checkout:oteldemo.CheckoutService/PlaceOrder 1208ms! > payment:oteldemo.PaymentService/Charge 0ms!
T-0010trace=6348bbe9 dur=16856ms ok path: accounting:order-consumed 16856ms
T-0011trace=bd7f05ce dur=2150ms ERROR path: load-generator:user_checkout_single 485ms! > frontend-proxy:POST 467ms! > frontend:POST /api/checkout 467ms! > checkout:oteldemo.CheckoutService/PlaceOrder 464ms! > payment:oteldemo.PaymentService/Charge 2ms!
T-0012trace=c6cb1f8e dur=16236ms ok path: accounting:order-consumed 16236ms
T-0013trace=14b8962d dur=2100ms ERROR path: load-generator:user_checkout_single 309ms! > frontend-proxy:POST 287ms! > frontend:POST /api/checkout 286ms! > checkout:oteldemo.CheckoutService/PlaceOrder 282ms! > payment:oteldemo.PaymentService/Charge 2ms!
T-0014trace=d5081077 dur=15872ms ok path: accounting:order-consumed 15872ms
T-0015trace=5d99707a dur=1984ms ERROR path: load-generator:user_checkout_single 168ms! > frontend-proxy:POST 149ms! > frontend:POST /api/checkout 148ms! > checkout:oteldemo.CheckoutService/PlaceOrder 146ms! > payment:oteldemo.PaymentService/Charge 1ms!
T-0016trace=fb551a55 dur=14130ms ok path: accounting:order-consumed 14130ms
T-0017trace=314f46ba dur=1971ms ERROR path: load-generator:user_checkout_single 117ms! > frontend-proxy:POST 99ms! > frontend:POST /api/checkout 98ms! > checkout:oteldemo.CheckoutService/PlaceOrder 96ms! > payment:oteldemo.PaymentService/Charge 1ms!
T-0018trace=09ef2f68 dur=12464ms ok path: accounting:order-consumed 12464ms
T-0019trace=ca75323b dur=1949ms ERROR path: load-generator:user_checkout_single 52ms! > frontend-proxy:POST 38ms! > frontend:POST /api/checkout 37ms! > checkout:oteldemo.CheckoutService/PlaceOrder 35ms! > payment:oteldemo.PaymentService/Charge 24ms!
T-0020trace=5d3755e3 dur=11457ms ok path: accounting:order-consumed 11457ms
T-0021trace=324e4a50 dur=1933ms ERROR path: load-generator:user_checkout_multi 243ms! > frontend-proxy:POST 189ms! > frontend:POST /api/checkout 188ms! > checkout:oteldemo.CheckoutService/PlaceOrder 183ms! > payment:oteldemo.PaymentService/Charge 1ms!
T-0022trace=ddc77104 dur=11361ms ok path: accounting:order-consumed 11361ms
T-0023trace=6dc3bc90 dur=1931ms ERROR path: load-generator:user_checkout_single 231ms! > frontend-proxy:POST 210ms! > frontend:POST /api/checkout 210ms! > checkout:oteldemo.CheckoutService/PlaceOrder 208ms! > payment:oteldemo.PaymentService/Charge 3ms!
T-0024trace=4830e300 dur=9928ms ok path: accounting:order-consumed 9928ms
T-0025trace=b8f62048 dur=1930ms ERROR path: load-generator:user_checkout_multi 81ms! > frontend-proxy:POST 31ms! > frontend:POST /api/checkout 31ms! > checkout:oteldemo.CheckoutService/PlaceOrder 29ms! > payment:oteldemo.PaymentService/Charge 2ms!
T-0026trace=8301911d dur=9917ms ok path: accounting:order-consumed 9917ms
T-0027trace=b52dd95b dur=1928ms ERROR path: load-generator:user_checkout_multi 81ms! > frontend-proxy:POST 31ms! > frontend:POST /api/checkout 31ms! > checkout:oteldemo.CheckoutService/PlaceOrder 28ms! > payment:oteldemo.PaymentService/Charge 4ms!
T-0028trace=a6d81f5c dur=8217ms ok path: accounting:order-consumed 8217ms
T-0029trace=c9db3856 dur=1928ms ERROR path: load-generator:user_checkout_multi 54ms! > frontend-proxy:POST 22ms! > frontend:POST /api/checkout 21ms! > checkout:oteldemo.CheckoutService/PlaceOrder 19ms! > payment:oteldemo.PaymentService/Charge 1ms!
T-0030trace=64c39b4f dur=7962ms ok path: accounting:order-consumed 7962ms
T-0031trace=af8e52c3 dur=1927ms ERROR path: load-generator:user_checkout_multi 85ms! > frontend-proxy:POST 22ms! > frontend:POST /api/checkout 21ms! > checkout:oteldemo.CheckoutService/PlaceOrder 19ms! > payment:oteldemo.PaymentService/Charge 1ms!
T-0032trace=d3f3c31f dur=7419ms ok path: accounting:order-consumed 7419ms
8 lower-ranked lines omitted
logs: log templates per service, rate before vs after fault onset, with examples
L-0001svc=frontend sev=INFO base=0.0/min fault=6.8/min tpl="Checkout payment declined" ex=['Checkout payment declined', 'Checkout payment declined']
L-0002svc=payment sev=WARN base=0.0/min fault=6.8/min tpl="Payment request failed. Invalid token. demo.user_context.loyalty_level=gold" ex=['Payment request failed. Invalid token. demo.user_context.loyalty_level=gold', 'Payment request failed. Invalid token. demo.user_context.loyalty_level=gold']
L-0003svc=load-generator sev=INFO base=10.4/min fault=16.8/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user 3252ec80-dbf8-11f1-8306-d6b650b5399b', 'Checkout completed for user 3252ec80-dbf8-11f1-8306-d6b650b5399b']
L-0004svc=checkout sev=INFO base=9.8/min fault=15.4/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0005svc=payment sev=INFO base=9.8/min fault=15.4/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0006svc=load-generator sev=INFO base=9.2/min fault=14.0/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user 2c6ec294-dbf8-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user 2c6ec294-dbf8-11f1-8306-d6b650b5399b']
L-0007svc=load-generator sev=INFO base=16.4/min fault=12.0/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0008svc=shipping sev=INFO base=1.4/min fault=0.8/min tpl="" ex=['', '']
L-0009svc=load-generator sev=INFO base=67.2/min fault=84.8/min tpl="User <UUID> adding <NUM> of product <*> to cart" ex=['User 2c6ec294-dbf8-11f1-8306-d6b650b5399b adding 10 of product OLJCESPC7Z to cart', 'User 2c6ec294-dbf8-11f1-8306-d6b650b5399b adding 10 of product OLJCESPC7Z to cart']
L-0010svc=quote sev=INFO base=25.8/min fault=32.2/min tpl="Calculated quote" ex=['Calculated quote', 'Calculated quote']
L-0011svc=shipping sev=INFO base=25.8/min fault=32.2/min tpl="Requesting quote" ex=['Requesting quote', 'Requesting quote']
L-0012svc=shipping sev=INFO base=25.8/min fault=32.2/min tpl="Sending Quote" ex=['Sending Quote', 'Sending Quote']
L-0013svc=checkout sev=INFO base=0.0/min fault=0.2/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0014svc=cart sev=INFORMATION base=49.6/min fault=59.2/min tpl="AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}" ex=['AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}', 'AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}']
L-0015svc=currency sev=INFO base=99.8/min fault=114.2/min tpl="conversion successful" ex=['conversion successful', 'conversion successful']
L-0016svc=accounting sev=INFORMATION base=9.8/min fault=8.6/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0017svc=cart sev=INFORMATION base=9.8/min fault=8.6/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0018svc=checkout sev=INFO base=9.8/min fault=8.6/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 275.739µs', 'Successful to write message. offset: 0, duration: 162.605µs']
L-0019svc=checkout sev=INFO base=9.8/min fault=8.6/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0020svc=checkout sev=INFO base=9.8/min fault=8.6/min tpl="order placed" ex=['order placed', 'order placed']
L-0021svc=checkout sev=INFO base=9.8/min fault=8.6/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0022svc=checkout sev=INFO base=9.8/min fault=8.6/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0023svc=email sev=INFO base=9.8/min fault=8.6/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0024svc=fraud-detection sev=INFO base=9.8/min fault=8.6/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: 2c75c99b-dbf8-11f1-82f3-d279f8fa1f6d, and updated total count to: 21984', 'Consumed record with orderId: 329ed55e-dbf8-11f1-82f3-d279f8fa1f6d, and updated total count to: 21985']
L-0025svc=frontend sev=INFO base=9.8/min fault=8.6/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0026svc=payment sev=INFO base=9.8/min fault=8.6/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0027svc=shipping sev=INFO base=9.8/min fault=8.6/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0028svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=68545, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=68695, siz…']
L-0029svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 68763 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 68880 breach: LogSegment(baseOff…']
33 lower-ranked lines omitted
Loading runs in your browser…