paymentFailure-r2
What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)
Service graph during the window
Evidence and answers
5737 prompt tokens
metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean
M-0001svc=ad metric=error_ratio base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0099,na,+0%,na]
M-0002svc=ad metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[+0%,na,+0%,na,new:2,na,+0%,na,+0%,na]
M-0003svc=ad metric=otel.sdk.processor.span.queue.size base=0 delta/30s=[+0%,na,+0%,na,new:4,na,+0%,na,+0%,na]
M-0004svc=ad metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0005svc=ad metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0006svc=ad metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0007svc=ad metric=traces.span.metrics.duration.p50 base=1 delta/30s=[-0%,na,+2%,na,-0%,na,+1493544%,na,-0%,na]
M-0008svc=ad metric=traces.span.metrics.duration.p95 base=1.91 delta/30s=[-0%,na,+2%,na,-0%,na,+786028%,na,-0%,na]
M-0009svc=ad metric=traces.span.metrics.duration.p99 base=2.19 delta/30s=[-10%,na,+38%,na,-10%,na,+684207%,na,-10%,na]
M-0010svc=cart metric=dotnet.thread_pool.thread.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0011svc=cart metric=process.thread.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0012svc=checkout metric=error_ratio base=0 delta/30s=[+0%,na,new:0.116,na,new:0.127,na,new:0.106,na,new:0.0588,na]
M-0013svc=fraud-detection metric=error_ratio base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0769,na,+0%,na]
M-0014svc=fraud-detection metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0015svc=fraud-detection metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0016svc=fraud-detection metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0017svc=frontend metric=error_ratio base=0 delta/30s=[+0%,na,new:0.00541,na,new:0.00546,na,new:0.00456,na,new:0.00265,na]
M-0018svc=frontend-proxy metric=error_ratio base=0 delta/30s=[+0%,na,new:0.00266,na,new:0.0029,na,new:0.00174,na,new:0.00124,na]
M-0019svc=kafka metric=jvm.thread.count.rate base=0 delta/30s=[+0%,na,+0%,na,new:0.0333,na,new:0.0333,na,+0%,na]
M-0020svc=load-generator metric=error_ratio base=0 delta/30s=[new:0.002,na,new:0.0218,na,new:0.0213,na,new:0.0185,na,new:0.0101,na]
M-0021svc=product-catalog metric=go.goroutine.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.133,na]
M-0022svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0 delta/30s=[new:0.0333,+0%,+0%,+0%,+0%,+0%,+0%,+0%,+0%,+0%]
M-0023svc=fraud-detection metric=traces.span.metrics.duration.p50 base=57.2 delta/30s=[-5%,na,-21%,na,-13%,na,+26139%,na,+5%,na]
M-0024svc=fraud-detection metric=traces.span.metrics.duration.p95 base=95.7 delta/30s=[-0%,na,-2%,na,-1%,na,+15571%,na,+0%,na]
M-0025svc=fraud-detection metric=traces.span.metrics.duration.p99 base=99.1 delta/30s=[-0%,na,-0%,na,-0%,na,+15030%,na,+0%,na]
M-0026svc=payment metric=error_ratio base=0.00741 delta/30s=[-100%,na,+7927%,na,+9543%,na,+8000%,na,+5092%,na]
M-0027svc=frontend metric=http.server.request.duration.p50 base=0.00652 delta/30s=[+2199%,na,+666%,na,+411%,na,+4690%,na,+666%,na]
M-0028svc=accounting metric=dotnet.gc.last_collection.heap.fragmentation.size base=8.1e+03 delta/30s=[+7%,na,+7%,na,+7%,na,+7%,na,+2547%,na]
M-0029svc=flagd metric=error_ratio base=0.00216 delta/30s=[+103%,na,-77%,na,-55%,na,+953%,na,+73%,na]
M-0030svc=ad metric=jvm.thread.count.rate base=0.00667 delta/30s=[+400%,na,-100%,na,-100%,na,+900%,na,-100%,na]
M-0031svc=quote metric=otel.sdk.processor.log.queue.size base=0.2 delta/30s=[-100%,na,-100%,na,-100%,na,+900%,na,-100%,na]
M-0032svc=load-generator metric=http.client.duration.p50 base=41.8 delta/30s=[-70%,na,+199%,na,+5%,na,+648%,na,+10%,na]
M-0033svc=load-generator metric=otel.sdk.processor.span.queue.size base=6 delta/30s=[-17%,na,-83%,na,-100%,na,+500%,na,+450%,na]
M-0034svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0.00667 delta/30s=[-100%,na,+400%,na,-100%,na,-100%,na,-100%,na]
M-0035svc=accounting metric=dns.lookup.duration.count.rate base=0.00667 delta/30s=[-100%,na,+400%,na,-100%,na,-100%,na,-100%,na]
M-0036svc=fraud-detection metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,-100%,na,+400%,na,-100%,na]
859 lower-ranked lines omitted
traces: slowest and errored traces after fault onset, critical path (! = error span)
T-0001trace=2f43f16b dur=2857ms ERROR path: load-generator:user_checkout_multi 1942ms! > frontend-proxy:POST 390ms! > frontend:POST /api/checkout 389ms! > checkout:oteldemo.CheckoutService/PlaceOrder 385ms! > payment:oteldemo.PaymentService/Charge 1ms!
T-0002trace=84501b37 dur=22745ms ok path: accounting:order-consumed 22745ms
T-0003trace=39928adf dur=2549ms ERROR path: load-generator:user_checkout_single 1646ms! > frontend-proxy:POST 1623ms! > frontend:POST /api/checkout 1623ms! > checkout:oteldemo.CheckoutService/PlaceOrder 1621ms! > payment:oteldemo.PaymentService/Charge 1ms!
T-0004trace=e411d505 dur=22387ms ok path: accounting:order-consumed 22387ms
T-0005trace=3c5a7789 dur=2063ms ERROR path: load-generator:user_checkout_single 1225ms! > frontend-proxy:POST 1207ms! > frontend:POST /api/checkout 1207ms! > checkout:oteldemo.CheckoutService/PlaceOrder 1204ms! > payment:oteldemo.PaymentService/Charge 7ms!
T-0006trace=e9d670b1 dur=21032ms ok path: accounting:order-consumed 21032ms
T-0007trace=ad69a638 dur=1992ms ERROR path: load-generator:user_checkout_single 1085ms! > frontend-proxy:POST 564ms! > frontend:POST /api/checkout 563ms! > checkout:oteldemo.CheckoutService/PlaceOrder 560ms! > payment:oteldemo.PaymentService/Charge 1ms!
T-0008trace=caf07337 dur=18806ms ok path: accounting:order-consumed 18806ms
T-0009trace=49b14d53 dur=1550ms ERROR path: load-generator:user_checkout_single 649ms! > frontend-proxy:POST 607ms! > frontend:POST /api/checkout 606ms! > checkout:oteldemo.CheckoutService/PlaceOrder 604ms! > payment:oteldemo.PaymentService/Charge 0ms!
T-0010trace=e66319bb dur=18141ms ok path: accounting:order-consumed 18141ms
T-0011trace=83ad9fe2 dur=1509ms ERROR path: load-generator:user_checkout_multi 605ms! > frontend-proxy:POST 38ms! > frontend:POST /api/checkout 38ms! > checkout:oteldemo.CheckoutService/PlaceOrder 35ms! > payment:oteldemo.PaymentService/Charge 0ms!
T-0012trace=46fc674d dur=17966ms ok path: accounting:order-consumed 17966ms
T-0013trace=ac5379cd dur=1449ms ERROR path: load-generator:user_checkout_multi 553ms! > frontend-proxy:POST 506ms! > frontend:POST /api/checkout 506ms! > checkout:oteldemo.CheckoutService/PlaceOrder 502ms! > payment:oteldemo.PaymentService/Charge 2ms!
T-0014trace=6c1f0f80 dur=17339ms ok path: accounting:order-consumed 17339ms
T-0015trace=3c94466e dur=1399ms ERROR path: load-generator:user_checkout_single 495ms! > frontend-proxy:POST 477ms! > frontend:POST /api/checkout 476ms! > checkout:oteldemo.CheckoutService/PlaceOrder 474ms! > payment:oteldemo.PaymentService/Charge 0ms!
T-0016trace=3ead7d60 dur=15777ms ok path: accounting:order-consumed 15777ms
T-0017trace=5b1280c8 dur=1310ms ERROR path: load-generator:user_checkout_single 452ms! > frontend-proxy:POST 423ms! > frontend:POST /api/checkout 421ms! > checkout:oteldemo.CheckoutService/PlaceOrder 417ms! > payment:oteldemo.PaymentService/Charge 1ms!
T-0018trace=026feec1 dur=13852ms ok path: accounting:order-consumed 13852ms
T-0019trace=476119de dur=1307ms ERROR path: load-generator:user_checkout_multi 401ms! > frontend-proxy:POST 32ms! > frontend:POST /api/checkout 31ms! > checkout:oteldemo.CheckoutService/PlaceOrder 29ms! > payment:oteldemo.PaymentService/Charge 1ms!
T-0020trace=382263d9 dur=12902ms ok path: accounting:order-consumed 12902ms
T-0021trace=bc2f4d02 dur=1265ms ERROR path: load-generator:user_checkout_multi 353ms! > frontend-proxy:POST 266ms! > frontend:POST /api/checkout 266ms! > checkout:oteldemo.CheckoutService/PlaceOrder 263ms! > payment:oteldemo.PaymentService/Charge 2ms!
T-0022trace=88b6f0d3 dur=11687ms ok path: accounting:order-consumed 11687ms
T-0023trace=31ee2694 dur=1187ms ERROR path: load-generator:user_checkout_single 280ms! > frontend-proxy:POST 259ms! > frontend:POST /api/checkout 257ms! > checkout:oteldemo.CheckoutService/PlaceOrder 252ms! > payment:oteldemo.PaymentService/Charge 0ms!
T-0024trace=d6bfe1d6 dur=11509ms ok path: accounting:order-consumed 11509ms
T-0025trace=e94bad11 dur=1175ms ERROR path: load-generator:user_checkout_multi 277ms! > frontend-proxy:POST 119ms! > frontend:POST /api/checkout 119ms! > checkout:oteldemo.CheckoutService/PlaceOrder 117ms! > payment:oteldemo.PaymentService/Charge 1ms!
T-0026trace=73d413db dur=8126ms ok path: accounting:order-consumed 8126ms
T-0027trace=f8134a8b dur=1153ms ERROR path: load-generator:user_checkout_multi 262ms! > frontend-proxy:POST 218ms! > frontend:POST /api/checkout 218ms! > checkout:oteldemo.CheckoutService/PlaceOrder 215ms! > payment:oteldemo.PaymentService/Charge 1ms!
T-0028trace=d891b449 dur=7442ms ok path: accounting:order-consumed 7442ms
T-0029trace=820a2580 dur=1134ms ERROR path: load-generator:user_checkout_single 365ms! > frontend-proxy:POST 344ms! > frontend:POST /api/checkout 343ms! > checkout:oteldemo.CheckoutService/PlaceOrder 341ms! > payment:oteldemo.PaymentService/Charge 0ms!
T-0030trace=ac2ecb6e dur=7223ms ok path: accounting:order-consumed 7223ms
T-0031trace=c0c00060 dur=1134ms ERROR path: load-generator:user_checkout_single 229ms! > frontend-proxy:POST 107ms! > frontend:POST /api/checkout 107ms! > checkout:oteldemo.CheckoutService/PlaceOrder 104ms! > payment:oteldemo.PaymentService/Charge 0ms!
T-0032trace=8236d644 dur=6862ms ok path: accounting:order-consumed 6862ms
8 lower-ranked lines omitted
logs: log templates per service, rate before vs after fault onset, with examples
L-0001svc=frontend sev=INFO base=0.0/min fault=8.2/min tpl="Checkout payment declined" ex=['Checkout payment declined', 'Checkout payment declined']
L-0002svc=payment sev=WARN base=0.0/min fault=8.2/min tpl="Payment request failed. Invalid token. demo.user_context.loyalty_level=gold" ex=['Payment request failed. Invalid token. demo.user_context.loyalty_level=gold', 'Payment request failed. Invalid token. demo.user_context.loyalty_level=gold']
L-0003svc=accounting sev=INFORMATION base=10.6/min fault=6.0/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0004svc=cart sev=INFORMATION base=10.6/min fault=6.0/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0005svc=checkout sev=INFO base=10.6/min fault=6.0/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 149.214µs', 'Successful to write message. offset: 0, duration: 233.621µs']
L-0006svc=checkout sev=INFO base=10.6/min fault=6.0/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0007svc=checkout sev=INFO base=10.6/min fault=6.0/min tpl="order placed" ex=['order placed', 'order placed']
L-0008svc=checkout sev=INFO base=10.6/min fault=6.0/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0009svc=checkout sev=INFO base=10.6/min fault=6.0/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0010svc=email sev=INFO base=10.6/min fault=6.0/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0011svc=fraud-detection sev=INFO base=10.6/min fault=6.0/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: 2e1d7b5f-dbcf-11f1-82f3-d279f8fa1f6d, and updated total count to: 10929', 'Consumed record with orderId: 35c226b0-dbcf-11f1-82f3-d279f8fa1f6d, and updated total count to: 10930']
L-0012svc=frontend sev=INFO base=10.6/min fault=6.0/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0013svc=payment sev=INFO base=10.6/min fault=6.0/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0014svc=shipping sev=INFO base=10.6/min fault=6.0/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0015svc=load-generator sev=INFO base=10.4/min fault=14.8/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user 24df42e8-dbcf-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user 24df42e8-dbcf-11f1-8306-d6b650b5399b']
L-0016svc=checkout sev=INFO base=10.6/min fault=14.2/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0017svc=payment sev=INFO base=10.6/min fault=14.2/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0018svc=load-generator sev=INFO base=12.8/min fault=9.6/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0019svc=load-generator sev=INFO base=10.8/min fault=13.6/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user 2c1a12ea-dbcf-11f1-8306-d6b650b5399b', 'Checkout completed for user 2c1a12ea-dbcf-11f1-8306-d6b650b5399b']
L-0020svc=load-generator sev=INFO base=70.8/min fault=87.2/min tpl="User <UUID> adding <NUM> of product <*> to cart" ex=['User 2272de52-dbcf-11f1-8306-d6b650b5399b adding 10 of product 0PUK6V6EV0 to cart', 'User 2272de52-dbcf-11f1-8306-d6b650b5399b adding 10 of product 0PUK6V6EV0 to cart']
L-0021svc=checkout sev=INFO base=0.2/min fault=0.0/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0022svc=ad sev=INFO base=2.2/min fault=2.8/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0023svc=currency sev=INFO base=101.4/min fault=116.8/min tpl="conversion successful" ex=['conversion successful', 'conversion successful']
L-0024svc=load-generator sev=INFO base=144.8/min fault=126.8/min tpl="User browsing product: <*>" ex=['User browsing product: HQTGWGPNH4', 'User browsing product: HQTGWGPNH4']
L-0025svc=cart sev=INFORMATION base=51.8/min fault=59.0/min tpl="AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}" ex=['AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}', 'AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}']
L-0026svc=kafka sev=INFO base=2.0/min fault=2.4/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 33468 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 33507 breach: LogSegment(baseOff…']
L-0027svc=kafka sev=INFO base=1.0/min fault=0.8/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=33364, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=33484, siz…']
36 lower-ranked lines omitted
Loading runs in your browser…