toxiReset-r1
What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)
Service graph during the window
Evidence and answers
4585 prompt tokens
metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean
M-0001svc=ad metric=jvm.gc.duration.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%]
M-0002svc=checkout metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[na,+0%,na,new:6,na,+0%,na,+0%,na,+0%]
M-0003svc=fraud-detection metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[na,new:3,na,+0%,na,+0%,na,+0%,na,new:3]
M-0004svc=kafka metric=jvm.gc.duration.count.rate base=0 delta/30s=[na,+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%]
M-0005svc=kafka metric=jvm.thread.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0006svc=load-generator metric=process.runtime.cpython.cpu.utilization base=0 delta/30s=[na,+0%,na,+0%,na,new:19.1,na,+0%,na,+0%]
M-0007svc=payment metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,new:0.0345,na,+0%,na,+0%]
M-0008svc=payment metric=traces.span.metrics.duration.p50 base=1.24 delta/30s=[na,+101%,na,+7%,na,+1208691%,na,-10%,na,-12%]
M-0009svc=recommendation metric=process.thread.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0010svc=shipping metric=http.server.active_requests base=0 delta/30s=[na,+0%,na,+0%,na,new:1,na,+0%,na,+0%]
M-0011svc=payment metric=traces.span.metrics.duration.p95 base=18.1 delta/30s=[na,+77%,na,-76%,na,+82697%,na,-83%,na,-80%]
M-0012svc=payment metric=traces.span.metrics.duration.p99 base=22.5 delta/30s=[na,+106%,na,-75%,na,+66557%,na,-83%,na,-58%]
M-0013svc=recommendation metric=otel.sdk.processor.log.queue.size base=0.2 delta/30s=[-100%,na,-100%,na,-100%,na,-100%,na,+1400%,na]
M-0014svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,-100%,-100%,-100%,-100%,+900%,-100%,-100%,-100%,-100%]
M-0015svc=otelcol-contrib metric=otelcol_exporter_queue_size base=1.8 delta/30s=[na,+11%,na,+67%,na,-100%,na,+678%,na,-44%]
M-0016svc=cart metric=traces.span.metrics.duration.p99 base=5.89 delta/30s=[na,+562%,na,+17%,na,+532%,na,-32%,na,-6%]
M-0017svc=email metric=traces.span.metrics.duration.p99 base=6.72 delta/30s=[na,+18%,na,+555%,na,+15%,na,-11%,na,+12%]
M-0018svc=recommendation metric=process.runtime.cpython.cpu.utilization base=2.18 delta/30s=[-100%,na,-100%,na,-100%,na,+521%,na,-100%,na]
M-0019svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,+400%,na,-100%,na,-100%]
M-0020svc=accounting metric=dns.lookup.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,+400%,na,-100%,na,-100%]
M-0021svc=checkout metric=go.goroutine.count.rate base=0.0267 delta/30s=[na,+150%,na,+150%,na,-100%,na,-100%,na,+400%]
M-0022svc=fraud-detection metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,+400%,na,-100%,na,-100%,na,-100%]
M-0023svc=unknown metric=system.disk.pending_operations base=0.2 delta/30s=[na,+400%,na,+400%,na,-100%,na,-100%,na,+400%]
M-0024svc=flagd metric=error_ratio base=0.00019 delta/30s=[na,-100%,na,-100%,na,+364%,na,-100%,na,-100%]
M-0025svc=frontend metric=http.client.request.duration.p50 base=0.04 delta/30s=[+275%,na,+337%,na,+300%,na,+41%,na,+212%,na]
M-0026svc=kafka metric=kafka.purgatory.size base=10.4 delta/30s=[na,+304%,na,+131%,na,+15%,na,+112%,na,+246%]
M-0027svc=cart metric=dotnet.gc.last_collection.heap.fragmentation.size base=1.39e+06 delta/30s=[na,-5%,na,+296%,na,+296%,na,+259%,na,+240%]
M-0028svc=product-catalog metric=otel.sdk.processor.log.queue.size base=2.8 delta/30s=[na,+114%,na,+114%,na,+293%,na,-100%,na,-100%]
M-0029svc=fraud-detection metric=otel.sdk.processor.span.queue.size base=1.6 delta/30s=[na,+275%,na,-100%,na,-100%,na,-100%,na,+275%]
M-0030svc=load-generator metric=otel.sdk.span.live base=0.6 delta/30s=[na,+67%,na,+233%,na,-100%,na,-100%,na,+67%]
M-0031svc=cart metric=demo.cart.add_item.latency.p99 base=0.00544 delta/30s=[na,+205%,na,+219%,na,+178%,na,-9%,na,-9%]
M-0032svc=email metric=traces.span.metrics.duration.p95 base=6.41 delta/30s=[na,+20%,na,+212%,na,+3%,na,-8%,na,-7%]
M-0033svc=load-generator metric=process.runtime.cpython.gc_count base=145.8 delta/30s=[na,-95%,na,+206%,na,+154%,na,-40%,na,-96%]
M-0034svc=cart metric=http.server.request.duration.p99 base=0.00697 delta/30s=[na,+198%,na,+149%,na,+116%,na,-29%,na,-29%]
M-0035svc=unknown metric=redis.commands base=8.4 delta/30s=[+198%,+7%,+7%,-29%,-17%,+43%,-17%,+162%,+31%,-17%]
M-0036svc=recommendation metric=otel.sdk.exporter.metric_data_point.exported base=1.83e+03 delta/30s=[+75%,na,+100%,na,+125%,na,+150%,na,+175%,na]
M-0037svc=accounting metric=db.client.operation.duration.count.rate base=0.34 delta/30s=[na,+8%,na,+27%,na,+37%,na,-22%,na,+165%]
870 lower-ranked lines omitted
traces: slowest and errored traces after fault onset, critical path (! = error span)
T-0001trace=35817081 dur=24959ms ok path: accounting:order-consumed 24959ms
T-0002trace=1dff63ed dur=12836ms ok path: accounting:order-consumed 12836ms
T-0003trace=884408b7 dur=12462ms ok path: accounting:order-consumed 12462ms
T-0004trace=a97ae5ed dur=12397ms ok path: accounting:order-consumed 12397ms
T-0005trace=ec21e0a8 dur=12301ms ok path: accounting:order-consumed 12301ms
T-0006trace=d2aecb16 dur=11256ms ok path: accounting:order-consumed 11256ms
T-0007trace=e6f50e41 dur=10235ms ok path: accounting:order-consumed 10235ms
T-0008trace=cab0bff9 dur=9581ms ok path: accounting:order-consumed 9581ms
T-0009trace=27f5b66a dur=9537ms ok path: accounting:order-consumed 9537ms
T-0010trace=41863aeb dur=9249ms ok path: accounting:order-consumed 9249ms
T-0011trace=eedfa427 dur=8808ms ok path: accounting:order-consumed 8808ms
T-0012trace=83904c89 dur=8738ms ok path: accounting:order-consumed 8738ms
T-0013trace=06c03517 dur=8518ms ok path: accounting:order-consumed 8518ms
T-0014trace=7361f848 dur=8504ms ok path: accounting:order-consumed 8504ms
T-0015trace=e6415e78 dur=8375ms ok path: accounting:order-consumed 8375ms
T-0016trace=3aa52fba dur=8227ms ok path: accounting:order-consumed 8227ms
T-0017trace=50e4f411 dur=8046ms ok path: accounting:order-consumed 8046ms
T-0018trace=0ddee1f4 dur=6556ms ok path: accounting:order-consumed 6556ms
T-0019trace=42c50b2b dur=6001ms ok path: accounting:order-consumed 6001ms
T-0020trace=2f44445b dur=5979ms ok path: accounting:order-consumed 5979ms
logs: log templates per service, rate before vs after fault onset, with examples
L-0001svc=ad sev=INFO base=1.8/min fault=3.2/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0002svc=load-generator sev=INFO base=11.2/min fault=14.0/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user e946916c-dbee-11f1-8306-d6b650b5399b', 'Checkout completed for user e946916c-dbee-11f1-8306-d6b650b5399b']
L-0003svc=load-generator sev=INFO base=74.8/min fault=91.2/min tpl="User getting <*> for <*> <*>" ex=['User getting ads for category: travel', 'User getting ads for category: travel']
L-0004svc=checkout sev=INFO base=0.2/min fault=0.0/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0005svc=accounting sev=INFORMATION base=12.0/min fault=14.4/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0006svc=cart sev=INFORMATION base=12.0/min fault=14.4/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0007svc=checkout sev=INFO base=12.0/min fault=14.4/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 225.122µs', 'Successful to write message. offset: 0, duration: 184.979µs']
L-0008svc=checkout sev=INFO base=12.0/min fault=14.4/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0009svc=checkout sev=INFO base=12.0/min fault=14.4/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0010svc=checkout sev=INFO base=12.0/min fault=14.4/min tpl="order placed" ex=['order placed', 'order placed']
L-0011svc=checkout sev=INFO base=12.0/min fault=14.4/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0012svc=checkout sev=INFO base=12.0/min fault=14.4/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0013svc=email sev=INFO base=12.0/min fault=14.4/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0014svc=fraud-detection sev=INFO base=12.0/min fault=14.4/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: e182b799-dbee-11f1-82f3-d279f8fa1f6d, and updated total count to: 21151', 'Consumed record with orderId: e1e017d7-dbee-11f1-82f3-d279f8fa1f6d, and updated total count to: 21152']
L-0015svc=frontend sev=INFO base=12.0/min fault=14.4/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0016svc=payment sev=INFO base=12.0/min fault=14.4/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0017svc=payment sev=INFO base=12.0/min fault=14.4/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0018svc=shipping sev=INFO base=12.0/min fault=14.4/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0019svc=load-generator sev=INFO base=12.8/min fault=14.8/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user e1782dec-dbee-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user e1782dec-dbee-11f1-8306-d6b650b5399b']
L-0020svc=currency sev=INFO base=111.2/min fault=126.8/min tpl="conversion successful" ex=['conversion successful', 'conversion successful']
L-0021svc=load-generator sev=INFO base=30.8/min fault=34.8/min tpl="Currency changed to CHF" ex=['Currency changed to CHF', 'Currency changed to CHF']
L-0022svc=load-generator sev=INFO base=15.2/min fault=17.2/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0023svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=60666, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=60786, siz…']
L-0024svc=shipping sev=INFO base=1.0/min fault=0.8/min tpl="" ex=['', '']
L-0025svc=load-generator sev=INFO base=82.4/min fault=90.4/min tpl="User <UUID> adding <NUM> of product <*> to cart" ex=['User e1782dec-dbee-11f1-8306-d6b650b5399b adding 2 of product 9SIQT8TOJO to cart', 'User e1782dec-dbee-11f1-8306-d6b650b5399b adding 2 of product 9SIQT8TOJO to cart']
L-0026svc=kafka sev=INFO base=7.6/min fault=8.4/min tpl="Deleted <*> index <*>" ex=['Deleted offset index /tmp/kafka-logs/__cluster_metadata-0/00000000000000060636.index.deleted.', 'Deleted time index /tmp/kafka-logs/__cluster_metadata-0/00000000000000060636.timeindex.deleted.']
L-0027svc=frontend-proxy sev=UNSET base=260.6/min fault=282.6/min tpl="<*> <*> <*> HTTP/<NUM>" <NUM> <*> <*> - "-" <NUM> <NUM> <NUM> <*> "-" <*> <*> <*> <*> <*> <*> <NUM>.<NUM>:<NUM> <NUM>.<NUM>:<NUM> - -" ex=['[2026-11-09T03:37:25.072Z] "GET /api/cart HTTP/1.1" 200 - via_upstream - "-" 0 24 3 3 "-" "python-requests/2.34.2" "0facebd9-e437-930a-96bb…', '[2026-11-09T03:37:25.666Z] "GET /api/products/9SIQT8TOJO HTTP/1.1" 200 - via_upstream - "-" 0 778 5 4 "-" "python-requests/2.34.2" "0795630…']
33 lower-ranked lines omitted
Loading runs in your browser…