toxiBandwidth-r2
What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)
Service graph during the window
Evidence and answers
5251 prompt tokens
metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean
M-0001svc=ad metric=error_ratio base=0 delta/30s=[+0%,na,new:0.012,na,+0%,na,+0%,na,+0%,na]
M-0002svc=ad metric=jvm.thread.count.rate base=0 delta/30s=[+0%,na,new:0.0667,na,+0%,na,+0%,na,+0%,na]
M-0003svc=ad metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0004svc=ad metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0005svc=ad metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0006svc=ad metric=traces.span.metrics.duration.p50 base=1.01 delta/30s=[-1%,na,+1487362%,na,-1%,na,+1%,na,-1%,na]
M-0007svc=ad metric=traces.span.metrics.duration.p95 base=1.92 delta/30s=[-1%,na,+782774%,na,-1%,na,+1%,na,-1%,na]
M-0008svc=ad metric=traces.span.metrics.duration.p99 base=2.8 delta/30s=[-29%,na,+535614%,na,-29%,na,+6%,na,-29%,na]
M-0009svc=fraud-detection metric=error_ratio base=0 delta/30s=[+0%,na,new:0.0435,na,+0%,na,+0%,na,+0%,na]
M-0010svc=fraud-detection metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0011svc=fraud-detection metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0012svc=fraud-detection metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0013svc=recommendation metric=process.runtime.cpython.cpu.utilization base=0 delta/30s=[+0%,na,new:11.1,na,new:12.5,na,new:12.0,na,+0%,na]
M-0014svc=fraud-detection metric=traces.span.metrics.duration.p50 base=44.7 delta/30s=[+3%,na,+33467%,na,+4%,na,-3%,na,+31%,na]
M-0015svc=fraud-detection metric=traces.span.metrics.duration.p95 base=111.1 delta/30s=[-15%,na,+13396%,na,-15%,na,-15%,na,-14%,na]
M-0016svc=fraud-detection metric=traces.span.metrics.duration.p99 base=134.2 delta/30s=[-26%,na,+11075%,na,-26%,na,-26%,na,-26%,na]
M-0017svc=unknown metric=system.disk.pending_operations base=0.4 delta/30s=[+2400%,na,-100%,na,-100%,na,-100%,na,+150%,na]
M-0018svc=quote metric=otel.sdk.processor.span.queue.size base=0.6 delta/30s=[-100%,na,-100%,na,+400%,na,+1400%,na,+1400%,na]
M-0019svc=shipping metric=http.server.request.duration.p50 base=0.0328 delta/30s=[+1140%,na,+53%,na,-27%,na,+91%,na,-27%,na]
M-0020svc=load-generator metric=otel.sdk.processor.log.queue.size base=1.2 delta/30s=[na,+733%,na,+67%,na,+567%,na,+1067%,na,+67%]
M-0021svc=fraud-detection metric=jvm.thread.count.rate base=0.00667 delta/30s=[-100%,na,+900%,na,-100%,na,-100%,na,-100%,na]
M-0022svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,-100%,-100%,-100%,+900%,-100%,-100%,-100%,-100%,-100%]
M-0023svc=product-catalog metric=otel.sdk.processor.log.queue.size base=0.6 delta/30s=[+733%,na,-100%,na,+400%,na,+67%,na,-100%,na]
M-0024svc=shipping metric=traces.span.metrics.duration.p50 base=50.0 delta/30s=[+700%,na,+50%,na,-10%,na,+0%,na,-11%,na]
M-0025svc=cart metric=traces.span.metrics.duration.p99 base=811.4 delta/30s=[+461%,na,+21%,na,-54%,na,+65%,na,-99%,na]
M-0026svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0.00667 delta/30s=[-100%,na,+400%,na,-100%,na,-100%,na,-100%,na]
M-0027svc=accounting metric=dns.lookup.duration.count.rate base=0.00667 delta/30s=[-100%,na,+400%,na,-100%,na,-100%,na,-100%,na]
M-0028svc=cart metric=dotnet.thread_pool.thread.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,-100%,na,+400%,na,-100%,na]
M-0029svc=cart metric=process.thread.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,-100%,na,+400%,na,-100%,na]
M-0030svc=checkout metric=go.goroutine.count.rate base=0.0133 delta/30s=[-100%,na,-100%,na,+400%,na,-100%,na,-100%,na]
M-0031svc=fraud-detection metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[+400%,na,-100%,na,-100%,na,-100%,na,+400%,na]
M-0032svc=fraud-detection metric=kafka.consumer.commit_latency_max base=1 delta/30s=[+0%,na,+0%,na,+400%,na,+0%,na,+0%,na]
M-0033svc=fraud-detection metric=kafka.consumer.heartbeat_response_time_max base=1 delta/30s=[+100%,na,+400%,na,+0%,na,+0%,na,+0%,na]
M-0034svc=quote metric=otel.sdk.processor.log.queue.size base=0.2 delta/30s=[-100%,na,+400%,na,+400%,na,+400%,na,-100%,na]
M-0035svc=checkout metric=rpc.server.call.duration.p50 base=0.313 delta/30s=[+352%,na,+20%,na,-27%,na,-44%,na,-87%,na]
M-0036svc=checkout metric=http.client.request.duration.p50 base=0.403 delta/30s=[-88%,na,-94%,na,-91%,na,+335%,na,-99%,na]
M-0037svc=load-generator metric=otel.sdk.span.live base=1.4 delta/30s=[na,-100%,na,+329%,na,+43%,na,+114%,na,-29%]
858 lower-ranked lines omitted
traces: slowest and errored traces after fault onset, critical path (! = error span)
T-0001trace=9c599642 dur=3134ms ERROR path: load-generator:user_checkout_multi 2850ms > frontend-proxy:POST 2770ms > frontend:POST /api/checkout 2770ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 399ms | errors off path: checkout:POST
T-0002trace=2bdbd2ca dur=19910ms ok path: accounting:order-consumed 19910ms
T-0003trace=c7cf5266 dur=54ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 54ms!
T-0004trace=fedc3c57 dur=17238ms ok path: accounting:order-consumed 17238ms
T-0005trace=73d9cf43 dur=33ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 33ms!
T-0006trace=d58223c8 dur=12495ms ok path: accounting:order-consumed 12495ms
T-0007trace=5e5d387b dur=32ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 32ms!
T-0008trace=87802dcd dur=12478ms ok path: accounting:order-consumed 12478ms
T-0009trace=174fa287 dur=31ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 31ms!
T-0010trace=31b3d7c1 dur=12244ms ok path: accounting:order-consumed 12244ms
T-0011trace=272d8f5d dur=31ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 31ms!
T-0012trace=fe2f3ef4 dur=11833ms ok path: accounting:order-consumed 11833ms
T-0013trace=369eb107 dur=30ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 30ms!
T-0014trace=c1b3cefa dur=11495ms ok path: accounting:order-consumed 11495ms
T-0015trace=c5fa70b6 dur=29ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 29ms!
T-0016trace=f4c9c213 dur=10827ms ok path: accounting:order-consumed 10827ms
T-0017trace=961bb844 dur=28ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 28ms!
T-0018trace=3412d4d7 dur=10771ms ok path: accounting:order-consumed 10771ms
T-0019trace=592ac13b dur=28ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 28ms!
T-0020trace=87ae9c1c dur=10166ms ok path: accounting:order-consumed 10166ms
T-0021trace=cb20a857 dur=25ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 25ms!
T-0022trace=786f4fc3 dur=10073ms ok path: accounting:order-consumed 10073ms
T-0023trace=575b9922 dur=25ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 25ms!
T-0024trace=ae3e29e7 dur=9643ms ok path: accounting:order-consumed 9643ms
T-0025trace=9486f7bf dur=24ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 24ms!
T-0026trace=365c9da9 dur=9616ms ok path: accounting:order-consumed 9616ms
T-0027trace=7e67fea1 dur=24ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 24ms!
T-0028trace=0f09d2b3 dur=9259ms ok path: accounting:order-consumed 9259ms
T-0029trace=4ad532ed dur=20ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 20ms!
T-0030trace=58cb0066 dur=8790ms ok path: accounting:order-consumed 8790ms
T-0031trace=3e025ce6 dur=20ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 20ms!
T-0032trace=e5a057fc dur=8657ms ok path: accounting:order-consumed 8657ms
T-0033trace=aed00dc8 dur=20ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 20ms!
T-0034trace=90bc5217 dur=8077ms ok path: accounting:order-consumed 8077ms
T-0035trace=1c5abd89 dur=19ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 19ms!
T-0036trace=56c43512 dur=6459ms ok path: accounting:order-consumed 6459ms
T-0037trace=464d9d69 dur=17ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 17ms!
T-0038trace=6968eea8 dur=6022ms ok path: accounting:order-consumed 6022ms
T-0039trace=bdb28e5d dur=17ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 17ms!
T-0040trace=6d94ed7c dur=5748ms ok path: accounting:order-consumed 5748ms
logs: log templates per service, rate before vs after fault onset, with examples
L-0001svc=checkout sev=WARN base=0.6/min fault=0.2/min tpl="failed to send order confirmation: failed POST to email service: Post "http://email:<NUM>/send_order_confirmation": context deadline exceed…" ex=['failed to send order confirmation: failed POST to email service: Post "http://email:6060/send_order_confirmation": context deadline exceeded']
L-0002svc=ad sev=INFO base=2.0/min fault=2.6/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0003svc=shipping sev=INFO base=1.4/min fault=1.0/min tpl="" ex=['', '']
L-0004svc=checkout sev=INFO base=0.0/min fault=0.2/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0005svc=load-generator sev=INFO base=14.8/min fault=12.8/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0006svc=checkout sev=INFO base=14.2/min fault=12.4/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0007svc=checkout sev=INFO base=14.2/min fault=12.4/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0008svc=payment sev=INFO base=14.2/min fault=12.4/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0009svc=payment sev=INFO base=14.2/min fault=12.4/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0010svc=shipping sev=INFO base=14.2/min fault=12.4/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0011svc=load-generator sev=INFO base=14.4/min fault=12.8/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user 45ec8226-dbe7-11f1-8306-d6b650b5399b', 'Checkout completed for user 45ec8226-dbe7-11f1-8306-d6b650b5399b']
L-0012svc=load-generator sev=INFO base=34.8/min fault=31.2/min tpl="User viewing cart" ex=['User viewing cart', 'User viewing cart']
L-0013svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=54036, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=54156, siz…']
L-0014svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 54216 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 54372 breach: LogSegment(baseOff…']
L-0015svc=accounting sev=INFORMATION base=14.0/min fault=12.6/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0016svc=cart sev=INFORMATION base=14.0/min fault=12.6/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0017svc=checkout sev=INFO base=14.0/min fault=12.6/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 256.072µs', 'Successful to write message. offset: 0, duration: 195.769µs']
L-0018svc=checkout sev=INFO base=14.0/min fault=12.6/min tpl="order placed" ex=['order placed', 'order placed']
L-0019svc=checkout sev=INFO base=14.0/min fault=12.6/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0020svc=email sev=INFO base=14.0/min fault=12.6/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0021svc=fraud-detection sev=INFO base=14.0/min fault=12.6/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: 463d186d-dbe7-11f1-82f3-d279f8fa1f6d, and updated total count to: 20486', 'Consumed record with orderId: 48350eeb-dbe7-11f1-82f3-d279f8fa1f6d, and updated total count to: 20487']
L-0022svc=frontend sev=INFO base=14.0/min fault=12.6/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0023svc=kafka sev=INFO base=7.6/min fault=8.4/min tpl="Deleted <*> index <*>" ex=['Deleted offset index /tmp/kafka-logs/__cluster_metadata-0/00000000000000054036.index.deleted.', 'Deleted offset index /tmp/kafka-logs/__cluster_metadata-0/00000000000000054066.index.deleted.']
L-0024svc=load-generator sev=INFO base=13.6/min fault=12.4/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user 47922338-dbe7-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user 47922338-dbe7-11f1-8306-d6b650b5399b']
L-0025svc=load-generator sev=INFO base=127.6/min fault=138.8/min tpl="User browsing product: <*>" ex=['User browsing product: 1YMWWN1N4O', 'User browsing product: 1YMWWN1N4O']
L-0026svc=kafka sev=INFO base=3.8/min fault=4.2/min tpl="Deleted log /tmp/kafka-logs/__cluster_metadata-<NUM>/<HEX>.log.deleted." ex=['Deleted log /tmp/kafka-logs/__cluster_metadata-0/00000000000000054036.log.deleted.', 'Deleted log /tmp/kafka-logs/__cluster_metadata-0/00000000000000054066.log.deleted.']
35 lower-ranked lines omitted
Loading runs in your browser…