toxiLatency-r2
What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)
Service graph during the window
Evidence and answers
5292 prompt tokens
metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean
M-0001svc=accounting metric=dotnet.thread_pool.thread.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,+0%,na,new:0.0333,na,+0%]
M-0002svc=accounting metric=process.thread.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,+0%,na,new:0.0333,na,+0%]
M-0003svc=ad metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0098]
M-0004svc=ad metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:1,na,+0%]
M-0005svc=ad metric=rpc.client.duration.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0006svc=ad metric=rpc.client.request.size.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0007svc=ad metric=rpc.client.response.size.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0008svc=ad metric=traces.span.metrics.duration.p50 base=1 delta/30s=[na,+0%,na,+7%,na,+2%,na,+0%,na,+1499900%]
M-0009svc=ad metric=traces.span.metrics.duration.p95 base=1.9 delta/30s=[na,+0%,na,+34%,na,+2%,na,+0%,na,+789374%]
M-0010svc=ad metric=traces.span.metrics.duration.p99 base=1.98 delta/30s=[na,+0%,na,+255%,na,+56%,na,+0%,na,+757476%]
M-0011svc=fraud-detection metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0345]
M-0012svc=fraud-detection metric=rpc.client.duration.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0013svc=fraud-detection metric=rpc.client.request.size.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0014svc=fraud-detection metric=rpc.client.response.size.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0015svc=load-generator metric=process.runtime.cpython.cpu.utilization base=0 delta/30s=[+0%,na,+0%,na,new:18.6,na,+0%,na,+0%,na]
M-0016svc=recommendation metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,new:0.000549,na,new:0.000358,na,new:0.000358]
M-0017svc=recommendation metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[na,+0%,na,new:4,na,+0%,na,new:2,na,+0%]
M-0018svc=recommendation metric=process.thread.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%]
M-0019svc=fraud-detection metric=traces.span.metrics.duration.p50 base=45.0 delta/30s=[na,+36%,na,-11%,na,-2%,na,+19%,na,+33216%]
M-0020svc=recommendation metric=traces.span.metrics.duration.p50 base=75.0 delta/30s=[na,+0%,na,+0%,na,+19900%,na,+19900%,na,+19900%]
M-0021svc=fraud-detection metric=traces.span.metrics.duration.p95 base=93.9 delta/30s=[na,+2%,na,-1%,na,+0%,na,+2%,na,+15882%]
M-0022svc=fraud-detection metric=traces.span.metrics.duration.p99 base=98.8 delta/30s=[na,+0%,na,-0%,na,+0%,na,+0%,na,+15087%]
M-0023svc=unknown metric=system.disk.pending_operations base=0.4 delta/30s=[na,-100%,na,+7900%,na,-100%,na,+3650%,na,+150%]
M-0024svc=recommendation metric=traces.span.metrics.duration.p95 base=334.0 delta/30s=[na,+0%,na,+0%,na,+4391%,na,+4391%,na,+4391%]
M-0025svc=unknown metric=system.processes.count.rate base=0.0533 delta/30s=[na,-37%,na,+1212%,na,+1088%,na,+2463%,na,-100%]
M-0026svc=recommendation metric=traces.span.metrics.duration.p99 base=687.0 delta/30s=[na,+0%,na,+0%,na,+2083%,na,+2083%,na,+2083%]
M-0027svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,+900%,-100%,-100%,-100%,-100%,-100%,-100%,-100%,-100%]
M-0028svc=ad metric=otel.sdk.processor.span.queue.size base=1.8 delta/30s=[na,+567%,na,-100%,na,+122%,na,+233%,na,+67%]
M-0029svc=load-generator metric=process.runtime.cpython.gc_count base=126.4 delta/30s=[+347%,na,-90%,na,+436%,na,-98%,na,-94%,na]
M-0030svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,-100%,na,-100%,na,+400%]
M-0031svc=accounting metric=dns.lookup.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,-100%,na,-100%,na,+400%]
M-0032svc=checkout metric=otel.sdk.span.live base=0.6 delta/30s=[-100%,na,-100%,na,-100%,na,-100%,na,+400%,na]
M-0033svc=fraud-detection metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,+400%,na,-100%,na,-100%]
M-0034svc=fraud-detection metric=jvm.thread.count.rate base=0.0133 delta/30s=[na,-100%,na,-100%,na,-100%,na,-100%,na,+400%]
M-0035svc=quote metric=otel.sdk.processor.span.queue.size base=1.8 delta/30s=[na,-100%,na,+233%,na,+233%,na,+400%,na,+233%]
M-0036svc=checkout metric=rpc.client.call.duration.p50 base=0.0828 delta/30s=[-37%,na,+353%,na,+353%,na,+353%,na,+353%,na]
M-0037svc=load-generator metric=otel.sdk.span.live base=1 delta/30s=[+100%,na,+300%,na,+100%,na,+100%,na,+300%,na]
858 lower-ranked lines omitted
traces: slowest and errored traces after fault onset, critical path (! = error span)
T-0001trace=dd335033 dur=1006ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 1006ms!
T-0002trace=f2519f97 dur=25477ms ok path: accounting:order-consumed 25477ms
T-0003trace=4d002dd3 dur=492ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 492ms!
T-0004trace=66b1ba46 dur=17587ms ok path: accounting:order-consumed 17587ms
T-0005trace=f90434c1 dur=430ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 430ms!
T-0006trace=2bc07090 dur=14280ms ok path: accounting:order-consumed 14280ms
T-0007trace=58c470a4 dur=325ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 325ms!
T-0008trace=8bd48ee1 dur=13935ms ok path: accounting:order-consumed 13935ms
T-0009trace=d983b274 dur=310ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 310ms!
T-0010trace=78e235d0 dur=12296ms ok path: accounting:order-consumed 12296ms
T-0011trace=a7a30c52 dur=138ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 138ms!
T-0012trace=6904d39b dur=9902ms ok path: accounting:order-consumed 9902ms
T-0013trace=274a3c2c dur=137ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 137ms!
T-0014trace=2a8c4b9b dur=9853ms ok path: accounting:order-consumed 9853ms
T-0015trace=d7d39841 dur=137ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 137ms!
T-0016trace=6d47ad62 dur=8639ms ok path: accounting:order-consumed 8639ms
T-0017trace=35f8ffd7 dur=125ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 125ms!
T-0018trace=4adba918 dur=8449ms ok path: accounting:order-consumed 8449ms
T-0019trace=b4f34b35 dur=113ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 113ms!
T-0020trace=8018dec9 dur=8257ms ok path: accounting:order-consumed 8257ms
T-0021trace=ea33c1c4 dur=86ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 86ms!
T-0022trace=eebfcdec dur=8241ms ok path: accounting:order-consumed 8241ms
T-0023trace=f6078ae9 dur=80ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 80ms!
T-0024trace=0cb98123 dur=7656ms ok path: accounting:order-consumed 7656ms
T-0025trace=99f58ac5 dur=77ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 77ms!
T-0026trace=a2cc9156 dur=7645ms ok path: accounting:order-consumed 7645ms
T-0027trace=f525bbd4 dur=74ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 74ms!
T-0028trace=3c3a4eed dur=6334ms ok path: accounting:order-consumed 6334ms
T-0029trace=ac383afa dur=74ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 74ms!
T-0030trace=1c46fa21 dur=6178ms ok path: accounting:order-consumed 6178ms
T-0031trace=7fc967e7 dur=72ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 72ms!
T-0032trace=c29f1e98 dur=5883ms ok path: accounting:order-consumed 5883ms
T-0033trace=0078658a dur=68ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 68ms!
T-0034trace=d65c4516 dur=5183ms ok path: accounting:order-consumed 5183ms
T-0035trace=b16777b5 dur=67ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 67ms!
T-0036trace=58e15215 dur=5124ms ok path: accounting:order-consumed 5124ms
T-0037trace=eee3d1ae dur=67ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 67ms!
T-0038trace=407883c8 dur=4960ms ok path: accounting:order-consumed 4960ms
T-0039trace=42cc90fa dur=64ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 64ms!
T-0040trace=5a8c3a3d dur=4898ms ok path: accounting:order-consumed 4898ms
logs: log templates per service, rate before vs after fault onset, with examples
L-0001svc=ad sev=INFO base=3.6/min fault=1.6/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0002svc=shipping sev=INFO base=1.2/min fault=2.0/min tpl="" ex=['', '']
L-0003svc=load-generator sev=INFO base=15.6/min fault=12.8/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0004svc=checkout sev=INFO base=0.2/min fault=0.0/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0005svc=checkout sev=WARN base=0.2/min fault=0.0/min tpl="failed to send order confirmation: failed POST to email service: Post "http://email:<NUM>/send_order_confirmation": context deadline exceed…" ex=['failed to send order confirmation: failed POST to email service: Post "http://email:6060/send_order_confirmation": context deadline exceeded']
L-0006svc=checkout sev=INFO base=12.4/min fault=14.4/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0007svc=load-generator sev=INFO base=12.8/min fault=14.8/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user 7c19efac-dbe5-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user 7c19efac-dbe5-11f1-8306-d6b650b5399b']
L-0008svc=accounting sev=INFORMATION base=12.6/min fault=14.4/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0009svc=cart sev=INFORMATION base=12.6/min fault=14.4/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0010svc=checkout sev=INFO base=12.6/min fault=14.4/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 272.425µs', 'Successful to write message. offset: 0, duration: 169.969µs']
L-0011svc=checkout sev=INFO base=12.6/min fault=14.4/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0012svc=checkout sev=INFO base=12.6/min fault=14.4/min tpl="order placed" ex=['order placed', 'order placed']
L-0013svc=checkout sev=INFO base=12.6/min fault=14.4/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0014svc=checkout sev=INFO base=12.6/min fault=14.4/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0015svc=email sev=INFO base=12.6/min fault=14.4/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0016svc=fraud-detection sev=INFO base=12.6/min fault=14.4/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: 6cc7e61c-dbe5-11f1-82f3-d279f8fa1f6d, and updated total count to: 20303', 'Consumed record with orderId: 6fa37853-dbe5-11f1-82f3-d279f8fa1f6d, and updated total count to: 20304']
L-0017svc=frontend sev=INFO base=12.6/min fault=14.4/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0018svc=payment sev=INFO base=12.6/min fault=14.4/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0019svc=payment sev=INFO base=12.6/min fault=14.4/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0020svc=shipping sev=INFO base=12.6/min fault=14.4/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0021svc=load-generator sev=INFO base=12.4/min fault=14.0/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user 6cc515fe-dbe5-11f1-8306-d6b650b5399b', 'Checkout completed for user 6cc515fe-dbe5-11f1-8306-d6b650b5399b']
L-0022svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=52476, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=52596, siz…']
L-0023svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 52656 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 52812 breach: LogSegment(baseOff…']
L-0024svc=load-generator sev=INFO base=141.2/min fault=127.2/min tpl="User browsing product: <*>" ex=['User browsing product: 0PUK6V6EV0', 'User browsing product: 0PUK6V6EV0']
L-0025svc=load-generator sev=INFO base=74.8/min fault=82.8/min tpl="User <UUID> adding <NUM> of product <*> to cart" ex=['User 6cc515fe-dbe5-11f1-8306-d6b650b5399b adding 2 of product 0PUK6V6EV0 to cart', 'User 6cc515fe-dbe5-11f1-8306-d6b650b5399b adding 2 of product 0PUK6V6EV0 to cart']
L-0026svc=currency sev=INFO base=107.6/min fault=117.8/min tpl="conversion successful" ex=['conversion successful', 'conversion successful']
L-0027svc=kafka sev=INFO base=7.6/min fault=8.4/min tpl="Deleted <*> index <*>" ex=['Deleted offset index /tmp/kafka-logs/__cluster_metadata-0/00000000000000052476.index.deleted.', 'Deleted offset index /tmp/kafka-logs/__cluster_metadata-0/00000000000000052506.index.deleted.']
34 lower-ranked lines omitted
Loading runs in your browser…