netemLoss-r1
What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)
Service graph during the window
Evidence and answers
5257 prompt tokens
metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean
M-0001svc=ad metric=otel.sdk.processor.span.queue.size base=0 delta/30s=[na,new:2,na,+0%,na,+0%,na,+0%,na,+0%]
M-0002svc=payment metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0526]
M-0003svc=payment metric=traces.span.metrics.duration.p50 base=1.17 delta/30s=[na,-7%,na,-4%,na,-4%,na,+285%,na,+1283423%]
M-0004svc=payment metric=traces.span.metrics.duration.p95 base=9.28 delta/30s=[na,-70%,na,-67%,na,-33%,na,+299%,na,+161538%]
M-0005svc=payment metric=traces.span.metrics.duration.p99 base=12.7 delta/30s=[na,-70%,na,-70%,na,-40%,na,+272%,na,+117676%]
M-0006svc=product-catalog metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[na,+0%,na,new:24.0,na,new:40.0,na,new:2,na,new:5]
M-0007svc=product-catalog metric=otel.sdk.span.live base=0 delta/30s=[na,+0%,na,new:1,na,+0%,na,+0%,na,+0%]
M-0008svc=recommendation metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,new:0.00299,na,+0%,na,+0%]
M-0009svc=recommendation metric=process.thread.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%]
M-0010svc=recommendation metric=traces.span.metrics.duration.p50 base=3.28 delta/30s=[na,+4%,na,+2%,na,+456716%,na,-1%,na,-0%]
M-0011svc=recommendation metric=traces.span.metrics.duration.p95 base=5.68 delta/30s=[na,+1%,na,+4%,na,+263955%,na,+1%,na,-0%]
M-0012svc=recommendation metric=traces.span.metrics.duration.p99 base=7.19 delta/30s=[na,+8%,na,+37%,na,+208407%,na,+400%,na,-3%]
M-0013svc=payment metric=nodejs.eventloop.delay.stddev base=0.000341 delta/30s=[na,-37%,na,+20%,na,-46%,na,+1189%,na,+159%]
M-0014svc=payment metric=v8js.gc.duration.p50 base=0.00833 delta/30s=[na,-10%,na,-55%,na,-70%,na,+950%,na,+65%]
M-0015svc=jaeger metric=otelcol_exporter_in_flight_requests base=0.1 delta/30s=[-100%,-100%,-100%,-100%,-100%,-100%,-100%,+900%,-100%,-100%]
M-0016svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,-100%,-100%,+900%,-100%,-100%,-100%,-100%,-100%,-100%]
M-0017svc=payment metric=v8js.gc.duration.p99 base=0.0304 delta/30s=[na,-67%,na,-21%,na,-84%,na,+702%,na,-19%]
M-0018svc=payment metric=v8js.gc.duration.p95 base=0.0282 delta/30s=[na,-65%,na,-27%,na,-83%,na,+680%,na,-15%]
M-0019svc=payment metric=nodejs.eventloop.utilization base=0.00489 delta/30s=[na,-15%,na,-3%,na,-14%,na,+676%,na,+34%]
M-0020svc=payment metric=nodejs.eventloop.delay.max base=0.0287 delta/30s=[na,-33%,na,+30%,na,-45%,na,+464%,na,+152%]
M-0021svc=otelcol-contrib metric=otelcol_exporter_queue_size base=1.6 delta/30s=[na,+87%,na,-38%,na,+87%,na,+87%,na,+462%]
M-0022svc=recommendation metric=otel.sdk.processor.span.queue.size base=6.6 delta/30s=[na,+173%,na,+445%,na,+264%,na,+218%,na,+173%]
M-0023svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,-100%,na,+400%,na,-100%]
M-0024svc=accounting metric=dns.lookup.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,-100%,na,+400%,na,-100%]
M-0025svc=fraud-detection metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,+400%,na,-100%,na,-100%,na,+400%]
M-0026svc=quote metric=otel.sdk.processor.log.queue.size base=0.2 delta/30s=[na,-100%,na,-100%,na,-100%,na,+400%,na,+400%]
M-0027svc=quote metric=otel.sdk.processor.span.queue.size base=4.2 delta/30s=[na,+400%,na,+257%,na,+43%,na,+43%,na,-29%]
M-0028svc=recommendation metric=process.runtime.cpython.gc_count base=147.8 delta/30s=[na,+120%,na,+344%,na,-53%,na,+104%,na,+119%]
M-0029svc=ad metric=rpc.server.duration.p99 base=4.95 delta/30s=[na,+0%,na,+0%,na,+0%,na,+302%,na,+0%]
M-0030svc=product-catalog metric=go.goroutine.count.rate base=0.0267 delta/30s=[na,-100%,na,+275%,na,-100%,na,-100%,na,-100%]
M-0031svc=unknown metric=system.processes.count.rate base=0.16 delta/30s=[na,+25%,na,+46%,na,-79%,na,+254%,na,+254%]
M-0032svc=checkout metric=go.goroutine.count.rate base=0.04 delta/30s=[na,+233%,na,-100%,na,+67%,na,-100%,na,-100%]
M-0033svc=load-generator metric=otel.sdk.processor.log.queue.size base=4.4 delta/30s=[na,-100%,na,+36%,na,+218%,na,-9%,na,+82%]
M-0034svc=cart metric=traces.span.metrics.duration.p99 base=1.47e+03 delta/30s=[na,-46%,na,-35%,na,+198%,na,-34%,na,-87%]
M-0035svc=quote metric=traces.span.metrics.duration.p50 base=7.65 delta/30s=[na,-22%,na,-48%,na,-43%,na,+5%,na,+188%]
M-0036svc=load-generator metric=process.runtime.cpython.gc_count base=240.0 delta/30s=[na,+182%,na,+71%,na,-98%,na,-29%,na,+89%]
M-0037svc=load-generator metric=http.client.duration.p50 base=31.6 delta/30s=[na,+19%,na,+19%,na,+71%,na,+177%,na,-29%]
876 lower-ranked lines omitted
traces: slowest and errored traces after fault onset, critical path (! = error span)
T-0001trace=9190a4cd dur=792ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 792ms!
T-0002trace=456c2ea1 dur=16053ms ok path: accounting:order-consumed 16053ms
T-0003trace=d162e7e5 dur=557ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 557ms!
T-0004trace=afbad5f7 dur=15694ms ok path: accounting:order-consumed 15694ms
T-0005trace=2c5ab6e6 dur=353ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 353ms!
T-0006trace=d63c50ca dur=12416ms ok path: accounting:order-consumed 12416ms
T-0007trace=1567b43b dur=162ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 162ms!
T-0008trace=6a240727 dur=12272ms ok path: accounting:order-consumed 12272ms
T-0009trace=6a1020e1 dur=150ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 150ms!
T-0010trace=5002b1e6 dur=11801ms ok path: accounting:order-consumed 11801ms
T-0011trace=c7f40b34 dur=138ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 138ms!
T-0012trace=81631b11 dur=11679ms ok path: accounting:order-consumed 11679ms
T-0013trace=cfc821b0 dur=137ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 137ms!
T-0014trace=77e025ed dur=11189ms ok path: accounting:order-consumed 11189ms
T-0015trace=f5a4522f dur=121ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 121ms!
T-0016trace=2b78993c dur=9617ms ok path: accounting:order-consumed 9617ms
T-0017trace=3fdbaff6 dur=117ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 117ms!
T-0018trace=2b7e7240 dur=9535ms ok path: accounting:order-consumed 9535ms
T-0019trace=85447e92 dur=117ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 117ms!
T-0020trace=13cf0645 dur=9514ms ok path: accounting:order-consumed 9514ms
T-0021trace=483ba746 dur=117ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 117ms!
T-0022trace=11b02d83 dur=8909ms ok path: accounting:order-consumed 8909ms
T-0023trace=fc108dda dur=116ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 116ms!
T-0024trace=8b45afa2 dur=8851ms ok path: accounting:order-consumed 8851ms
T-0025trace=1702ab37 dur=109ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 109ms!
T-0026trace=4564fc88 dur=7843ms ok path: accounting:order-consumed 7843ms
T-0027trace=79fb4810 dur=99ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 99ms!
T-0028trace=d2fc13c9 dur=7515ms ok path: accounting:order-consumed 7515ms
T-0029trace=5b086756 dur=98ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 98ms!
T-0030trace=1ff52edc dur=7436ms ok path: accounting:order-consumed 7436ms
T-0031trace=bd9d6c1e dur=91ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 91ms!
T-0032trace=6c63e0fc dur=6732ms ok path: accounting:order-consumed 6732ms
T-0033trace=751b608a dur=79ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 79ms!
T-0034trace=26150cfb dur=6551ms ok path: accounting:order-consumed 6551ms
T-0035trace=355252bc dur=79ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 79ms!
T-0036trace=5a8ae902 dur=6226ms ok path: frontend-proxy:POST 6225ms > flagd:flagd.evaluation.v1.Service/EventStream 6223ms
T-0037trace=7201fc61 dur=76ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 76ms!
T-0038trace=20db868b dur=6127ms ok path: frontend-proxy:POST 6120ms > flagd:flagd.evaluation.v1.Service/EventStream 6122ms
T-0039trace=593cfa9e dur=62ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 62ms!
T-0040trace=da54ff0f dur=6123ms ok path: accounting:order-consumed 6123ms
logs: log templates per service, rate before vs after fault onset, with examples
L-0001svc=otelcol-contrib sev=ERROR base=15.0/min fault=0.0/min tpl="Exporting failed. Dropping data." ex=['Exporting failed. Dropping data.', 'Exporting failed. Dropping data.']
L-0002svc=load-generator sev=INFO base=16.0/min fault=8.8/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0003svc=load-generator sev=INFO base=25.2/min fault=32.4/min tpl="Currency changed to CHF" ex=['Currency changed to CHF', 'Currency changed to CHF']
L-0004svc=ad sev=INFO base=49.8/min fault=39.2/min tpl="Targeted ad request received for <*>" ex=['Targeted ad request received for [binoculars]', 'Targeted ad request received for [telescopes]']
L-0005svc=shipping sev=INFO base=1.4/min fault=2.0/min tpl="" ex=['', '']
L-0006svc=load-generator sev=INFO base=34.4/min fault=28.4/min tpl="Product added to cart successfully" ex=['Product added to cart successfully', 'Product added to cart successfully']
L-0007svc=checkout sev=INFO base=0.0/min fault=0.2/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0008svc=load-generator sev=INFO base=88.4/min fault=76.4/min tpl="User getting <*> for <*> <*>" ex=['User getting recommendations for product: HQTGWGPNH4', 'User getting recommendations for product: HQTGWGPNH4']
L-0009svc=load-generator sev=INFO base=41.6/min fault=47.6/min tpl="User viewing cart" ex=['User viewing cart', 'User viewing cart']
L-0010svc=currency sev=INFO base=107.0/min fault=121.8/min tpl="conversion successful" ex=['conversion successful', 'conversion successful']
L-0011svc=kafka sev=INFO base=2.0/min fault=2.4/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=34984, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=35014, siz…']
L-0012svc=kafka sev=INFO base=2.0/min fault=2.4/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 35145 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 35184 breach: LogSegment(baseOff…']
L-0013svc=kafka sev=INFO base=1.0/min fault=0.8/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=35044, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=35194, siz…']
L-0014svc=kafka sev=INFO base=1.0/min fault=0.8/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 35262 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 35379 breach: LogSegment(baseOff…']
L-0015svc=quote sev=INFO base=31.6/min fault=28.4/min tpl="Calculated quote" ex=['Calculated quote', 'Calculated quote']
L-0016svc=shipping sev=INFO base=31.6/min fault=28.4/min tpl="Requesting quote" ex=['Requesting quote', 'Requesting quote']
L-0017svc=shipping sev=INFO base=31.6/min fault=28.4/min tpl="Sending Quote" ex=['Sending Quote', 'Sending Quote']
L-0018svc=frontend-proxy sev=UNSET base=1838.6/min fault=1716.6/min tpl="<*> <*> <*> HTTP/<NUM>" <NUM> <*> <*> - "-" <NUM> <NUM> <NUM> <*> "-" "Mozilla/<NUM> (X11; Linux x86_64) AppleWebKit/<NUM> (KHTML, like Gec…" ex=['[2026-11-09T00:03:27.208Z] "GET /icons/Chevron.svg HTTP/1.1" 200 - via_upstream - "-" 0 793 4 3 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleW…', '[2026-11-09T00:03:27.209Z] "GET /api/shipping?itemList=%5B%7B%22productId%22%3A%222ZYFJ3GM2N%22%2C%22quantity%22%3A1%7D%5D¤cyCode=USD…']
L-0019svc=product-catalog sev=INFO base=144.6/min fault=135.0/min tpl="Found <NUM> products from database" ex=['Found 10 products from database', 'Found 10 products from database']
L-0020svc=load-generator sev=INFO base=126.8/min fault=134.8/min tpl="User browsing product: <*>" ex=['User browsing product: 9SIQT8TOJO', 'User browsing product: 9SIQT8TOJO']
L-0021svc=load-generator sev=INFO base=14.4/min fault=13.6/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user fe5cb20c-dbd0-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user fe5cb20c-dbd0-11f1-8306-d6b650b5399b']
L-0022svc=load-generator sev=INFO base=14.0/min fault=14.8/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user 0644a812-dbd1-11f1-8306-d6b650b5399b', 'Checkout completed for user 0644a812-dbd1-11f1-8306-d6b650b5399b']
L-0023svc=ad sev=INFO base=3.0/min fault=2.8/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
38 lower-ranked lines omitted
Loading runs in your browser…