healthy-r1
What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)
Service graph during the window
Evidence and answers
5362 prompt tokens
metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean
M-0001svc=ad metric=error_ratio base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.00862,na]
M-0002svc=ad metric=jvm.gc.duration.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0003svc=ad metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[new:2,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0004svc=ad metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333,na]
M-0005svc=ad metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333,na]
M-0006svc=ad metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333,na]
M-0007svc=ad metric=traces.span.metrics.duration.p50 base=1 delta/30s=[+5%,na,+0%,na,+2%,na,+3%,na,+1499900%,na]
M-0008svc=ad metric=traces.span.metrics.duration.p95 base=1.9 delta/30s=[+5%,na,+0%,na,+2%,na,+3%,na,+789374%,na]
M-0009svc=ad metric=traces.span.metrics.duration.p99 base=1.98 delta/30s=[+79%,na,+0%,na,+54%,na,+62%,na,+757476%,na]
M-0010svc=checkout metric=go.goroutine.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,new:0.0667,na,new:0.0667,na]
M-0011svc=checkout metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[+0%,na,new:1,na,+0%,na,+0%,na,+0%,na]
M-0012svc=fraud-detection metric=error_ratio base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.037,na]
M-0013svc=fraud-detection metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[+0%,na,+0%,na,new:1,na,+0%,na,new:1,na]
M-0014svc=fraud-detection metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333,na]
M-0015svc=fraud-detection metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333,na]
M-0016svc=fraud-detection metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333,na]
M-0017svc=kafka metric=jvm.gc.duration.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0018svc=kafka metric=jvm.thread.count.rate base=0 delta/30s=[new:0.0333,na,new:0.0333,na,new:0.0333,na,+0%,na,new:0.0333,na]
M-0019svc=otelcol-contrib metric=otelcol_exporter_in_flight_requests base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:1,na,+0%,na]
M-0020svc=fraud-detection metric=traces.span.metrics.duration.p50 base=52.3 delta/30s=[-14%,na,-24%,na,+2%,na,+0%,na,+28567%,na]
M-0021svc=fraud-detection metric=traces.span.metrics.duration.p95 base=103.4 delta/30s=[-9%,na,-10%,na,-8%,na,-8%,na,+14409%,na]
M-0022svc=fraud-detection metric=traces.span.metrics.duration.p99 base=116.7 delta/30s=[-15%,na,-15%,na,-15%,na,-15%,na,+12756%,na]
M-0023svc=product-catalog metric=otel.sdk.processor.log.queue.size base=0.6 delta/30s=[-100%,na,-100%,na,-100%,na,+4067%,na,+67%,na]
M-0024svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[+900%,-100%,-100%,-100%,-100%,-100%,-100%,-100%,-100%,-100%]
M-0025svc=fraud-detection metric=jvm.thread.count.rate base=0.0133 delta/30s=[-100%,na,-100%,na,-100%,na,+150%,na,+650%,na]
M-0026svc=payment metric=v8js.gc.duration.p50 base=0.0126 delta/30s=[-60%,na,+595%,na,+39%,na,+24%,na,+39%,na]
M-0027svc=payment metric=v8js.gc.duration.p95 base=0.0335 delta/30s=[+557%,na,+580%,na,-28%,na,-28%,na,+535%,na]
M-0028svc=payment metric=v8js.gc.duration.p99 base=0.0363 delta/30s=[+572%,na,+576%,na,-32%,na,-32%,na,+568%,na]
M-0029svc=flagd metric=error_ratio base=0.00336 delta/30s=[+150%,na,+480%,na,+58%,na,-87%,na,+4%,na]
M-0030svc=load-generator metric=process.runtime.cpython.gc_count base=125.4 delta/30s=[na,+430%,na,+260%,na,+374%,na,+115%,na,-33%]
M-0031svc=checkout metric=rpc.client.call.duration.p99 base=0.911 delta/30s=[-49%,na,+416%,na,+153%,na,-23%,na,-48%,na]
M-0032svc=cart metric=http.server.request.duration.p99 base=0.91 delta/30s=[+157%,na,+414%,na,-21%,na,-23%,na,-49%,na]
M-0033svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,-100%,na,+400%,na,-100%,na]
M-0034svc=accounting metric=dns.lookup.duration.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,-100%,na,+400%,na,-100%,na]
M-0035svc=accounting metric=dotnet.thread_pool.thread.count.rate base=0.00667 delta/30s=[-100%,na,+400%,na,-100%,na,-100%,na,+400%,na]
M-0036svc=accounting metric=process.thread.count.rate base=0.00667 delta/30s=[-100%,na,+400%,na,-100%,na,-100%,na,+400%,na]
M-0037svc=cart metric=dotnet.thread_pool.thread.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,-100%,na,-100%,na,+400%,na]
852 lower-ranked lines omitted
traces: slowest and errored traces after fault onset, critical path (! = error span)
T-0001trace=f766c535 dur=4633ms ERROR path: load-generator:user_checkout_multi 4633ms > frontend-proxy:POST 4427ms > frontend:POST /api/checkout 4426ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 508ms | errors off path: checkout:POST
T-0002trace=29d3829d dur=25057ms ok path: accounting:order-consumed 25057ms
T-0003trace=af5103bc dur=652ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 652ms!
T-0004trace=5c6b2f9f dur=17937ms ok path: accounting:order-consumed 17937ms
T-0005trace=543159cb dur=238ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 238ms!
T-0006trace=70113d39 dur=14061ms ok path: accounting:order-consumed 14061ms
T-0007trace=db1cab3c dur=228ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 228ms!
T-0008trace=d76baea3 dur=9742ms ok path: accounting:order-consumed 9742ms
T-0009trace=b628e4f2 dur=157ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 157ms!
T-0010trace=f6104e1e dur=9671ms ok path: accounting:order-consumed 9671ms
T-0011trace=c7526466 dur=144ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 144ms!
T-0012trace=4dda6422 dur=9174ms ok path: accounting:order-consumed 9174ms
T-0013trace=f25c1456 dur=59ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 59ms!
T-0014trace=0cc4a9a1 dur=9124ms ok path: accounting:order-consumed 9124ms
T-0015trace=e260a71a dur=44ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 44ms!
T-0016trace=9e808891 dur=8689ms ok path: accounting:order-consumed 8689ms
T-0017trace=5b0dcc4b dur=43ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 43ms!
T-0018trace=b7893339 dur=8313ms ok path: accounting:order-consumed 8313ms
T-0019trace=f13ab1b0 dur=42ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 42ms!
T-0020trace=321bdc2d dur=8100ms ok path: accounting:order-consumed 8100ms
T-0021trace=f9c2d27b dur=41ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 41ms!
T-0022trace=e41538a8 dur=7794ms ok path: accounting:order-consumed 7794ms
T-0023trace=a8c5ebcf dur=36ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 36ms!
T-0024trace=699a9db9 dur=7628ms ok path: accounting:order-consumed 7628ms
T-0025trace=047a0e73 dur=32ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 32ms!
T-0026trace=71586ac5 dur=7562ms ok path: accounting:order-consumed 7562ms
T-0027trace=c1c68e1c dur=32ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 32ms!
T-0028trace=0ba10451 dur=6830ms ok path: load-generator:user_checkout_multi 6830ms > frontend-proxy:POST 5705ms > frontend:POST /api/checkout 5704ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 48ms
T-0029trace=6376fb07 dur=29ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 29ms!
T-0030trace=0beb3e67 dur=6067ms ok path: accounting:order-consumed 6067ms
T-0031trace=cee37c15 dur=28ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 28ms!
T-0032trace=bd8cb6c3 dur=5992ms ok path: accounting:order-consumed 5992ms
T-0033trace=3d424a93 dur=26ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 26ms!
T-0034trace=c0392045 dur=5883ms ok path: accounting:order-consumed 5883ms
T-0035trace=ddfac667 dur=25ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 25ms!
T-0036trace=829826c3 dur=5850ms ok path: accounting:order-consumed 5850ms
T-0037trace=2bfe08fc dur=23ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 23ms!
T-0038trace=724b0006 dur=5719ms ok path: frontend-proxy:POST 5710ms > flagd:flagd.evaluation.v1.Service/EventStream 4994ms
T-0039trace=f1856af9 dur=22ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 22ms!
T-0040trace=570b70fa dur=5705ms ok path: accounting:order-consumed 5705ms
logs: log templates per service, rate before vs after fault onset, with examples
L-0001svc=load-generator sev=INFO base=38.4/min fault=28.4/min tpl="User viewing cart" ex=['User viewing cart', 'User viewing cart']
L-0002svc=shipping sev=INFO base=2.2/min fault=1.4/min tpl="" ex=['', '']
L-0003svc=ad sev=INFO base=2.8/min fault=2.0/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0004svc=checkout sev=INFO base=0.2/min fault=0.0/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0005svc=shipping sev=ERROR base=0.2/min fault=0.4/min tpl="failed to resolve integer flag" ex=['failed to resolve integer flag', 'failed to resolve integer flag']
L-0006svc=checkout sev=WARN base=0.4/min fault=0.2/min tpl="failed to send order confirmation: failed POST to email service: Post "http://email:<NUM>/send_order_confirmation": context deadline exceed…" ex=['failed to send order confirmation: failed POST to email service: Post "http://email:6060/send_order_confirmation": context deadline exceeded']
L-0007svc=load-generator sev=INFO base=29.6/min fault=25.6/min tpl="Currency changed to CHF" ex=['Currency changed to CHF', 'Currency changed to CHF']
L-0008svc=currency sev=INFO base=117.6/min fault=105.4/min tpl="conversion successful" ex=['conversion successful', 'conversion successful']
L-0009svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=47674, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=47794, siz…']
L-0010svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 47859 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 47976 breach: LogSegment(baseOff…']
L-0011svc=load-generator sev=INFO base=15.6/min fault=14.0/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user e7d345dc-dbdf-11f1-8306-d6b650b5399b', 'Checkout completed for user e7d345dc-dbdf-11f1-8306-d6b650b5399b']
L-0012svc=frontend sev=INFO base=15.2/min fault=13.8/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0013svc=load-generator sev=INFO base=87.2/min fault=79.6/min tpl="User <UUID> adding <NUM> of product <*> to cart" ex=['User ddbf1cf6-dbdf-11f1-8306-d6b650b5399b adding 1 of product OLJCESPC7Z to cart', 'User ddbf1cf6-dbdf-11f1-8306-d6b650b5399b adding 1 of product OLJCESPC7Z to cart']
L-0014svc=kafka sev=INFO base=7.6/min fault=8.4/min tpl="Deleted <*> index <*>" ex=['Deleted offset index /tmp/kafka-logs/__cluster_metadata-0/00000000000000047674.index.deleted.', 'Deleted time index /tmp/kafka-logs/__cluster_metadata-0/00000000000000047674.timeindex.deleted.']
L-0015svc=load-generator sev=INFO base=30.8/min fault=33.6/min tpl="Product added to cart successfully" ex=['Product added to cart successfully', 'Product added to cart successfully']
L-0016svc=kafka sev=INFO base=3.8/min fault=4.2/min tpl="Deleted log /tmp/kafka-logs/__cluster_metadata-<NUM>/<HEX>.log.deleted." ex=['Deleted log /tmp/kafka-logs/__cluster_metadata-0/00000000000000047674.log.deleted.', 'Deleted log /tmp/kafka-logs/__cluster_metadata-0/00000000000000047704.log.deleted.']
L-0017svc=kafka sev=INFO base=3.8/min fault=4.2/min tpl="Deleted producer state snapshot /tmp/kafka-logs/__cluster_metadata-<NUM>/<HEX>.snapshot.deleted" ex=['Deleted producer state snapshot /tmp/kafka-logs/__cluster_metadata-0/00000000000000047674.snapshot.deleted', 'Deleted producer state snapshot /tmp/kafka-logs/__cluster_metadata-0/00000000000000047704.snapshot.deleted']
L-0018svc=load-generator sev=INFO base=14.8/min fault=13.6/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user de20b40c-dbdf-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user de20b40c-dbdf-11f1-8306-d6b650b5399b']
L-0019svc=cart sev=INFORMATION base=184.4/min fault=172.6/min tpl="GetCartAsync called with userId={userId}" ex=['GetCartAsync called with userId={userId}', 'GetCartAsync called with userId={userId}']
L-0020svc=accounting sev=INFORMATION base=14.8/min fault=13.8/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0021svc=cart sev=INFORMATION base=14.8/min fault=13.8/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0022svc=checkout sev=INFO base=14.8/min fault=13.8/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 185.108µs', 'Successful to write message. offset: 0, duration: 142.496µs']
L-0023svc=checkout sev=INFO base=14.8/min fault=13.8/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0024svc=checkout sev=INFO base=14.8/min fault=13.8/min tpl="order placed" ex=['order placed', 'order placed']
L-0025svc=checkout sev=INFO base=14.8/min fault=13.8/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0026svc=checkout sev=INFO base=14.8/min fault=13.8/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
36 lower-ranked lines omitted
Loading runs in your browser…