productCatalogFailure-r2
What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)
Service graph during the window
Evidence and answers
5348 prompt tokens
metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean
M-0001svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%]
M-0002svc=accounting metric=dns.lookup.duration.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%]
M-0003svc=checkout metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,new:0.00521,na,new:0.00595,na,+0%]
M-0004svc=checkout metric=go.goroutine.count.rate base=0 delta/30s=[na,+0%,na,new:0.133,na,+0%,na,+0%,na,+0%]
M-0005svc=fraud-detection metric=otel.sdk.processor.span.queue.size base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:2]
M-0006svc=load-generator metric=process.runtime.cpython.cpu.utilization base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:18.9,na,+0%]
M-0007svc=payment metric=error_ratio base=0 delta/30s=[na,+0%,na,new:0.0286,na,+0%,na,+0%,na,+0%]
M-0008svc=payment metric=traces.span.metrics.duration.p50 base=1.15 delta/30s=[na,+6%,na,+1302899%,na,+508%,na,+25%,na,+16%]
M-0009svc=payment metric=traces.span.metrics.duration.p95 base=3.27 delta/30s=[na,+6%,na,+459084%,na,+1890%,na,+13%,na,+59%]
M-0010svc=payment metric=traces.span.metrics.duration.p99 base=3.85 delta/30s=[na,+1%,na,+389173%,na,+2313%,na,+2%,na,+52%]
M-0011svc=quote metric=otel.sdk.processor.span.queue.size base=0 delta/30s=[na,+0%,na,+0%,na,new:15.0,na,new:12.0,na,new:9]
M-0012svc=recommendation metric=error_ratio base=0 delta/30s=[na,new:0.00299,na,+0%,na,+0%,na,+0%,na,+0%]
M-0013svc=recommendation metric=process.thread.count.rate base=0 delta/30s=[na,+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%]
M-0014svc=recommendation metric=traces.span.metrics.duration.p50 base=3.68 delta/30s=[na,+408037%,na,-8%,na,-5%,na,+6%,na,-11%]
M-0015svc=recommendation metric=traces.span.metrics.duration.p95 base=6.85 delta/30s=[na,+218955%,na,-14%,na,-0%,na,+5%,na,-17%]
M-0016svc=shipping metric=http.server.active_requests base=0 delta/30s=[na,+0%,na,+0%,na,new:1,na,+0%,na,+0%]
M-0017svc=recommendation metric=traces.span.metrics.duration.p99 base=23.5 delta/30s=[na,+63793%,na,-68%,na,-62%,na,+54%,na,-67%]
M-0018svc=checkout metric=http.client.request.duration.p50 base=0.0418 delta/30s=[na,-18%,na,+10%,na,+4091%,na,+4091%,na,+80%]
M-0019svc=product-catalog metric=go.goroutine.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,-100%,na,+3900%,na,-100%]
M-0020svc=product-catalog metric=otel.sdk.span.live base=0.2 delta/30s=[na,-100%,na,-100%,na,-100%,na,+1900%,na,-100%]
M-0021svc=product-catalog metric=otel.sdk.processor.log.queue.size base=0.8 delta/30s=[na,-100%,na,-100%,na,-100%,na,+650%,na,+1775%]
M-0022svc=recommendation metric=otel.sdk.processor.span.queue.size base=2.4 delta/30s=[na,-100%,na,-100%,na,-100%,na,-100%,na,+1275%]
M-0023svc=email metric=traces.span.metrics.duration.p99 base=432.5 delta/30s=[na,-13%,na,+70%,na,+959%,na,+212%,na,-9%]
M-0024svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,-100%,-100%,-100%,-100%,-100%,-100%,+900%,-100%,-100%]
M-0025svc=cart metric=traces.span.metrics.duration.p99 base=503.5 delta/30s=[na,+50%,na,-27%,na,+810%,na,-26%,na,-24%]
M-0026svc=email metric=traces.span.metrics.duration.p95 base=322.7 delta/30s=[na,-10%,na,+49%,na,+799%,na,+253%,na,+12%]
M-0027svc=payment metric=v8js.gc.duration.p99 base=0.0294 delta/30s=[na,+729%,na,-67%,na,+153%,na,+66%,na,+236%]
M-0028svc=payment metric=v8js.gc.duration.p95 base=0.0272 delta/30s=[na,+709%,na,-68%,na,+167%,na,+65%,na,+249%]
M-0029svc=cart metric=http.server.request.duration.p95 base=0.302 delta/30s=[na,+24%,na,-5%,na,+579%,na,-25%,na,+24%]
M-0030svc=cart metric=traces.span.metrics.duration.p95 base=437.5 delta/30s=[na,+33%,na,-45%,na,+563%,na,-38%,na,-27%]
M-0031svc=checkout metric=rpc.client.call.duration.p95 base=0.299 delta/30s=[na,+21%,na,+84%,na,+559%,na,-25%,na,+34%]
M-0032svc=cart metric=http.server.request.duration.p99 base=0.38 delta/30s=[na,+25%,na,+20%,na,+534%,na,-36%,na,+25%]
M-0033svc=checkout metric=rpc.client.call.duration.p99 base=0.38 delta/30s=[na,+24%,na,+87%,na,+530%,na,-35%,na,+26%]
M-0034svc=load-generator metric=otel.sdk.span.live base=1 delta/30s=[na,+500%,na,+0%,na,+100%,na,+100%,na,+300%]
M-0035svc=payment metric=v8js.gc.duration.p50 base=0.00858 delta/30s=[na,-61%,na,-64%,na,+483%,na,+104%,na,+104%]
M-0036svc=frontend-web metric=traces.span.metrics.duration.p99 base=1.55e+03 delta/30s=[na,+25%,na,+163%,na,+14%,na,+169%,na,+456%]
M-0037svc=payment metric=nodejs.eventloop.delay.stddev base=0.000481 delta/30s=[na,+236%,na,+132%,na,+418%,na,+29%,na,+147%]
876 lower-ranked lines omitted
traces: slowest and errored traces after fault onset, critical path (! = error span)
T-0001trace=1bc8bce1 dur=3640ms ERROR path: load-generator:user_checkout_multi 2771ms > frontend-proxy:POST 2110ms > frontend:POST /api/checkout 2109ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 502ms | errors off path: checkout:POST
T-0002trace=6edd4fc3 dur=16497ms ok path: accounting:order-consumed 16497ms
T-0003trace=21225175 dur=2187ms ERROR path: load-generator:user_checkout_multi 2187ms > frontend-proxy:POST 2131ms > frontend:POST /api/checkout 2130ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 211ms | errors off path: checkout:POST
T-0004trace=867e5f17 dur=15376ms ok path: accounting:order-consumed 15376ms
T-0005trace=3a9bc4fa dur=491ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 491ms!
T-0006trace=2a0791bd dur=14360ms ok path: accounting:order-consumed 14360ms
T-0007trace=56d2ca59 dur=489ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 489ms!
T-0008trace=6d84aa53 dur=13482ms ok path: accounting:order-consumed 13482ms
T-0009trace=226cde1b dur=351ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 351ms!
T-0010trace=56769f50 dur=12695ms ok path: accounting:order-consumed 12695ms
T-0011trace=dd85dab6 dur=317ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 317ms!
T-0012trace=5682ce4a dur=12120ms ok path: accounting:order-consumed 12120ms
T-0013trace=9b5a0c19 dur=316ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 316ms!
T-0014trace=9cf4d4c2 dur=11854ms ok path: accounting:order-consumed 11854ms
T-0015trace=f63d0626 dur=315ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 315ms!
T-0016trace=ad9c93ba dur=11669ms ok path: accounting:order-consumed 11669ms
T-0017trace=345ea378 dur=314ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 314ms!
T-0018trace=a5316974 dur=11234ms ok path: accounting:order-consumed 11234ms
T-0019trace=6472c99d dur=314ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 314ms!
T-0020trace=40c196e9 dur=10814ms ok path: accounting:order-consumed 10814ms
T-0021trace=7d4c2eee dur=313ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 313ms!
T-0022trace=ccebe950 dur=10494ms ok path: accounting:order-consumed 10494ms
T-0023trace=5ddb5d2d dur=313ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 313ms!
T-0024trace=1dd6079a dur=8567ms ok path: accounting:order-consumed 8567ms
T-0025trace=eca5a688 dur=193ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 193ms!
T-0026trace=02ebc123 dur=8348ms ok path: accounting:order-consumed 8348ms
T-0027trace=4b6ecc37 dur=138ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 138ms!
T-0028trace=d55b59bb dur=8143ms ok path: accounting:order-consumed 8143ms
T-0029trace=a561271f dur=133ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 133ms!
T-0030trace=24928dcb dur=7760ms ok path: accounting:order-consumed 7760ms
T-0031trace=bd1de3fb dur=112ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 112ms!
T-0032trace=f5c9f719 dur=7491ms ok path: load-generator:browser_add_to_cart 7491ms
T-0033trace=0f904ec1 dur=62ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 62ms!
T-0034trace=22b15df9 dur=7262ms ok path: accounting:order-consumed 7262ms
T-0035trace=81bf45a7 dur=33ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 33ms!
T-0036trace=eb0b6a30 dur=6783ms ok path: accounting:order-consumed 6783ms
T-0037trace=aeb890d2 dur=22ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 22ms!
T-0038trace=5b0f7faa dur=6658ms ok path: accounting:order-consumed 6658ms
T-0039trace=3965adc0 dur=19ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 19ms!
T-0040trace=746a3cf9 dur=6574ms ok path: frontend-proxy:POST 6560ms > flagd:flagd.evaluation.v1.Service/EventStream 6550ms
logs: log templates per service, rate before vs after fault onset, with examples
L-0001svc=load-generator sev=INFO base=12.0/min fault=17.6/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0002svc=load-generator sev=INFO base=10.4/min fault=15.2/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user 6d500e14-dbd8-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user 6d500e14-dbd8-11f1-8306-d6b650b5399b']
L-0003svc=checkout sev=WARN base=0.0/min fault=0.4/min tpl="failed to send order confirmation: failed POST to email service: Post "http://email:<NUM>/send_order_confirmation": context deadline exceed…" ex=['failed to send order confirmation: failed POST to email service: Post "http://email:6060/send_order_confirmation": context deadline exceeded', 'failed to send order confirmation: failed POST to email service: Post "http://email:6060/send_order_confirmation": context deadline exceeded']
L-0004svc=checkout sev=INFO base=0.2/min fault=0.0/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0005svc=accounting sev=INFORMATION base=11.0/min fault=12.8/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0006svc=cart sev=INFORMATION base=11.0/min fault=12.8/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0007svc=checkout sev=INFO base=11.0/min fault=12.8/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 142.188µs', 'Successful to write message. offset: 0, duration: 187.93µs']
L-0008svc=checkout sev=INFO base=11.0/min fault=12.8/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0009svc=checkout sev=INFO base=11.0/min fault=12.8/min tpl="order placed" ex=['order placed', 'order placed']
L-0010svc=checkout sev=INFO base=11.0/min fault=12.8/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0011svc=checkout sev=INFO base=11.0/min fault=12.8/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0012svc=email sev=INFO base=11.0/min fault=12.8/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0013svc=fraud-detection sev=INFO base=11.0/min fault=12.8/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: 6d56beff-dbd8-11f1-82f3-d279f8fa1f6d, and updated total count to: 19096', 'Consumed record with orderId: 6df64f7b-dbd8-11f1-82f3-d279f8fa1f6d, and updated total count to: 19097']
L-0014svc=frontend sev=INFO base=11.0/min fault=12.8/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0015svc=payment sev=INFO base=11.0/min fault=12.8/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0016svc=payment sev=INFO base=11.0/min fault=12.8/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0017svc=shipping sev=INFO base=11.0/min fault=12.8/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0018svc=kafka sev=INFO base=2.0/min fault=2.4/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=41343, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=41373, siz…']
L-0019svc=kafka sev=INFO base=2.0/min fault=2.4/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 41502 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 41541 breach: LogSegment(baseOff…']
L-0020svc=load-generator sev=INFO base=69.2/min fault=77.6/min tpl="User <UUID> adding <NUM> of product <*> to cart" ex=['User 6d500e14-dbd8-11f1-8306-d6b650b5399b adding 2 of product 9SIQT8TOJO to cart', 'User 6d500e14-dbd8-11f1-8306-d6b650b5399b adding 2 of product 9SIQT8TOJO to cart']
L-0021svc=checkout sev=INFO base=11.0/min fault=12.4/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0022svc=load-generator sev=INFO base=30.8/min fault=27.6/min tpl="Currency changed to CHF" ex=['Currency changed to CHF', 'Currency changed to CHF']
L-0023svc=kafka sev=INFO base=1.0/min fault=0.8/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=41403, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=41553, siz…']
L-0024svc=kafka sev=INFO base=1.0/min fault=0.8/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 41619 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 41736 breach: LogSegment(baseOff…']
37 lower-ranked lines omitted
Loading runs in your browser…