kafkaQueueProblems-r2
What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)
Service graph during the window
Evidence and answers
5116 prompt tokens
metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean
M-0001svc=accounting metric=error_ratio base=0 delta/30s=[+0%,na,+0%,na,new:0.491,na,new:0.33,na,new:0.33,na]
M-0002svc=ad metric=error_ratio base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.00917,na,+0%,na]
M-0003svc=ad metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0004svc=ad metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0005svc=ad metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0006svc=ad metric=traces.span.metrics.duration.p50 base=1 delta/30s=[-0%,na,+2%,na,+1%,na,+1494562%,na,-0%,na]
M-0007svc=ad metric=traces.span.metrics.duration.p95 base=1.91 delta/30s=[-0%,na,+2%,na,+1%,na,+786564%,na,-0%,na]
M-0008svc=ad metric=traces.span.metrics.duration.p99 base=2.16 delta/30s=[-8%,na,+40%,na,+127%,na,+695633%,na,-8%,na]
M-0009svc=fraud-detection metric=error_ratio base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0164,na,+0%,na]
M-0010svc=fraud-detection metric=kafka.consumer.last_heartbeat_seconds_ago base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:2,na,new:2,na]
M-0011svc=fraud-detection metric=kafka.consumer.last_poll_seconds_ago base=0 delta/30s=[new:6,na,new:66.0,na,new:27.0,na,new:87.0,na,new:147.0,na]
M-0012svc=fraud-detection metric=kafka.consumer.records_lag base=0 delta/30s=[+0%,na,+0%,na,new:1.92e+03,na,new:1.92e+03,na,new:1.92e+03,na]
M-0013svc=fraud-detection metric=kafka.consumer.records_lag_avg base=0 delta/30s=[+0%,na,na,na,new:1.92e+03,na,na,na,na,na]
M-0014svc=fraud-detection metric=kafka.consumer.records_lag_max base=0 delta/30s=[+0%,na,na,na,new:1.92e+03,na,na,na,na,na]
M-0015svc=fraud-detection metric=rpc.client.duration.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0016svc=fraud-detection metric=rpc.client.request.size.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0017svc=fraud-detection metric=rpc.client.response.size.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0018svc=product-catalog metric=go.goroutine.count.rate base=0 delta/30s=[new:0.0667,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0019svc=unknown metric=kafka.consumer_group.lag base=0.7 delta/30s=[+100900%,+216329%,+259614%,+389471%,+519329%,+663614%,+822329%,+894471%,+1053186%,+1183043%]
M-0020svc=unknown metric=kafka.consumer_group.lag_sum base=0.7 delta/30s=[+100900%,+216329%,+259614%,+389471%,+519329%,+663614%,+822329%,+894471%,+1053186%,+1183043%]
M-0021svc=unknown metric=postgresql.rollbacks base=0 delta/30s=[new:600.0,new:1.3e+03,new:1.6e+03,new:2.7e+03,new:3.6e+03,new:4.6e+03,new:5.6e+03,new:6.2e+03,new:7.33e+03,new:8.2e+03]
M-0022svc=fraud-detection metric=kafka.consumer.time_between_poll_avg base=99.3 delta/30s=[+3%,na,na,na,+99685%,na,na,na,na,na]
M-0023svc=fraud-detection metric=kafka.consumer.time_between_poll_max base=101.4 delta/30s=[+928%,na,na,na,+97581%,na,na,na,na,na]
M-0024svc=fraud-detection metric=traces.span.metrics.duration.p50 base=55.3 delta/30s=[-19%,na,+2071%,na,+2071%,na,+27042%,na,+2071%,na]
M-0025svc=accounting metric=request_rate base=1.17 delta/30s=[+3%,na,+8600%,na,+17769%,na,+16349%,na,+11426%,na]
M-0026svc=accounting metric=db.client.operation.duration.count.rate base=0.38 delta/30s=[+2654%,na,+11418%,na,+16733%,na,+15847%,na,+11418%,na]
M-0027svc=accounting metric=traces.span.metrics.duration.count.rate base=1.17 delta/30s=[+3%,na,+8600%,na,+13483%,na,+16349%,na,+11426%,na]
M-0028svc=fraud-detection metric=traces.span.metrics.duration.p95 base=95.4 delta/30s=[-1%,na,+1347%,na,+1347%,na,+15624%,na,+1347%,na]
M-0029svc=fraud-detection metric=traces.span.metrics.duration.p99 base=99.1 delta/30s=[-0%,na,+1309%,na,+1309%,na,+15039%,na,+1309%,na]
M-0030svc=accounting metric=dotnet.thread_pool.work_item.count.rate base=3.44 delta/30s=[+1720%,na,+7493%,na,+10947%,na,+10366%,na,+7442%,na]
M-0031svc=fraud-detection metric=kafka.consumer.incoming_byte_total.rate base=273.3 delta/30s=[+432%,na,-96%,na,+10813%,na,-96%,na,-96%,na]
M-0032svc=fraud-detection metric=kafka.consumer.heartbeat_response_time_max base=1 delta/30s=[+9900%,na,+10000%,na,+10000%,na,+10000%,na,+10000%,na]
M-0033svc=fraud-detection metric=kafka.consumer.incoming_byte_rate base=162.5 delta/30s=[+278%,na,-96%,na,+9072%,na,-97%,na,-97%,na]
861 lower-ranked lines omitted
traces: slowest and errored traces after fault onset, critical path (! = error span)
T-0001trace=e037a85b dur=873ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 873ms!
T-0002trace=074e8af0 dur=196114ms ok path: fraud-detection:receive orders 7ms
T-0003trace=bb594235 dur=867ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 867ms!
T-0004trace=d3e56dc8 dur=99052ms ok path: fraud-detection:receive orders 0ms
T-0005trace=33b058fa dur=866ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 866ms!
T-0006trace=6a289fa9 dur=18015ms ok path: accounting:order-consumed 18015ms
T-0007trace=959e7b68 dur=865ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 865ms!
T-0008trace=f76d2b24 dur=14874ms ok path: accounting:order-consumed 14874ms
T-0009trace=712132b2 dur=858ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 858ms!
T-0010trace=7b903b91 dur=13165ms ok path: accounting:order-consumed 13165ms
T-0011trace=1b4829f8 dur=581ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 581ms!
T-0012trace=0ceb6422 dur=11576ms ok path: accounting:order-consumed 11576ms
T-0013trace=bc9940e2 dur=113ms ERROR path: accounting:order-consumed 113ms!
T-0014trace=7429cebd dur=11390ms ok path: accounting:order-consumed 11390ms
T-0015trace=0d9ebe80 dur=66ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 66ms!
T-0016trace=c3ea41c7 dur=10503ms ok path: accounting:order-consumed 10503ms
T-0017trace=59828568 dur=30ms ERROR path: accounting:order-consumed 30ms!
T-0018trace=2a607c15 dur=8699ms ok path: accounting:order-consumed 8699ms
T-0019trace=08e4bb43 dur=13ms ERROR path: accounting:order-consumed 13ms!
T-0020trace=6a1e726e dur=7396ms ok path: accounting:order-consumed 7396ms
T-0021trace=a5e70dca dur=10ms ERROR path: accounting:order-consumed 10ms!
T-0022trace=48a70187 dur=6799ms ok path: accounting:order-consumed 6799ms
T-0023trace=d9a93a12 dur=9ms ERROR path: accounting:order-consumed 9ms!
T-0024trace=aa996097 dur=6604ms ok path: accounting:order-consumed 6604ms
T-0025trace=11735a9c dur=9ms ERROR path: accounting:order-consumed 9ms!
T-0026trace=7b4722c0 dur=6473ms ok path: accounting:order-consumed 6473ms
T-0027trace=13934600 dur=8ms ERROR path: accounting:order-consumed 8ms!
T-0028trace=d870e949 dur=6374ms ok path: accounting:order-consumed 6374ms
T-0029trace=286bf4a0 dur=7ms ERROR path: accounting:order-consumed 7ms!
T-0030trace=3d3ab286 dur=6065ms ok path: accounting:order-consumed 6065ms
T-0031trace=baecaf49 dur=7ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 7ms!
T-0032trace=c3b08f27 dur=6043ms ok path: accounting:order-consumed 6043ms
T-0033trace=abb3bb67 dur=7ms ERROR path: accounting:order-consumed 7ms!
T-0034trace=39b3d808 dur=5999ms ok path: accounting:order-consumed 5999ms
T-0035trace=dbcb0c29 dur=7ms ERROR path: accounting:order-consumed 7ms!
T-0036trace=71b68457 dur=5991ms ok path: accounting:order-consumed 5991ms
T-0037trace=32479ea7 dur=6ms ERROR path: accounting:order-consumed 6ms!
T-0038trace=bc5d15c4 dur=5875ms ok path: accounting:order-consumed 5875ms
T-0039trace=b255349e dur=6ms ERROR path: accounting:order-consumed 6ms!
T-0040trace=eb53a3cb dur=5766ms ok path: accounting:order-consumed 5766ms
logs: log templates per service, rate before vs after fault onset, with examples
L-0001svc=accounting sev=INFORMATION base=0.0/min fault=1680.0/min tpl="Duplicate order received, skipping." ex=['Duplicate order received, skipping.', 'Duplicate order received, skipping.']
L-0002svc=accounting sev=INFORMATION base=13.8/min fault=1697.0/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0003svc=fraud-detection sev=INFO base=0.0/min fault=59.4/min tpl="FeatureFlag '<redacted>' is enabled, sleeping <NUM> second" ex=["FeatureFlag '<redacted>' is enabled, sleeping 1 second", "FeatureFlag '<redacted>' is enabled, sleeping 1 second"]
L-0004svc=checkout sev=INFO base=0.0/min fault=16.8/min tpl="Done with #<NUM> messages for overload simulation." ex=['Done with #100 messages for overload simulation.', 'Done with #100 messages for overload simulation.']
L-0005svc=checkout sev=INFO base=0.0/min fault=16.8/min tpl="Warning: FeatureFlag '<redacted>' is activated, overloading queue now." ex=["Warning: FeatureFlag '<redacted>' is activated, overloading queue now.", "Warning: FeatureFlag '<redacted>' is activated, overloading queue now."]
L-0006svc=fraud-detection sev=INFO base=13.8/min fault=59.4/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: b4f3eec6-dbb5-11f1-82f3-d279f8fa1f6d, and updated total count to: 1054', 'Consumed record with orderId: b69126dd-dbb5-11f1-82f3-d279f8fa1f6d, and updated total count to: 1055']
L-0007svc=shipping sev=INFO base=1.8/min fault=3.2/min tpl="" ex=['', '']
L-0008svc=load-generator sev=INFO base=12.8/min fault=19.6/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user b679274c-dbb5-11f1-8306-d6b650b5399b', 'Checkout completed for user b679274c-dbb5-11f1-8306-d6b650b5399b']
L-0009svc=checkout sev=INFO base=13.6/min fault=17.0/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0010svc=cart sev=INFORMATION base=13.8/min fault=17.0/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0011svc=checkout sev=INFO base=13.8/min fault=17.0/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 179.288µs', 'Successful to write message. offset: 0, duration: 213.369µs']
L-0012svc=checkout sev=INFO base=13.8/min fault=17.0/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0013svc=checkout sev=INFO base=13.8/min fault=17.0/min tpl="order placed" ex=['order placed', 'order placed']
L-0014svc=checkout sev=INFO base=13.8/min fault=17.0/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0015svc=checkout sev=INFO base=13.8/min fault=17.0/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0016svc=email sev=INFO base=13.8/min fault=17.0/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0017svc=frontend sev=INFO base=13.8/min fault=17.0/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0018svc=payment sev=INFO base=13.8/min fault=17.0/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0019svc=payment sev=INFO base=13.8/min fault=17.0/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0020svc=shipping sev=INFO base=13.8/min fault=17.0/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0021svc=checkout sev=INFO base=0.2/min fault=0.0/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0022svc=checkout sev=WARN base=0.2/min fault=0.0/min tpl="failed to send order confirmation: failed POST to email service: Post "http://email:<NUM>/send_order_confirmation": context deadline exceed…" ex=['failed to send order confirmation: failed POST to email service: Post "http://email:6060/send_order_confirmation": context deadline exceeded']
L-0023svc=load-generator sev=INFO base=38.0/min fault=45.6/min tpl="User viewing cart" ex=['User viewing cart', 'User viewing cart']
L-0024svc=quote sev=INFO base=29.8/min fault=34.6/min tpl="Calculated quote" ex=['Calculated quote', 'Calculated quote']
L-0025svc=shipping sev=INFO base=29.8/min fault=34.6/min tpl="Requesting quote" ex=['Requesting quote', 'Requesting quote']
L-0026svc=shipping sev=INFO base=29.8/min fault=34.6/min tpl="Sending Quote" ex=['Sending Quote', 'Sending Quote']
L-0027svc=load-generator sev=INFO base=14.0/min fault=12.0/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0028svc=kafka sev=INFO base=2.0/min fault=2.4/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=11469, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=11499, siz…']
L-0029svc=kafka sev=INFO base=2.0/min fault=2.4/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 11628 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 11667 breach: LogSegment(baseOff…']
37 lower-ranked lines omitted
Loading runs in your browser…