kafkaQueueProblems-r1
What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)
Service graph during the window
Evidence and answers
5261 prompt tokens
metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean
M-0001svc=accounting metric=db.client.connection.count.rate base=0 delta/30s=[na,new:0.0333,na,new:0.0333,na,+0%,na,+0%,na,+0%]
M-0002svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%]
M-0003svc=accounting metric=dns.lookup.duration.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%]
M-0004svc=accounting metric=dotnet.thread_pool.thread.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%]
M-0005svc=accounting metric=error_ratio base=0 delta/30s=[na,new:0.328,na,new:0.33,na,new:0.33,na,new:0.33,na,new:0.33]
M-0006svc=accounting metric=process.thread.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,new:0.0667,na,+0%,na,new:0.0333]
M-0007svc=ad metric=otel.sdk.processor.span.queue.size base=0 delta/30s=[na,new:2,na,new:14.0,na,new:12.0,na,new:8,na,new:8]
M-0008svc=checkout metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:0.00588,na,new:0.00725]
M-0009svc=checkout metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[new:8,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0010svc=fraud-detection metric=jvm.class.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%]
M-0011svc=fraud-detection metric=kafka.consumer.last_poll_seconds_ago base=0 delta/30s=[na,new:28.0,na,new:88.0,na,new:49.0,na,new:7,na,new:67.0]
M-0012svc=fraud-detection metric=kafka.consumer.records_lag base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:2.02e+03,na,new:2.02e+03]
M-0013svc=fraud-detection metric=kafka.consumer.records_lag_avg base=0 delta/30s=[na,+0%,na,na,na,+0%,na,new:2.02e+03,na,na]
M-0014svc=fraud-detection metric=kafka.consumer.records_lag_max base=0 delta/30s=[na,+0%,na,na,na,+0%,na,new:2.02e+03,na,na]
M-0015svc=fraud-detection metric=kafka.consumer.time_between_poll_avg base=99.2 delta/30s=[na,+5%,na,na,na,+99744%,na,+101763%,na,na]
M-0016svc=fraud-detection metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[na,new:2,na,new:2,na,new:2,na,+0%,na,+0%]
M-0017svc=fraud-detection metric=otel.sdk.processor.span.queue.size base=0 delta/30s=[na,new:1,na,new:1,na,new:1,na,+0%,na,+0%]
M-0018svc=fraud-detection metric=rpc.client.request.size.p50 base=0 delta/30s=[na,new:87.5,na,na,na,na,na,na,na,na]
M-0019svc=fraud-detection metric=rpc.client.request.size.p95 base=0 delta/30s=[na,new:98.8,na,na,na,na,na,na,na,na]
M-0020svc=fraud-detection metric=rpc.client.request.size.p99 base=0 delta/30s=[na,new:99.8,na,na,na,na,na,na,na,na]
M-0021svc=kafka metric=jvm.gc.duration.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0022svc=kafka metric=jvm.thread.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333,na]
M-0023svc=payment metric=error_ratio base=0 delta/30s=[na,+0%,na,new:0.0286,na,+0%,na,+0%,na,+0%]
M-0024svc=payment metric=traces.span.metrics.duration.p50 base=1.21 delta/30s=[na,-3%,na,+1236649%,na,+14%,na,+147%,na,+65%]
M-0025svc=payment metric=traces.span.metrics.duration.p95 base=5.07 delta/30s=[na,-7%,na,+296011%,na,+532%,na,+10560%,na,+9%]
M-0026svc=payment metric=traces.span.metrics.duration.p99 base=5.81 delta/30s=[na,-1%,na,+258105%,na,+699%,na,+12776%,na,+2%]
M-0027svc=product-catalog metric=go.goroutine.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.0667,na]
M-0028svc=product-catalog metric=otel.sdk.span.live base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:1,na,new:1,na]
M-0029svc=recommendation metric=process.runtime.cpython.cpu.utilization base=0 delta/30s=[+0%,na,+0%,na,new:11.8,na,+0%,na,+0%,na]
M-0030svc=unknown metric=kafka.consumer_group.lag base=1 delta/30s=[+70600%,+151400%,+232200%,+302900%,+383700%,+454400%,+525100%,+545300%,+605900%,+676600%]
M-0031svc=unknown metric=kafka.consumer_group.lag_sum base=1 delta/30s=[+70600%,+151400%,+232200%,+302900%,+383700%,+454400%,+525100%,+545300%,+605900%,+676600%]
M-0032svc=fraud-detection metric=kafka.consumer.time_between_poll_max base=103.0 delta/30s=[na,+918%,na,na,na,+96049%,na,+97992%,na,na]
M-0033svc=unknown metric=postgresql.rollbacks base=8 delta/30s=[+6250%,+15000%,+27500%,+36250%,+47775%,+56250%,+65000%,+68750%,+76250%,+85000%]
M-0034svc=accounting metric=db.client.operation.duration.count.rate base=0.26 delta/30s=[na,+12362%,na,+19887%,na,+24503%,na,+14144%,na,+12849%]
M-0035svc=accounting metric=request_rate base=0.78 delta/30s=[na,+11289%,na,+18374%,na,+24503%,na,+15438%,na,+11554%]
875 lower-ranked lines omitted
traces: slowest and errored traces after fault onset, critical path (! = error span)
T-0001trace=181e78f2 dur=4518ms ERROR path: load-generator:user_checkout_single 3860ms > frontend-proxy:POST 3839ms > frontend:POST /api/checkout 3839ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 501ms | errors off path: checkout:POST
T-0002trace=fe7a7936 dur=101040ms ok path: fraud-detection:receive orders 0ms
T-0003trace=f2013ff9 dur=1931ms ERROR path: load-generator:user_checkout_multi 1931ms > frontend-proxy:POST 1853ms > frontend:POST /api/checkout 1853ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 179ms | errors off path: checkout:POST
T-0004trace=72551814 dur=99039ms ok path: fraud-detection:receive orders 0ms
T-0005trace=1671ea68 dur=1062ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 1062ms!
T-0006trace=30c75b7e dur=92038ms ok path: fraud-detection:receive orders 0ms
T-0007trace=9d23da81 dur=833ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 833ms!
T-0008trace=053f4939 dur=17692ms ok path: accounting:order-consumed 17692ms
T-0009trace=4630c52f dur=787ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 787ms!
T-0010trace=694328eb dur=15754ms ok path: accounting:order-consumed 15754ms
T-0011trace=d0766f8d dur=686ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 686ms!
T-0012trace=8395dd4e dur=13559ms ok path: accounting:order-consumed 13559ms
T-0013trace=0a39b9a4 dur=557ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 557ms!
T-0014trace=546a596d dur=10785ms ok path: accounting:order-consumed 10785ms
T-0015trace=2c9b9605 dur=545ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 545ms!
T-0016trace=87c1cfc4 dur=10173ms ok path: accounting:order-consumed 10173ms
T-0017trace=45d7148e dur=507ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 507ms!
T-0018trace=262fa0c7 dur=9146ms ok path: load-generator:browser_add_to_cart 9146ms
T-0019trace=a1864494 dur=360ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 360ms!
T-0020trace=5717c1f7 dur=8657ms ok path: accounting:order-consumed 8657ms
T-0021trace=99a3f6ca dur=333ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 333ms!
T-0022trace=338f1174 dur=8624ms ok path: accounting:order-consumed 8624ms
T-0023trace=e6ec1b52 dur=246ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 246ms!
T-0024trace=1043f01e dur=8542ms ok path: accounting:order-consumed 8542ms
T-0025trace=b94d4613 dur=206ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 206ms!
T-0026trace=23f449a7 dur=8316ms ok path: load-generator:browser_change_currency 8316ms
T-0027trace=72d93dd2 dur=205ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 205ms!
T-0028trace=b6915924 dur=7962ms ok path: accounting:order-consumed 7962ms
T-0029trace=1bd0e8a8 dur=205ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 205ms!
T-0030trace=0949a5b6 dur=7815ms ok path: accounting:order-consumed 7815ms
T-0031trace=fe89f06c dur=182ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 182ms!
T-0032trace=31831d61 dur=7455ms ok path: accounting:order-consumed 7455ms
T-0033trace=a8b0bbd4 dur=164ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 164ms!
T-0034trace=0f5485f9 dur=7268ms ok path: accounting:order-consumed 7268ms
T-0035trace=452f14d4 dur=163ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 163ms!
T-0036trace=3e5327f8 dur=7188ms ok path: accounting:order-consumed 7188ms
T-0037trace=e7a922bd dur=160ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 160ms!
T-0038trace=ed63e36f dur=7175ms ok path: accounting:order-consumed 7175ms
T-0039trace=6e807449 dur=149ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 149ms!
T-0040trace=dc66ea1f dur=7058ms ok path: accounting:order-consumed 7058ms
logs: log templates per service, rate before vs after fault onset, with examples
L-0001svc=accounting sev=INFORMATION base=0.0/min fault=1400.0/min tpl="Duplicate order received, skipping." ex=['Duplicate order received, skipping.', 'Duplicate order received, skipping.']
L-0002svc=accounting sev=INFORMATION base=12.0/min fault=1414.0/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0003svc=fraud-detection sev=INFO base=0.0/min fault=58.8/min tpl="FeatureFlag '<redacted>' is enabled, sleeping <NUM> second" ex=["FeatureFlag '<redacted>' is enabled, sleeping 1 second", "FeatureFlag '<redacted>' is enabled, sleeping 1 second"]
L-0004svc=checkout sev=INFO base=0.0/min fault=14.0/min tpl="Done with #<NUM> messages for overload simulation." ex=['Done with #100 messages for overload simulation.', 'Done with #100 messages for overload simulation.']
L-0005svc=checkout sev=INFO base=0.0/min fault=14.0/min tpl="Warning: FeatureFlag '<redacted>' is activated, overloading queue now." ex=["Warning: FeatureFlag '<redacted>' is activated, overloading queue now.", "Warning: FeatureFlag '<redacted>' is activated, overloading queue now."]
L-0006svc=fraud-detection sev=INFO base=12.0/min fault=58.6/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: dbf1fbe1-dbd2-11f1-82f3-d279f8fa1f6d, and updated total count to: 11263', 'Consumed record with orderId: dbf1fbe1-dbd2-11f1-82f3-d279f8fa1f6d, and updated total count to: 11264']
L-0007svc=ad sev=INFO base=2.2/min fault=4.0/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0008svc=checkout sev=WARN base=0.0/min fault=0.4/min tpl="failed to send order confirmation: failed POST to email service: Post "http://email:<NUM>/send_order_confirmation": context deadline exceed…" ex=['failed to send order confirmation: failed POST to email service: Post "http://email:6060/send_order_confirmation": context deadline exceeded', 'failed to send order confirmation: failed POST to email service: Post "http://email:6060/send_order_confirmation": context deadline exceeded']
L-0009svc=load-generator sev=INFO base=11.2/min fault=14.4/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0010svc=shipping sev=ERROR base=0.0/min fault=0.2/min tpl="failed to resolve integer flag" ex=['failed to resolve integer flag']
L-0011svc=checkout sev=INFO base=0.2/min fault=0.0/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0012svc=load-generator sev=INFO base=13.2/min fault=16.0/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user e0f45808-dbd2-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user e0f45808-dbd2-11f1-8306-d6b650b5399b']
L-0013svc=cart sev=INFORMATION base=12.0/min fault=14.0/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0014svc=checkout sev=INFO base=12.0/min fault=14.0/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 181.759µs', 'Successful to write message. offset: 0, duration: 356.427µs']
L-0015svc=checkout sev=INFO base=12.0/min fault=14.0/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0016svc=checkout sev=INFO base=12.0/min fault=14.0/min tpl="order placed" ex=['order placed', 'order placed']
L-0017svc=checkout sev=INFO base=12.0/min fault=14.0/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0018svc=checkout sev=INFO base=12.0/min fault=14.0/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0019svc=email sev=INFO base=12.0/min fault=14.0/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0020svc=frontend sev=INFO base=12.0/min fault=14.0/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0021svc=payment sev=INFO base=12.0/min fault=14.0/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0022svc=payment sev=INFO base=12.0/min fault=14.0/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0023svc=shipping sev=INFO base=12.0/min fault=14.0/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0024svc=kafka sev=INFO base=2.0/min fault=2.4/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=36544, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=36574, siz…']
L-0025svc=kafka sev=INFO base=2.0/min fault=2.4/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 36705 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 36744 breach: LogSegment(baseOff…']
L-0026svc=checkout sev=INFO base=12.0/min fault=13.6/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
40 lower-ranked lines omitted
Loading runs in your browser…