emailMemoryLeak-r2
What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)
Service graph during the window
Evidence and answers
5816 prompt tokens
metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean
M-0001svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0002svc=accounting metric=dotnet.thread_pool.thread.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0003svc=ad metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[new:1,na,new:1,na,new:1,na,new:1,na,+0%,na]
M-0004svc=flagd metric=error_ratio base=0 delta/30s=[+0%,na,new:0.75,na,+0%,na,+0%,na,+0%,na]
M-0005svc=flagd metric=request_rate base=0 delta/30s=[+0%,na,new:0.133,na,new:0.0667,na,+0%,na,+0%,na]
M-0006svc=flagd metric=traces.span.metrics.duration.count.rate base=0 delta/30s=[+0%,na,new:0.133,na,new:0.0667,na,+0%,na,+0%,na]
M-0007svc=kafka metric=jvm.gc.duration.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0008svc=load-generator metric=process.thread.count.rate base=0 delta/30s=[+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%,na]
M-0009svc=payment metric=error_ratio base=0 delta/30s=[+0%,na,+0%,na,new:0.0909,na,+0%,na,+0%,na]
M-0010svc=payment metric=traces.span.metrics.duration.p50 base=1 delta/30s=[+0%,na,+0%,na,+1499900%,na,+1499900%,na,+1499900%,na]
M-0011svc=payment metric=traces.span.metrics.duration.p95 base=1.9 delta/30s=[+0%,na,+0%,na,+789374%,na,+789374%,na,+789374%,na]
M-0012svc=payment metric=traces.span.metrics.duration.p99 base=1.98 delta/30s=[+0%,na,+0%,na,+757476%,na,+757476%,na,+757476%,na]
M-0013svc=quote metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:1,na]
M-0014svc=ad metric=otel.sdk.processor.span.queue.size base=1 delta/30s=[+100%,na,+1800%,na,+1000%,na,+2000%,na,+3700%,na]
M-0015svc=product-catalog metric=go.goroutine.count.rate base=0.133 delta/30s=[+1550%,na,-100%,na,-100%,na,+200%,na,-100%,na]
M-0016svc=load-generator metric=process.open_file_descriptor.count.rate base=0.00667 delta/30s=[+400%,na,-100%,na,+1400%,na,-100%,na,-100%,na]
M-0017svc=recommendation metric=otel.sdk.processor.log.queue.size base=0.2 delta/30s=[+900%,na,+900%,na,+1400%,na,+900%,na,+400%,na]
M-0018svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,-100%,-100%,-100%,-100%,-100%,+900%,-100%,-100%,-100%]
M-0019svc=unknown metric=system.processes.count.rate base=0.113 delta/30s=[+106%,na,+429%,na,+312%,na,+694%,na,+47%,na]
M-0020svc=fraud-detection metric=otel.sdk.processor.span.queue.size base=2.2 delta/30s=[+627%,na,+82%,na,+82%,na,-100%,na,-100%,na]
M-0021svc=ad metric=jvm.class.count.rate base=0.02 delta/30s=[+567%,na,+567%,na,-100%,na,-100%,na,-100%,na]
M-0022svc=product-catalog metric=otel.sdk.span.live base=3.8 delta/30s=[+532%,na,-74%,na,-74%,na,+32%,na,-100%,na]
M-0023svc=unknown metric=kafka.consumer_group.lag base=1 delta/30s=[-100%,-100%,-100%,-100%,-100%,-100%,-100%,-100%,-100%,+500%]
M-0024svc=unknown metric=kafka.consumer_group.lag_sum base=1 delta/30s=[-100%,-100%,-100%,-100%,-100%,-100%,-100%,-100%,-100%,+500%]
M-0025svc=frontend-web metric=traces.span.metrics.duration.p95 base=856.2 delta/30s=[+423%,na,+85%,na,-8%,na,-8%,na,-9%,na]
M-0026svc=kafka metric=kafka.purgatory.size base=75.6 delta/30s=[-47%,na,-76%,na,-42%,na,+98%,na,+416%,na]
M-0027svc=accounting metric=http.client.connection.duration.count.rate base=0.0133 delta/30s=[-100%,na,-100%,na,-100%,na,-100%,na,+400%,na]
M-0028svc=ad metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,+400%,na,-100%,na,-100%,na]
M-0029svc=cart metric=dotnet.timer.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,+400%,na,-100%,na,-100%,na]
M-0030svc=fraud-detection metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[-100%,na,+400%,na,-100%,na,-100%,na,-100%,na]
M-0031svc=fraud-detection metric=jvm.thread.count.rate base=0.00667 delta/30s=[+400%,na,-100%,na,+400%,na,+400%,na,-100%,na]
M-0032svc=product-catalog metric=otel.sdk.processor.log.queue.size base=3.6 delta/30s=[-100%,na,+344%,na,+39%,na,+289%,na,+39%,na]
M-0033svc=frontend-web metric=traces.span.metrics.duration.p99 base=2.19e+03 delta/30s=[+296%,na,-12%,na,-23%,na,-22%,na,-16%,na]
M-0034svc=checkout metric=go.goroutine.count.rate base=0.1 delta/30s=[-100%,na,+67%,na,-100%,na,+133%,na,+267%,na]
M-0035svc=otelcol-contrib metric=otelcol_exporter_in_flight_requests base=2.8 delta/30s=[+7%,na,-64%,na,+43%,na,+7%,na,+257%,na]
M-0036svc=ad metric=jvm.thread.count.rate base=0.02 delta/30s=[+233%,na,-100%,na,-100%,na,+67%,na,+67%,na]
M-0037svc=load-generator metric=process.runtime.cpython.gc_count base=169.4 delta/30s=[-41%,na,+56%,na,-96%,na,+221%,na,+23%,na]
875 lower-ranked lines omitted
traces: slowest and errored traces after fault onset, critical path (! = error span)
T-0001trace=4be29074 dur=165136ms ERROR path: load-generator:user_checkout_multi 16552ms! > frontend-proxy:POST 15003ms! > frontend:POST 15003ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 507ms | errors off path: checkout:POST
T-0002trace=33b49386 dur=117643ms ok path: accounting:order-consumed 117643ms
T-0003trace=9f285188 dur=145609ms ERROR path: load-generator:user_checkout_multi 16539ms! > frontend-proxy:POST 14997ms! > frontend:POST 14999ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 507ms | errors off path: checkout:POST
T-0004trace=fada3b4a dur=82995ms ok path: cart:flagd.evaluation.v2.Service/ResolveBoolean 82995ms
T-0005trace=652a6906 dur=144680ms ERROR path: load-generator:user_checkout_multi 16550ms! > frontend-proxy:POST 14995ms! > frontend:POST 14997ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 507ms | errors off path: checkout:POST
T-0006trace=6cd83cf5 dur=63642ms ERROR path: load-generator:user_checkout_multi 17067ms! > frontend-proxy:POST 15004ms! > frontend:POST 15006ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 509ms | errors off path: checkout:POST
T-0007trace=45cae813 dur=144652ms ERROR path: load-generator:user_checkout_single 15515ms! > frontend-proxy:POST 15000ms! > frontend:POST 15002ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 507ms | errors off path: checkout:POST
T-0008trace=a6fcabc7 dur=63338ms ERROR path: load-generator:user_checkout_single 15516ms! > frontend-proxy:POST 14999ms! > frontend:POST 14999ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 501ms | errors off path: checkout:POST
T-0009trace=a8aa6bce dur=141627ms ERROR path: load-generator:user_checkout_single 15517ms! > frontend-proxy:POST 14997ms! > frontend:POST 14998ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 500ms | errors off path: checkout:POST
T-0010trace=c5490b9f dur=62840ms ERROR path: load-generator:user_checkout_single 15458ms > frontend-proxy:POST 14944ms > frontend:POST /api/checkout 14943ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 501ms | errors off path: checkout:POST
T-0011trace=9c2c3c74 dur=139069ms ERROR path: load-generator:user_checkout_single 15517ms! > frontend-proxy:POST 14999ms! > frontend:POST 14999ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 509ms | errors off path: checkout:POST
T-0012trace=194b6ac0 dur=61685ms ok path: cart:flagd.evaluation.v2.Service/ResolveBoolean 61685ms
T-0013trace=fe39972d dur=133776ms ERROR path: load-generator:user_checkout_single 15521ms! > frontend-proxy:POST 14999ms! > frontend:POST 14999ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 501ms | errors off path: checkout:POST
T-0014trace=d10142e8 dur=61658ms ERROR path: load-generator:user_checkout_multi 14269ms > frontend-proxy:POST 13241ms > frontend:POST /api/checkout 13241ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 506ms | errors off path: checkout:POST
T-0015trace=c426a639 dur=117094ms ERROR path: load-generator:user_checkout_multi 17061ms! > frontend-proxy:POST 15000ms! > frontend:POST 15000ms > checkout:oteldemo.CheckoutService/PlaceOrder 114522ms > cart:POST /oteldemo.CartService/EmptyCart 41690ms
T-0016trace=d69f3e2f dur=57568ms ERROR path: load-generator:user_checkout_single 15514ms! > frontend-proxy:POST 15000ms! > frontend:POST 15001ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 501ms | errors off path: checkout:POST
T-0017trace=5d9951b9 dur=104007ms ERROR path: load-generator:user_checkout_multi 16029ms! > frontend-proxy:POST 14998ms! > frontend:POST 15000ms > checkout:oteldemo.CheckoutService/PlaceOrder 102471ms > cart:POST /oteldemo.CartService/EmptyCart 100402ms
T-0018trace=d2a2037c dur=49630ms ok path: accounting:order-consumed 49630ms
T-0019trace=fd0d6ccb dur=91042ms ERROR path: load-generator:user_checkout_single 15516ms! > frontend-proxy:POST 14996ms! > frontend:POST 14998ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 501ms | errors off path: checkout:POST
T-0020trace=49a88ed2 dur=48541ms ERROR path: load-generator:user_checkout_single 15515ms! > frontend-proxy:POST 14999ms! > frontend:POST 15002ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 501ms | errors off path: checkout:POST
T-0021trace=0ed64072 dur=86586ms ERROR path: load-generator:user_checkout_multi 16541ms! > frontend-proxy:POST 14997ms! > frontend:POST 14999ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 507ms | errors off path: checkout:POST
T-0022trace=7e53cf01 dur=46573ms ok path: cart:flagd.evaluation.v2.Service/ResolveBoolean 46573ms
T-0023trace=22f1a556 dur=85922ms ERROR path: load-generator:user_checkout_single 15513ms! > frontend-proxy:POST 14997ms! > frontend:POST 14999ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 500ms | errors off path: checkout:POST
T-0024trace=89b64939 dur=41858ms ERROR path: load-generator:user_checkout_single 15517ms! > frontend-proxy:POST 15003ms! > frontend:POST 15005ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 501ms | errors off path: checkout:POST
T-0025trace=cefabc14 dur=82041ms ERROR path: load-generator:user_checkout_multi 16538ms! > frontend-proxy:POST 14997ms! > frontend:POST 14998ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 501ms | errors off path: checkout:POST
T-0026trace=39e004c8 dur=39994ms ERROR path: load-generator:user_checkout_multi 16614ms! > frontend-proxy:POST 15000ms! > frontend:POST 15000ms
T-0027trace=6a748d0b dur=80106ms ERROR path: load-generator:user_checkout_single 15514ms! > frontend-proxy:POST 14999ms! > frontend:POST 14999ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 501ms | errors off path: checkout:POST
T-0028trace=e947a081 dur=39005ms ok path: accounting:order-consumed 39005ms
12 lower-ranked lines omitted
logs: log templates per service, rate before vs after fault onset, with examples
L-0001svc=ad sev=INFO base=0.0/min fault=1.6/min tpl="[id: 0x6a96a892, L:/<NUM>.<NUM>:<NUM> - R:flagd/<NUM>.<NUM>:<NUM>] ignoring <*> frame for stream RST_STREAM sent." ex=['[id: 0x6a96a892, L:/172.18.0.25:58138 - R:flagd/172.18.0.15:8013] ignoring HEADERS frame for stream RST_STREAM sent.', '[id: 0x6a96a892, L:/172.18.0.25:58138 - R:flagd/172.18.0.15:8013] ignoring DATA frame for stream RST_STREAM sent.']
L-0002svc=kafka sev=INFO base=2.2/min fault=0.6/min tpl="Recorded new KRaft controller, from now on will use node kafka:<NUM> (id: <NUM> rack: null isFenced: false)" ex=['Recorded new KRaft controller, from now on will use node kafka:9093 (id: 1 rack: null isFenced: false)', 'Recorded new KRaft controller, from now on will use node kafka:9093 (id: 1 rack: null isFenced: false)']
L-0003svc=kafka sev=INFO base=2.2/min fault=0.6/min tpl="[BrokerLifecycleManager id=<NUM>] Unable to send a heartbeat because the RPC got timed out before it could be sent." ex=['[BrokerLifecycleManager id=1] Unable to send a heartbeat because the RPC got timed out before it could be sent.', '[BrokerLifecycleManager id=1] Unable to send a heartbeat because the RPC got timed out before it could be sent.']
L-0004svc=kafka sev=INFO base=2.2/min fault=0.6/min tpl="[NodeToControllerChannelManager id=<NUM> name=heartbeat] Cancelled in-flight BROKER_HEARTBEAT request with correlation id <NUM> due to node…" ex=['[NodeToControllerChannelManager id=1 name=heartbeat] Cancelled in-flight BROKER_HEARTBEAT request with correlation id 858 due to node 1 bei…', '[NodeToControllerChannelManager id=1 name=heartbeat] Cancelled in-flight BROKER_HEARTBEAT request with correlation id 863 due to node 1 bei…']
L-0005svc=kafka sev=INFO base=2.2/min fault=0.6/min tpl="[NodeToControllerChannelManager id=<NUM> name=heartbeat] Disconnecting from node <NUM> due to request timeout." ex=['[NodeToControllerChannelManager id=1 name=heartbeat] Disconnecting from node 1 due to request timeout.', '[NodeToControllerChannelManager id=1 name=heartbeat] Disconnecting from node 1 due to request timeout.']
L-0006svc=load-generator sev=INFO base=12.4/min fault=7.2/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user 22aa401c-dc43-11f1-944a-626ba00a1477', 'Multi-item checkout completed for user 22aa401c-dc43-11f1-944a-626ba00a1477']
L-0007svc=accounting sev=INFORMATION base=10.2/min fault=6.4/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0008svc=checkout sev=INFO base=10.2/min fault=6.4/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 214.204µs', 'Successful to write message. offset: 0, duration: 256.777µs']
L-0009svc=checkout sev=WARN base=10.2/min fault=6.4/min tpl="failed to send order confirmation: failed POST to email service: Post "http://email:<NUM>/send_order_confirmation": context deadline exceed…" ex=['failed to send order confirmation: failed POST to email service: Post "http://email:6060/send_order_confirmation": context deadline exceeded', 'failed to send order confirmation: failed POST to email service: Post "http://email:6060/send_order_confirmation": context deadline exceeded']
L-0010svc=checkout sev=INFO base=10.2/min fault=6.4/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0011svc=fraud-detection sev=INFO base=10.2/min fault=6.4/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: 1642795d-dc43-11f1-9880-0a00946e758e, and updated total count to: 265', 'Consumed record with orderId: 18c8b389-dc43-11f1-9880-0a00946e758e, and updated total count to: 266']
L-0012svc=frontend sev=INFO base=10.2/min fault=6.4/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0013svc=cart sev=INFORMATION base=9.8/min fault=6.4/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0014svc=checkout sev=INFO base=9.8/min fault=6.4/min tpl="order placed" ex=['order placed', 'order placed']
L-0015svc=kafka sev=ERROR base=21.4/min fault=15.4/min tpl="[QuorumController id=<NUM>] Exceptionally slow controller event <*> took <NUM> ms." ex=['[QuorumController id=1] Exceptionally slow controller event writeNoOpRecord(1665637428) took 2181 ms.', '[QuorumController id=1] Exceptionally slow controller event writeNoOpRecord(1275701757) took 2425 ms.']
L-0016svc=shipping sev=WARN base=1.0/min fault=1.6/min tpl="" ex=['', '']
L-0017svc=shipping sev=ERROR base=1.0/min fault=1.6/min tpl="failed to resolve integer flag" ex=['failed to resolve integer flag', 'failed to resolve integer flag']
L-0018svc=load-generator sev=INFO base=64.0/min fault=49.6/min tpl="User <UUID> adding <NUM> of product <*> to cart" ex=['User 29d2ca58-dc43-11f1-944a-626ba00a1477 adding 10 of product 0PUK6V6EV0 to cart', 'User 29d2ca58-dc43-11f1-944a-626ba00a1477 adding 10 of product 0PUK6V6EV0 to cart']
L-0019svc=checkout sev=INFO base=10.0/min fault=7.8/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0020svc=payment sev=INFO base=10.0/min fault=7.8/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0021svc=payment sev=INFO base=10.0/min fault=7.8/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0022svc=shipping sev=INFO base=10.0/min fault=7.8/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0023svc=checkout sev=INFO base=10.2/min fault=8.0/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
47 lower-ranked lines omitted
Loading runs in your browser…