imageSlowLoad-r1
What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)
Service graph during the window
Evidence and answers
5712 prompt tokens
metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean
M-0001svc=ad metric=rpc.client.response.size.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%]
M-0002svc=fraud-detection metric=rpc.client.response.size.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%]
M-0003svc=kafka metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:2]
M-0004svc=load-generator metric=process.thread.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,new:0.0333,na,+0%,na,+0%]
M-0005svc=recommendation metric=error_ratio base=0 delta/30s=[+0%,na,+0%,na,new:0.00326,na,+0%,na,+0%,na]
M-0006svc=recommendation metric=process.thread.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333]
M-0007svc=recommendation metric=traces.span.metrics.duration.p50 base=3.12 delta/30s=[+3%,na,+7%,na,+481275%,na,+1%,na,+3%,na]
M-0008svc=recommendation metric=traces.span.metrics.duration.p95 base=4.97 delta/30s=[+19%,na,+19%,na,+301546%,na,+6%,na,+17%,na]
M-0009svc=recommendation metric=traces.span.metrics.duration.p99 base=11.1 delta/30s=[-12%,na,-3%,na,+134870%,na,-47%,na,+225%,na]
M-0010svc=fraud-detection metric=traces.span.metrics.duration.p50 base=600.0 delta/30s=[+0%,na,+0%,na,+2400%,na,+0%,na,+0%,na]
M-0011svc=fraud-detection metric=traces.span.metrics.duration.p95 base=780.0 delta/30s=[+0%,na,+0%,na,+1823%,na,+0%,na,+0%,na]
M-0012svc=fraud-detection metric=traces.span.metrics.duration.p99 base=796.0 delta/30s=[+0%,na,+0%,na,+1784%,na,+0%,na,+0%,na]
M-0013svc=ad metric=rpc.client.duration.p50 base=625.0 delta/30s=[na,+0%,na,-0%,na,+1500%,na,+0%,na,+0%]
M-0014svc=fraud-detection metric=rpc.client.duration.p50 base=625.0 delta/30s=[na,+0%,na,+0%,na,+1500%,na,+0%,na,+0%]
M-0015svc=ad metric=rpc.client.duration.p95 base=737.5 delta/30s=[na,+0%,na,-0%,na,+1256%,na,+0%,na,+0%]
M-0016svc=fraud-detection metric=rpc.client.duration.p95 base=737.5 delta/30s=[na,+0%,na,+0%,na,+1256%,na,+0%,na,+0%]
M-0017svc=ad metric=rpc.client.duration.p99 base=747.5 delta/30s=[na,+0%,na,-0%,na,+1238%,na,+0%,na,+0%]
M-0018svc=fraud-detection metric=rpc.client.duration.p99 base=747.5 delta/30s=[na,+0%,na,+0%,na,+1238%,na,+0%,na,+0%]
M-0019svc=frontend metric=http.client.request.duration.p50 base=0.0187 delta/30s=[na,-7%,na,-64%,na,+1034%,na,-47%,na,-7%]
M-0020svc=fraud-detection metric=jvm.class.count.rate base=0.0133 delta/30s=[na,-100%,na,-100%,na,+900%,na,-100%,na,-100%]
M-0021svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,-100%,-100%,+900%,-100%,-100%,-100%,-100%,-100%,-100%]
M-0022svc=ad metric=traces.span.metrics.duration.p50 base=1.7e+03 delta/30s=[+0%,na,+0%,na,+782%,na,+0%,na,+0%,na]
M-0023svc=ad metric=traces.span.metrics.duration.p95 base=1.97e+03 delta/30s=[+0%,na,+0%,na,+661%,na,+0%,na,+0%,na]
M-0024svc=ad metric=traces.span.metrics.duration.p99 base=1.99e+03 delta/30s=[+0%,na,+0%,na,+652%,na,+0%,na,+0%,na]
M-0025svc=product-catalog metric=go.goroutine.count.rate base=0.22 delta/30s=[na,-100%,na,+627%,na,-100%,na,-100%,na,+445%]
M-0026svc=recommendation metric=otel.sdk.processor.log.queue.size base=0.8 delta/30s=[na,+525%,na,+25%,na,+150%,na,-100%,na,+275%]
M-0027svc=product-catalog metric=otel.sdk.processor.log.queue.size base=4.4 delta/30s=[na,+445%,na,+127%,na,+36%,na,-77%,na,-32%]
M-0028svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,-100%,na,+400%,na,-100%]
M-0029svc=accounting metric=process.thread.count.rate base=0.00667 delta/30s=[na,+400%,na,-100%,na,-100%,na,-100%,na,-100%]
M-0030svc=ad metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,+400%,na,-100%,na,-100%,na,+400%]
M-0031svc=ad metric=jvm.thread.count.rate base=0.02 delta/30s=[na,+67%,na,+400%,na,+400%,na,-100%,na,+67%]
M-0032svc=fraud-detection metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,+400%,na,-100%,na,-100%]
M-0033svc=fraud-detection metric=jvm.thread.count.rate base=0.0133 delta/30s=[na,-100%,na,-100%,na,+400%,na,-100%,na,-100%]
M-0034svc=unknown metric=kafka.consumer_group.lag base=1.6 delta/30s=[+400%,+25%,-38%,-38%,-100%,+87%,-100%,+25%,-100%,-38%]
M-0035svc=unknown metric=kafka.consumer_group.lag_sum base=1.6 delta/30s=[+400%,+25%,-38%,-38%,-100%,+87%,-100%,+25%,-100%,-38%]
M-0036svc=checkout metric=go.goroutine.count.rate base=0.0867 delta/30s=[na,-100%,na,+362%,na,+15%,na,-100%,na,+169%]
879 lower-ranked lines omitted
traces: slowest and errored traces after fault onset, critical path (! = error span)
T-0001trace=6739d28b dur=148236ms ERROR path: load-generator:user_checkout_multi 16538ms! > frontend-proxy:POST 14998ms! > frontend:POST 14999ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 512ms | errors off path: checkout:POST
T-0002trace=14fcab44 dur=87219ms ok path: cart:flagd.evaluation.v2.Service/ResolveBoolean 87219ms
T-0003trace=b72858c0 dur=128454ms ERROR path: load-generator:user_checkout_single 15513ms! > frontend-proxy:POST 14999ms! > frontend:POST 14999ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 501ms | errors off path: checkout:POST
T-0004trace=8f7546a7 dur=80177ms ERROR path: load-generator:user_checkout_single 15516ms! > frontend-proxy:POST 15002ms! > frontend:POST 15002ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 501ms | errors off path: checkout:POST
T-0005trace=1c2ca628 dur=109401ms ERROR path: load-generator:user_checkout_multi 16026ms! > frontend-proxy:POST 15000ms! > frontend:POST 15001ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 501ms | errors off path: checkout:POST
T-0006trace=a1c2f8fc dur=69764ms ERROR path: load-generator:user_checkout_single 15529ms! > frontend-proxy:POST 15004ms! > frontend:POST 14999ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 501ms | errors off path: checkout:POST
T-0007trace=e9bf8a9b dur=109115ms ERROR path: load-generator:user_checkout_multi 16550ms! > frontend-proxy:POST 14998ms! > frontend:POST 14998ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 511ms | errors off path: checkout:POST
T-0008trace=cbc56d29 dur=68808ms ERROR path: load-generator:user_checkout_single 15528ms! > frontend-proxy:POST 15006ms! > frontend:POST 15006ms > checkout:oteldemo.CheckoutService/PlaceOrder 67772ms > cart:POST /oteldemo.CartService/EmptyCart 65758ms
T-0009trace=94c8cf1c dur=104848ms ERROR path: load-generator:user_checkout_multi 16042ms! > frontend-proxy:POST 14999ms! > frontend:POST 14999ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 513ms | errors off path: checkout:POST
T-0010trace=612ddd3a dur=68101ms ERROR path: load-generator:user_checkout_single 15512ms! > frontend-proxy:POST 14998ms! > frontend:POST 14997ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 507ms | errors off path: checkout:POST
T-0011trace=59c749a1 dur=104812ms ERROR path: load-generator:user_checkout_single 15513ms! > frontend-proxy:POST 14999ms! > frontend:POST 14999ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 501ms | errors off path: checkout:POST
T-0012trace=e5b3e073 dur=64389ms ERROR path: load-generator:user_checkout_single 15513ms! > frontend-proxy:POST 14999ms! > frontend:POST 15000ms > checkout:oteldemo.CheckoutService/PlaceOrder 63359ms > cart:POST /oteldemo.CartService/EmptyCart 61774ms
T-0013trace=fbdb3287 dur=104516ms ERROR path: load-generator:user_checkout_single 15519ms! > frontend-proxy:POST 14998ms! > frontend:POST 14997ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 512ms | errors off path: checkout:POST
T-0014trace=4d2ae9b5 dur=63403ms ERROR path: load-generator:user_checkout_single 15523ms! > frontend-proxy:POST 14998ms! > frontend:POST 14999ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 507ms | errors off path: checkout:POST
T-0015trace=229a2c80 dur=103425ms ERROR path: load-generator:user_checkout_single 15516ms! > frontend-proxy:POST 15001ms! > frontend:POST 15001ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 509ms | errors off path: checkout:POST
T-0016trace=8eafe7aa dur=61518ms ERROR path: load-generator:user_checkout_single 15514ms! > frontend-proxy:POST 14998ms! > frontend:POST 14999ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 501ms | errors off path: checkout:POST
T-0017trace=cefbf61e dur=100438ms ERROR path: load-generator:user_checkout_multi 16546ms! > frontend-proxy:POST 15004ms! > frontend:POST 15004ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 506ms | errors off path: checkout:POST
T-0018trace=31a01b26 dur=60489ms ERROR path: load-generator:user_checkout_multi 16543ms! > frontend-proxy:POST 14999ms! > frontend:POST 15000ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 506ms | errors off path: checkout:POST
T-0019trace=df67d706 dur=98474ms ERROR path: load-generator:user_checkout_multi 16542ms! > frontend-proxy:POST 15003ms! > frontend:POST 15003ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 510ms | errors off path: checkout:POST
T-0020trace=3a8444a1 dur=59947ms ERROR path: load-generator:user_checkout_single 15511ms! > frontend-proxy:POST 14997ms! > frontend:POST 14998ms > checkout:oteldemo.CheckoutService/PlaceOrder 58925ms > cart:POST /oteldemo.CartService/EmptyCart 57234ms
T-0021trace=e3dfff8c dur=95957ms ERROR path: load-generator:user_checkout_single 15512ms! > frontend-proxy:POST 14997ms! > frontend:POST 14999ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 509ms | errors off path: checkout:POST
T-0022trace=c47bf02f dur=58413ms ok path: cart:flagd.evaluation.v2.Service/ResolveBoolean 58413ms
T-0023trace=25fa5d33 dur=90690ms ERROR path: load-generator:user_checkout_single 15525ms! > frontend-proxy:POST 15001ms! > frontend:POST 15002ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 501ms | errors off path: checkout:POST
T-0024trace=9e9b0394 dur=57217ms ok path: cart:flagd.evaluation.v2.Service/ResolveBoolean 57217ms
T-0025trace=a52ab167 dur=88370ms ERROR path: load-generator:user_checkout_multi 16541ms! > frontend-proxy:POST 14998ms! > frontend:POST 14999ms > checkout:oteldemo.CheckoutService/PlaceOrder 86308ms > cart:POST /oteldemo.CartService/EmptyCart 83787ms
15 lower-ranked lines omitted
logs: log templates per service, rate before vs after fault onset, with examples
L-0001svc=load-generator sev=INFO base=6.8/min fault=11.2/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0002svc=email sev=INFO base=3.4/min fault=5.4/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0003svc=shipping sev=WARN base=1.4/min fault=0.8/min tpl="" ex=['', '']
L-0004svc=shipping sev=ERROR base=1.4/min fault=0.8/min tpl="failed to resolve integer flag" ex=['failed to resolve integer flag', 'failed to resolve integer flag']
L-0005svc=load-generator sev=INFO base=8.8/min fault=11.6/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user 4b751df2-dc41-11f1-944a-626ba00a1477', 'Checkout completed for user 4b751df2-dc41-11f1-944a-626ba00a1477']
L-0006svc=accounting sev=INFORMATION base=7.4/min fault=9.6/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0007svc=cart sev=INFORMATION base=7.4/min fault=9.6/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0008svc=checkout sev=INFO base=7.4/min fault=9.6/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 117.134µs', 'Successful to write message. offset: 0, duration: 159.136µs']
L-0009svc=checkout sev=WARN base=7.4/min fault=9.6/min tpl="failed to send order confirmation: failed POST to email service: Post "http://email:<NUM>/send_order_confirmation": context deadline exceed…" ex=['failed to send order confirmation: failed POST to email service: Post "http://email:6060/send_order_confirmation": context deadline exceeded', 'failed to send order confirmation: failed POST to email service: Post "http://email:6060/send_order_confirmation": context deadline exceeded']
L-0010svc=checkout sev=INFO base=7.4/min fault=9.6/min tpl="order placed" ex=['order placed', 'order placed']
L-0011svc=checkout sev=INFO base=7.4/min fault=9.6/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0012svc=fraud-detection sev=INFO base=7.4/min fault=9.6/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: 44bfc81c-dc41-11f1-9880-0a00946e758e, and updated total count to: 132', 'Consumed record with orderId: 3d6f0c28-dc41-11f1-9880-0a00946e758e, and updated total count to: 133']
L-0013svc=frontend sev=INFO base=7.4/min fault=9.6/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0014svc=checkout sev=INFO base=0.0/min fault=0.2/min tpl="Closed connection to broker kafka:<NUM>" ex=['Closed connection to broker kafka:9092']
L-0015svc=checkout sev=INFO base=0.0/min fault=0.2/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0016svc=checkout sev=INFO base=0.0/min fault=0.2/min tpl="client/seedbrokers close seed broker #-<NUM> at kafka:<NUM> due to socket error: broken pipe" ex=['client/seedbrokers close seed broker #-1 at kafka:9092 due to socket error: broken pipe']
L-0017svc=kafka sev=ERROR base=11.2/min fault=13.6/min tpl="[QuorumController id=<NUM>] Exceptionally slow controller event <*> took <NUM> ms." ex=['[QuorumController id=1] Exceptionally slow controller event writeNoOpRecord(1283005802) took 2073 ms.', '[QuorumController id=1] Exceptionally slow controller event writeNoOpRecord(298325160) took 2573 ms.']
L-0018svc=kafka sev=INFO base=1.8/min fault=1.4/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=92564, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=92655, siz…']
L-0019svc=checkout sev=INFO base=9.0/min fault=10.6/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0020svc=checkout sev=INFO base=9.0/min fault=10.6/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0021svc=payment sev=INFO base=9.0/min fault=10.6/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0022svc=payment sev=INFO base=9.0/min fault=10.6/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0023svc=shipping sev=INFO base=9.0/min fault=10.6/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0024svc=load-generator sev=INFO base=53.2/min fault=61.6/min tpl="User <UUID> adding <NUM> of product <*> to cart" ex=['User 4f1424a8-dc41-11f1-944a-626ba00a1477 adding 10 of product L9ECAV7KIM to cart', 'User 4f1424a8-dc41-11f1-944a-626ba00a1477 adding 10 of product L9ECAV7KIM to cart']
L-0025svc=kafka sev=INFO base=8.0/min fault=6.8/min tpl="Deleted <*> index <*>" ex=['Deleted offset index /tmp/kafka-logs/__cluster_metadata-0/00000000000000092564.index.deleted.', 'Deleted time index /tmp/kafka-logs/__cluster_metadata-0/00000000000000092564.timeindex.deleted.']
L-0026svc=kafka sev=INFO base=0.4/min fault=0.6/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=92625, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=92840, siz…']
42 lower-ranked lines omitted
Loading runs in your browser…