hwContention-r1
What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)
Service graph during the window
Evidence and answers
4537 prompt tokens
metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean
M-0001svc=ad metric=error_ratio base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.00909,na,+0%,na]
M-0002svc=ad metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0003svc=ad metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0004svc=ad metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0005svc=ad metric=traces.span.metrics.duration.p50 base=1.01 delta/30s=[-1%,na,-1%,na,-1%,na,+1487995%,na,-1%,na]
M-0006svc=ad metric=traces.span.metrics.duration.p95 base=1.92 delta/30s=[-1%,na,-1%,na,-1%,na,+783108%,na,-1%,na]
M-0007svc=ad metric=traces.span.metrics.duration.p99 base=2.28 delta/30s=[-13%,na,-13%,na,-13%,na,+657795%,na,-13%,na]
M-0008svc=flagd metric=error_ratio base=0 delta/30s=[new:0.000447,na,+0%,na,+0%,na,new:0.000926,na,+0%,na]
M-0009svc=fraud-detection metric=error_ratio base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0476,na,+0%,na]
M-0010svc=fraud-detection metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0011svc=fraud-detection metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0012svc=fraud-detection metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0013svc=kafka metric=jvm.gc.duration.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%]
M-0014svc=otelcol-contrib metric=otelcol_exporter_in_flight_requests base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:1,na,+0%,na]
M-0015svc=recommendation metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[na,new:1,na,+0%,na,new:1,na,+0%,na,new:1]
M-0016svc=recommendation metric=process.runtime.cpython.cpu.utilization base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:11.1,na,+0%]
M-0017svc=fraud-detection metric=traces.span.metrics.duration.p50 base=49.4 delta/30s=[-19%,na,-33%,na,-10%,na,+30247%,na,+1%,na]
M-0018svc=fraud-detection metric=traces.span.metrics.duration.p95 base=107.4 delta/30s=[-13%,na,+16%,na,-12%,na,+13864%,na,-12%,na]
M-0019svc=fraud-detection metric=traces.span.metrics.duration.p99 base=133.5 delta/30s=[-26%,na,+39%,na,-26%,na,+11137%,na,-26%,na]
M-0020svc=product-catalog metric=otel.sdk.processor.log.queue.size base=2.4 delta/30s=[na,+983%,na,+1150%,na,+567%,na,+108%,na,+108%]
M-0021svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,-100%,-100%,-100%,+900%,-100%,-100%,-100%,-100%,-100%]
M-0022svc=otelcol-contrib metric=otelcol_exporter_queue_size base=0.6 delta/30s=[+233%,na,+400%,na,+67%,na,+733%,na,+233%,na]
M-0023svc=load-generator metric=otel.sdk.processor.log.queue.size base=0.8 delta/30s=[+400%,na,+650%,na,+400%,na,-100%,na,+150%,na]
M-0024svc=quote metric=otel.sdk.processor.log.queue.size base=0.4 delta/30s=[-100%,na,-100%,na,+650%,na,-100%,na,-100%,na]
M-0025svc=frontend-proxy metric=traces.span.metrics.duration.p50 base=414.1 delta/30s=[-99%,na,-99%,na,-99%,na,-99%,na,+409%,na]
M-0026svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,-100%,na,+400%,na,-100%,na]
M-0027svc=accounting metric=dns.lookup.duration.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,-100%,na,+400%,na,-100%,na]
M-0028svc=ad metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,-100%,na,-100%,na,+400%,na]
M-0029svc=ad metric=jvm.thread.count.rate base=0.0133 delta/30s=[+150%,na,-100%,na,-100%,na,+400%,na,+150%,na]
M-0030svc=checkout metric=go.goroutine.count.rate base=0.0133 delta/30s=[-100%,na,+400%,na,-100%,na,-100%,na,+400%,na]
M-0031svc=fraud-detection metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[+400%,na,-100%,na,-100%,na,-100%,na,+400%,na]
M-0032svc=fraud-detection metric=jvm.thread.count.rate base=0.0133 delta/30s=[-100%,na,-100%,na,-100%,na,+400%,na,-100%,na]
M-0033svc=checkout metric=rpc.client.call.duration.p50 base=0.0172 delta/30s=[+14%,na,+214%,na,+118%,na,+360%,na,+384%,na]
M-0034svc=cart metric=traces.span.metrics.duration.p99 base=6.07 delta/30s=[+236%,na,-4%,na,-6%,na,-35%,na,+381%,na]
M-0035svc=checkout metric=rpc.server.call.duration.p50 base=0.0746 delta/30s=[-29%,na,+120%,na,+68%,na,+263%,na,+120%,na]
M-0036svc=recommendation metric=otel.sdk.processor.span.queue.size base=4.2 delta/30s=[na,+257%,na,+257%,na,-29%,na,-100%,na,+43%]
M-0037svc=checkout metric=rpc.client.call.duration.p95 base=0.102 delta/30s=[-45%,na,+28%,na,+50%,na,+257%,na,+125%,na]
864 lower-ranked lines omitted
traces: slowest and errored traces after fault onset, critical path (! = error span)
T-0001trace=01067b32 dur=19556ms ok path: accounting:order-consumed 19556ms
T-0002trace=2f984a89 dur=17712ms ok path: accounting:order-consumed 17712ms
T-0003trace=e4d79061 dur=13503ms ok path: accounting:order-consumed 13503ms
T-0004trace=766a299e dur=13134ms ok path: accounting:order-consumed 13134ms
T-0005trace=f22a1413 dur=11412ms ok path: accounting:order-consumed 11412ms
T-0006trace=501b7c74 dur=10574ms ok path: accounting:order-consumed 10574ms
T-0007trace=74eca0cb dur=10108ms ok path: accounting:order-consumed 10108ms
T-0008trace=9dd8f252 dur=9084ms ok path: accounting:order-consumed 9084ms
T-0009trace=b26844c2 dur=9060ms ok path: accounting:order-consumed 9060ms
T-0010trace=279037a3 dur=9022ms ok path: accounting:order-consumed 9022ms
T-0011trace=8a4e3026 dur=8846ms ok path: accounting:order-consumed 8846ms
T-0012trace=9549d8bb dur=8625ms ok path: accounting:order-consumed 8625ms
T-0013trace=794751c3 dur=8039ms ok path: accounting:order-consumed 8039ms
T-0014trace=15f73bcb dur=7133ms ok path: accounting:order-consumed 7133ms
T-0015trace=4be82517 dur=6857ms ok path: accounting:order-consumed 6857ms
T-0016trace=e90940eb dur=6831ms ok path: accounting:order-consumed 6831ms
T-0017trace=e51dd8a4 dur=6729ms ok path: accounting:order-consumed 6729ms
T-0018trace=0af996b3 dur=6697ms ok path: accounting:order-consumed 6697ms
T-0019trace=64889b08 dur=6634ms ok path: accounting:order-consumed 6634ms
T-0020trace=1555a9f2 dur=6561ms ok path: accounting:order-consumed 6561ms
logs: log templates per service, rate before vs after fault onset, with examples
L-0001svc=load-generator sev=INFO base=10.0/min fault=14.8/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0002svc=load-generator sev=INFO base=32.0/min fault=25.6/min tpl="Currency changed to CHF" ex=['Currency changed to CHF', 'Currency changed to CHF']
L-0003svc=load-generator sev=INFO base=93.6/min fault=77.6/min tpl="User <UUID> adding <NUM> of product <*> to cart" ex=['User 1265e5ce-dbff-11f1-8306-d6b650b5399b adding 1 of product 2ZYFJ3GM2N to cart', 'User 1265e5ce-dbff-11f1-8306-d6b650b5399b adding 1 of product 2ZYFJ3GM2N to cart']
L-0004svc=ad sev=INFO base=45.0/min fault=54.2/min tpl="Targeted ad request received for <*>" ex=['Targeted ad request received for [telescopes]', 'Targeted ad request received for [telescopes]']
L-0005svc=checkout sev=INFO base=0.2/min fault=0.0/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0006svc=currency sev=INFO base=123.0/min fault=104.4/min tpl="conversion successful" ex=['conversion successful', 'conversion successful']
L-0007svc=load-generator sev=INFO base=15.2/min fault=12.8/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user 19e106ee-dbff-11f1-8306-d6b650b5399b', 'Checkout completed for user 19e106ee-dbff-11f1-8306-d6b650b5399b']
L-0008svc=load-generator sev=INFO base=29.6/min fault=34.4/min tpl="Product added to cart successfully" ex=['Product added to cart successfully', 'Product added to cart successfully']
L-0009svc=accounting sev=INFORMATION base=15.4/min fault=13.2/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0010svc=cart sev=INFORMATION base=15.4/min fault=13.2/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0011svc=checkout sev=INFO base=15.4/min fault=13.2/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 234.729µs', 'Successful to write message. offset: 0, duration: 166.387µs']
L-0012svc=checkout sev=INFO base=15.4/min fault=13.2/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0013svc=checkout sev=INFO base=15.4/min fault=13.2/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0014svc=checkout sev=INFO base=15.4/min fault=13.2/min tpl="order placed" ex=['order placed', 'order placed']
L-0015svc=checkout sev=INFO base=15.4/min fault=13.2/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0016svc=checkout sev=INFO base=15.4/min fault=13.2/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0017svc=email sev=INFO base=15.4/min fault=13.2/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0018svc=fraud-detection sev=INFO base=15.4/min fault=13.2/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: 1493b2b1-dbff-11f1-9d0e-3288f2b1222e, and updated total count to: 22592', 'Consumed record with orderId: 16d60418-dbff-11f1-9d0e-3288f2b1222e, and updated total count to: 22593']
L-0019svc=frontend sev=INFO base=15.4/min fault=13.2/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0020svc=payment sev=INFO base=15.4/min fault=13.2/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0021svc=payment sev=INFO base=15.4/min fault=13.2/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0022svc=shipping sev=INFO base=15.4/min fault=13.2/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0023svc=load-generator sev=INFO base=15.6/min fault=13.6/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user 148c5cb6-dbff-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user 148c5cb6-dbff-11f1-8306-d6b650b5399b']
L-0024svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=74545, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=74665, siz…']
L-0025svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 74730 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 74847 breach: LogSegment(baseOff…']
L-0026svc=cart sev=INFORMATION base=61.4/min fault=56.2/min tpl="AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}" ex=['AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}', 'AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}']
L-0027svc=load-generator sev=INFO base=38.4/min fault=41.2/min tpl="User viewing cart" ex=['User viewing cart', 'User viewing cart']
33 lower-ranked lines omitted
Loading runs in your browser…