hwContention-r2
What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)
Service graph during the window
Evidence and answers
4521 prompt tokens
metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean
M-0001svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0002svc=accounting metric=dns.lookup.duration.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0003svc=accounting metric=dotnet.thread_pool.thread.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333,na]
M-0004svc=accounting metric=process.thread.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333,na]
M-0005svc=ad metric=error_ratio base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.00952,na]
M-0006svc=ad metric=jvm.gc.duration.count.rate base=0 delta/30s=[+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%,na]
M-0007svc=ad metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333,na]
M-0008svc=ad metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333,na]
M-0009svc=ad metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333,na]
M-0010svc=ad metric=traces.span.metrics.duration.p50 base=1 delta/30s=[-0%,na,-0%,na,-0%,na,-0%,na,+1494562%,na]
M-0011svc=ad metric=traces.span.metrics.duration.p95 base=1.91 delta/30s=[-0%,na,-0%,na,-0%,na,-0%,na,+786564%,na]
M-0012svc=ad metric=traces.span.metrics.duration.p99 base=2.56 delta/30s=[-23%,na,-23%,na,-23%,na,-23%,na,+586754%,na]
M-0013svc=flagd metric=error_ratio base=0 delta/30s=[+0%,na,+0%,na,new:0.000453,na,+0%,na,new:0.000815,na]
M-0014svc=fraud-detection metric=error_ratio base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.0256,na]
M-0015svc=fraud-detection metric=otel.sdk.processor.span.queue.size base=0 delta/30s=[new:2,na,+0%,na,+0%,na,new:2,na,+0%,na]
M-0016svc=fraud-detection metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333,na]
M-0017svc=fraud-detection metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333,na]
M-0018svc=fraud-detection metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.0333,na]
M-0019svc=kafka metric=jvm.gc.duration.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0020svc=kafka metric=jvm.thread.count.rate base=0 delta/30s=[+0%,na,+0%,na,new:0.0667,na,new:0.0667,na,+0%,na]
M-0021svc=kafka metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[new:2,na,new:2,na,new:2,na,+0%,na,+0%,na]
M-0022svc=load-generator metric=process.runtime.cpython.cpu.utilization base=0 delta/30s=[na,new:12.0,na,+0%,na,+0%,na,+0%,na,+0%]
M-0023svc=otelcol-contrib metric=otelcol_exporter_in_flight_requests base=0 delta/30s=[new:1,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0024svc=quote metric=otel.sdk.processor.span.queue.size base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:6,na]
M-0025svc=recommendation metric=process.runtime.cpython.cpu.utilization base=0 delta/30s=[+0%,na,new:7.83,na,+0%,na,+0%,na,+0%,na]
M-0026svc=fraud-detection metric=traces.span.metrics.duration.p50 base=52.1 delta/30s=[-23%,na,-14%,na,+4%,na,+17%,na,+28683%,na]
M-0027svc=fraud-detection metric=traces.span.metrics.duration.p95 base=95.0 delta/30s=[-3%,na,-1%,na,+0%,na,+1%,na,+15683%,na]
M-0028svc=fraud-detection metric=traces.span.metrics.duration.p99 base=99.0 delta/30s=[-1%,na,-0%,na,+0%,na,+0%,na,+15050%,na]
M-0029svc=ad metric=jvm.thread.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,-100%,na,-100%,na,+1400%,na]
M-0030svc=checkout metric=rpc.client.call.duration.p95 base=0.0355 delta/30s=[+95%,na,+520%,na,+85%,na,+478%,na,+957%,na]
M-0031svc=checkout metric=rpc.client.call.duration.p99 base=0.0471 delta/30s=[+57%,na,+418%,na,+55%,na,+412%,na,+909%,na]
M-0032svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,-100%,-100%,-100%,-100%,-100%,+900%,-100%,-100%,-100%]
M-0033svc=fraud-detection metric=jvm.thread.count.rate base=0.0133 delta/30s=[-100%,na,-100%,na,-100%,na,-100%,na,+650%,na]
M-0034svc=product-catalog metric=otel.sdk.processor.log.queue.size base=5 delta/30s=[-100%,na,-80%,na,-100%,na,-100%,na,+580%,na]
M-0035svc=ad metric=otel.sdk.processor.log.queue.size base=0.2 delta/30s=[-100%,na,+400%,na,-100%,na,+400%,na,-100%,na]
M-0036svc=cart metric=dotnet.thread_pool.thread.count.rate base=0.00667 delta/30s=[+400%,na,-100%,na,-100%,na,-100%,na,-100%,na]
M-0037svc=cart metric=process.thread.count.rate base=0.00667 delta/30s=[+400%,na,-100%,na,+400%,na,-100%,na,-100%,na]
861 lower-ranked lines omitted
traces: slowest and errored traces after fault onset, critical path (! = error span)
T-0001trace=56a2af6b dur=27184ms ok path: accounting:order-consumed 27184ms
T-0002trace=f7b95736 dur=22085ms ok path: accounting:order-consumed 22085ms
T-0003trace=dc2f643e dur=11795ms ok path: accounting:order-consumed 11795ms
T-0004trace=93abb58c dur=10022ms ok path: accounting:order-consumed 10022ms
T-0005trace=7141b431 dur=9322ms ok path: accounting:order-consumed 9322ms
T-0006trace=42c36b2c dur=9148ms ok path: accounting:order-consumed 9148ms
T-0007trace=678bb20b dur=8920ms ok path: accounting:order-consumed 8920ms
T-0008trace=a5d497ee dur=8691ms ok path: accounting:order-consumed 8691ms
T-0009trace=f6512ac8 dur=8524ms ok path: accounting:order-consumed 8524ms
T-0010trace=8d0943d4 dur=8369ms ok path: accounting:order-consumed 8369ms
T-0011trace=956d1b78 dur=8182ms ok path: accounting:order-consumed 8182ms
T-0012trace=a42dd813 dur=8001ms ok path: accounting:order-consumed 8001ms
T-0013trace=13c594e9 dur=7996ms ok path: accounting:order-consumed 7996ms
T-0014trace=09360232 dur=7875ms ok path: accounting:order-consumed 7875ms
T-0015trace=f79d0a99 dur=7371ms ok path: accounting:order-consumed 7371ms
T-0016trace=6c9080b2 dur=6851ms ok path: accounting:order-consumed 6851ms
T-0017trace=5bc4b950 dur=6513ms ok path: accounting:order-consumed 6513ms
T-0018trace=d88f87af dur=6468ms ok path: accounting:order-consumed 6468ms
T-0019trace=b67d9c46 dur=6427ms ok path: accounting:order-consumed 6427ms
T-0020trace=091988eb dur=5672ms ok path: accounting:order-consumed 5672ms
logs: log templates per service, rate before vs after fault onset, with examples
L-0001svc=shipping sev=INFO base=1.8/min fault=0.6/min tpl="" ex=['', '']
L-0002svc=load-generator sev=INFO base=10.0/min fault=14.4/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0003svc=checkout sev=INFO base=0.0/min fault=0.2/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0004svc=load-generator sev=INFO base=14.0/min fault=11.6/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user 1d7e233a-dc0a-11f1-8306-d6b650b5399b', 'Checkout completed for user 1d7e233a-dc0a-11f1-8306-d6b650b5399b']
L-0005svc=ad sev=INFO base=3.6/min fault=4.4/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0006svc=load-generator sev=INFO base=26.0/min fault=29.6/min tpl="Currency changed to CHF" ex=['Currency changed to CHF', 'Currency changed to CHF']
L-0007svc=accounting sev=INFORMATION base=13.8/min fault=12.2/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0008svc=cart sev=INFORMATION base=13.8/min fault=12.2/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0009svc=checkout sev=INFO base=13.8/min fault=12.2/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 159.116µs', 'Successful to write message. offset: 0, duration: 179.127µs']
L-0010svc=checkout sev=INFO base=13.8/min fault=12.2/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0011svc=checkout sev=INFO base=13.8/min fault=12.2/min tpl="order placed" ex=['order placed', 'order placed']
L-0012svc=checkout sev=INFO base=13.8/min fault=12.2/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0013svc=checkout sev=INFO base=13.8/min fault=12.2/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0014svc=email sev=INFO base=13.8/min fault=12.2/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0015svc=fraud-detection sev=INFO base=13.8/min fault=12.2/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: 1d80fd4c-dc0a-11f1-9d0e-3288f2b1222e, and updated total count to: 23635', 'Consumed record with orderId: 1ea7c0ef-dc0a-11f1-9d0e-3288f2b1222e, and updated total count to: 23636']
L-0016svc=frontend sev=INFO base=13.8/min fault=12.2/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0017svc=payment sev=INFO base=13.8/min fault=12.2/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0018svc=payment sev=INFO base=13.8/min fault=12.2/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0019svc=shipping sev=INFO base=13.8/min fault=12.2/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0020svc=quote sev=INFO base=31.0/min fault=27.6/min tpl="Calculated quote" ex=['Calculated quote', 'Calculated quote']
L-0021svc=shipping sev=INFO base=31.0/min fault=27.6/min tpl="Sending Quote" ex=['Sending Quote', 'Sending Quote']
L-0022svc=load-generator sev=INFO base=125.2/min fault=140.0/min tpl="User browsing product: <*>" ex=['User browsing product: 0PUK6V6EV0', 'User browsing product: 0PUK6V6EV0']
L-0023svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=83905, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=84025, siz…']
L-0024svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 84090 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 84207 breach: LogSegment(baseOff…']
L-0025svc=shipping sev=INFO base=31.0/min fault=27.8/min tpl="Requesting quote" ex=['Requesting quote', 'Requesting quote']
L-0026svc=checkout sev=INFO base=13.8/min fault=12.4/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0027svc=load-generator sev=INFO base=34.4/min fault=31.2/min tpl="Product added to cart successfully" ex=['Product added to cart successfully', 'Product added to cart successfully']
L-0028svc=kafka sev=INFO base=2.2/min fault=2.0/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=83965, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=83995, siz…']
32 lower-ranked lines omitted
Loading runs in your browser…