hwContention-r4
What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)
Service graph during the window
Evidence and answers
4601 prompt tokens
metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean
M-0001svc=checkout metric=go.goroutine.count.rate base=0 delta/30s=[na,+0%,na,new:0.133,na,+0%,na,new:0.1,na,+0%]
M-0002svc=checkout metric=otel.sdk.span.live base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:3,na,+0%]
M-0003svc=product-catalog metric=otel.sdk.span.live base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:2]
M-0004svc=recommendation metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:1]
M-0005svc=product-catalog metric=go.goroutine.count.rate base=0.0133 delta/30s=[na,-100%,na,+400%,na,-100%,na,-100%,na,+900%]
M-0006svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,-100%,-100%,-100%,-100%,-100%,-100%,-100%,-100%,+900%]
M-0007svc=unknown metric=system.disk.pending_operations base=1.2 delta/30s=[na,+733%,na,-17%,na,-100%,na,-100%,na,-100%]
M-0008svc=cart metric=http.server.request.duration.p95 base=0.0185 delta/30s=[na,-70%,na,+31%,na,+604%,na,-11%,na,-74%]
M-0009svc=load-generator metric=process.runtime.cpython.gc_count base=55.8 delta/30s=[-59%,na,-73%,na,+525%,na,-89%,na,-59%,na]
M-0010svc=cart metric=traces.span.metrics.duration.p95 base=22.1 delta/30s=[na,-72%,na,+31%,na,+420%,na,+18%,na,-83%]
M-0011svc=checkout metric=rpc.client.call.duration.p95 base=0.0435 delta/30s=[na,+81%,na,+336%,na,+388%,na,+67%,na,+414%]
M-0012svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0.00667 delta/30s=[na,-100%,na,+400%,na,-100%,na,-100%,na,-100%]
M-0013svc=accounting metric=dns.lookup.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,+400%,na,-100%,na,-100%,na,-100%]
M-0014svc=ad metric=otel.sdk.processor.log.queue.size base=0.4 delta/30s=[na,+400%,na,-100%,na,+150%,na,-100%,na,-100%]
M-0015svc=cart metric=dotnet.thread_pool.thread.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,+400%,na,-100%,na,-100%]
M-0016svc=cart metric=process.thread.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,+400%,na,-100%,na,-100%]
M-0017svc=fraud-detection metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,+400%,na,-100%,na,-100%,na,-100%]
M-0018svc=checkout metric=rpc.client.call.duration.p99 base=0.0527 delta/30s=[na,+82%,na,+352%,na,+360%,na,+41%,na,+364%]
M-0019svc=quote metric=otel.sdk.processor.span.queue.size base=3.6 delta/30s=[na,-17%,na,+150%,na,+150%,na,+67%,na,+317%]
M-0020svc=cart metric=http.server.request.duration.p99 base=0.0571 delta/30s=[na,-38%,na,+177%,na,+296%,na,+8%,na,-91%]
M-0021svc=load-generator metric=http.client.duration.p95 base=281.6 delta/30s=[-14%,na,-14%,na,+251%,na,+69%,na,-13%,na]
M-0022svc=cart metric=demo.cart.add_item.latency.p99 base=0.047 delta/30s=[na,-25%,na,+237%,na,-56%,na,+31%,na,-89%]
M-0023svc=load-generator metric=otel.sdk.processor.log.queue.size base=3.6 delta/30s=[+67%,na,+11%,na,-100%,na,+67%,na,+233%,na]
M-0024svc=payment metric=error_ratio base=0.00315 delta/30s=[na,+70%,na,+68%,na,+69%,na,+66%,na,+226%]
M-0025svc=checkout metric=rpc.client.call.duration.p50 base=0.0173 delta/30s=[na,+105%,na,+189%,na,+138%,na,+201%,na,+153%]
M-0026svc=cart metric=traces.span.metrics.duration.p99 base=61.3 delta/30s=[na,-56%,na,+127%,na,+198%,na,+17%,na,-92%]
M-0027svc=frontend metric=http.server.request.duration.p95 base=0.156 delta/30s=[na,+104%,na,+196%,na,+76%,na,+172%,na,+32%]
M-0028svc=quote metric=traces.span.metrics.duration.p99 base=496.4 delta/30s=[na,-24%,na,-23%,na,-24%,na,+160%,na,-32%]
M-0029svc=flagd metric=traces.span.metrics.duration.p50 base=5.81e+03 delta/30s=[na,-40%,na,-40%,na,-40%,na,+158%,na,+158%]
M-0030svc=accounting metric=dotnet.thread_pool.thread.count.rate base=0.0133 delta/30s=[na,-100%,na,-100%,na,+150%,na,-100%,na,-100%]
M-0031svc=accounting metric=process.thread.count.rate base=0.0133 delta/30s=[na,-100%,na,+150%,na,-100%,na,-100%,na,-100%]
M-0032svc=ad metric=jvm.thread.count.rate base=0.0133 delta/30s=[na,-100%,na,-100%,na,-100%,na,-100%,na,+150%]
M-0033svc=frontend metric=http.server.request.duration.p99 base=0.769 delta/30s=[na,+8%,na,+144%,na,+77%,na,+9%,na,-55%]
M-0034svc=unknown metric=httpcheck.duration base=5.4 delta/30s=[na,+141%,na,+104%,na,+67%,na,+48%,na,+67%]
M-0035svc=frontend metric=v8js.gc.duration.p99 base=0.00992 delta/30s=[na,+0%,na,-0%,na,+134%,na,+134%,na,-0%]
M-0036svc=checkout metric=request_rate base=3.87 delta/30s=[na,+126%,na,+40%,na,+93%,na,+38%,na,+47%]
M-0037svc=checkout metric=traces.span.metrics.duration.count.rate base=3.87 delta/30s=[na,+126%,na,+40%,na,+93%,na,+38%,na,+47%]
M-0038svc=shipping metric=traces.span.metrics.duration.p95 base=723.9 delta/30s=[na,-6%,na,+32%,na,+0%,na,+124%,na,-31%]
881 lower-ranked lines omitted
traces: slowest and errored traces after fault onset, critical path (! = error span)
T-0001trace=985f94a5 dur=17569ms ok path: accounting:order-consumed 17569ms
T-0002trace=a89bf7a6 dur=12850ms ok path: accounting:order-consumed 12850ms
T-0003trace=5e7adc06 dur=11454ms ok path: accounting:order-consumed 11454ms
T-0004trace=f418123a dur=9904ms ok path: accounting:order-consumed 9904ms
T-0005trace=1ca57afa dur=9840ms ok path: accounting:order-consumed 9840ms
T-0006trace=d190baf1 dur=9515ms ok path: accounting:order-consumed 9515ms
T-0007trace=d40eabc4 dur=9250ms ok path: accounting:order-consumed 9250ms
T-0008trace=a9d186b6 dur=9049ms ok path: accounting:order-consumed 9049ms
T-0009trace=7fa7818b dur=8806ms ok path: accounting:order-consumed 8806ms
T-0010trace=e04c4612 dur=8452ms ok path: accounting:order-consumed 8452ms
T-0011trace=e758379d dur=8412ms ok path: accounting:order-consumed 8412ms
T-0012trace=6e0cdedb dur=8339ms ok path: accounting:order-consumed 8339ms
T-0013trace=fff91ace dur=7826ms ok path: accounting:order-consumed 7826ms
T-0014trace=0532aca2 dur=7810ms ok path: accounting:order-consumed 7810ms
T-0015trace=4022fac4 dur=7689ms ok path: accounting:order-consumed 7689ms
T-0016trace=42ebb107 dur=7109ms ok path: accounting:order-consumed 7109ms
T-0017trace=2214a8ea dur=6394ms ok path: accounting:order-consumed 6394ms
T-0018trace=a2cf5309 dur=6310ms ok path: accounting:order-consumed 6310ms
T-0019trace=8e15bfe7 dur=6254ms ok path: accounting:order-consumed 6254ms
T-0020trace=06ba2cd3 dur=6196ms ok path: accounting:order-consumed 6196ms
logs: log templates per service, rate before vs after fault onset, with examples
L-0001svc=load-generator sev=INFO base=10.4/min fault=15.6/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user 7c8c3696-dc0f-11f1-8306-d6b650b5399b', 'Checkout completed for user 7c8c3696-dc0f-11f1-8306-d6b650b5399b']
L-0002svc=shipping sev=INFO base=1.6/min fault=2.2/min tpl="" ex=['', '']
L-0003svc=accounting sev=INFORMATION base=12.0/min fault=14.6/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0004svc=cart sev=INFORMATION base=12.0/min fault=14.6/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0005svc=checkout sev=INFO base=12.0/min fault=14.6/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 181.517µs', 'Successful to write message. offset: 0, duration: 309.593µs']
L-0006svc=checkout sev=INFO base=12.0/min fault=14.6/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0007svc=checkout sev=INFO base=12.0/min fault=14.6/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0008svc=checkout sev=INFO base=12.0/min fault=14.6/min tpl="order placed" ex=['order placed', 'order placed']
L-0009svc=checkout sev=INFO base=12.0/min fault=14.6/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0010svc=checkout sev=INFO base=12.0/min fault=14.6/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0011svc=email sev=INFO base=12.0/min fault=14.6/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0012svc=fraud-detection sev=INFO base=12.0/min fault=14.6/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: 79af565c-dc0f-11f1-9d0e-3288f2b1222e, and updated total count to: 24132', 'Consumed record with orderId: 7b18f571-dc0f-11f1-9d0e-3288f2b1222e, and updated total count to: 24133']
L-0013svc=frontend sev=INFO base=12.0/min fault=14.6/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0014svc=payment sev=INFO base=12.0/min fault=14.6/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0015svc=payment sev=INFO base=12.0/min fault=14.6/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0016svc=shipping sev=INFO base=12.0/min fault=14.6/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0017svc=checkout sev=INFO base=0.0/min fault=0.2/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0018svc=quote sev=INFO base=26.0/min fault=29.6/min tpl="Calculated quote" ex=['Calculated quote', 'Calculated quote']
L-0019svc=shipping sev=INFO base=26.0/min fault=29.6/min tpl="Requesting quote" ex=['Requesting quote', 'Requesting quote']
L-0020svc=shipping sev=INFO base=26.0/min fault=29.6/min tpl="Sending Quote" ex=['Sending Quote', 'Sending Quote']
L-0021svc=load-generator sev=INFO base=27.2/min fault=30.8/min tpl="Product added to cart successfully" ex=['Product added to cart successfully', 'Product added to cart successfully']
L-0022svc=load-generator sev=INFO base=34.4/min fault=30.8/min tpl="Currency changed to CHF" ex=['Currency changed to CHF', 'Currency changed to CHF']
L-0023svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 88770 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 88887 breach: LogSegment(baseOff…']
L-0024svc=ad sev=INFO base=3.0/min fault=2.6/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0025svc=kafka sev=INFO base=1.0/min fault=0.8/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=88704, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=88824, siz…']
L-0026svc=kafka sev=INFO base=8.4/min fault=7.6/min tpl="Deleted <*> index <*>" ex=['Deleted offset index /tmp/kafka-logs/__cluster_metadata-0/00000000000000088614.index.deleted.', 'Deleted time index /tmp/kafka-logs/__cluster_metadata-0/00000000000000088614.timeindex.deleted.']
L-0027svc=kafka sev=INFO base=4.2/min fault=3.8/min tpl="Deleted log /tmp/kafka-logs/__cluster_metadata-<NUM>/<HEX>.log.deleted." ex=['Deleted log /tmp/kafka-logs/__cluster_metadata-0/00000000000000088614.log.deleted.', 'Deleted log /tmp/kafka-logs/__cluster_metadata-0/00000000000000088644.log.deleted.']
L-0028svc=kafka sev=INFO base=4.2/min fault=3.8/min tpl="Deleted producer state snapshot /tmp/kafka-logs/__cluster_metadata-<NUM>/<HEX>.snapshot.deleted" ex=['Deleted producer state snapshot /tmp/kafka-logs/__cluster_metadata-0/00000000000000088614.snapshot.deleted', 'Deleted producer state snapshot /tmp/kafka-logs/__cluster_metadata-0/00000000000000088644.snapshot.deleted']
L-0029svc=load-generator sev=INFO base=82.4/min fault=76.0/min tpl="User getting <*> for <*> <*>" ex=['User getting ads for category: accessories', 'User getting ads for category: accessories']
31 lower-ranked lines omitted
Loading runs in your browser…