hwWeakSignal-r1
What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)
Service graph during the window
Evidence and answers
4556 prompt tokens
metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean
M-0001svc=ad metric=jvm.gc.duration.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0002svc=cart metric=dotnet.thread_pool.thread.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0003svc=cart metric=process.thread.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0004svc=kafka metric=jvm.class.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%]
M-0005svc=kafka metric=jvm.thread.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%]
M-0006svc=product-catalog metric=go.goroutine.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:0.0667,na,+0%]
M-0007svc=shipping metric=http.server.active_requests base=0 delta/30s=[+0%,na,new:3,na,+0%,na,+0%,na,+0%,na]
M-0008svc=product-catalog metric=otel.sdk.processor.log.queue.size base=0.4 delta/30s=[na,+150%,na,-100%,na,+150%,na,+3900%,na,+2650%]
M-0009svc=unknown metric=system.processes.count.rate base=0.06 delta/30s=[+233%,na,+178%,na,+678%,na,+956%,na,+233%,na]
M-0010svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,-100%,-100%,-100%,+900%,-100%,-100%,-100%,-100%,-100%]
M-0011svc=accounting metric=dotnet.gc.last_collection.heap.fragmentation.size base=1.16e+05 delta/30s=[+546%,na,+546%,na,+546%,na,+546%,na,+546%,na]
M-0012svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,-100%,na,+400%,na,-100%,na]
M-0013svc=accounting metric=dns.lookup.duration.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,-100%,na,+400%,na,-100%,na]
M-0014svc=ad metric=otel.sdk.processor.log.queue.size base=0.2 delta/30s=[-100%,na,+400%,na,-100%,na,-100%,na,-100%,na]
M-0015svc=fraud-detection metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,-100%,na,+400%,na,-100%,na]
M-0016svc=kafka metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,-100%,na,-100%,na,+400%]
M-0017svc=load-generator metric=otel.sdk.span.live base=0.4 delta/30s=[+150%,na,+400%,na,+400%,na,+400%,na,-100%,na]
M-0018svc=unknown metric=system.disk.pending_operations base=0.2 delta/30s=[-100%,na,+400%,na,-100%,na,-100%,na,-100%,na]
M-0019svc=kafka metric=kafka.purgatory.size base=18.4 delta/30s=[na,-89%,na,+280%,na,+357%,na,+41%,na,-13%]
M-0020svc=quote metric=traces.span.metrics.duration.p50 base=7.31 delta/30s=[+310%,na,-34%,na,-38%,na,+3%,na,+37%,na]
M-0021svc=checkout metric=go.goroutine.count.rate base=0.0333 delta/30s=[+300%,na,-100%,na,-100%,na,-100%,na,-100%,na]
M-0022svc=frontend metric=http.server.request.duration.p95 base=0.279 delta/30s=[+222%,na,-37%,na,-32%,na,-43%,na,+70%,na]
M-0023svc=shipping metric=traces.span.metrics.duration.p50 base=53.0 delta/30s=[+136%,na,-25%,na,+221%,na,+53%,na,+89%,na]
M-0024svc=checkout metric=rpc.server.call.duration.p50 base=0.0856 delta/30s=[+170%,na,-27%,na,-45%,na,+2%,na,+192%,na]
M-0025svc=shipping metric=http.server.request.duration.p50 base=0.0471 delta/30s=[+112%,na,-53%,na,+192%,na,+112%,na,+165%,na]
M-0026svc=load-generator metric=http.client.duration.p95 base=362.5 delta/30s=[+190%,na,-43%,na,-16%,na,-53%,na,+69%,na]
M-0027svc=checkout metric=http.client.request.duration.p95 base=0.68 delta/30s=[+157%,na,-45%,na,-60%,na,-64%,na,+135%,na]
M-0028svc=quote metric=otel.sdk.processor.span.queue.size base=3.6 delta/30s=[+150%,na,-17%,na,-17%,na,-100%,na,-17%,na]
M-0029svc=recommendation metric=otel.sdk.processor.log.queue.size base=1.2 delta/30s=[na,-17%,na,-100%,na,-17%,na,+150%,na,-17%]
M-0030svc=kafka metric=otel.sdk.processor.log.queue.size base=0.8 delta/30s=[na,+150%,na,+150%,na,+150%,na,+150%,na,+150%]
M-0031svc=frontend-proxy metric=traces.span.metrics.duration.p50 base=840.0 delta/30s=[-99%,na,+144%,na,-99%,na,-99%,na,+144%,na]
M-0032svc=load-generator metric=process.runtime.cpython.gc_count base=196.6 delta/30s=[-98%,na,-99%,na,+118%,na,-92%,na,+37%,na]
M-0033svc=unknown metric=redis.commands base=9.9 delta/30s=[-60%,+72%,-49%,-60%,+21%,-9%,+52%,+112%,-39%,+52%]
M-0034svc=unknown metric=redis.memory.fragmentation_ratio base=1.1 delta/30s=[+8%,+8%,+8%,+8%,+8%,+8%,+8%,+8%,+100%,+69%]
M-0035svc=unknown metric=redis.memory.rss base=1.93e+06 delta/30s=[+8%,+8%,+8%,+8%,+8%,+8%,+8%,+8%,+100%,+69%]
M-0036svc=ad metric=error_ratio base=0.00167 delta/30s=[-100%,na,-100%,na,-100%,na,-100%,na,-100%,na]
M-0037svc=ad metric=jvm.thread.count.rate base=0.0133 delta/30s=[-100%,na,-100%,na,-100%,na,-100%,na,-100%,na]
879 lower-ranked lines omitted
traces: slowest and errored traces after fault onset, critical path (! = error span)
T-0001trace=0dece709 dur=28384ms ok path: accounting:order-consumed 28384ms
T-0002trace=33d615a5 dur=28163ms ok path: accounting:order-consumed 28163ms
T-0003trace=c26075ba dur=19394ms ok path: accounting:order-consumed 19394ms
T-0004trace=157d20a8 dur=16063ms ok path: accounting:order-consumed 16063ms
T-0005trace=219686e8 dur=15747ms ok path: accounting:order-consumed 15747ms
T-0006trace=dcb70908 dur=14369ms ok path: accounting:order-consumed 14369ms
T-0007trace=4d487470 dur=14273ms ok path: accounting:order-consumed 14273ms
T-0008trace=b38dc1ed dur=9243ms ok path: accounting:order-consumed 9243ms
T-0009trace=5d8032d2 dur=7256ms ok path: accounting:order-consumed 7256ms
T-0010trace=5916ffe9 dur=7188ms ok path: accounting:order-consumed 7188ms
T-0011trace=e712398c dur=7171ms ok path: accounting:order-consumed 7171ms
T-0012trace=9ed3726c dur=6836ms ok path: accounting:order-consumed 6836ms
T-0013trace=ca3ac23b dur=6710ms ok path: accounting:order-consumed 6710ms
T-0014trace=25e37101 dur=6677ms ok path: accounting:order-consumed 6677ms
T-0015trace=501e7775 dur=6451ms ok path: accounting:order-consumed 6451ms
T-0016trace=4877180b dur=6424ms ok path: accounting:order-consumed 6424ms
T-0017trace=096717eb dur=6376ms ok path: accounting:order-consumed 6376ms
T-0018trace=ab865a1b dur=6013ms ok path: accounting:order-consumed 6013ms
T-0019trace=8a352b50 dur=5788ms ok path: accounting:order-consumed 5788ms
T-0020trace=5d41bc21 dur=5705ms ok path: accounting:order-consumed 5705ms
logs: log templates per service, rate before vs after fault onset, with examples
L-0001svc=shipping sev=INFO base=1.2/min fault=2.0/min tpl="" ex=['', '']
L-0002svc=checkout sev=INFO base=0.0/min fault=0.2/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0003svc=load-generator sev=INFO base=12.8/min fault=10.8/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user 8e8eb088-dc02-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user 8e8eb088-dc02-11f1-8306-d6b650b5399b']
L-0004svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 77733 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 77850 breach: LogSegment(baseOff…']
L-0005svc=kafka sev=INFO base=1.0/min fault=0.8/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=77665, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=77785, siz…']
L-0006svc=load-generator sev=INFO base=10.0/min fault=11.2/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user 9167d46a-dc02-11f1-8306-d6b650b5399b', 'Checkout completed for user 9167d46a-dc02-11f1-8306-d6b650b5399b']
L-0007svc=kafka sev=INFO base=8.4/min fault=7.6/min tpl="Deleted <*> index <*>" ex=['Deleted offset index /tmp/kafka-logs/__cluster_metadata-0/00000000000000077575.index.deleted.', 'Deleted time index /tmp/kafka-logs/__cluster_metadata-0/00000000000000077575.timeindex.deleted.']
L-0008svc=kafka sev=INFO base=4.2/min fault=3.8/min tpl="Deleted log /tmp/kafka-logs/__cluster_metadata-<NUM>/<HEX>.log.deleted." ex=['Deleted log /tmp/kafka-logs/__cluster_metadata-0/00000000000000077575.log.deleted.', 'Deleted log /tmp/kafka-logs/__cluster_metadata-0/00000000000000077605.log.deleted.']
L-0009svc=kafka sev=INFO base=4.2/min fault=3.8/min tpl="Deleted producer state snapshot /tmp/kafka-logs/__cluster_metadata-<NUM>/<HEX>.snapshot.deleted" ex=['Deleted producer state snapshot /tmp/kafka-logs/__cluster_metadata-0/00000000000000077575.snapshot.deleted', 'Deleted producer state snapshot /tmp/kafka-logs/__cluster_metadata-0/00000000000000077605.snapshot.deleted']
L-0010svc=load-generator sev=INFO base=40.0/min fault=43.2/min tpl="User viewing cart" ex=['User viewing cart', 'User viewing cart']
L-0011svc=kafka sev=INFO base=2.2/min fault=2.0/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 77772 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 77811 breach: LogSegment(baseOff…']
L-0012svc=kafka sev=INFO base=3.2/min fault=3.0/min tpl="Deleted snapshot files for snapshot OffsetAndEpoch[offset=<NUM>, epoch=<NUM>]." ex=['Deleted snapshot files for snapshot OffsetAndEpoch[offset=77577, epoch=1].', 'Deleted snapshot files for snapshot OffsetAndEpoch[offset=77616, epoch=1].']
L-0013svc=ad sev=INFO base=3.4/min fault=3.2/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0014svc=accounting sev=INFORMATION base=11.4/min fault=11.0/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0015svc=cart sev=INFORMATION base=11.4/min fault=11.0/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0016svc=checkout sev=INFO base=11.4/min fault=11.0/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 162.647µs', 'Successful to write message. offset: 0, duration: 163.318µs']
L-0017svc=checkout sev=INFO base=11.4/min fault=11.0/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0018svc=checkout sev=INFO base=11.4/min fault=11.0/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0019svc=checkout sev=INFO base=11.4/min fault=11.0/min tpl="order placed" ex=['order placed', 'order placed']
L-0020svc=checkout sev=INFO base=11.4/min fault=11.0/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0021svc=checkout sev=INFO base=11.4/min fault=11.0/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0022svc=email sev=INFO base=11.4/min fault=11.0/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0023svc=fraud-detection sev=INFO base=11.4/min fault=11.0/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: 8e97aa0b-dc02-11f1-9d0e-3288f2b1222e, and updated total count to: 22943', 'Consumed record with orderId: 916b54e2-dc02-11f1-9d0e-3288f2b1222e, and updated total count to: 22944']
L-0024svc=frontend sev=INFO base=11.4/min fault=11.0/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
36 lower-ranked lines omitted
Loading runs in your browser…