hwWifiOutage-r2
What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)
Service graph during the window
Evidence and answers
4822 prompt tokens
metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean
M-0001svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0002svc=accounting metric=dns.lookup.duration.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0003svc=ad metric=error_ratio base=0 delta/30s=[new:0.00855,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0004svc=ad metric=jvm.thread.count.rate base=0 delta/30s=[new:0.0667,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0005svc=ad metric=rpc.client.duration.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0006svc=ad metric=rpc.client.request.size.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0007svc=ad metric=rpc.client.response.size.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0008svc=ad metric=traces.span.metrics.duration.p50 base=1.01 delta/30s=[+1478621%,na,-1%,na,-1%,na,-1%,na,-1%,na]
M-0009svc=ad metric=traces.span.metrics.duration.p95 base=1.99 delta/30s=[+755100%,na,-4%,na,-4%,na,-4%,na,-4%,na]
M-0010svc=ad metric=traces.span.metrics.duration.p99 base=3.23 delta/30s=[+464872%,na,-39%,na,-39%,na,-39%,na,-39%,na]
M-0011svc=cart metric=dotnet.thread_pool.thread.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,new:0.0333,na,+0%,na,+0%,na]
M-0012svc=cart metric=process.thread.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,new:0.0333,na,+0%,na,+0%,na]
M-0013svc=checkout metric=error_ratio base=0 delta/30s=[+0%,na,new:0.0552,na,+0%,na,+0%,na,+0%,na]
M-0014svc=checkout metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:1,na]
M-0015svc=checkout metric=otel.sdk.span.live base=0 delta/30s=[new:2,na,+0%,na,+0%,na,+0%,na,new:3,na]
M-0016svc=edge-relay metric=relay.connection.duration.count.rate base=0 delta/30s=[+0%,+0%,+0%,+0%,+0%,new:0.0333,+0%,+0%,+0%,+0%]
M-0017svc=flagd metric=error_ratio base=0 delta/30s=[new:0.000923,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0018svc=fraud-detection metric=error_ratio base=0 delta/30s=[new:0.0476,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0019svc=fraud-detection metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:1,na]
M-0020svc=fraud-detection metric=otel.sdk.processor.span.queue.size base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:2,na]
M-0021svc=fraud-detection metric=rpc.client.duration.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0022svc=fraud-detection metric=rpc.client.request.size.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0023svc=fraud-detection metric=rpc.client.response.size.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0024svc=frontend metric=error_ratio base=0 delta/30s=[+0%,na,new:0.00381,na,+0%,na,+0%,na,+0%,na]
M-0025svc=frontend metric=http.server.request.duration.p50 base=0.00512 delta/30s=[-19%,na,+158736%,na,-3%,na,-0%,na,+15%,na]
M-0026svc=frontend-proxy metric=error_ratio base=0 delta/30s=[+0%,na,new:0.00169,na,+0%,na,+0%,na,+0%,na]
M-0027svc=load-generator metric=error_ratio base=0 delta/30s=[+0%,na,new:0.00877,na,+0%,na,+0%,na,+0%,na]
M-0028svc=load-generator metric=http.client.duration.p50 base=7.53 delta/30s=[na,+132615%,na,-2%,na,+4%,na,+1%,na,-4%]
M-0029svc=product-catalog metric=go.goroutine.count.rate base=0 delta/30s=[new:0.0667,na,+0%,na,+0%,na,+0%,na,new:0.0667,na]
M-0030svc=recommendation metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[+0%,na,+0%,na,new:1,na,+0%,na,+0%,na]
M-0031svc=recommendation metric=process.runtime.cpython.cpu.utilization base=0 delta/30s=[new:3.47,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0032svc=checkout metric=rpc.client.call.duration.p50 base=0.0161 delta/30s=[+22%,na,+54365%,na,+9%,na,+4%,na,-0%,na]
M-0033svc=fraud-detection metric=traces.span.metrics.duration.p50 base=44.7 delta/30s=[+33465%,na,+1%,na,-23%,na,-27%,na,+12%,na]
M-0034svc=checkout metric=rpc.client.call.duration.p95 base=0.0565 delta/30s=[+13%,na,+17599%,na,-49%,na,-40%,na,-42%,na]
M-0035svc=frontend metric=traces.span.metrics.duration.p50 base=87.8 delta/30s=[+82%,na,+16992%,na,+33%,na,+7%,na,+57%,na]
M-0036svc=fraud-detection metric=traces.span.metrics.duration.p95 base=99.1 delta/30s=[+15041%,na,-5%,na,-10%,na,-8%,na,-4%,na]
M-0037svc=checkout metric=rpc.server.call.duration.p50 base=0.0665 delta/30s=[+3%,na,+13058%,na,+73%,na,+41%,na,+107%,na]
864 lower-ranked lines omitted
traces: slowest and errored traces after fault onset, critical path (! = error span)
T-0001trace=593f4495 dur=20804ms ERROR path: load-generator:user_checkout_single 15016ms! > frontend-proxy:POST 15001ms! > frontend:POST 15002ms! > checkout:oteldemo.CheckoutService/PlaceOrder 19358ms!
T-0002trace=1a5b1335 dur=38495ms ok path: accounting:order-consumed 38495ms
T-0003trace=8daa69e9 dur=11273ms ERROR path: load-generator:user_checkout_multi 10034ms! > frontend-proxy:POST 9987ms! > frontend:POST /api/checkout 9988ms! > checkout:oteldemo.CheckoutService/PlaceOrder 9986ms!
T-0004trace=e8dc8a19 dur=12150ms ok path: accounting:order-consumed 12150ms
T-0005trace=1098938e dur=10159ms ERROR path: load-generator:user_checkout_multi 8863ms! > frontend-proxy:POST 8830ms! > frontend:POST /api/checkout 8830ms! > checkout:oteldemo.CheckoutService/PlaceOrder 8826ms!
T-0006trace=36d3b65f dur=12138ms ok path: accounting:order-consumed 12138ms
T-0007trace=ca034e88 dur=6072ms ERROR path: load-generator:user_checkout_single 4606ms! > frontend-proxy:POST 4590ms! > frontend:POST /api/checkout 4589ms! > checkout:oteldemo.CheckoutService/PlaceOrder 4586ms!
T-0008trace=42da2c77 dur=11377ms ok path: accounting:order-consumed 11377ms
T-0009trace=dc7015da dur=10757ms ok path: accounting:order-consumed 10757ms
T-0010trace=944368af dur=10748ms ok path: accounting:order-consumed 10748ms
T-0011trace=0f156683 dur=9995ms ok path: accounting:order-consumed 9995ms
T-0012trace=74164a46 dur=9639ms ok path: accounting:order-consumed 9639ms
T-0013trace=94367b9a dur=9568ms ok path: accounting:order-consumed 9568ms
T-0014trace=75a36592 dur=9037ms ok path: accounting:order-consumed 9037ms
T-0015trace=89865716 dur=8434ms ok path: accounting:order-consumed 8434ms
T-0016trace=0a044434 dur=8003ms ok path: accounting:order-consumed 8003ms
T-0017trace=94b645de dur=7475ms ok path: accounting:order-consumed 7475ms
T-0018trace=0017cace dur=7159ms ok path: accounting:order-consumed 7159ms
T-0019trace=97bd178e dur=6420ms ok path: accounting:order-consumed 6420ms
T-0020trace=c91772b7 dur=6412ms ok path: accounting:order-consumed 6412ms
T-0021trace=e560c197 dur=6370ms ok path: frontend-proxy:POST 6370ms > flagd:flagd.evaluation.v1.Service/EventStream 6369ms
T-0022trace=6f6436d4 dur=6058ms ok path: accounting:order-consumed 6058ms
T-0023trace=4f05e4d1 dur=5849ms ok path: accounting:order-consumed 5849ms
T-0024trace=5678f2d5 dur=5755ms ok path: accounting:order-consumed 5755ms
logs: log templates per service, rate before vs after fault onset, with examples
L-0001svc=frontend sev=ERROR base=0.0/min fault=0.8/min tpl="Checkout failed to place order" ex=['Checkout failed to place order', 'Checkout failed to place order']
L-0002svc=shipping sev=INFO base=1.2/min fault=2.2/min tpl="" ex=['', '']
L-0003svc=load-generator sev=INFO base=10.0/min fault=13.6/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user 97dce38e-dc0d-11f1-8306-d6b650b5399b', 'Checkout completed for user 97dce38e-dc0d-11f1-8306-d6b650b5399b']
L-0004svc=load-generator sev=INFO base=14.0/min fault=10.4/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0005svc=checkout sev=INFO base=10.6/min fault=13.8/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0006svc=load-generator sev=INFO base=69.2/min fault=88.0/min tpl="User <UUID> adding <NUM> of product <*> to cart" ex=['User 97dce38e-dc0d-11f1-8306-d6b650b5399b adding 2 of product LS4PSXUNUM to cart', 'User 97dce38e-dc0d-11f1-8306-d6b650b5399b adding 2 of product LS4PSXUNUM to cart']
L-0007svc=load-generator sev=INFO base=11.2/min fault=14.0/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user 98d19302-dc0d-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user 98d19302-dc0d-11f1-8306-d6b650b5399b']
L-0008svc=load-generator sev=INFO base=42.4/min fault=34.8/min tpl="User viewing cart" ex=['User viewing cart', 'User viewing cart']
L-0009svc=ad sev=INFO base=2.8/min fault=3.6/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0010svc=accounting sev=INFORMATION base=10.6/min fault=13.0/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0011svc=cart sev=INFORMATION base=10.6/min fault=13.0/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0012svc=checkout sev=INFO base=10.6/min fault=13.0/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 166.858µs', 'Successful to write message. offset: 0, duration: 256.221µs']
L-0013svc=checkout sev=INFO base=10.6/min fault=13.0/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0014svc=checkout sev=INFO base=10.6/min fault=13.0/min tpl="order placed" ex=['order placed', 'order placed']
L-0015svc=checkout sev=INFO base=10.6/min fault=13.0/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0016svc=checkout sev=INFO base=10.6/min fault=13.0/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0017svc=email sev=INFO base=10.6/min fault=13.0/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0018svc=fraud-detection sev=INFO base=10.6/min fault=13.0/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: 97df4fbf-dc0d-11f1-9d0e-3288f2b1222e, and updated total count to: 23957', 'Consumed record with orderId: 98dad965-dc0d-11f1-9d0e-3288f2b1222e, and updated total count to: 23958']
L-0019svc=frontend sev=INFO base=10.6/min fault=13.0/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0020svc=payment sev=INFO base=10.6/min fault=13.0/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0021svc=payment sev=INFO base=10.6/min fault=13.0/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0022svc=shipping sev=INFO base=10.6/min fault=13.0/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0023svc=cart sev=INFORMATION base=50.2/min fault=60.6/min tpl="AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}" ex=['AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}', 'AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}']
L-0024svc=checkout sev=INFO base=0.2/min fault=0.0/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0025svc=load-generator sev=INFO base=94.4/min fault=81.6/min tpl="User getting <*> for <*> <*>" ex=['User getting recommendations for product: 66VCHSJNUP', 'User getting recommendations for product: 66VCHSJNUP']
L-0026svc=quote sev=INFO base=26.2/min fault=30.4/min tpl="Calculated quote" ex=['Calculated quote', 'Calculated quote']
L-0027svc=shipping sev=INFO base=26.2/min fault=30.4/min tpl="Requesting quote" ex=['Requesting quote', 'Requesting quote']
L-0028svc=shipping sev=INFO base=26.2/min fault=30.4/min tpl="Sending Quote" ex=['Sending Quote', 'Sending Quote']
L-0029svc=kafka sev=INFO base=1.0/min fault=0.8/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=86994, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=87144, siz…']
32 lower-ranked lines omitted
Loading runs in your browser…