intlShippingSlowdown-r2
What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)
Service graph during the window
Evidence and answers
5433 prompt tokens
metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean
M-0001svc=ad metric=error_ratio base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0103,na,+0%,na]
M-0002svc=ad metric=jvm.class.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.1,na]
M-0003svc=ad metric=jvm.gc.duration.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0004svc=ad metric=jvm.thread.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,+0%,na,new:0.0667,na,+0%,na]
M-0005svc=ad metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0006svc=ad metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0007svc=ad metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0008svc=ad metric=traces.span.metrics.duration.p50 base=1 delta/30s=[+0%,na,+0%,na,+0%,na,+1499900%,na,+0%,na]
M-0009svc=ad metric=traces.span.metrics.duration.p95 base=1.9 delta/30s=[+0%,na,+0%,na,+0%,na,+789374%,na,+0%,na]
M-0010svc=ad metric=traces.span.metrics.duration.p99 base=1.98 delta/30s=[+0%,na,+0%,na,+0%,na,+757476%,na,+0%,na]
M-0011svc=fraud-detection metric=error_ratio base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0303,na,+0%,na]
M-0012svc=fraud-detection metric=jvm.thread.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,new:0.0333,na,new:0.1,na,+0%,na]
M-0013svc=fraud-detection metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0014svc=fraud-detection metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0015svc=fraud-detection metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,+0%,na]
M-0016svc=kafka metric=jvm.gc.duration.count.rate base=0 delta/30s=[+0%,na,new:0.0667,na,+0%,na,+0%,na,+0%,na]
M-0017svc=product-catalog metric=go.goroutine.count.rate base=0 delta/30s=[+0%,na,+0%,na,new:0.133,na,+0%,na,+0%,na]
M-0018svc=quote metric=otel.sdk.processor.span.queue.size base=0 delta/30s=[new:15.0,na,new:9,na,new:6,na,new:9,na,new:3,na]
M-0019svc=recommendation metric=process.runtime.cpython.cpu.utilization base=0 delta/30s=[+0%,na,+0%,na,new:13.0,na,+0%,na,new:12.9,na]
M-0020svc=fraud-detection metric=kafka.consumer.heartbeat_response_time_max base=1.8 delta/30s=[-44%,na,+48844%,na,+11%,na,-44%,na,-44%,na]
M-0021svc=fraud-detection metric=traces.span.metrics.duration.p50 base=46.1 delta/30s=[+16%,na,+36%,na,-6%,na,+32466%,na,+9%,na]
M-0022svc=fraud-detection metric=traces.span.metrics.duration.p95 base=91.5 delta/30s=[+4%,na,+5%,na,+3%,na,+16294%,na,+4%,na]
M-0023svc=fraud-detection metric=traces.span.metrics.duration.p99 base=98.3 delta/30s=[+1%,na,+1%,na,+1%,na,+15159%,na,+1%,na]
M-0024svc=product-catalog metric=otel.sdk.processor.log.queue.size base=1 delta/30s=[+0%,na,-100%,na,-100%,na,+1500%,na,+100%,na]
M-0025svc=checkout metric=http.client.request.duration.p95 base=0.575 delta/30s=[+43%,na,+204%,na,+1074%,na,-22%,na,+39%,na]
M-0026svc=shipping metric=traces.span.metrics.duration.p95 base=887.6 delta/30s=[-14%,na,+604%,na,+942%,na,+576%,na,+520%,na]
M-0027svc=quote metric=otel.sdk.processor.log.queue.size base=0.2 delta/30s=[-100%,na,-100%,na,+900%,na,-100%,na,-100%,na]
M-0028svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[+900%,-100%,-100%,-100%,-100%,-100%,-100%,-100%,-100%,-100%]
M-0029svc=shipping metric=http.server.request.duration.p95 base=0.91 delta/30s=[-21%,na,+518%,na,+683%,na,+463%,na,+504%,na]
M-0030svc=checkout metric=http.client.request.duration.p99 base=1.04 delta/30s=[-7%,na,+549%,na,+606%,na,+544%,na,+534%,na]
M-0031svc=shipping metric=traces.span.metrics.duration.p99 base=1.41e+03 delta/30s=[-34%,na,+554%,na,+597%,na,+551%,na,+544%,na]
M-0032svc=cart metric=dotnet.gc.last_collection.heap.fragmentation.size base=7.61e+05 delta/30s=[-21%,na,-21%,na,-21%,na,+590%,na,+555%,na]
M-0033svc=frontend metric=http.server.request.duration.p95 base=0.735 delta/30s=[+17%,na,+220%,na,+586%,na,-11%,na,+46%,na]
M-0034svc=recommendation metric=otel.sdk.processor.span.queue.size base=4.2 delta/30s=[-100%,na,-100%,na,+543%,na,+329%,na,+376%,na]
M-0035svc=checkout metric=traces.span.metrics.duration.p95 base=1.67e+03 delta/30s=[+141%,na,+274%,na,+453%,na,+259%,na,+229%,na]
M-0036svc=otelcol-contrib metric=otelcol_exporter_queue_size base=2.2 delta/30s=[-9%,na,-55%,na,-9%,na,+36%,na,+445%,na]
853 lower-ranked lines omitted
traces: slowest and errored traces after fault onset, critical path (! = error span)
T-0001trace=d0eb623f dur=198ms ERROR path: frontend-proxy:POST 83ms > flagd:flagd.evaluation.v1.Service/ResolveAll 2ms!
T-0002trace=76bdb1fd dur=17384ms ok path: accounting:order-consumed 17384ms
T-0003trace=db6b0c16 dur=144ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 144ms!
T-0004trace=1a1093a7 dur=15922ms ok path: accounting:order-consumed 15922ms
T-0005trace=7ec05559 dur=133ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 133ms!
T-0006trace=555a8caa dur=15603ms ok path: accounting:order-consumed 15603ms
T-0007trace=ce2161b1 dur=120ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 120ms!
T-0008trace=6f1efa78 dur=12730ms ok path: accounting:order-consumed 12730ms
T-0009trace=e98c30dd dur=62ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 62ms!
T-0010trace=c72ab8c1 dur=12374ms ok path: accounting:order-consumed 12374ms
T-0011trace=94c1bcf3 dur=54ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 54ms!
T-0012trace=acd35020 dur=10288ms ok path: accounting:order-consumed 10288ms
T-0013trace=ca18f879 dur=26ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 26ms!
T-0014trace=8738ee67 dur=8874ms ok path: accounting:order-consumed 8874ms
T-0015trace=a6899c1d dur=26ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 26ms!
T-0016trace=eda76aee dur=8781ms ok path: accounting:order-consumed 8781ms
T-0017trace=f8e306ad dur=26ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 26ms!
T-0018trace=4b111914 dur=8670ms ok path: accounting:order-consumed 8670ms
T-0019trace=becc7aff dur=26ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 26ms!
T-0020trace=6d2fc9f1 dur=8485ms ok path: accounting:order-consumed 8485ms
T-0021trace=7ce041f4 dur=26ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 26ms!
T-0022trace=2c02ac8e dur=8301ms ok path: accounting:order-consumed 8301ms
T-0023trace=ebc2ad74 dur=25ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 25ms!
T-0024trace=c13f32b3 dur=7908ms ok path: accounting:order-consumed 7908ms
T-0025trace=4c5bbf8d dur=24ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 24ms!
T-0026trace=72db4000 dur=7551ms ok path: accounting:order-consumed 7551ms
T-0027trace=b9fde385 dur=24ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 24ms!
T-0028trace=e36c3cc6 dur=7174ms ok path: accounting:order-consumed 7174ms
T-0029trace=83dc6dc3 dur=23ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 23ms!
T-0030trace=5273e275 dur=6191ms ok path: accounting:order-consumed 6191ms
T-0031trace=76808e89 dur=23ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 23ms!
T-0032trace=90db839c dur=6096ms ok path: load-generator:user_checkout_single 6096ms > frontend-proxy:POST 6069ms > frontend:POST /api/checkout 6067ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 44ms
T-0033trace=20bda7da dur=23ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 23ms!
T-0034trace=4e8c4a12 dur=5897ms ok path: accounting:order-consumed 5897ms
T-0035trace=9d8a5ec0 dur=23ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 23ms!
T-0036trace=f4e3fca4 dur=5790ms ok path: load-generator:user_checkout_multi 5790ms > frontend-proxy:POST 5736ms > frontend:POST /api/checkout 5737ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 36ms
T-0037trace=d4a25d3b dur=22ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 22ms!
T-0038trace=25595dd5 dur=5710ms ok path: load-generator:user_checkout_multi 5710ms > frontend-proxy:POST 5676ms > frontend:POST /api/checkout 5675ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 220ms
T-0039trace=ca942140 dur=22ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 22ms!
T-0040trace=8d6e566f dur=5604ms ok path: load-generator:user_checkout_single 5604ms > frontend-proxy:POST 5586ms > frontend:POST /api/checkout 5585ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 27ms
logs: log templates per service, rate before vs after fault onset, with examples
L-0001svc=shipping sev=INFO base=0.0/min fault=1.4/min tpl="Delaying international shipment due to <redacted> feature flag" ex=['Delaying international shipment due to <redacted> feature flag', 'Delaying international shipment due to <redacted> feature flag']
L-0002svc=shipping sev=INFO base=0.6/min fault=1.4/min tpl="" ex=['', '']
L-0003svc=ad sev=INFO base=4.0/min fault=2.6/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0004svc=checkout sev=INFO base=0.2/min fault=0.0/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0005svc=load-generator sev=INFO base=14.0/min fault=16.8/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user c0eefbce-dbd4-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user c0eefbce-dbd4-11f1-8306-d6b650b5399b']
L-0006svc=load-generator sev=INFO base=80.0/min fault=94.8/min tpl="User <UUID> adding <NUM> of product <*> to cart" ex=['User b3b0165a-dbd4-11f1-8306-d6b650b5399b adding 3 of product 6E92ZMYYFZ to cart', 'User b3b0165a-dbd4-11f1-8306-d6b650b5399b adding 3 of product 6E92ZMYYFZ to cart']
L-0007svc=load-generator sev=INFO base=79.6/min fault=68.4/min tpl="User getting <*> for <*> <*>" ex=['User getting ads for category: binoculars', 'User getting ads for category: binoculars']
L-0008svc=cart sev=INFORMATION base=54.8/min fault=63.0/min tpl="AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}" ex=['AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}', 'AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}']
L-0009svc=kafka sev=INFO base=2.0/min fault=2.4/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 38265 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 38304 breach: LogSegment(baseOff…']
L-0010svc=load-generator sev=INFO base=14.4/min fault=12.8/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0011svc=accounting sev=INFORMATION base=13.2/min fault=14.8/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0012svc=cart sev=INFORMATION base=13.2/min fault=14.8/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0013svc=checkout sev=INFO base=13.2/min fault=14.8/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 187.645µs', 'Successful to write message. offset: 0, duration: 183.266µs']
L-0014svc=checkout sev=INFO base=13.2/min fault=14.8/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0015svc=checkout sev=INFO base=13.2/min fault=14.8/min tpl="order placed" ex=['order placed', 'order placed']
L-0016svc=checkout sev=INFO base=13.2/min fault=14.8/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0017svc=email sev=INFO base=13.2/min fault=14.8/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0018svc=fraud-detection sev=INFO base=13.2/min fault=14.8/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: b380d361-dbd4-11f1-82f3-d279f8fa1f6d, and updated total count to: 18736', 'Consumed record with orderId: b3d6a103-dbd4-11f1-82f3-d279f8fa1f6d, and updated total count to: 18737']
L-0019svc=frontend sev=INFO base=13.2/min fault=14.8/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0020svc=kafka sev=INFO base=1.0/min fault=0.8/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=38164, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=38314, siz…']
L-0021svc=kafka sev=INFO base=1.0/min fault=0.8/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 38382 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 38499 breach: LogSegment(baseOff…']
L-0022svc=checkout sev=INFO base=13.4/min fault=14.6/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0023svc=checkout sev=INFO base=13.4/min fault=14.6/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0024svc=payment sev=INFO base=13.4/min fault=14.6/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0025svc=payment sev=INFO base=13.4/min fault=14.6/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0026svc=shipping sev=INFO base=13.4/min fault=14.6/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0027svc=currency sev=INFO base=111.6/min fault=119.4/min tpl="conversion successful" ex=['conversion successful', 'conversion successful']
34 lower-ranked lines omitted
Loading runs in your browser…