imageSlowLoad-r2
What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)
Service graph during the window
Evidence and answers
4557 prompt tokens
metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean
M-0001svc=ad metric=error_ratio base=0 delta/30s=[+0%,na,+0%,na,new:0.0164,na,+0%,na,+0%,na]
M-0002svc=ad metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%,na]
M-0003svc=ad metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%,na]
M-0004svc=ad metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%,na]
M-0005svc=ad metric=traces.span.metrics.duration.p50 base=1.01 delta/30s=[-1%,na,-1%,na,+1492047%,na,+2%,na,-1%,na]
M-0006svc=ad metric=traces.span.metrics.duration.p95 base=1.91 delta/30s=[-1%,na,-1%,na,+785240%,na,+2%,na,-1%,na]
M-0007svc=ad metric=traces.span.metrics.duration.p99 base=2.23 delta/30s=[-11%,na,-11%,na,+673150%,na,+44%,na,-11%,na]
M-0008svc=checkout metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:5,na]
M-0009svc=fraud-detection metric=error_ratio base=0 delta/30s=[+0%,na,+0%,na,new:0.0588,na,+0%,na,+0%,na]
M-0010svc=fraud-detection metric=jvm.thread.count.rate base=0 delta/30s=[new:0.0333,na,new:0.0667,na,new:0.0667,na,+0%,na,+0%,na]
M-0011svc=fraud-detection metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0012svc=fraud-detection metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0013svc=fraud-detection metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0014svc=kafka metric=jvm.gc.duration.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0015svc=quote metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[new:1,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0016svc=recommendation metric=error_ratio base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:0.00431,na]
M-0017svc=recommendation metric=traces.span.metrics.duration.p50 base=3.28 delta/30s=[+3%,na,-2%,na,-5%,na,-3%,na,+457565%,na]
M-0018svc=recommendation metric=traces.span.metrics.duration.p95 base=5.79 delta/30s=[+3%,na,-2%,na,-12%,na,-6%,na,+258878%,na]
M-0019svc=recommendation metric=traces.span.metrics.duration.p99 base=7.53 delta/30s=[+327%,na,+160%,na,-11%,na,-12%,na,+199039%,na]
M-0020svc=fraud-detection metric=traces.span.metrics.duration.p50 base=52.5 delta/30s=[+9%,na,-24%,na,+28471%,na,-37%,na,-16%,na]
M-0021svc=unknown metric=system.disk.pending_operations base=0.2 delta/30s=[-100%,na,-100%,na,+16900%,na,-100%,na,-100%,na]
M-0022svc=fraud-detection metric=traces.span.metrics.duration.p95 base=95.2 delta/30s=[+1%,na,-2%,na,+15662%,na,-4%,na,-1%,na]
M-0023svc=fraud-detection metric=traces.span.metrics.duration.p99 base=99.0 delta/30s=[+0%,na,-0%,na,+15046%,na,-1%,na,-0%,na]
M-0024svc=checkout metric=go.goroutine.count.rate base=0.0133 delta/30s=[+400%,na,+400%,na,-100%,na,-100%,na,+2400%,na]
M-0025svc=product-catalog metric=otel.sdk.processor.log.queue.size base=0.4 delta/30s=[+2150%,na,+2400%,na,+900%,na,+150%,na,-100%,na]
M-0026svc=accounting metric=dotnet.gc.last_collection.heap.fragmentation.size base=2.14e+05 delta/30s=[+0%,na,+0%,na,+0%,na,+1955%,na,+1955%,na]
M-0027svc=ad metric=jvm.thread.count.rate base=0.00667 delta/30s=[+900%,na,-100%,na,+900%,na,-100%,na,-100%,na]
M-0028svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,-100%,-100%,+900%,-100%,-100%,-100%,-100%,-100%,-100%]
M-0029svc=shipping metric=http.server.request.duration.p50 base=0.0248 delta/30s=[-29%,na,+405%,na,+102%,na,-44%,na,+546%,na]
M-0030svc=load-generator metric=process.runtime.cpython.gc_count base=118.6 delta/30s=[+448%,na,+310%,na,-94%,na,-98%,na,+141%,na]
M-0031svc=frontend-web metric=traces.span.metrics.duration.p95 base=1.2e+03 delta/30s=[+17%,na,+377%,na,+384%,na,+434%,na,+437%,na]
M-0032svc=payment metric=v8js.gc.duration.p99 base=0.0463 delta/30s=[+427%,na,+61%,na,-46%,na,+7%,na,-79%,na]
M-0033svc=payment metric=v8js.gc.duration.p50 base=0.00713 delta/30s=[-53%,na,+426%,na,+146%,na,+216%,na,+5%,na]
M-0034svc=payment metric=v8js.gc.duration.p95 base=0.0436 delta/30s=[+405%,na,+65%,na,-44%,na,+8%,na,-78%,na]
M-0035svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,+400%,na,-100%,na,-100%,na]
M-0036svc=accounting metric=dns.lookup.duration.count.rate base=0.00667 delta/30s=[-100%,na,-100%,na,+400%,na,-100%,na,-100%,na]
859 lower-ranked lines omitted
traces: slowest and errored traces after fault onset, critical path (! = error span)
T-0001trace=f0143cb3 dur=24036ms ok path: accounting:order-consumed 24036ms
T-0002trace=10a65540 dur=19171ms ok path: accounting:order-consumed 19171ms
T-0003trace=af778d22 dur=15377ms ok path: accounting:order-consumed 15377ms
T-0004trace=bc4e01b0 dur=14745ms ok path: accounting:order-consumed 14745ms
T-0005trace=e2c9aa07 dur=13558ms ok path: accounting:order-consumed 13558ms
T-0006trace=19f048ce dur=13180ms ok path: load-generator:browser_add_to_cart 13180ms
T-0007trace=25a21411 dur=13087ms ok path: load-generator:browser_add_to_cart 13087ms
T-0008trace=3c5002a0 dur=12971ms ok path: load-generator:browser_add_to_cart 12971ms
T-0009trace=b3d333bf dur=12899ms ok path: load-generator:browser_add_to_cart 12899ms
T-0010trace=5e20495a dur=12739ms ok path: load-generator:browser_add_to_cart 12739ms
T-0011trace=60c74777 dur=12690ms ok path: load-generator:browser_add_to_cart 12690ms
T-0012trace=1189ceb4 dur=12601ms ok path: load-generator:browser_add_to_cart 12601ms
T-0013trace=055262df dur=12577ms ok path: load-generator:browser_add_to_cart 12577ms
T-0014trace=f4b3557d dur=12571ms ok path: load-generator:browser_add_to_cart 12571ms
T-0015trace=7f194aae dur=12546ms ok path: load-generator:browser_add_to_cart 12546ms
T-0016trace=445c0133 dur=12540ms ok path: load-generator:browser_add_to_cart 12540ms
T-0017trace=b46ba42b dur=12531ms ok path: load-generator:browser_add_to_cart 12531ms
T-0018trace=74cbe9e8 dur=12530ms ok path: load-generator:browser_add_to_cart 12530ms
T-0019trace=68e3cc43 dur=12506ms ok path: load-generator:browser_add_to_cart 12506ms
T-0020trace=b7bd4d01 dur=12499ms ok path: load-generator:browser_add_to_cart 12499ms
logs: log templates per service, rate before vs after fault onset, with examples
L-0001svc=load-generator sev=INFO base=28.4/min fault=16.8/min tpl="Product added to cart successfully" ex=['Product added to cart successfully', 'Product added to cart successfully']
L-0002svc=load-generator sev=INFO base=27.2/min fault=17.2/min tpl="Currency changed to CHF" ex=['Currency changed to CHF', 'Currency changed to CHF']
L-0003svc=product-catalog sev=INFO base=129.0/min fault=87.8/min tpl="Found <NUM> products from database" ex=['Found 10 products from database', 'Found 10 products from database']
L-0004svc=quote sev=INFO base=27.6/min fault=18.8/min tpl="Calculated quote" ex=['Calculated quote', 'Calculated quote']
L-0005svc=shipping sev=INFO base=27.6/min fault=18.8/min tpl="Requesting quote" ex=['Requesting quote', 'Requesting quote']
L-0006svc=shipping sev=INFO base=27.6/min fault=18.8/min tpl="Sending Quote" ex=['Sending Quote', 'Sending Quote']
L-0007svc=shipping sev=INFO base=1.6/min fault=0.8/min tpl="" ex=['', '']
L-0008svc=ad sev=INFO base=42.6/min fault=29.2/min tpl="Targeted ad request received for <*>" ex=['Targeted ad request received for [accessories]', 'Targeted ad request received for [binoculars]']
L-0009svc=recommendation sev=INFO base=201.2/min fault=140.4/min tpl="Receive ListRecommendations for product <*> <*> <*> <*> <*>" ex=["Receive ListRecommendations for product ids:['66VCHSJNUP', '2ZYFJ3GM2N', 'LS4PSXUNUM', '9SIQT8TOJO', 'L9ECAV7KIM']", "Receive ListRecommendations for product ids:['66VCHSJNUP', '2ZYFJ3GM2N', 'LS4PSXUNUM', '9SIQT8TOJO', 'L9ECAV7KIM']"]
L-0010svc=load-generator sev=ERROR base=0.4/min fault=0.0/min tpl="Error in change currency task: Page.select_option: Timeout <NUM> exceeded. Call log: - waiting for locator("[name=\"currency_code\"]") - lo…" ex=['Error in change currency task: Page.select_option: Timeout 60000ms exceeded. Call log: - waiting for locator("[name=\\"currency_code\\"]") - …', 'Error in change currency task: Page.select_option: Timeout 60000ms exceeded. Call log: - waiting for locator("[name=\\"currency_code\\"]") - …']
L-0011svc=currency sev=INFO base=108.0/min fault=77.2/min tpl="conversion successful" ex=['conversion successful', 'conversion successful']
L-0012svc=product-catalog sev=INFO base=598.0/min fault=456.6/min tpl="Product Found" ex=['Product Found', 'Product Found']
L-0013svc=frontend-proxy sev=UNSET base=1614.6/min fault=1243.2/min tpl="<*> <*> <*> HTTP/<NUM>" <NUM> <*> <*> - "-" <NUM> <NUM> <NUM> <*> "-" "Mozilla/<NUM> (X11; Linux x86_64) AppleWebKit/<NUM> (KHTML, like Gec…" ex=['[2026-11-08T23:10:57.330Z] "GET /cart HTTP/1.1" 200 - via_upstream - "-" 0 2972 5 4 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36…', '[2026-11-08T23:10:57.350Z] "GET /_next/static/chunks/12ch15y12__j9.css HTTP/1.1" 200 - via_upstream - "-" 0 279 1 1 "-" "Mozilla/5.0 (X11; …']
L-0014svc=frontend sev=INFO base=512.4/min fault=413.8/min tpl="API request completed" ex=['API request completed', 'API request completed']
L-0015svc=checkout sev=INFO base=0.0/min fault=0.2/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0016svc=cart sev=INFORMATION base=170.0/min fault=141.6/min tpl="GetCartAsync called with userId={userId}" ex=['GetCartAsync called with userId={userId}', 'GetCartAsync called with userId={userId}']
L-0017svc=load-generator sev=INFO base=14.0/min fault=16.8/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0018svc=cart sev=INFORMATION base=55.4/min fault=46.6/min tpl="AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}" ex=['AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}', 'AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}']
L-0019svc=kafka sev=INFO base=0.6/min fault=0.4/min tpl="[QuorumController id=<NUM>] Periodic task <*> generated <NUM> records in <NUM> microseconds." ex=['[QuorumController id=1] Periodic task electPreferred generated 0 records in 15 microseconds.', '[QuorumController id=1] Periodic task electUnclean generated 0 records in 10 microseconds.']
L-0020svc=kafka sev=INFO base=2.0/min fault=2.4/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 28788 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 28827 breach: LogSegment(baseOff…']
L-0021svc=kafka sev=INFO base=1.0/min fault=0.8/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=28684, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=28804, siz…']
L-0022svc=kafka sev=INFO base=1.0/min fault=0.8/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 28866 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 29022 breach: LogSegment(baseOff…']
L-0023svc=load-generator sev=INFO base=141.6/min fault=128.8/min tpl="User browsing product: <*>" ex=['User browsing product: LS4PSXUNUM', 'User browsing product: LS4PSXUNUM']
39 lower-ranked lines omitted
Loading runs in your browser…