recommendationCacheFailure-r2
What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)
Service graph during the window
Evidence and answers
5401 prompt tokens
metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean
M-0001svc=ad metric=error_ratio base=0 delta/30s=[+0%,na,+0%,na,new:0.0103,na,+0%,na,+0%,na]
M-0002svc=ad metric=jvm.gc.duration.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0003svc=ad metric=jvm.thread.count.rate base=0 delta/30s=[new:0.0333,na,new:0.0333,na,new:0.0333,na,+0%,na,+0%,na]
M-0004svc=ad metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%,na]
M-0005svc=ad metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%,na]
M-0006svc=ad metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%,na]
M-0007svc=ad metric=traces.span.metrics.duration.p50 base=1 delta/30s=[+0%,na,+0%,na,+1499900%,na,+0%,na,+0%,na]
M-0008svc=ad metric=traces.span.metrics.duration.p95 base=1.9 delta/30s=[+0%,na,+0%,na,+789374%,na,+0%,na,+0%,na]
M-0009svc=ad metric=traces.span.metrics.duration.p99 base=1.98 delta/30s=[+0%,na,+0%,na,+757476%,na,+0%,na,+0%,na]
M-0010svc=fraud-detection metric=error_ratio base=0 delta/30s=[+0%,na,+0%,na,new:0.04,na,+0%,na,+0%,na]
M-0011svc=fraud-detection metric=jvm.thread.count.rate base=0 delta/30s=[new:0.0333,na,+0%,na,new:0.0667,na,+0%,na,+0%,na]
M-0012svc=fraud-detection metric=rpc.client.duration.count.rate base=0 delta/30s=[+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%,na]
M-0013svc=fraud-detection metric=rpc.client.request.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%,na]
M-0014svc=fraud-detection metric=rpc.client.response.size.count.rate base=0 delta/30s=[+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%,na]
M-0015svc=frontend metric=error_ratio base=0 delta/30s=[+0%,na,+0%,na,new:0.151,na,+0%,na,new:0.0383,na]
M-0016svc=frontend-proxy metric=error_ratio base=0 delta/30s=[+0%,na,new:0.00209,na,new:0.0102,na,+0%,na,new:0.00325,na]
M-0017svc=frontend-web metric=error_ratio base=0 delta/30s=[+0%,na,+0%,na,new:0.00608,na,+0%,na,+0%,na]
M-0018svc=kafka metric=jvm.gc.duration.count.rate base=0 delta/30s=[+0%,na,new:0.0333,na,+0%,na,+0%,na,+0%,na]
M-0019svc=kafka metric=jvm.thread.count.rate base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:0.0333,na,new:0.0333,na]
M-0020svc=kafka metric=otel.sdk.exporter.log.inflight base=0 delta/30s=[+0%,na,+0%,na,+0%,na,new:2,na,+0%,na]
M-0021svc=kafka metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[+0%,na,+0%,na,+0%,na,+0%,na,new:4,na]
M-0022svc=load-generator metric=error_ratio base=0 delta/30s=[+0%,na,new:0.0185,na,new:0.0446,na,+0%,na,new:0.0293,na]
M-0023svc=load-generator metric=http.client.duration.p50 base=8.44 delta/30s=[+0%,na,+118331%,na,+118331%,na,+118331%,na,+118331%,na]
M-0024svc=product-catalog metric=otel.sdk.span.live base=0 delta/30s=[new:2,na,+0%,na,+0%,na,+0%,na,+0%,na]
M-0025svc=fraud-detection metric=traces.span.metrics.duration.p50 base=45.7 delta/30s=[+31%,na,+9%,na,+32747%,na,+32%,na,+3%,na]
M-0026svc=fraud-detection metric=traces.span.metrics.duration.p95 base=93.8 delta/30s=[+2%,na,+1%,na,+15900%,na,+2%,na,+1%,na]
M-0027svc=fraud-detection metric=traces.span.metrics.duration.p99 base=98.8 delta/30s=[+0%,na,+0%,na,+15090%,na,+0%,na,+0%,na]
M-0028svc=frontend metric=http.server.request.duration.p50 base=0.00693 delta/30s=[na,+6%,na,+7%,na,+12531%,na,+0%,na,+18%]
M-0029svc=frontend metric=traces.span.metrics.duration.p50 base=122.7 delta/30s=[+63%,na,+145%,na,+12128%,na,-59%,na,+12128%,na]
M-0030svc=frontend metric=http.server.request.duration.p95 base=0.215 delta/30s=[na,+284%,na,+1363%,na,+4557%,na,-50%,na,+1476%]
M-0031svc=frontend-proxy metric=traces.span.metrics.duration.p50 base=417.2 delta/30s=[-99%,na,+3345%,na,+2746%,na,-98%,na,+3324%,na]
M-0032svc=load-generator metric=http.client.duration.p95 base=322.4 delta/30s=[+162%,na,+3001%,na,+3001%,na,+3001%,na,+3001%,na]
M-0033svc=flagd metric=error_ratio base=9e-05 delta/30s=[-100%,na,-100%,na,+2887%,na,-100%,na,-100%,na]
M-0034svc=frontend metric=traces.span.metrics.duration.p95 base=1.2e+03 delta/30s=[+46%,na,+1146%,na,+1146%,na,-49%,na,+1146%,na]
M-0035svc=frontend metric=traces.span.metrics.duration.p99 base=1.39e+03 delta/30s=[+40%,na,+977%,na,+977%,na,+0%,na,+977%,na]
M-0036svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,-100%,-100%,-100%,+900%,-100%,-100%,-100%,-100%,-100%]
853 lower-ranked lines omitted
traces: slowest and errored traces after fault onset, critical path (! = error span)
T-0001trace=f84556db dur=98587ms ERROR path: load-generator:user_get_recommendations 15005ms! > frontend-proxy:GET 15002ms! > frontend:GET 15002ms!
T-0002trace=c9fb0f39 dur=63607ms ok path: frontend-proxy:GET 2957ms > frontend:GET 2957ms > product-catalog:oteldemo.ProductCatalogService/GetProduct 12ms
T-0003trace=ce933c2c dur=95857ms ERROR path: frontend-proxy:GET 3110ms > frontend:GET 3110ms!
T-0004trace=3cfe7abe dur=55897ms ERROR path: frontend-proxy:GET 3071ms > frontend:GET 3071ms!
T-0005trace=9669b05b dur=95728ms ERROR path: frontend-proxy:GET 2981ms > frontend:GET 2981ms!
T-0006trace=83ce633b dur=55876ms ERROR path: frontend-proxy:GET 3050ms > frontend:GET 3050ms!
T-0007trace=4aa6e921 dur=63705ms ERROR path: frontend-proxy:GET 3114ms > frontend:GET 3115ms!
T-0008trace=69fba971 dur=55851ms ERROR path: frontend-proxy:GET 3024ms > frontend:GET 3025ms!
T-0009trace=70456535 dur=63656ms ERROR path: frontend-proxy:GET 3066ms > frontend:GET 3067ms!
T-0010trace=d399a3b3 dur=55836ms ERROR path: frontend-proxy:GET 3082ms > frontend:GET 3082ms!
T-0011trace=df592f64 dur=63613ms ERROR path: frontend-proxy:GET 3023ms > frontend:GET 3022ms!
T-0012trace=2b5d7928 dur=55813ms ERROR path: frontend-proxy:GET 3060ms > frontend:GET 3060ms!
T-0013trace=4ddcc233 dur=63299ms ERROR path: frontend-proxy:GET 3508ms > frontend:GET 3508ms!
T-0014trace=cc1e853e dur=55779ms ERROR path: frontend-proxy:GET 3026ms > frontend:GET 3022ms!
T-0015trace=cb5064fa dur=63202ms ERROR path: frontend-proxy:GET 3411ms > frontend:GET 3410ms!
T-0016trace=50fbe777 dur=55641ms ERROR path: frontend-proxy:GET 3374ms > frontend:GET 3374ms!
T-0017trace=d35d4b50 dur=59992ms ERROR path: frontend-proxy:GET 3085ms > frontend:GET 3084ms!
T-0018trace=2ccca019 dur=55569ms ERROR path: frontend-proxy:GET 3301ms > frontend:GET 3301ms!
T-0019trace=4d7b2eb2 dur=59962ms ERROR path: frontend-proxy:GET 3053ms > frontend:GET 3053ms!
T-0020trace=a3d0c7bc dur=55358ms ERROR path: frontend-proxy:GET 3218ms > frontend:GET 3218ms!
T-0021trace=089ffac4 dur=59941ms ERROR path: frontend-proxy:GET 3031ms > frontend:GET 3031ms!
T-0022trace=aa580d8d dur=55104ms ERROR path: frontend-proxy:GET 2965ms > frontend:GET 2965ms!
T-0023trace=75b4cfa2 dur=59476ms ERROR path: frontend-proxy:GET 3106ms > frontend:GET 3106ms!
T-0024trace=b6730c8a dur=54988ms ERROR path: load-generator:user_get_recommendations 15005ms! > frontend-proxy:GET 15002ms! > frontend:GET 15001ms
T-0025trace=68a72c56 dur=59432ms ERROR path: frontend-proxy:GET 3062ms > frontend:GET 3062ms!
T-0026trace=ef90a329 dur=53909ms ok path: frontend-proxy:GET 3109ms > frontend:GET 3109ms
T-0027trace=cc0231db dur=59396ms ERROR path: frontend-proxy:GET 3027ms > frontend:GET 3027ms!
T-0028trace=d151285e dur=52319ms ERROR path: frontend-proxy:GET 3069ms > frontend:GET 3070ms!
T-0029trace=105b5dd9 dur=59316ms ERROR path: frontend-proxy:GET 3061ms > frontend:GET 3063ms!
T-0030trace=5c064f43 dur=52301ms ERROR path: frontend-proxy:GET 3051ms > frontend:GET 3051ms!
T-0031trace=03512d6d dur=59303ms ERROR path: frontend-proxy:GET 3047ms > frontend:GET 3049ms!
T-0032trace=d300dd63 dur=52272ms ERROR path: frontend-proxy:GET 3022ms > frontend:GET 3023ms!
T-0033trace=fe6863f7 dur=59279ms ERROR path: frontend-proxy:GET 3024ms > frontend:GET 3025ms!
T-0034trace=9060141c dur=51955ms ERROR path: frontend-proxy:GET 3115ms > frontend:GET 3116ms!
T-0035trace=85585dde dur=56930ms ERROR path: load-generator:user_get_recommendations 15001ms! > frontend-proxy:GET 14998ms! > frontend:GET 14997ms!
T-0036trace=227684d4 dur=51912ms ERROR path: frontend-proxy:GET 3070ms > frontend:GET 3071ms!
T-0037trace=b7984b63 dur=56608ms ERROR path: load-generator:user_get_recommendations 15000ms! > frontend-proxy:GET 14998ms! > frontend:GET 14999ms!
T-0038trace=b80f2f39 dur=51877ms ERROR path: frontend-proxy:GET 3034ms > frontend:GET 3036ms!
T-0039trace=0e0b8e18 dur=56009ms ERROR path: load-generator:user_get_recommendations 15002ms! > frontend-proxy:GET 14998ms! > frontend:GET 14998ms!
T-0040trace=81281f7b dur=51449ms ERROR path: frontend-proxy:GET 3120ms > frontend:GET 3122ms!
logs: log templates per service, rate before vs after fault onset, with examples
L-0001svc=recommendation sev=INFO base=0.0/min fault=70.8/min tpl="get_product_list: cache <*>" ex=['get_product_list: cache miss', 'get_product_list: cache miss']
L-0002svc=recommendation sev=INFO base=216.0/min fault=70.0/min tpl="Receive ListRecommendations for product <*> <*> <*> <*> <*>" ex=["Receive ListRecommendations for product ids:['HQTGWGPNH4', '1YMWWN1N4O', '0PUK6V6EV0', 'LS4PSXUNUM', 'OLJCESPC7Z']", "Receive ListRecommendations for product ids:['HQTGWGPNH4', '1YMWWN1N4O', '0PUK6V6EV0', 'LS4PSXUNUM', 'OLJCESPC7Z']"]
L-0003svc=product-catalog sev=INFO base=142.0/min fault=51.8/min tpl="Found <NUM> products from database" ex=['Found 10 products from database', 'Found 10 products from database']
L-0004svc=product-catalog sev=INFO base=639.0/min fault=305.0/min tpl="Product Found" ex=['Product Found', 'Product Found']
L-0005svc=load-generator sev=INFO base=17.6/min fault=9.6/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user e41cd680-dbec-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user e41cd680-dbec-11f1-8306-d6b650b5399b']
L-0006svc=load-generator sev=INFO base=9.6/min fault=16.8/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user e156a480-dbec-11f1-8306-d6b650b5399b', 'Checkout completed for user e156a480-dbec-11f1-8306-d6b650b5399b']
L-0007svc=currency sev=INFO base=109.4/min fault=69.6/min tpl="conversion successful" ex=['conversion successful', 'conversion successful']
L-0008svc=recommendation sev=INFO base=0.0/min fault=0.4/min tpl="Recommendation service started, listening on port <NUM>" ex=['Recommendation service started, listening on port 9001', 'Recommendation service started, listening on port 9001']
L-0009svc=load-generator sev=INFO base=92.8/min fault=66.4/min tpl="User <UUID> adding <NUM> of product <*> to cart" ex=['User dab1a648-dbec-11f1-8306-d6b650b5399b adding 4 of product 2ZYFJ3GM2N to cart', 'User dab1a648-dbec-11f1-8306-d6b650b5399b adding 4 of product 2ZYFJ3GM2N to cart']
L-0010svc=ad sev=INFO base=2.6/min fault=3.8/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0011svc=cart sev=INFORMATION base=63.4/min fault=48.8/min tpl="AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}" ex=['AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}', 'AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}']
L-0012svc=load-generator sev=INFO base=134.0/min fault=106.8/min tpl="User browsing product: <*>" ex=['User browsing product: 1YMWWN1N4O', 'User browsing product: 1YMWWN1N4O']
L-0013svc=checkout sev=INFO base=0.0/min fault=0.2/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0014svc=recommendation sev=WARN base=0.0/min fault=0.2/min tpl="Overriding of current LoggerProvider is not allowed" ex=['Overriding of current LoggerProvider is not allowed']
L-0015svc=frontend-proxy sev=UNSET base=273.8/min fault=229.2/min tpl="<*> <*> <*> HTTP/<NUM>" <NUM> <*> <*> - "-" <NUM> <NUM> <NUM> <*> "-" <*> <*> <*> <*> <*> <*> <NUM>.<NUM>:<NUM> <NUM>.<NUM>:<NUM> - -" ex=['[2026-11-09T03:22:53.317Z] "GET /api/recommendations?productIds=2ZYFJ3GM2N HTTP/1.1" 200 - via_upstream - "-" 0 1299 15 15 "-" "python-requ…', '[2026-11-09T03:22:54.164Z] "GET /api/data/?contextKeys=assembly HTTP/1.1" 308 - via_upstream - "-" 0 30 2 1 "-" "python-requests/2.34.2" "3…']
L-0016svc=load-generator sev=INFO base=24.4/min fault=29.2/min tpl="Currency changed to CHF" ex=['Currency changed to CHF', 'Currency changed to CHF']
L-0017svc=frontend sev=INFO base=557.2/min fault=474.6/min tpl="API request <*>" ex=['API request completed', 'API request completed']
L-0018svc=frontend-proxy sev=UNSET base=1828.0/min fault=1598.2/min tpl="<*> <*> <*> HTTP/<NUM>" <NUM> <*> <*> - "-" <NUM> <NUM> <NUM> <*> "-" "Mozilla/<NUM> (X11; Linux x86_64) AppleWebKit/<NUM> (KHTML, like Gec…" ex=['[2026-11-09T03:22:53.554Z] "POST /flagservice/flagd.evaluation.v1.Service/ResolveAll HTTP/1.1" 200 - via_upstream - "-" 136 365 1 0 "-" "Mo…', '[2026-11-09T03:22:53.554Z] "POST /flagservice/flagd.evaluation.v1.Service/ResolveAll HTTP/1.1" 200 - via_upstream - "-" 136 365 1 0 "-" "Mo…']
L-0019svc=ad sev=INFO base=53.2/min fault=47.6/min tpl="Targeted ad request received for <*>" ex=['Targeted ad request received for [assembly]', 'Targeted ad request received for [assembly]']
L-0020svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=58836, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=58986, siz…']
43 lower-ranked lines omitted
Loading runs in your browser…