recommendationCacheFailure-r1
What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)
Service graph during the window
Evidence and answers
5403 prompt tokens
metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean
M-0001svc=frontend metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:0.146,na,new:0.000344]
M-0002svc=frontend-web metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:0.00534,na,+0%]
M-0003svc=kafka metric=jvm.gc.duration.count.rate base=0 delta/30s=[na,+0%,na,+0%,na,new:0.0333,na,+0%,na,+0%]
M-0004svc=payment metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,new:0.04,na,+0%,na,+0%]
M-0005svc=payment metric=traces.span.metrics.duration.p50 base=1.36 delta/30s=[na,-20%,na,-21%,na,+1103872%,na,-26%,na,-26%]
M-0006svc=recommendation metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[na,new:4,na,na,na,na,na,na,+0%,na]
M-0007svc=recommendation metric=process.runtime.cpython.cpu.utilization base=0 delta/30s=[na,new:13.7,na,na,na,na,na,na,new:0.854,na]
M-0008svc=payment metric=traces.span.metrics.duration.p95 base=17.9 delta/30s=[na,-84%,na,-86%,na,+83512%,na,-89%,na,-89%]
M-0009svc=payment metric=traces.span.metrics.duration.p99 base=22.5 delta/30s=[na,-83%,na,-84%,na,+66662%,na,-91%,na,-91%]
M-0010svc=recommendation metric=traces.span.metrics.duration.p95 base=5.9 delta/30s=[na,+714%,na,+43984%,na,na,na,+1553%,na,+5563%]
M-0011svc=frontend metric=http.server.request.duration.p50 base=0.00684 delta/30s=[-0%,na,+5%,na,+5%,na,+6%,na,+43781%,na]
M-0012svc=recommendation metric=traces.span.metrics.duration.count.rate base=8.86 delta/30s=[na,+14%,na,-70%,na,-100%,na,-99%,na,+41886%]
M-0013svc=recommendation metric=traces.span.metrics.duration.p99 base=11.7 delta/30s=[na,+513%,na,+38400%,na,na,na,+748%,na,+5752%]
M-0014svc=load-generator metric=http.client.duration.p50 base=30.0 delta/30s=[-67%,na,-35%,na,+33212%,na,+33212%,na,-46%,na]
M-0015svc=recommendation metric=request_rate base=8.86 delta/30s=[na,+14%,na,-70%,na,-100%,na,+28032%,na,+27716%]
M-0016svc=frontend-proxy metric=error_ratio base=5.35e-05 delta/30s=[na,-100%,na,+4486%,na,+11485%,na,+19652%,na,+3157%]
M-0017svc=load-generator metric=error_ratio base=0.000474 delta/30s=[na,-100%,na,+4532%,na,+12452%,na,+5151%,na,+3658%]
M-0018svc=frontend metric=traces.span.metrics.duration.p50 base=396.3 delta/30s=[na,-12%,na,-52%,na,-24%,na,+3685%,na,+4%]
M-0019svc=recommendation metric=traces.span.metrics.duration.p50 base=3.29 delta/30s=[na,+812%,na,+2940%,na,na,na,+2180%,na,+2180%]
M-0020svc=frontend metric=nodejs.eventloop.delay.max base=0.0195 delta/30s=[-17%,na,-17%,na,+52%,na,+114%,na,+1823%,na]
M-0021svc=frontend-proxy metric=traces.span.metrics.duration.p50 base=1.16e+03 delta/30s=[na,-74%,na,+1150%,na,+1139%,na,+253%,na,+1193%]
M-0022svc=frontend metric=nodejs.eventloop.delay.stddev base=0.000414 delta/30s=[-12%,na,-5%,na,+20%,na,+75%,na,+1112%,na]
M-0023svc=frontend metric=http.server.request.duration.p95 base=0.874 delta/30s=[-52%,na,-45%,na,+211%,na,+316%,na,+1044%,na]
M-0024svc=load-generator metric=http.client.duration.p95 base=972.1 delta/30s=[-58%,na,-26%,na,+929%,na,+929%,na,+14%,na]
M-0025svc=ad metric=otel.sdk.processor.log.queue.size base=0.2 delta/30s=[na,-100%,na,+900%,na,-100%,na,+400%,na,-100%]
M-0026svc=checkout metric=go.goroutine.count.rate base=0.0133 delta/30s=[na,+900%,na,-100%,na,+400%,na,+400%,na,-100%]
M-0027svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,-100%,-100%,-100%,-100%,-100%,-100%,+900%,-100%,-100%]
M-0028svc=frontend metric=traces.span.metrics.duration.p95 base=2.02e+03 delta/30s=[na,-38%,na,+642%,na,+87%,na,+642%,na,+185%]
M-0029svc=frontend metric=traces.span.metrics.duration.p99 base=2.39e+03 delta/30s=[na,-43%,na,+528%,na,+528%,na,+528%,na,+283%]
M-0030svc=frontend metric=http.server.request.duration.p99 base=1.69 delta/30s=[-49%,na,+22%,na,+492%,na,+492%,na,+492%,na]
M-0031svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,+400%,na,-100%,na,-100%]
M-0032svc=accounting metric=dns.lookup.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,+400%,na,-100%,na,-100%]
M-0033svc=accounting metric=dotnet.thread_pool.thread.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,+400%,na,-100%,na,-100%]
M-0034svc=accounting metric=process.thread.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,+400%,na,-100%,na,-100%]
M-0035svc=ad metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,-100%,na,-100%,na,+400%]
M-0036svc=cart metric=dotnet.thread_pool.thread.count.rate base=0.00667 delta/30s=[na,+400%,na,-100%,na,-100%,na,+400%,na,-100%]
M-0037svc=cart metric=process.thread.count.rate base=0.00667 delta/30s=[na,+400%,na,-100%,na,-100%,na,+400%,na,-100%]
876 lower-ranked lines omitted
traces: slowest and errored traces after fault onset, critical path (! = error span)
T-0001trace=9322fbbf dur=115730ms ERROR path: load-generator:user_get_recommendations 15008ms! > frontend-proxy:GET 15006ms! > frontend:GET 15005ms!
T-0002trace=6e087785 dur=104729ms ok path: flagd:flagd.evaluation.v2.Service/EventStream 104729ms
T-0003trace=f97f8b92 dur=114259ms ERROR path: load-generator:user_get_recommendations 15006ms! > frontend-proxy:GET 15004ms! > frontend:GET 15003ms!
T-0004trace=d848099f dur=101152ms ERROR path: load-generator:user_get_recommendations 15000ms! > frontend-proxy:GET 14997ms! > frontend:GET 14998ms!
T-0005trace=c0c8dffc dur=113669ms ERROR path: frontend-proxy:GET 3114ms > frontend:GET 3114ms!
T-0006trace=3c3be82a dur=101147ms ERROR path: frontend-proxy:GET 3063ms > frontend:GET 3062ms!
T-0007trace=c240c6d8 dur=113541ms ERROR path: frontend-proxy:GET 2985ms > frontend:GET 2986ms!
T-0008trace=c100396b dur=101059ms ERROR path: frontend-proxy:GET 2974ms > frontend:GET 2974ms!
T-0009trace=d153c8d2 dur=113497ms ERROR path: frontend-proxy:GET 3117ms > frontend:GET 3117ms!
T-0010trace=80272c02 dur=98277ms ERROR path: load-generator:user_get_recommendations 14999ms! > frontend-proxy:GET 14997ms! > frontend:GET 14999ms!
T-0011trace=1c7ac541 dur=113343ms ERROR path: frontend-proxy:GET 2964ms > frontend:GET 2963ms!
T-0012trace=ba40a5a2 dur=97656ms ERROR path: frontend-proxy:GET 3079ms > frontend:GET 3081ms!
T-0013trace=88040e5d dur=111841ms ERROR path: load-generator:user_get_recommendations 15006ms! > frontend-proxy:GET 15004ms! > frontend:GET 15003ms!
T-0014trace=2e23b0e4 dur=97651ms ERROR path: frontend-proxy:GET 3079ms > frontend:GET 3075ms!
T-0015trace=85d660b3 dur=110086ms ERROR path: frontend-proxy:GET 3083ms > frontend:GET 3083ms!
T-0016trace=626d20e8 dur=97622ms ERROR path: frontend-proxy:GET 3045ms > frontend:GET 3046ms!
T-0017trace=f704b408 dur=110061ms ERROR path: frontend-proxy:GET 3058ms > frontend:GET 3058ms!
T-0018trace=bf830154 dur=97618ms ERROR path: frontend-proxy:GET 3045ms > frontend:GET 3045ms!
T-0019trace=60e8455e dur=110028ms ERROR path: frontend-proxy:GET 3025ms > frontend:GET 3025ms!
T-0020trace=cdf81e98 dur=97597ms ERROR path: frontend-proxy:GET 3024ms > frontend:GET 3026ms!
T-0021trace=e390fdaf dur=109486ms ERROR path: frontend-proxy:GET 3365ms > frontend:GET 3365ms!
T-0022trace=6d920597 dur=97596ms ERROR path: frontend-proxy:GET 3019ms > frontend:GET 3021ms!
T-0023trace=54198f56 dur=109364ms ERROR path: frontend-proxy:GET 3244ms > frontend:GET 3243ms!
T-0024trace=ac6b213a dur=97310ms ERROR path: load-generator:user_get_recommendations 15000ms! > frontend-proxy:GET 14997ms! > frontend:GET 14999ms!
T-0025trace=72f39251 dur=105796ms ERROR path: frontend-proxy:GET 3089ms > frontend:GET 3089ms!
T-0026trace=26a30da7 dur=97019ms ERROR path: load-generator:user_get_recommendations 14999ms! > frontend-proxy:GET 14996ms! > frontend:GET 14998ms!
T-0027trace=b8d2be57 dur=105775ms ERROR path: frontend-proxy:GET 3069ms > frontend:GET 3069ms!
T-0028trace=20d1e592 dur=96500ms ERROR path: load-generator:user_get_recommendations 15000ms! > frontend-proxy:GET 14998ms! > frontend:GET 15000ms!
T-0029trace=c56594bc dur=105747ms ERROR path: frontend-proxy:GET 3040ms > frontend:GET 3038ms!
T-0030trace=fa1518c2 dur=92935ms ERROR path: frontend-proxy:GET 3125ms > frontend:GET 3125ms!
T-0031trace=de3faf99 dur=104998ms ERROR path: frontend-proxy:GET 3104ms > frontend:GET 3103ms!
T-0032trace=59d98d92 dur=92823ms ERROR path: frontend-proxy:GET 3198ms > frontend:GET 3197ms!
T-0033trace=07f679c8 dur=104886ms ERROR path: frontend-proxy:GET 2991ms > frontend:GET 2991ms!
T-0034trace=27f7b7c6 dur=92786ms ERROR path: frontend-proxy:GET 2976ms > frontend:GET 2976ms!
T-0035trace=b67ae2e6 dur=101737ms ERROR path: frontend-proxy:GET 3089ms > frontend:GET 3088ms!
T-0036trace=677189df dur=92528ms ERROR path: frontend-proxy:GET 2902ms > frontend:GET 2903ms!
T-0037trace=da40ac09 dur=101712ms ERROR path: frontend-proxy:GET 3062ms > frontend:GET 3062ms!
T-0038trace=0f93753a dur=89346ms ERROR path: frontend-proxy:GET 3078ms > frontend:GET 3079ms!
T-0039trace=058cfb7a dur=101672ms ERROR path: frontend-proxy:GET 3023ms > frontend:GET 3022ms!
T-0040trace=96bf4fc2 dur=89322ms ERROR path: frontend-proxy:GET 3056ms > frontend:GET 3056ms!
logs: log templates per service, rate before vs after fault onset, with examples
L-0001svc=recommendation sev=INFO base=0.0/min fault=93.8/min tpl="get_product_list: cache <*>" ex=['get_product_list: cache miss', 'get_product_list: cache miss']
L-0002svc=recommendation sev=INFO base=222.4/min fault=95.2/min tpl="Receive ListRecommendations for product <*> <*> <*> <*> <*>" ex=["Receive ListRecommendations for product ids:['6E92ZMYYFZ', '0PUK6V6EV0', 'HQTGWGPNH4', 'OLJCESPC7Z', '1YMWWN1N4O']", "Receive ListRecommendations for product ids:['6E92ZMYYFZ', '0PUK6V6EV0', 'HQTGWGPNH4', 'OLJCESPC7Z', '1YMWWN1N4O']"]
L-0003svc=product-catalog sev=INFO base=140.0/min fault=60.0/min tpl="Found <NUM> products from database" ex=['Found 10 products from database', 'Found 10 products from database']
L-0004svc=recommendation sev=INFO base=0.0/min fault=0.8/min tpl="Recommendation service started, listening on port <NUM>" ex=['Recommendation service started, listening on port 9001', 'Recommendation service started, listening on port 9001']
L-0005svc=product-catalog sev=INFO base=615.0/min fault=357.6/min tpl="Product Found" ex=['Product Found', 'Product Found']
L-0006svc=currency sev=INFO base=108.2/min fault=70.8/min tpl="conversion successful" ex=['conversion successful', 'conversion successful']
L-0007svc=shipping sev=INFO base=1.8/min fault=1.0/min tpl="" ex=['', '']
L-0008svc=recommendation sev=WARN base=0.0/min fault=0.4/min tpl="Overriding of current LoggerProvider is not allowed" ex=['Overriding of current LoggerProvider is not allowed', 'Overriding of current LoggerProvider is not allowed']
L-0009svc=load-generator sev=INFO base=134.4/min fault=106.0/min tpl="User browsing product: <*>" ex=['User browsing product: 1YMWWN1N4O', 'User browsing product: 1YMWWN1N4O']
L-0010svc=ad sev=INFO base=3.4/min fault=2.6/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0011svc=load-generator sev=INFO base=10.8/min fault=8.8/min tpl="Checkout completed for user <UUID>" ex=['Checkout completed for user 983276e8-dbe3-11f1-8306-d6b650b5399b', 'Checkout completed for user 983276e8-dbe3-11f1-8306-d6b650b5399b']
L-0012svc=checkout sev=INFO base=0.2/min fault=0.0/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0013svc=load-generator sev=INFO base=31.2/min fault=26.4/min tpl="Currency changed to CHF" ex=['Currency changed to CHF', 'Currency changed to CHF']
L-0014svc=load-generator sev=INFO base=11.6/min fault=13.2/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user 95f630c2-dbe3-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user 95f630c2-dbe3-11f1-8306-d6b650b5399b']
L-0015svc=load-generator sev=INFO base=28.8/min fault=32.4/min tpl="Product added to cart successfully" ex=['Product added to cart successfully', 'Product added to cart successfully']
L-0016svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=50916, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=51036, siz…']
L-0017svc=kafka sev=INFO base=0.8/min fault=1.0/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 51096 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 51252 breach: LogSegment(baseOff…']
L-0018svc=kafka sev=INFO base=7.6/min fault=8.4/min tpl="Deleted <*> index <*>" ex=['Deleted offset index /tmp/kafka-logs/__cluster_metadata-0/00000000000000050916.index.deleted.', 'Deleted time index /tmp/kafka-logs/__cluster_metadata-0/00000000000000050916.timeindex.deleted.']
L-0019svc=load-generator sev=INFO base=88.4/min fault=81.2/min tpl="User getting <*> for <*> <*>" ex=['User getting recommendations for product: 2ZYFJ3GM2N', 'User getting recommendations for product: 2ZYFJ3GM2N']
L-0020svc=cart sev=INFORMATION base=48.2/min fault=52.4/min tpl="AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}" ex=['AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}', 'AddItemAsync called with userId={userId}, productId={productId}, quantity={quantity}']
L-0021svc=kafka sev=INFO base=3.8/min fault=4.2/min tpl="Deleted log /tmp/kafka-logs/__cluster_metadata-<NUM>/<HEX>.log.deleted." ex=['Deleted log /tmp/kafka-logs/__cluster_metadata-0/00000000000000050916.log.deleted.', 'Deleted log /tmp/kafka-logs/__cluster_metadata-0/00000000000000050946.log.deleted.']
L-0022svc=kafka sev=INFO base=3.8/min fault=4.2/min tpl="Deleted producer state snapshot /tmp/kafka-logs/__cluster_metadata-<NUM>/<HEX>.snapshot.deleted" ex=['Deleted producer state snapshot /tmp/kafka-logs/__cluster_metadata-0/00000000000000050916.snapshot.deleted', 'Deleted producer state snapshot /tmp/kafka-logs/__cluster_metadata-0/00000000000000050946.snapshot.deleted']
41 lower-ranked lines omitted
Loading runs in your browser…