Rootline v0.1 pilot

← All incidents

productCatalogFailure-r1

What each model saw and answered. The true root cause stays hidden until you reveal it. pilot (n=42 snapshots)

Service graph during the window

Evidence and answers

4846 prompt tokens

metrics: per-service metrics, 30 s buckets from fault onset, % change vs pre-onset mean

M-0001svc=accounting metric=db.client.connection.npgsql.create_time.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%]
M-0002svc=accounting metric=dns.lookup.duration.count.rate base=0 delta/30s=[na,new:0.0333,na,+0%,na,+0%,na,+0%,na,+0%]
M-0003svc=ad metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[na,new:2,na,+0%,na,+0%,na,+0%,na,new:2]
M-0004svc=fraud-detection metric=error_ratio base=0 delta/30s=[na,new:0.0435,na,+0%,na,+0%,na,+0%,na,+0%]
M-0005svc=kafka metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[na,+0%,na,new:2,na,+0%,na,+0%,na,+0%]
M-0006svc=payment metric=error_ratio base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,+0%,na,new:0.0286]
M-0007svc=quote metric=otel.sdk.processor.log.queue.size base=0 delta/30s=[na,+0%,na,+0%,na,+0%,na,new:1,na,+0%]
M-0008svc=recommendation metric=error_ratio base=0 delta/30s=[na,+0%,na,new:0.00299,na,+0%,na,+0%,na,+0%]
M-0009svc=recommendation metric=traces.span.metrics.duration.p50 base=3.3 delta/30s=[na,-4%,na,+454661%,na,+0%,na,-4%,na,+1%]
M-0010svc=recommendation metric=traces.span.metrics.duration.p95 base=5.82 delta/30s=[na,-4%,na,+257673%,na,-3%,na,-8%,na,-4%]
M-0011svc=recommendation metric=traces.span.metrics.duration.p99 base=11.3 delta/30s=[na,-31%,na,+132124%,na,-47%,na,-47%,na,-48%]
M-0012svc=payment metric=traces.span.metrics.duration.p50 base=15.9 delta/30s=[na,-92%,na,-93%,na,-87%,na,-93%,na,+94402%]
M-0013svc=payment metric=traces.span.metrics.duration.p95 base=26.5 delta/30s=[na,-67%,na,-89%,na,-65%,na,-76%,na,+56461%]
M-0014svc=payment metric=traces.span.metrics.duration.p99 base=46.8 delta/30s=[na,-79%,na,-92%,na,-79%,na,-84%,na,+31932%]
M-0015svc=fraud-detection metric=traces.span.metrics.duration.p50 base=52.9 delta/30s=[na,+28268%,na,-18%,na,-43%,na,-31%,na,-18%]
M-0016svc=fraud-detection metric=traces.span.metrics.duration.p95 base=95.1 delta/30s=[na,+15673%,na,-1%,na,-3%,na,-3%,na,-1%]
M-0017svc=fraud-detection metric=traces.span.metrics.duration.p99 base=99.0 delta/30s=[na,+15049%,na,-0%,na,-1%,na,-1%,na,-0%]
M-0018svc=quote metric=otel.sdk.processor.span.queue.size base=0.6 delta/30s=[na,+1400%,na,+400%,na,-100%,na,+400%,na,+900%]
M-0019svc=checkout metric=go.goroutine.count.rate base=0.0133 delta/30s=[na,-100%,na,-100%,na,+400%,na,+900%,na,-100%]
M-0020svc=unknown metric=postgresql.bgwriter.checkpoint.count.rate base=0.00333 delta/30s=[-100%,-100%,-100%,-100%,-100%,-100%,-100%,+900%,-100%,-100%]
M-0021svc=ad metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[na,-100%,na,-100%,na,-100%,na,+400%,na,-100%]
M-0022svc=fraud-detection metric=jvm.gc.duration.count.rate base=0.00667 delta/30s=[na,+400%,na,-100%,na,-100%,na,-100%,na,+400%]
M-0023svc=checkout metric=http.client.request.duration.p95 base=0.352 delta/30s=[na,+22%,na,+35%,na,+354%,na,+6%,na,+17%]
M-0024svc=flagd metric=traces.span.metrics.duration.p50 base=3.5e+03 delta/30s=[na,+329%,na,+329%,na,+0%,na,+0%,na,+329%]
M-0025svc=shipping metric=http.server.request.duration.p50 base=0.0386 delta/30s=[na,+192%,na,-75%,na,+289%,na,-48%,na,-55%]
M-0026svc=load-generator metric=otel.sdk.processor.log.queue.size base=1.6 delta/30s=[+275%,na,+275%,na,+150%,na,+25%,na,+25%,na]
M-0027svc=quote metric=traces.span.metrics.duration.p50 base=5.41 delta/30s=[na,+269%,na,-34%,na,+11%,na,-8%,na,-31%]
M-0028svc=shipping metric=traces.span.metrics.duration.p50 base=38.9 delta/30s=[na,+234%,na,-78%,na,+242%,na,+3%,na,-23%]
M-0029svc=flagd metric=traces.span.metrics.duration.p95 base=4.85e+03 delta/30s=[na,+209%,na,+209%,na,+0%,na,+0%,na,+209%]
M-0030svc=load-generator metric=otel.sdk.processor.span.queue.size base=20.2 delta/30s=[+73%,na,+207%,na,+39%,na,+49%,na,+44%,na]
M-0031svc=flagd metric=traces.span.metrics.duration.p99 base=4.97e+03 delta/30s=[na,+202%,na,+202%,na,+0%,na,+0%,na,+202%]
M-0032svc=ad metric=jvm.cpu.recent_utilization base=0.0002 delta/30s=[na,+39%,na,-13%,na,+4%,na,+182%,na,-20%]
M-0033svc=checkout metric=traces.span.metrics.duration.p50 base=221.7 delta/30s=[na,+119%,na,-10%,na,+171%,na,+13%,na,-10%]
M-0034svc=accounting metric=traces.span.metrics.duration.p50 base=3.79e+03 delta/30s=[na,-47%,na,-8%,na,+164%,na,-12%,na,-47%]
M-0035svc=ad metric=jvm.thread.count.rate base=0.0267 delta/30s=[na,+150%,na,-100%,na,-100%,na,+25%,na,-100%]
M-0036svc=product-catalog metric=go.goroutine.count.rate base=0.0267 delta/30s=[na,+150%,na,-100%,na,-100%,na,-100%,na,-100%]
M-0037svc=fraud-detection metric=otel.sdk.processor.span.queue.size base=0.8 delta/30s=[na,+150%,na,-100%,na,-100%,na,-100%,na,+150%]
876 lower-ranked lines omitted

traces: slowest and errored traces after fault onset, critical path (! = error span)

T-0001trace=f6bb7cf5 dur=123ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 123ms!
T-0002trace=2715e9af dur=31510ms ok path: accounting:order-consumed 31510ms
T-0003trace=7e5a77e6 dur=59ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 59ms!
T-0004trace=06edebb2 dur=26368ms ok path: accounting:order-consumed 26368ms
T-0005trace=101b79fb dur=38ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 38ms!
T-0006trace=f60fde93 dur=20103ms ok path: accounting:order-consumed 20103ms
T-0007trace=4dff4d5b dur=30ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 30ms!
T-0008trace=f59382e3 dur=16867ms ok path: accounting:order-consumed 16867ms
T-0009trace=f3dc7cb9 dur=30ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 30ms!
T-0010trace=d3f63c76 dur=15775ms ok path: accounting:order-consumed 15775ms
T-0011trace=78b53453 dur=27ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 27ms!
T-0012trace=5d3fa984 dur=15569ms ok path: accounting:order-consumed 15569ms
T-0013trace=50c671b6 dur=10ms ERROR path: flagd:flagd.evaluation.v2.Service/ResolveBoolean 10ms!
T-0014trace=7cf351ee dur=11372ms ok path: accounting:order-consumed 11372ms
T-0015trace=b6213c03 dur=11061ms ok path: accounting:order-consumed 11061ms
T-0016trace=500d02c1 dur=10741ms ok path: accounting:order-consumed 10741ms
T-0017trace=9475009b dur=10468ms ok path: accounting:order-consumed 10468ms
T-0018trace=d248ff01 dur=8699ms ok path: accounting:order-consumed 8699ms
T-0019trace=569726c6 dur=8581ms ok path: accounting:order-consumed 8581ms
T-0020trace=a5dc3008 dur=8048ms ok path: accounting:order-consumed 8048ms
T-0021trace=5e8a59a8 dur=7320ms ok path: accounting:order-consumed 7320ms
T-0022trace=415c3867 dur=6848ms ok path: accounting:order-consumed 6848ms
T-0023trace=af0509dc dur=5665ms ok path: accounting:order-consumed 5665ms
T-0024trace=b0433abc dur=5610ms ok path: accounting:order-consumed 5610ms
T-0025trace=0a9fba96 dur=5382ms ok path: accounting:order-consumed 5382ms
T-0026trace=3dac4591 dur=4724ms ok path: accounting:order-consumed 4724ms
T-0027trace=f66e11ed dur=4584ms ok path: accounting:order-consumed 4584ms

logs: log templates per service, rate before vs after fault onset, with examples

L-0001svc=shipping sev=INFO base=2.2/min fault=0.8/min tpl="" ex=['', '']
L-0002svc=ad sev=INFO base=1.4/min fault=2.6/min tpl="Non-targeted ad request received, preparing random response." ex=['Non-targeted ad request received, preparing random response.', 'Non-targeted ad request received, preparing random response.']
L-0003svc=load-generator sev=INFO base=16.4/min fault=10.8/min tpl="Multi-item checkout completed for user <UUID>" ex=['Multi-item checkout completed for user 91e8e974-dbb7-11f1-8306-d6b650b5399b', 'Multi-item checkout completed for user 91e8e974-dbb7-11f1-8306-d6b650b5399b']
L-0004svc=accounting sev=INFORMATION base=14.6/min fault=11.4/min tpl="Order details: {@OrderResult}." ex=['Order details: {@OrderResult}.', 'Order details: {@OrderResult}.']
L-0005svc=cart sev=INFORMATION base=14.6/min fault=11.4/min tpl="EmptyCartAsync called with userId={userId}" ex=['EmptyCartAsync called with userId={userId}', 'EmptyCartAsync called with userId={userId}']
L-0006svc=checkout sev=INFO base=14.6/min fault=11.4/min tpl="Successful to write message. offset: <NUM>, duration: <*>" ex=['Successful to write message. offset: 0, duration: 199.98µs', 'Successful to write message. offset: 0, duration: 191.709µs']
L-0007svc=checkout sev=INFO base=14.6/min fault=11.4/min tpl="[PlaceOrder]" ex=['[PlaceOrder]', '[PlaceOrder]']
L-0008svc=checkout sev=INFO base=14.6/min fault=11.4/min tpl="order confirmation email sent" ex=['order confirmation email sent', 'order confirmation email sent']
L-0009svc=checkout sev=INFO base=14.6/min fault=11.4/min tpl="order placed" ex=['order placed', 'order placed']
L-0010svc=checkout sev=INFO base=14.6/min fault=11.4/min tpl="payment went through" ex=['payment went through', 'payment went through']
L-0011svc=checkout sev=INFO base=14.6/min fault=11.4/min tpl="sending to postProcessor" ex=['sending to postProcessor', 'sending to postProcessor']
L-0012svc=email sev=INFO base=14.6/min fault=11.4/min tpl="Order confirmation email sent" ex=['Order confirmation email sent', 'Order confirmation email sent']
L-0013svc=fraud-detection sev=INFO base=14.6/min fault=11.4/min tpl="Consumed record with orderId: <UUID>, and updated total count to: <NUM>" ex=['Consumed record with orderId: 91f0bfd9-dbb7-11f1-82f3-d279f8fa1f6d, and updated total count to: 10054', 'Consumed record with orderId: 91e4133e-dbb7-11f1-82f3-d279f8fa1f6d, and updated total count to: 10055']
L-0014svc=frontend sev=INFO base=14.6/min fault=11.4/min tpl="Order placed successfully" ex=['Order placed successfully', 'Order placed successfully']
L-0015svc=payment sev=INFO base=14.6/min fault=11.4/min tpl="Charge request received." ex=['Charge request received.', 'Charge request received.']
L-0016svc=payment sev=INFO base=14.6/min fault=11.4/min tpl="Transaction complete." ex=['Transaction complete.', 'Transaction complete.']
L-0017svc=shipping sev=INFO base=14.6/min fault=11.4/min tpl="Tracking ID Created" ex=['Tracking ID Created', 'Tracking ID Created']
L-0018svc=checkout sev=INFO base=0.0/min fault=0.2/min tpl="client/metadata fetching metadata for all topics from broker kafka:<NUM>" ex=['client/metadata fetching metadata for all topics from broker kafka:9092']
L-0019svc=load-generator sev=INFO base=12.4/min fault=14.4/min tpl="User accessing index page" ex=['User accessing index page', 'User accessing index page']
L-0020svc=kafka sev=INFO base=2.0/min fault=2.4/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=13149, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=13179, siz…']
L-0021svc=currency sev=INFO base=125.0/min fault=111.2/min tpl="conversion successful" ex=['conversion successful', 'conversion successful']
L-0022svc=kafka sev=INFO base=1.0/min fault=0.8/min tpl="[LocalLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs/__cluster_metadata-<NUM>] Deleting segment files LogSegment(baseOffset=<N…" ex=['[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=13239, siz…', '[LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Deleting segment files LogSegment(baseOffset=13359, siz…']
L-0023svc=kafka sev=INFO base=1.0/min fault=0.8/min tpl="[UnifiedLog partition=__cluster_metadata-<NUM>, dir=/tmp/kafka-logs] Deleting segments due to log start offset <NUM> breach: LogSegment(bas…" ex=['[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 13422 breach: LogSegment(baseOff…', '[UnifiedLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs] Deleting segments due to log start offset 13539 breach: LogSegment(baseOff…']
L-0024svc=load-generator sev=INFO base=39.6/min fault=35.6/min tpl="User viewing cart" ex=['User viewing cart', 'User viewing cart']
L-0025svc=load-generator sev=INFO base=89.6/min fault=81.2/min tpl="User <UUID> adding <NUM> of product <*> to cart" ex=['User 90585ed2-dbb7-11f1-8306-d6b650b5399b adding 5 of product 6E92ZMYYFZ to cart', 'User 90585ed2-dbb7-11f1-8306-d6b650b5399b adding 5 of product 6E92ZMYYFZ to cart']
L-0026svc=quote sev=INFO base=29.6/min fault=27.2/min tpl="Calculated quote" ex=['Calculated quote', 'Calculated quote']
L-0027svc=shipping sev=INFO base=29.6/min fault=27.2/min tpl="Requesting quote" ex=['Requesting quote', 'Requesting quote']
L-0028svc=shipping sev=INFO base=29.6/min fault=27.2/min tpl="Sending Quote" ex=['Sending Quote', 'Sending Quote']
L-0029svc=load-generator sev=INFO base=29.6/min fault=32.0/min tpl="Product added to cart successfully" ex=['Product added to cart successfully', 'Product added to cart successfully']
32 lower-ranked lines omitted
Loading runs in your browser…