A transit RAG team already tracks relevance scores but riders still abandon lookups that feel slow. What should they add to retrieval quality evaluation?

Adding retrieval latency SLOs alongside relevance scores so slow-but-relevant answers are still treated as failures is what a transit RAG team should add when riders abandon lookups that feel slow despite healthy relevance metrics. Relevance without timely delivery fails the rider journey: a perfect stop-disruption snippet that arrives after the train left is operationally useless. Pairing latency SLOs with relevance makes “right but late” a first-class failure mode in retrieval quality evaluation. Measuring relevance only, with no timing measurements, because latency never affects rider experience fails by concept—latency directly shapes abandonment and trust during transit disruptions. Measuring only the color theme of the transit mobile app fails by concept: UI theming is unrelated to retrieval quality. Measuring only the count of unused IAM access keys fails by concept: IAM hygiene is a security metric, not retrieval performance for riders. Exam caveat: retrieval quality for user-facing civic apps includes latency SLOs plus relevance, not relevance alone or unrelated UI/IAM metrics. Operational check: define p95 retrieval latency SLOs by channel, chart latency with relevance, alert when either breaches, optimize indexes/caching before festival or peak commute events, and include abandonment correlated with slow retrieval in transit product reviews.