A company runs a generative AI assistant for customer support. Leadership wants sustained reliability in production. Which practice is a key part of responsible ongoing problem-solving for that system?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Let's get real about reliability. Shipping a support bot isn't a one-and-done trophy. You need regular audits—sample real chats, score them—and A/B tests when you change prompts or models so you know if you're helping or hurting. Relying on one offline number? That's how teams miss angry customers who "technically" match accuracy. Never training again after a shiny demo? Recipe for drift disaster. Ignoring user comments? You're throwing away free bug reports. Imagine your boss walks in after a viral bad reply: "Why didn't we catch this?" Hopefully you answer with audits and experiments. Exam angle: reliability = continuous evaluation in the wild, not a single lab score. Keep measuring, compare carefully, listen to users. You've got this—treat production like a living system.
Full explanation below image
Full Explanation
Reliability for customer-facing AI is multidimensional. The system must remain accurate enough for the task, safe and policy-compliant, responsive under load, and stable as products, policies, and user language evolve. Achieving that in production requires continuous problem-solving: sample production conversations for human audit, track automated quality and safety metrics, watch for distribution shift, and evaluate proposed changes with controlled experiments such as A/B or champion/challenger tests before full traffic migration. Those practices convert reliability from a launch checklist into an operating discipline.
Depending on a single evaluation metric is a frequent failure mode. Accuracy or an automatic similarity score may look strong while latency spikes, refusal behavior breaks, or certain customer segments receive worse outcomes. Multiple complementary metrics—task success, escalation rate, customer satisfaction, toxicity or policy violations, and operational SLOs—give a fuller picture. Freezing measurement design forever ignores new failure modes discovered in the field.
Never training or improving after an early demo confuses prototype success with production readiness. Demos often use curated prompts and lack adversarial users, peak load, and long-tail intents. Production systems need refresh cycles for retrieval corpora, prompts, fine-tunes, or guardrails when audits show gaps. Ignoring user comments and tickets discards high-value labeled signal: repeated complaints map directly to failure clusters that offline test sets may underrepresent.
Best practices include documented evaluation rubrics for auditors, statistically sound experiment design, clear rollback criteria, feedback loops from CRM or ticketing into the evaluation set, and governance that assigns owners for model risk. Memory aid: demo proves possibility; audits and A/B tests prove reliability under change. For certification and real deployments, prefer ongoing multi-signal evaluation over one-shot metrics, frozen models, or silence toward customers.