A team is building a customer-facing chatbot for a financial services firm. The system must handle 10,000 simultaneous users, respond within 500ms, and answer FAQs from a knowledge base. Cost is a primary constraint. Which model selection strategy is most architecturally sound?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal — a tiered model routing strategy — Haiku for high-volume, low-complexity FAQ retrieval and Sonnet for complex regulatory analysis — optimizes both cost and quality. Haiku offers the lowest latency and cost per token, making it viable for the bulk of repetitive FAQ queries.
Full explanation below image
Full Explanation
A tiered model routing strategy — Haiku for high-volume, low-complexity FAQ retrieval and Sonnet for complex regulatory analysis — optimizes both cost and quality. Haiku offers the lowest latency and cost per token, making it viable for the bulk of repetitive FAQ queries. Escalating to Sonnet only when complexity warrants it keeps average cost low while maintaining quality where it matters. Option A (Opus for all) is prohibitively expensive at 10K concurrent users and overkill for FAQs. Option B (Sonnet for all) improves on Opus but still overspends on simple queries. Option D misunderstands the relationship between prompting and model capability — prompt engineering cannot substitute for model reasoning capacity.