A team is building a customer-facing chatbot for a financial services firm. The system must handle 10,000 simultaneous users, respond within 500ms, and answer FAQs from a knowledge base. Cost is a primary constraint. Which model selection strategy is most architecturally sound?
Select an answer to reveal the explanation.
Short Explanation
Here's the deal — you need a tiered routing strategy: use Haiku for high-volume FAQ and routine queries, then escalate to Sonnet for complex regulatory questions. Haiku gives you the lowest latency and cost per token for the bulk of traffic, while Sonnet preserves quality where it matters.
Full Explanation
A tiered model routing strategy — Claude 3 Haiku for high-volume, low-complexity FAQ retrieval and routine queries, with automatic escalation to Claude 3.5 Sonnet for complex regulatory questions — optimizes both cost and quality. Haiku offers the lowest latency and cost per token, making it viable for the bulk of repetitive FAQ queries and helping meet the 500ms response target for 10,000 simultaneous users. Escalating to Sonnet only when complexity warrants it keeps average cost low while maintaining quality where it matters. The option that uses Claude 3 Opus for all requests is prohibitively expensive at 10K concurrent users and overkill for FAQs. The option that uses Claude 3.5 Sonnet for all requests improves on Opus but still overspends on simple queries. The option that uses Claude 3 Haiku with a fine-tuned system prompt misunderstands the relationship between prompting and model capability — prompt engineering cannot substitute for model reasoning capacity.