A fintech startup is building a document analysis pipeline that processes scanned bank statements and extracts structured data. Processing volume is 50,000 documents per day, and each document averages 8 pages. The team is evaluating whether to use Claude Haiku or Claude Sonnet for the vision extraction task. Accuracy testing shows Haiku achieves 94% field extraction accuracy and Sonnet achieves 97%. The downstream reconciliation system can tolerate up to 5% error rate before requiring human review. Which model selection strategy minimizes total cost while meeting accuracy requirements?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal — a tiered routing strategy (Haiku first, Sonnet escalation on low confidence) is architecturally optimal here. It captures Haiku's cost advantage on the majority of documents while using Sonnet only where its accuracy premium is needed, staying within the 5% tolerance.
Full explanation below image
Full Explanation
A tiered routing strategy (Haiku first, Sonnet escalation on low confidence) is architecturally optimal here. It captures Haiku's cost advantage on the majority of documents while using Sonnet only where its accuracy premium is needed, staying within the 5% tolerance. Option B is tempting — 94% accuracy does meet the 5% tolerance — but ignores that errors are not uniformly distributed; complex documents will cluster failures and push certain document classes above tolerance. Option A uses Sonnet universally, paying a ~5x cost premium for marginal improvement on documents Haiku handles correctly. Option D (Opus) is cost-prohibitive at this volume and solves a problem that doesn't exist given the reconciliation system already handles errors.