A city IT team is weighing a single large foundation model against a cascade of smaller, specialized models for a resident-facing assistant, trying to balance response quality against inference cost. What is the key tradeoff the team should reason through?
Select an answer to reveal the explanation.
Short Explanation
Think of it like triage at a help desk: routine questions go to whoever's fastest and cheapest to consult, and only the tricky cases get escalated to the expert. A cascade of smaller models does exactly that — cheap models field simple resident questions, the big model handles what's genuinely hard, and average cost drops. The tradeoff is you now have more moving parts to build and keep working together.
Full Explanation
Architecting a multi-model FM solution means weighing average inference cost against added system complexity: a cascade routes simple, high-volume requests to smaller, cheaper models and escalates only complex cases to the large model, which can meaningfully lower blended cost per request compared with sending everything to the large model, but it introduces routing logic, additional monitoring surfaces, and more failure points to manage. Claiming a single large model's per-request cost is independent of request complexity ignores that inference cost for generative models generally scales with the amount of context and output generated, which does vary with how complex a request is, even under a single model. Claiming a cascade eliminates the need to monitor response quality because smaller models are inherently more accurate reverses the usual tradeoff — smaller specialized models are typically narrower and can underperform the large model on cases outside their specialty, which is exactly why quality monitoring across the cascade matters more, not less. Claiming the model architecture choice affects only development time and not inference cost misses the entire point of the comparison — cost and quality are the two axes the team is explicitly weighing. Scope caveat: a cascade's cost savings depend on request volume actually skewing toward the simple cases the cheaper models can handle; a workload dominated by complex requests erodes the savings. Operational check: track what fraction of resident requests get resolved by the smaller models versus escalated, and monitor cost per resolved request against the single-model baseline.