A content moderation pipeline processes 2 million short social media posts per day using Claude. The posts average 50 tokens each. The system prompt is 800 tokens. An architect wants to reduce inference cost. Which optimization has the greatest leverage?
Select an answer to reveal the explanation.
Short Explanation
Here's the deal — you want the option that caches the 800-token system prompt, because it's the dominant cost across 2M daily posts. It removes about 1.6B redundant input tokens each day, and cache hits are typically much cheaper, so you get the biggest cost cut without hurting moderation quality.
Full Explanation
The greatest leverage comes from using prompt caching to cache the 800-token system prompt, eliminating redundant processing across the 2M daily requests. The system prompt represents about 94% of the total input tokens for each 50-token post (800/(800+50)). With 2M requests/day, caching eliminates roughly 1.6B redundant system prompt tokens daily. Cache hit pricing is typically 10x cheaper than standard input pricing, making this a massive cost reduction with no quality impact. Switching from Claude 3.5 Sonnet to Claude 3 Haiku is wrong because model choice affects per-token pricing, but Haiku's lower quality on nuanced moderation decisions may require additional review passes that offset savings; the system prompt caching optimization is independent and larger in magnitude. Compressing each 50-token post to 30 tokens using a text summarization preprocessing step is wrong because summarizing already very short posts risks losing the exact content needed for accurate moderation; the savings (20 tokens × 2M = 40M tokens) are far smaller than the system prompt caching savings (800 tokens × 2M = 1.6B tokens). Batching posts into groups of 10 and processing them in a single API call is wrong because the Anthropic API does not support multi-document batching within a single messages API call in the way described; and system prompt caching already addresses the dominant cost driver.