An enterprise platform processes 50,000 API requests per day to Claude. The engineering team discovers that 35% of these requests have identical system prompts (10,000 tokens each) combined with unique user messages (500-2,000 tokens each). Without caching, the monthly bill is $180,000. The team implements prompt caching using cache_control with type: 'ephemeral' on the system prompt. However, after implementation, they observe only a 12% cost reduction instead of the expected ~70% reduction. Which root cause is MOST likely responsible for this underperformance?
Select an answer to reveal the explanation.
Short Explanation
Here's the deal — you're probably injecting dynamic content into the supposedly static system prompt, so the cached prefix isn't byte-identical. That changes the cache key, causes cache misses, and leaves you with only a small savings instead of the expected large one.
Full Explanation
The most likely root cause for a 12% actual reduction versus the expected roughly 70% reduction is dynamic content injection into the supposedly static system prompt. Prompt caching only applies to input tokens processed before the cached breakpoint, and Claude's prompt cache key is computed from the exact byte sequence of the cached prefix. If the code injects anything variable — a request timestamp, session ID, user tier, dynamic date reference, or other user-specific content — into the system prompt before the user message, every request generates a unique cache key and results in a cache miss. The 35% of requests with 'identical' system prompts may not be byte-identical, so the cache cannot reuse the expensive prefix processing. The explanation that cache_control is applied to the user message rather than the system prompt is unlikely because cached user messages provide no savings since they are always unique, and that mistake would more plausibly produce 0% savings, not 12%. The explanation that the ephemeral cache TTL is shorter than the inter-request interval is also less likely: it assumes most requests are spaced more than 5 minutes apart, but at 50,000 requests per day with a 35% overlap group, the interval for the repeated system prompt is likely under 5 minutes. The explanation that the 10,000-token system prompt falls below the minimum cacheable token threshold misidentifies the issue because 10,000 tokens is well above the minimum, and the related point that cache_control must be set as the last block before variable content would cause cache misses, not partial savings.