A codebase assistant uses prompt caching to reduce costs. The system prompt is 4,000 tokens and is shared across all users. User-specific conversation history follows the system prompt. After deploying cache_control breakpoints, the team discovers cache hit rates are below 10%. What is the most likely root cause?
Select an answer to reveal the explanation.
Short Explanation
Here's the deal — prompt caching hashes the token sequence up to and including the cache_control breakpoint. If you put that breakpoint after user-specific conversation history, every user's request creates a different hash, so the cache misses. Put it right after the shared system prompt so all users share the same cached prefix.
Full Explanation
The correct root cause is that the cache_control breakpoint is placed after the user-specific conversation history rather than immediately after the shared system prompt, causing the cached prefix to include user-specific content that differs per request. Prompt caching hashes the token sequence up to and including the cache_control breakpoint. If the breakpoint is placed after user-specific content, each user's request produces a different hash because their conversation history differs, so the cache never hits. The breakpoint must be placed at the end of the longest stable prefix — immediately after the shared system prompt — so every user's request shares the same cached prefix. The claim about a 10,000-token minimum is wrong because the minimum threshold for claude-3-5 models is 1,024 tokens, so a 4,000-token system prompt qualifies regardless of breakpoint placement. The claim about system prompt changes exceeding a per-minute invalidation limit is wrong because invalidation is based on token content changes, not a time-based frequency limit, and there is no documented per-minute invalidation limit. The claim that claude-3-5-sonnet does not support system prompt caching is wrong because that model fully supports prompt caching for system prompts, a primary documented use case.