A developer wants to add prompt caching to a RAG pipeline where the system prompt contains static instructions (2,000 tokens), followed by dynamically retrieved documents that change per query (average 8,000 tokens), followed by the user query (average 200 tokens). Where should cache_control breakpoints be placed to maximize cache hit rate?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal — a is correct because cache_control breakpoints should be placed at the end of stable, shared prefixes. The 2,000-token static system prompt is identical across all queries; caching it saves 2,000 tokens of processing on every call.
Full explanation below image
Full Explanation
A is correct because cache_control breakpoints should be placed at the end of stable, shared prefixes. The 2,000-token static system prompt is identical across all queries; caching it saves 2,000 tokens of processing on every call. The dynamically retrieved documents change per query, so caching them produces near-zero hit rates and wastes cache write costs. The optimal single breakpoint is at the system-prompt boundary. B is wrong because caching the dynamic documents adds cache write overhead (billed at input token cost) on every miss with negligible savings — unless the same exact document set is retrieved for many identical queries, which is rare in a RAG system. C is wrong because the API supports a limited number of cache breakpoints; placing them at every chunk boundary is not supported and adds complexity without benefit for content that changes per query. D is wrong because partial caching of stable prefixes (even a 2,000-token static system prompt) produces real savings at scale; the static/dynamic distinction is what determines where to break, not an all-or-nothing rule.