A customer service platform uses Claude with a 12,000-token system prompt containing comprehensive product knowledge, tone guidelines, and escalation rules. Response latency at P95 is 8.2 seconds, which is above the 5-second SLA. The team identifies that 70% of the system prompt is product catalog data that changes monthly. They're evaluating: (A) prompt caching on the full system prompt, (B) moving product catalog to RAG and shrinking the system prompt, (C) splitting into a 2,000-token core system prompt plus dynamic context injection per query. Which optimization strategy should be prioritized FIRST, and why?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal — the two-phase approach (caching first, then RAG) is architecturally correct given the constraints. Prompt caching is a tactical, low-effort change that can be implemented immediately — it reduces effective input token processing for the static 12,000-token system prompt, with cache hits saving approximately 90% of input token costs and meaningfully reducing TTFT.
Full explanation below image
Full Explanation
The two-phase approach (caching first, then RAG) is architecturally correct given the constraints. Prompt caching is a tactical, low-effort change that can be implemented immediately — it reduces effective input token processing for the static 12,000-token system prompt, with cache hits saving approximately 90% of input token costs and meaningfully reducing TTFT. However, caching alone may not move P95 from 8.2s to under 5s if the output generation time (not input processing) is the bottleneck. RAG migration addresses the architecture more fundamentally — a 2,000-token system prompt with dynamically retrieved relevant product context also improves response relevance. The two-phase approach respects engineering reality: caching is a one-day implementation, RAG migration is a multi-week project. Option A (caching only) is correct as a first step but may be insufficient alone. Option B (RAG first) is strategically sound but delays any latency improvement by weeks while engineering builds the retrieval system. Option C (core + dynamic injection without RAG) is actually a stepping stone architecture toward RAG — it's not a complete solution and still requires the same 8,400 product tokens in context per query if injected dynamically.