An architect is designing a prompt caching strategy for a system with a 3,000-token system prompt. The system processes 5,000 requests per day but has a quiet overnight period of 6 hours with no traffic. What is the implication for cache hit rates?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal — anthropic's prompt caching has a TTL of approximately 5 minutes of inactivity — but crucially, each cache hit resets the TTL. For a system processing 5,000 requests/day, during peak hours the cache will remain warm because requests arrive faster than the TTL expires.
Full explanation below image
Full Explanation
Anthropic's prompt caching has a TTL of approximately 5 minutes of inactivity — but crucially, each cache hit resets the TTL. For a system processing 5,000 requests/day, during peak hours the cache will remain warm because requests arrive faster than the TTL expires. However, during the 6-hour overnight gap, the cache will expire. The first request after the gap is a cache miss at full price and rebuilds the cache. Architects must factor in cache miss rates during traffic lulls when calculating expected cost savings. Option A (24-hour expiry) is incorrect. Option B (permanent cache) is incorrect — cache entries do expire and must be actively maintained by ongoing traffic. Option D (1-hour expiry) is also incorrect; the actual TTL is approximately 5 minutes of inactivity.