A city permitting-chatbot team must serve sub-second interactive responses to residents typing permit questions throughout business hours, with continuous, predictable traffic. Which inference option best fits this requirement?
Select an answer to reveal the explanation.
Short Explanation
Residents typing questions during business hours want an answer right now, not a callback later. A real-time endpoint that's kept warm delivers that consistent, sub-second response because it's already provisioned and waiting, not spinning up on demand. That's the tradeoff you're paying for: always-on compute for always-on responsiveness.
Full Explanation
A real-time SageMaker endpoint keeps model instances warm and provisioned, so it can return predictions with low, consistent latency the moment a request arrives — exactly what an interactive chat experience during business hours needs. Batch transform processes a full dataset as one offline job with no interactive request/response cycle, so it structurally cannot answer a resident mid-conversation; it fits reporting workloads, not chat. Asynchronous inference is built for requests that can tolerate a delay while a queue drains, which breaks the expectation of an immediate reply that a chatbot's UX depends on. A serverless endpoint sounds appealing for cost, but it scales down to zero between bursts and pays a cold-start penalty on the next request, which directly conflicts with a sub-second latency requirement during a steady, predictable business-hours load — the very traffic pattern serverless is a worse fit for than a provisioned real-time endpoint. Scope note: real-time endpoints cost more per idle hour than serverless, so this tradeoff only pays off when traffic is steady enough to justify staying warm. Operational check: monitor p99 latency against the sub-second target and confirm auto scaling keeps enough instances warm through peak permitting hours.