311 Run:ai inference must stay responsive during Monday morning ticket storms. What should ops configure?
Select an answer to reveal the explanation.
Short Explanation
Keep the Run:ai inference endpoint warm so you avoid making each 311 ticket wait for a cold start during Monday’s surge. If you shut it down Sunday night or send peaks only to CPU, you risk slow responses when demand spikes.
Full Explanation
Keep the Run:ai inference endpoint warm so 311 stays responsive during Monday morning ticket storms. Warm replicas retain loaded model weights and GPU context, avoiding cold-start penalties when civic demand spikes. Autoscaling on queue depth can help meet bursts without keeping a fixed, oversized fleet all week. Configuring every incoming ticket to trigger a fresh cold start puts model-loading time into every request and can be catastrophic under burst. Shutting the endpoint down each Sunday night with no warm pool makes Monday’s first wave pay the full cold-start cost. Routing Monday peaks only to CPU fallback with no GPU replicas may work for tiny models, but typically misses latency SLOs for neural classifiers. Exam caveat: warm capacity still costs GPUs, so balance minReplicas or warm-pool size against the off-peak budget; scale-to-zero suits truly quiet services. Run:ai scheduling delays can still occur if guarantees are undersized for Monday peaks, and warm empty quotas do not help. Sunday maintenance node drains can also unintentionally empty the warm pool. Before Monday opens, confirm warm replica count and GPU allocation, load-test on Friday at Monday-like QPS, and alert on cold-start rate and p95 latency from 07:00–10:00 local time for the 311 classifier.