A public-health analytics team runs large nightly batch-inference jobs and wants to cut compute spend without changing the model or missing the overnight turnaround window. What should the team do?
Select an answer to reveal the explanation.
Short Explanation
Batch work is a slow freight train, not a sports car — it doesn't need the same expensive engine a real-time endpoint does. Match the instance family to the job's actual shape and you keep the same overnight schedule for a lot less money. Paying for latency you don't need is the waste here.
Full Explanation
Real-time endpoints are provisioned on low-latency, often GPU-accelerated instance families because a resident-facing chatbot can't tolerate multi-second delays, but a nightly batch job has no such constraint — its only deadline is finishing before the next business day. Matching the batch job to a throughput-oriented, typically CPU-based instance family lowers the per-hour cost of the compute doing the same total amount of work, without touching the model or missing the overnight window, because batch jobs process records asynchronously rather than serving live requests. Running the batch job on the same family used for real-time chat pays for latency characteristics the batch workload never uses. Splitting the job into many small real-time endpoints multiplies the number of always-on, latency-optimized resources instead of consolidating throughput work into fewer, cheaper instances — this moves cost in the wrong direction. Retraining on a smaller dataset changes model quality and is an entirely separate lever from instance selection; it doesn't address the stated constraint that the model must stay the same. Scope caveat: instance family choice interacts with parallelism settings, so re-validate throughput after switching to make sure the smaller instance still finishes within the overnight window. Operational check: compare cost-per-record before and after the switch alongside job completion time.