A transit authority trains its ridership-demand model on GPU instances but only needs to serve predictions afterward. The platform team is picking compute for each phase separately. Which approach best matches cost to workload?
Select an answer to reveal the explanation.
Short Explanation
Training and serving are two very different jobs wearing the same model's name — training is the marathon that benefits from a team of GPUs crunching gradients, while serving one rider's prediction is more like answering a single question, which a right-sized CPU instance can often do just fine. Matching each phase's compute to its actual workload is what keeps the transit authority from paying marathon prices for a quick answer.
Full Explanation
Training involves repeated forward and backward passes over large batches of data, work that benefits heavily from GPU parallelism and finishes far faster than it would on CPU alone; serving, by contrast, is typically single-example or small-batch inference where the per-request compute is much lighter, so CPU-based instances often deliver adequate latency at a lower cost per hour. Assuming the training compute is automatically right for serving ignores that these are separate cost-and-latency profiles — many production endpoints run comfortably on CPU while their training jobs ran on GPU. Training on CPU-only instances to save money can backfire: a training job that takes many times longer to converge often ends up costing more in wall-clock compute time than a shorter GPU-accelerated run would have. Over-provisioning the largest GPU instance for every serving request ignores the endpoint's actual latency requirement and per-request compute cost, paying for capacity the workload doesn't need. Scope caveat: if the serving workload later grows to require GPU acceleration (for example, a much larger model), that decision should be revisited against updated latency and cost data rather than assumed up front. Operational check: benchmark inference latency on a CPU instance against the transit authority's latency requirement before committing to the CPU-based serving choice.