A data center manager is optimizing infrastructure resources for a machine learning cluster. How should the key performance indicators (KPIs) differ when evaluating the performance of the model training phase versus the model deployment (inference serving) phase?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal: training a model and serving a model are two completely different beasts. When you're training, you've got massive datasets and you're running GPUs at 100% capacity for days or weeks. You care about how fast that model converges and how efficiently you're using those expensive cluster resources. But once that model is trained and you push it out to the real world (inference), nobody cares about epoch time! Now, it's all about how fast the system can respond to a user request (latency), how many queries it can handle at once (throughput), and how much each request costs you. Keep these two phases distinct in your mind!
Full explanation below image
Full Explanation
The machine learning lifecycle is divided into two distinct operational phases: training (development) and inference (serving). Each phase places different demands on the hardware infrastructure, requiring distinct performance metrics to optimize operations.
Let's look at the metrics that define success in each phase: - Model Training (Correct Answer B): Training is an offline, compute-heavy, iterative process where a model learns from a dataset. The goal is to produce a high-accuracy model as quickly and cheaply as possible. The primary KPIs include: - Time to Convergence: The total time required for the model to reach a target accuracy level. - Hardware Utilization: Maximizing GPU engine and Tensor Core utilization to ensure expensive assets are not sitting idle. - Throughput (Samples per Second): How quickly training data can be fed into the model. - Model Serving (Correct Answer B): Serving is an online, transaction-oriented process where the trained model processes live user queries. The key metrics focus on user experience and cost-efficiency: - Inference Latency: The time it takes to return a prediction (typically measured as p99 or p95 response times). - Throughput (Queries per Second / QPS): The volume of requests the serving infrastructure can handle simultaneously. - Cost per Inference: Minimizing the compute cost required to serve each prediction.
The distractors are incorrect because they mix up these phases (A), claim they share identical metrics (C), or focus on overly narrow metrics like cold-start times or data ingest alone (D).