You're architecting an AI infrastructure that needs to support both model training and model inference, and your boss asks, "Do training and inference have the same infrastructure requirements?" What's the honest answer about the architectural differences?
Select an answer to reveal the explanation.
Short Explanation
Training and inference are fundamentally different workloads with different optimization goals. Training is throughput-centric (process data as fast as possible), inference is latency-centric (respond quickly to requests). They can share infrastructure, but you need to understand the tradeoffs.
Full Explanation
Training and inference are different enough that they benefit from different infrastructure strategies. In training, you're processing massive batches of data, doing forward passes, backprop, and weight updates. Throughput is king — how much data can you process per second? You want GPUs with large memory pools, high bandwidth, and massive parallelism. Batch sizes are large (256, 512, 1024) to amortize communication overhead. In inference, you typically process individual requests or small batches. Latency matters: "This customer query arrived; how fast can I return a prediction?" You might tolerate slightly lower throughput if you can reduce latency. Optimal batch size might be 1 or 8. They can share infrastructure (you can run both on H100 clusters), but you're making a tradeoff. If you optimize for throughput training with large batches, inference might experience higher latency. If you configure for low-latency inference, training might run slower. In practice, large organizations often have separate infrastructure: training clusters optimized for throughput (multiple H100s per job, large batches) and inference clusters optimized for latency (lower-power GPUs, small batch serving). The question tests whether you grasp this fundamental architectural distinction.