While a high-end server CPU has a few dozen powerful cores, a modern GPU has thousands of smaller cores. However, raw core count isn't the only bottleneck when training deep learning models. What critical memory-related architectural characteristic allows GPUs to feed data to these thousands of cores fast enough to sustain high computational throughput?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Let's dive in here. Think of a CPU like a Ferrari—it's super fast, and it can move a few passengers down the highway at high speed. A GPU is like a massive train—it moves slower per core, but it carries thousands of passengers at once. But if you have a train that holds thousands of people, you need a massive train station door so they can all get on and off at the same time. That "door" is memory bandwidth. While a CPU gets bottlenecked transferring data at around 50 to 100 GB/s, a modern GPU uses High Bandwidth Memory (HBM) to blast data at up to 7.8 terabytes per second! Without that massive bandwidth, those thousands of GPU cores would just sit around starving for data.
Full explanation below image
Full Explanation
Deep learning workloads are highly parallelizable but extremely data-intensive. Training a neural network involves performing billions of matrix multiplications, which require constantly fetching weights, activations, and input data from memory. If the processor cannot fetch data quickly enough, the execution cores will experience "starvation" and sit idle, regardless of how many cores are available. To address this bottleneck, GPUs are designed with highly optimized memory architectures, such as High Bandwidth Memory (HBM3/HBM3e) or Graphics Double Data Rate (GDDR) memory, which offer memory bandwidths of several terabytes per second (e.g., up to 7.8 TB/s on NVIDIA H200/B200). In comparison, typical enterprise CPUs utilizing standard DDR5 memory channels yield a bandwidth of only around 50 to 100 GB/s.
Let's look at why the other options are incorrect: Option A is incorrect because GPU cores have significantly smaller cache sizes per core compared to CPUs. CPUs devote a large portion of their die area to massive cache hierarchies (L1, L2, L3) to minimize latency for sequential processing. GPUs, on the other hand, rely on massive multi-threading and parallelism to hide memory access latency rather than large per-core caches. Option C is incorrect because GPU memory architectures actually have higher access latency than CPU memory architectures. The latency is compensated for by the GPU's ability to switch execution context to other threads while waiting for memory fetches to complete. Option D is incorrect because GPUs do not store entire large-scale training datasets directly on the silicon die; they rely on on-board VRAM (like HBM or GDDR) and transfer data from system RAM or storage over the PCIe bus as needed during training.