A dataset is far too large to fit entirely into GPU memory at once. What is the standard, memory-efficient approach for training a neural network on data like this?
Select an answer to reveal the explanation.
Short Explanation and Infographic
This is exactly what mini-batch training was built for. Instead of trying to cram your whole dataset into GPU memory at once — which just isn't going to happen with a large dataset — you load a small chunk, say 32 or 64 samples, run a forward and backward pass on just that chunk, update the weights, then load the next chunk. A DataLoader with workers handles fetching the next batch efficiently, often while the GPU is still busy with the current one. So 'train using mini-batches' is the answer. Deleting most of your data to make it fit is throwing away signal you need, not a real solution. Loading everything into memory at once is precisely the thing you can't do here — that's the whole problem. And turning samples into scalars would destroy the actual information in your data; that's not a memory technique, it's data destruction.
Full explanation below image
Full Explanation
The standard, memory-efficient technique for training on datasets too large to fit entirely into GPU (or even CPU) memory is mini-batch training. Rather than attempting to load the full dataset at once, the data is divided into small batches — commonly ranging from 16 to a few hundred samples depending on hardware and model size — and only one batch is loaded into memory and processed through the network at a time. For each mini-batch, a forward pass computes predictions, a loss is calculated, gradients are computed via backpropagation, and the optimizer updates the weights, after which that batch is released from memory and the next one is loaded. This approach, implemented in practice via constructs like PyTorch's DataLoader (built on a Dataset that lazily reads samples, often from disk, only when requested) or TensorFlow's tf.data pipelines, keeps memory usage bounded regardless of total dataset size, since only a small, fixed-size batch needs to reside in memory at any given moment. Additionally, using multiple worker processes to prefetch and preprocess upcoming batches while the current batch is being processed on the GPU further improves efficiency without increasing peak memory usage.
Permanently deleting most of the training samples to make the dataset fit in memory is not an acceptable or standard solution; it discards potentially valuable information and would typically harm model performance and generalization by starving the model of training signal, especially for tasks where more data usually helps.
Loading the entire dataset into memory once and never touching disk again is precisely what is infeasible when the dataset exceeds available memory; this is the constraint the question describes, not a workaround for it. For datasets that do comfortably fit in memory, this approach can be reasonable, but it is not applicable to the scenario described.
Converting every sample into a scalar value before training would eliminate essentially all the meaningful structure and information in the data (a scalar carries only a single number, whereas real samples like images or feature vectors carry many dimensions of information), making the model unable to learn anything useful; this is not a legitimate memory-management technique.
Mini-batch training is so foundational that virtually all modern deep learning frameworks are built around it by default, and it also has secondary benefits beyond memory efficiency, such as introducing a small amount of noise into gradient estimates that can help escape poor local minima and improve generalization.