A team trains a model on a dataset with tens of millions of examples using standard batch gradient descent. What is the main drawback they will run into?
Select an answer to reveal the explanation.
Short Explanation
Here's the deal with batch gradient descent: to take just one single step, it has to run through your ENTIRE dataset first, compute the gradient over all of it, and only then update the weights. With tens of millions of examples, that means every step is slow and you need enough memory to hold — or at least process — the whole dataset at once. That's the real drawback, and it's answer C. It's not that batch GD converges to a worse minimum — actually its gradient estimate is more stable than SGD's. It doesn't require validation data, that's a separate concept for tuning and overfitting checks. And it works fine with any loss function, not just MSE. The tradeoff is purely computational: stability and accuracy of the gradient versus speed and memory footprint.
Full Explanation
Batch gradient descent computes the gradient of the loss function using the entire training set before performing a single parameter update. On huge datasets, this means every iteration requires a full pass through millions of examples, making each step computationally expensive and slow, and it often demands holding large amounts of data in memory (or repeatedly streaming it from disk), which becomes impractical at scale. This is why batch gradient descent is rarely used in its pure form for large-scale deep learning; mini-batch gradient descent is used instead to get frequent, cheaper updates.
The first distractor is incorrect because batch gradient descent's gradient estimate is actually more accurate and its convergence path smoother than SGD's — the tradeoff is speed, not final solution quality. The second distractor is incorrect because gradient descent methods only require labeled training data (with a loss function to compare predictions against ground truth); validation data is a separate holdout set used for tuning hyperparameters and monitoring generalization, not for computing training gradients. The third distractor is incorrect because batch gradient descent, like all gradient-based optimization, works with any differentiable loss function — cross-entropy, MSE, hinge loss, and others — it is not restricted to MSE.
The practical fix for the scalability problem is mini-batch gradient descent, which splits the dataset into smaller batches (commonly 32, 64, 128, or 256 examples) and updates the weights after each batch. This gives many more updates per epoch, uses a fraction of the memory, and still benefits from vectorized computation on GPUs, striking a balance between the noisy-but-fast updates of pure stochastic gradient descent and the stable-but-slow updates of full-batch gradient descent.