How does stochastic gradient descent (SGD), updating on one example at a time, typically compare to full batch gradient descent?
Select an answer to reveal the explanation.
Short Explanation and Infographic
SGD's superpower is frequency — it updates the weights after every single example instead of waiting to chew through the whole dataset like batch gradient descent does. That means way more updates per epoch, so even though each individual step is based on noisy, single-example information, SGD often gets to a good solution faster in real wall-clock time. That's answer A. It's the opposite of needing the entire dataset in memory upfront — that's batch gradient descent's constraint, not SGD's; SGD needs just one example at a time. SGD's loss curve is famously choppy and noisy, not smooth, because each update is based on a single noisy gradient estimate — smoothness is what full batch gives you. And SGD absolutely can be, and constantly is, combined with adaptive optimizers like Adam and RMSprop; those optimizers were literally built to make SGD-style per-example or per-mini-batch updates less noisy and more effective.
Full explanation below image
Full Explanation
Stochastic gradient descent updates model weights after computing the gradient from a single training example, in contrast to batch gradient descent, which computes the gradient over the entire dataset before performing one update. Because SGD updates so frequently, it can make rapid progress through the loss landscape and often reaches a good solution in less wall-clock time than batch gradient descent, especially on large datasets where a single full-batch pass is expensive. The tradeoff is that each individual update is based on a noisy, high-variance gradient estimate from just one example, so the loss curve for pure SGD is characteristically jagged rather than smooth, though this noise can actually help the optimizer escape shallow local minima.
The first distractor is incorrect because requiring the full dataset to be loaded before any update describes batch gradient descent's memory profile, not SGD's; SGD's defining characteristic is that it needs only one example at a time to compute a gradient and perform an update, making it far lighter on memory. The second distractor is incorrect because SGD's loss curve is noisy and non-monotonic by nature — a smooth, steadily decreasing loss curve is characteristic of batch gradient descent's more stable, averaged gradient estimates, not SGD's single-example updates. The third distractor is incorrect because adaptive optimizers such as Adam and RMSprop are explicitly designed to work with SGD-style or mini-batch-style updates; in fact, they were developed to smooth out and adapt the learning rate for exactly this kind of noisy, frequent-update training, and are commonly described as variants or extensions of SGD rather than incompatible with it.
In modern deep learning practice, pure single-example SGD is rarely used directly; instead, mini-batch gradient descent (processing small batches of examples per update) is the standard compromise, retaining much of SGD's frequent-update advantage and noise-driven ability to escape poor local minima, while reducing variance and enabling efficient vectorized computation on GPU hardware.