What is a key distinction between stochastic gradient descent (SGD) and the Adam optimizer?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Okay, optimizer face-off. Vanilla SGD is straightforward: grab a gradient, step downhill with one learning rate (maybe with a schedule or momentum bolted on). Adam is the adaptive gadget—it keeps running averages of the gradient and the squared gradient so each weight gets its own effective step size. Think of SGD as one thermostat for the whole building; Adam as per-room climate control. Trap alert: Adam is not "unsupervised learning," and it's not cheaper than SGD—it carries more state. And SGD absolutely runs on big data with mini-batches. On the job and the exam, if they ask main difference, say adaptive per-parameter rates versus a single shared rate. Pick what fits your training dynamics, but know that contrast cold.
Full explanation below image
Full Explanation
Stochastic gradient descent updates parameters in the opposite direction of the gradient of the loss computed on a mini-batch (or single example). In its classic form, a single learning rate multiplies that gradient for every parameter, possibly with momentum or a global schedule. That simplicity makes SGD memory-light and well understood, but a rate that works for one subset of weights may be too aggressive or too timid for others, especially when gradient scales differ across layers. Despite that limitation, carefully tuned SGD with momentum remains competitive on many large vision and language training runs.
Adam (Adaptive Moment Estimation) maintains exponential moving averages of the first moment (mean) and second moment (uncentered variance) of gradients. Those statistics yield per-parameter adaptive step sizes, with bias correction in the original formulation. The practical effect is faster progress on many deep learning problems with less manual learning-rate fiddling, at the cost of extra memory for moment buffers and slightly more compute per step. Variants and debates exist—some tasks still favor tuned SGD with momentum for final generalization—but the definitional difference for exams remains adaptive per-parameter rates versus a shared base rate. Knowing that contrast is more important than memorizing every hyperparameter default.
Incorrect options reverse cost comparisons, misclassify Adam as unsupervised learning, or invent a small-data-only rule for SGD. Adam is an optimizer for gradient-based training, used with labeled, unlabeled, or self-supervised losses alike. SGD and mini-batch SGD are workhorses of large-scale training, not niche methods for tiny tables. Compute cost usually favors plain SGD per step; adaptivity is Adam's selling point, not lower FLOPs.
Best practice: understand default betas and epsilon in Adam, watch for issues with sparse updates or sudden distribution shift, and remember that learning-rate warmup and decay still matter. Memory aid: SGD equals one shared dial; Adam equals many adaptive dials driven by gradient moments. Choose based on training stability needs, hardware memory, and the generalization behavior you observe on a held-out set.