A newcomer is picking an optimizer for a fresh image classification project and wants a solid default that adapts its step size per parameter using running averages of both the gradients and their squares. Which optimizer are they describing?
Select an answer to reveal the explanation.
Short Explanation
This one's describing Adam almost word for word, even without naming it. Adam keeps two running averages as it trains: a decaying average of past gradients (the momentum piece, keeping it moving smoothly) and a decaying average of the squared gradients (the adaptive per-parameter step-size piece, borrowed from RMSprop). Combine those and you get an optimizer that adjusts its own learning rate for every parameter while still cruising smoothly instead of jittering around. That's why Adam is the go-to default for so many projects — it just works out of the box. Plain SGD with a fixed rate doesn't adapt anything, same step size forever. Vanilla batch gradient descent is the same story, just over the whole dataset. And Polyak averaging is different — it averages the weights themselves after training, not gradients during it.
Full Explanation
Adam (Adaptive Moment Estimation) maintains two exponentially decaying moving averages during training: the first moment, which is a running average of the raw gradients and functions like momentum by smoothing out the update direction, and the second moment, which is a running average of the squared gradients and is used to scale the learning rate individually for each parameter, similar to RMSprop. After applying bias-correction terms to account for the fact that both averages start at zero, Adam divides the first moment by the square root of the second moment (plus a small epsilon for numerical stability) to compute each parameter's update. This combination gives Adam per-parameter adaptive learning rates plus momentum-driven smoothing, which is why it converges quickly and reliably across a wide range of architectures and datasets without extensive manual tuning, making it a common default choice.
Plain stochastic gradient descent with a fixed learning rate is incorrect because it applies one single, unchanging step size to every parameter and every update, with no mechanism for tracking gradient history or adapting per parameter. Vanilla batch gradient descent is incorrect for the same core reason: it also uses one global learning rate uniformly across all parameters, the only difference from SGD being that it computes the gradient over the entire dataset rather than one example at a time, which has nothing to do with adaptive moment estimation. Polyak averaging is incorrect because it is a post-hoc technique that averages a model's parameter values across recent training iterations to produce a smoother final weight estimate; it operates on the weights themselves rather than tracking moving averages of gradients or squared gradients during the optimization process, so it does not provide per-parameter adaptive step sizes.
A useful memory aid: Adam essentially fuses momentum (average of gradients, the 'first moment') with RMSprop (average of squared gradients, the 'second moment'), which is precisely where its name comes from and why it tends to be the sensible first optimizer to reach for on a new project before considering more specialized alternatives.