How does the Adam optimizer relate to RMSprop in terms of what it borrows from each technique?
Select an answer to reveal the explanation.
Short Explanation
Think of Adam as the optimizer that took the best idea from two earlier approaches and mashed them together. From momentum-based SGD, it borrows the idea of keeping a running, decaying average of the raw gradients — this smooths out the direction of travel and helps push through small bumps and noise. From RMSprop, it borrows the idea of keeping a running, decaying average of the squared gradients — this lets it scale the step size per parameter, taking bigger steps where gradients have been small and smaller steps where they've been large. Put those two moving averages together, do some bias correction early in training, and you get Adam. That's answer C. The other options are all wrong: Adam doesn't drop RMSprop's squared-gradient tracking, it keeps it; it doesn't remove epsilon, that's still there to prevent division by zero; and it definitely adds its own gradient-averaging term on top of the RMSprop-style piece, not just squared gradients alone.
Full Explanation
Adam, short for Adaptive Moment Estimation, maintains two separate exponentially decaying moving averages during training: the first moment estimate, which is a decaying average of the raw gradients (conceptually the same idea as classical momentum, smoothing the direction of updates over time), and the second moment estimate, which is a decaying average of the squared gradients (the same core mechanism RMSprop uses to adapt the effective learning rate per parameter). Adam then uses both moments together, after applying bias-correction terms to counteract the fact that the moving averages start biased toward zero early in training, to compute each parameter's update as roughly the first-moment estimate divided by the square root of the second-moment estimate plus a small epsilon for numerical stability.
The first distractor is incorrect because Adam explicitly keeps and relies on the squared-gradient moving average mechanism from RMSprop; it does not discard it in favor of a fixed schedule, and in fact this adaptive per-parameter scaling is one of Adam's defining features. The second distractor is incorrect because epsilon remains a necessary component of Adam's update rule, preventing division by a near-zero denominator when the squared-gradient average is very small, exactly as in RMSprop; Adam does not remove it. The fourth distractor is incorrect because it describes plain RMSprop, not Adam; Adam's defining innovation over RMSprop specifically is the addition of the first-moment (momentum-style) gradient average, which RMSprop alone does not maintain.
A useful memory aid: RMSprop answers 'how big should this step be,' scaling by recent squared-gradient magnitude, while momentum answers 'which direction should this step go,' smoothing over recent raw gradients. Adam answers both questions simultaneously by tracking both moving averages at once, which is why it's often described as combining the direction-smoothing benefit of momentum with the per-parameter step-scaling benefit of RMSprop into a single, widely-used optimizer.