Which optimizer gives each parameter its own learning rate but has a well-known drawback of that per-parameter rate shrinking too aggressively over long training runs?
Select an answer to reveal the explanation.
Short Explanation
Adagrad's whole idea is giving every single parameter its own personalized learning rate, based on how much that parameter's gradient has accumulated over training. Sounds great in theory — frequently-updated parameters get smaller steps, rarely-updated ones get bigger steps. The catch: Adagrad keeps an ever-growing SUM of squared gradients since the beginning of training, and that sum only ever gets bigger, never resets or decays. So over a long training run, that denominator grows huge and the effective learning rate for every parameter eventually shrinks toward zero, effectively stalling training. That's answer A, and it's exactly the problem RMSprop and Adam were built to fix by using a decaying moving average instead of an endless running sum. Plain SGD with momentum, vanilla batch gradient descent, and Polyak averaging don't have per-parameter adaptive rates at all, so this specific shrinking-rate drawback doesn't apply to them.
Full Explanation
Adagrad (Adaptive Gradient Algorithm) adapts the learning rate individually for each parameter by dividing the global learning rate by the square root of the sum of all squared gradients that parameter has received since the start of training: weight -= (learning_rate / sqrt(G + epsilon)) * gradient, where G accumulates gradient^2 values across every training step seen so far. This design gives infrequently updated parameters (such as those tied to rare features) relatively larger effective steps and frequently updated parameters relatively smaller ones, which is valuable for sparse data problems like certain NLP tasks. However, because G is a monotonically increasing sum with no decay mechanism, it grows without bound as training continues, causing the effective learning rate for every parameter to shrink continuously and eventually approach zero, which can cause training to stall prematurely well before an optimal solution is reached.
Plain SGD with momentum uses a single global learning rate (typically fixed or governed by an external decay schedule) combined with a velocity term that accumulates a fraction of past gradients to smooth the update direction; it does not maintain any per-parameter adaptive rate, so it cannot exhibit Adagrad's specific shrinking-denominator problem. Vanilla batch gradient descent likewise uses one global learning rate applied uniformly to all parameters and has no per-parameter adaptation mechanism at all. Polyak averaging is a technique that averages the parameter values (not gradients) across recent iterations to produce a smoother final estimate of the weights; it is not an adaptive per-parameter learning-rate method and does not exhibit Adagrad's issue.
The fix for Adagrad's aggressive decay problem is to replace the ever-growing sum of squared gradients with an exponentially decaying moving average, which is precisely what RMSprop does, allowing the effective learning rate to recover if recent gradients shrink rather than being permanently suppressed by ancient gradient history. Adam later combined this decaying-average-of-squared-gradients idea from RMSprop with a decaying average of the raw gradients (momentum), producing an optimizer that adapts per-parameter learning rates without Adagrad's premature-stalling drawback.