What is a key defining feature of the RMSprop optimizer?
Select an answer to reveal the explanation.
Short Explanation
RMSprop's whole trick is keeping a moving average of the squared gradients for each parameter, then dividing the learning rate by the square root of that average before applying the update. That's answer C. The effect: parameters that have been getting big, consistent gradients get their effective step size shrunk down, and parameters getting small gradients keep taking relatively larger steps. It's not about updating only once per epoch — RMSprop updates every batch, same as other mini-batch optimizers. It doesn't permanently freeze anything either — that moving average keeps adapting throughout training, it's dynamic, not a one-time freeze. And it definitely doesn't touch or replace your loss function; the loss function is what you're trying to minimize, RMSprop is just the method for adjusting weights based on the gradients of that loss. RMSprop is all about per-parameter, adaptive step sizing.
Full Explanation
RMSprop (Root Mean Square Propagation) adapts the learning rate for each parameter individually by dividing the raw gradient at each step by the square root of an exponentially decaying moving average of that parameter's squared gradients. Concretely, it maintains a running average v of squared gradients (v = decay_rate v + (1 - decay_rate) gradient^2) and then updates weights using something like weight -= (learning_rate / sqrt(v + epsilon)) * gradient. Parameters that have consistently large gradients accumulate a large v and therefore receive a smaller effective step, while parameters with small or infrequent gradients keep a relatively larger effective step, which helps stabilize training especially in the presence of noisy or sparse gradients and helps overcome the aggressive learning-rate decay problem seen in Adagrad.
The first distractor is incorrect because RMSprop, like other mini-batch-friendly optimizers, updates weights after each processed batch (or even each example), not once per full epoch — waiting a full epoch to update would defeat the purpose of using mini-batches for frequent progress. The second distractor is incorrect because RMSprop's adaptation is continuous and dynamic throughout training via the moving average; it never permanently freezes the learning rate for any parameter, and the effective step size for every parameter can rise or fall again later in training as gradient statistics change. The third distractor is incorrect because RMSprop operates entirely at the optimizer level, adjusting how gradients are used to update weights — it has no interaction with or influence over the loss function itself, which is defined independently by the task (e.g., cross-entropy or MSE).
RMSprop was introduced (informally, by Geoffrey Hinton in lecture notes) as a fix for Adagrad's tendency to shrink learning rates too aggressively over long training runs, since Adagrad accumulates squared gradients over the entire training history without decay. By using an exponentially decaying average instead of an ever-growing sum, RMSprop lets the effective learning rate recover if gradients become smaller later in training, making it well suited to non-stationary objectives such as those found in recurrent neural networks. RMSprop's core idea — a per-parameter adaptive learning rate driven by squared-gradient statistics — was later combined with momentum (an average of the raw gradients) to produce the Adam optimizer.