What does L2 regularization add to a model's loss function?
Select an answer to reveal the explanation.
Short Explanation
L2 regularization tacks an extra term onto your loss function based on the squared value of every weight in the network. Square them, sum them up, multiply by a small coefficient (often called lambda), and add that to the loss. That's the correct answer. Because the penalty grows with the square of the weight, big weights get punished disproportionately hard, which nudges the optimizer to keep weights small and spread out rather than letting a handful of weights blow up. Using absolute values instead of squares describes L1 regularization — related idea, different math, different behavior. A random mask zeroing out weights each step is dropout, not L2 at all. And a term that rewards bigger weights would be the exact opposite of regularization — regularization exists specifically to discourage large weights, not encourage them.
Full Explanation
L2 regularization, also known as weight decay or ridge regularization, adds a penalty term to the loss function equal to a regularization coefficient (lambda) multiplied by the sum of the squares of all the model's weights: Loss_total = Loss_original + lambda * sum(w_i^2). Because the penalty scales with the square of each weight's magnitude, the optimizer is pushed to keep weights small in aggregate during training, since large weights would otherwise incur a disproportionately large penalty. This tends to produce smoother, more evenly distributed weight values and helps reduce overfitting by discouraging the model from relying too heavily on any single feature or connection.
Using absolute values instead of squared values describes L1 regularization, which has a qualitatively different effect: because the penalty's derivative is constant (not proportional to the weight's magnitude), L1 regularization can push some weights all the way to exactly zero, producing sparse models — a property L2 generally does not share, since L2's gradient shrinks as the weight approaches zero, causing weights to shrink toward but rarely reach exactly zero. Randomly zeroing a fraction of neuron activations during each forward pass at training time describes dropout, a different regularization technique entirely that does not add any penalty term to the loss function based on weight values. A term that rewards large weights or encourages larger weight magnitudes would increase overfitting risk, the opposite of regularization's purpose — regularization techniques exist specifically to constrain model complexity and reduce overfitting by discouraging large weights.
L2 regularization is mathematically equivalent, under standard SGD, to weight decay, where each update subtracts a small fraction of the current weight value in addition to the gradient-based update. It is one of the most widely used regularization techniques in deep learning because it is simple to implement, differentiable everywhere, and effective at controlling model complexity without eliminating any particular weight or feature outright, in contrast to L1's tendency toward sparsity.