A training team notices their deep network memorizes the training set and fails to generalize. They enable weight decay (L2 regularization). What is the primary purpose of this technique?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal: your network's weights can get huge and start memorizing every weird detail in the training data. Weight decay (L2) says, "Pay a little extra in the loss for big weights," so the model prefers simpler solutions. Think of it like a fine for stuffing your backpack—you still carry what matters, but you stop packing junk. Don't confuse this with jacking up the learning rate, chopping layers, or normalizing inputs. Those are different knobs. On the exam, if you see "penalty for large weights" or "prevent overfitting," pick L2 / weight decay. Got it? Sweet—remember: bigger weights get taxed so generalization wins.
Full explanation below image
Full Explanation
Weight decay, commonly implemented as L2 regularization, modifies the training objective so that model complexity is controlled alongside fit to the data. In practice, the loss becomes the original task loss plus a coefficient λ times the sum of squared parameter values. Gradient updates then pull weights toward smaller magnitudes unless those magnitudes are clearly justified by reduced error. Smaller, better-distributed weights tend to yield smoother decision boundaries and less sensitivity to idiosyncrasies in the training set, which is exactly what teams need when training loss falls while held-out performance stalls or worsens.
This is not the same as increasing the learning rate. The learning rate scales how large each optimizer step is; weight decay changes the objective itself. Confusing the two leads to unstable training without addressing capacity control. Likewise, shrinking the architecture by removing layers is a hard capacity cut that may underfit or discard useful hierarchical features. Regularization is a softer, continuous control that keeps the architecture intact while discouraging unnecessary complexity. Input normalization—mapping features to a consistent range—improves conditioning and optimizer behavior, but it does not penalize large weights after they form.
From a practical standpoint, λ is a hyperparameter: too small and overfitting continues; too large and the model underfits. Weight decay pairs well with early stopping, dropout, data augmentation, and careful validation monitoring. In optimizers such as AdamW, decoupled weight decay applies the shrinkage more cleanly than naive L2-in-loss formulations with adaptive gradients. Memory aid: L2 = "large weights get a squared tax." When a question asks for the purpose of weight decay in neural networks, the answer centers on penalizing large weights to combat overfitting, not on learning rates, layer counts, or input rescaling.