What is the effect of applying L1 (lasso) regularization to a model's weights during training?
Select an answer to reveal the explanation.
Short Explanation
L1, or lasso, regularization adds a penalty to the loss based on the absolute value of each weight, and the practical effect of that penalty is exactly what its nickname suggests — it lassos in the weights, shrinking them toward zero, and for a good chunk of them, it drives them all the way to exactly zero. That gives you a sparser model where some features are effectively dropped out of the equation entirely, since their weight is now literally nothing. That's answer B. It definitely doesn't increase weight magnitude — regularization exists specifically to discourage weights from growing too large, not to encourage bigger ones. Randomly disabling neurons each pass is dropout, a totally different regularization method with a totally different mechanism. And rescaling activations to zero mean and unit variance within a mini-batch is batch normalization, again unrelated to L1's weight-shrinking penalty.
Full Explanation
L1 (lasso) regularization adds a term to the loss function proportional to the sum of the absolute values of the model's weights, scaled by a regularization strength hyperparameter (commonly called lambda). During gradient-based optimization, this penalty exerts a constant-magnitude pull toward zero on every weight, and because the absolute value function is non-differentiable at exactly zero, this pull is strong enough in practice to drive many weights all the way to exactly zero rather than merely shrinking them asymptotically. The practical consequence is a sparser model: some features effectively drop out of the model's decision-making entirely because their associated weight becomes zero, which can act as a form of automatic feature selection and often produces a more interpretable and compact model.
Increasing every weight's magnitude proportionally is incorrect and describes the opposite of what any standard regularization technique does; regularization terms exist specifically to counteract the tendency of unconstrained optimization to grow weights arbitrarily large in pursuit of minimizing training loss, and L1 in particular actively shrinks, never grows, weight magnitudes. Randomly disabling a fraction of neurons on each training pass describes dropout, a distinct regularization technique that operates by zeroing out neuron activations stochastically during training, not by adding any penalty term to the loss based on weight magnitude; L1 regularization has no random component and works purely through the loss function. Rescaling activations within each mini-batch to zero mean and unit variance describes batch normalization, a technique aimed at stabilizing and accelerating training by normalizing layer inputs, which is conceptually unrelated to L1's weight-penalty mechanism and does not by itself function as a sparsity-inducing regularizer.
A useful memory aid distinguishing L1 from L2: L1's absolute-value penalty has sharp corners in its geometry that let optimization land exactly on zero for some weights, producing sparse, feature-selecting models, whereas L2's squared penalty is smooth everywhere and only asymptotically shrinks weights, rarely if ever reaching exactly zero, producing dense models with generally smaller but non-zero weights throughout.