Your team is deploying a machine learning model with hundreds of input features. To reduce model complexity and make it more interpretable, you want to perform feature selection directly during training by forcing some weights to zero. Which regularization approach should you choose, and how does its mathematical penalty differ from the other main approach?
Select an answer to reveal the explanation.
Short Explanation
Here's the deal: when you're dealing with overfitting, regularization is your best friend. But L1 and L2 don't behave the same way under the hood. Think of L1—also known as Lasso—like a strict editor who isn't afraid to cross out words entirely. It adds an absolute value penalty to the loss function, which actually drives less useful feature weights all the way to zero. This is awesome because it acts as built-in feature selection. L2, or Ridge regularization, is more like a gentle volume knob. It adds a squared penalty, shrinking those weights down so they don't dominate, but it never actually turns them off to absolute zero. Pay close attention here, because this difference is a favorite exam topic and will save your skin in production when you need to clean up a messy, high-dimensional dataset!
Full Explanation
Regularization is a fundamental technique used in machine learning to prevent overfitting by penalizing large weights in the model's objective function. The two primary types of regularization are L1 (Lasso) and L2 (Ridge) regularization, which differ in their penalty terms and their effect on model parameters. L1 regularization adds a penalty proportional to the sum of the absolute values of the weights (known as the L1 norm). Mathematically, this creates a sharp, diamond-shaped constraint region in the parameter space. Because the optimal solution often hits the corners of this constraint region, L1 regularization tends to yield sparse models where many feature weights are driven to exactly zero. Consequently, L1 acts as an intrinsic feature selection mechanism, making the final model more interpretable and computationally efficient during inference. In contrast, L2 regularization adds a penalty proportional to the sum of the squared values of the weights (known as the L2 norm or weight decay). This creates a spherical constraint region. The optimal solution is pulled toward the origin but rarely lands exactly on any axis, meaning the weights are shrunk toward zero but remain non-zero. L2 regularization is highly effective when all features contribute small amounts of predictive power and we want to prevent any single feature from dominating the model's predictions. Therefore, the key distinction is that L1 regularization can drive weights to exactly zero (resulting in feature selection), while L2 regularization only shrinks weights toward zero asymptotically, retaining all features in the model.