What does regularisation do to a model, and how do L1 and L2 differ in their effect?

The penalty makes coefficients cost something

Unregularised fitting minimises error alone, so any coefficient that reduces training error by even a hair is worth setting non-zero, including coefficients that are fitting noise. Regularisation adds a second term to the objective, so the optimiser minimises error + λ × penalty(w). Now every coefficient has a price. A feature only earns a large weight if it reduces error by more than the penalty charges for that weight, and λ sets the exchange rate.

This is why the mechanism is worth stating precisely rather than described as "it stops overfitting". It changes what the fit is optimising, which is why it works at all and also why it is scale-sensitive. If income is measured in pounds and age in years, the coefficient on income is numerically tiny to produce the same effect on the prediction, so a penalty on raw coefficient size charges almost nothing for income and a great deal for age. Standardise the features first, or the penalty is quietly a statement about your units. The intercept is normally excluded from the penalty for the same class of reason: shrinking it towards zero is a claim that the mean outcome is near zero, which is not something you meant to assert.

Where sparsity comes from

L2, the ridge penalty, is the sum of squared coefficients. Its gradient with respect to a coefficient is proportional to that coefficient, so as a coefficient approaches zero the pull towards zero also approaches zero. Coefficients therefore shrink asymptotically and land small but non-zero.

L1, the lasso penalty, is the sum of absolute values. Its gradient is a constant of magnitude λ regardless of how small the coefficient already is. A coefficient whose contribution to error reduction is worth less than λ gets pushed all the way to exactly zero and stays there. Sparsity is not an emergent statistical accident, it is what a constant-magnitude gradient does at the origin.

The difference is easiest to see with duplicated information. Take two features that are identical, x1 = x2, and suppose the best fit requires their coefficients to sum to 2.

Split of the weight	L2 penalty w1² + w2²	L1 penalty abs(w1) + abs(w2)
(1.0, 1.0)	1 + 1 = 2	1 + 1 = 2
(1.5, 0.5)	2.25 + 0.25 = 2.5	1.5 + 0.5 = 2
(2.0, 0.0)	4 + 0 = 4	2 + 0 = 2

Every row fits the data equally well, because the features are interchangeable. L2 strictly prefers the even split, which is why ridge spreads weight across correlated features and produces stable, jointly shrunken coefficients. L1 is exactly indifferent between all three, so the optimiser is free to land on a corner and typically keeps one feature and zeroes the other. Which one it keeps is decided by noise, and that is the interpretive hazard: a lasso model that drops x2 is not evidence that x2 does not matter.

Feature selection is a by-product, not a guarantee

Because L1 produces exact zeros, sweeping λ upwards gives you a nested sequence of ever-smaller feature sets, and the surviving set is often treated as "the important features". Treat it as a compression result rather than a causal one. It tells you a smaller set suffices to predict about as well, which is genuinely useful when you are paying to compute or acquire each feature at serving time. It does not tell you the discarded features are irrelevant, and with correlated inputs the set is unstable across bootstrap resamples of the same data. Elastic net combines both penalties to keep the sparsity while restoring some of L2's stability, so correlated features tend to enter or leave as a group.

Stating the bias-variance effect in numbers you can observe

The textbook phrasing, "regularisation increases bias and reduces variance", is worth nothing in an interview unless you can say what you would see on a screen. Say it as a monotone claim about two curves. As λ rises from zero, the training score falls monotonically, because you are strictly restricting the fit and it can no longer chase every training row. The validation score rises first, because the coefficients it gives up were fitting noise, then peaks, then falls once the penalty starts destroying real signal. The gap between the two curves narrows throughout.

So a concrete diagnosis follows. Training 0.97 and validation 0.78 is a wide gap and λ is too low. Training 0.74 and validation 0.73, both below what a simpler model achieved, is a closed gap at a bad level and λ is too high. Pick λ by cross-validation at the peak, and read the shape of the curve as well as its argmax: a flat plateau means taking the strongest λ inside the plateau, which buys a simpler model for a difference that is inside the noise.

Confusing the penalty with the remedy

The failure that separates a strong answer here is treating regularisation as the fix for overfitting rather than as one instrument with a specific mechanism. It restricts coefficient magnitude, which helps precisely when the model overfits by assigning large weights to weak features. It does nothing about a leaked feature, because a leaked feature reduces error enormously and will happily pay any penalty you set, and nothing about a validation split that put the same customer on both sides. On a boosted ensemble the levers that matter most are structural, max_depth, min_child_weight, the number of rounds and subsampling, with the L1 and L2 terms on leaf weights being secondary.

A penalty term makes each coefficient buy its own weight with error. L2 charges less as the coefficient shrinks, so weights get small; L1 charges the same all the way down, so weights hit zero.