A team training a deep feedforward network notices that training loss oscillates wildly and convergence is painfully slow, even after tuning the learning rate. A colleague suggests inserting a normalization layer after each hidden layer's activations. Why would this help?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal: as data flows through a deep net, the distribution of activations at each layer keeps shifting during training — that's called internal covariate shift, and it makes the next layer chase a moving target. Batch normalization fixes that by re-centering and re-scaling each layer's inputs to a stable distribution (roughly zero mean, unit variance) before the next layer sees them. That stability lets you use a higher learning rate and converge faster, with less oscillation. It's not dropout (that randomly zeroes neurons), it doesn't replace your activation function, and it doesn't magically add depth — it just keeps the numbers well-behaved layer to layer.
Full explanation below image
Full Explanation
Batch normalization (BatchNorm) standardizes the inputs to a layer by normalizing each mini-batch to have zero mean and unit variance, then applying a learned scale (gamma) and shift (beta) so the network can still represent the optimal distribution if needed. The core benefit is addressing internal covariate shift: as earlier layers' weights update during training, the distribution of activations feeding later layers keeps changing, forcing those layers to continuously readapt. By normalizing at each layer, BatchNorm keeps this distribution stable, which in practice allows higher learning rates, reduces sensitivity to weight initialization, and speeds up convergence — exactly the oscillating, slow-training symptom described in the question.
The first distractor describes dropout, a separate regularization technique that randomly deactivates a fraction of neurons during each training pass to reduce overfitting; it does not standardize activation distributions and is not about training stability in the same sense. The second distractor is wrong because batch normalization is applied in addition to an activation function (typically before or after it, depending on architecture choice) — it never substitutes for the nonlinearity that gives the network its representational power. The third distractor confuses depth with normalization; BatchNorm layers do add a small number of learnable parameters (gamma and beta per feature) but do not increase the network's representational depth in the sense of adding new transformation capacity comparable to a hidden layer.
As a memory aid: think of BatchNorm as giving every layer a fresh, consistent starting point each pass — like re-leveling a table before each round of a game, so no player has to compensate for a wobble introduced by the previous round.