During the training of a deep convolutional neural network, you notice that as weights in early layers change, the distribution of inputs to later layers shifts dramatically, causing training to be slow and unstable. Which technique should you implement to standardize the inputs of each layer, thereby stabilizing and accelerating the training process?
Select an answer to reveal the explanation.
Short Explanation
Here's the deal: when early layers keep changing, later layers chase moving targets. Batch normalization fixes this by standardizing each layer's inputs per mini-batch, stabilizing training and letting you crank up the learning rate. Dropout, learning rate decay, and gradient clipping don't standardize layer inputs.
Full Explanation
The correct technique is Batch Normalization. During training, the distribution of each layer's inputs changes as the parameters of the preceding layers change. This phenomenon, known as internal covariate shift, requires subsequent layers to continuously adapt to new distributions, slowing down training. Batch normalization addresses this by normalizing the activations of a layer across each mini-batch. Specifically, it scales and shifts the activations to have a mean of zero and a variance of one, using learned parameters (gamma and beta) to preserve the network's representational power. This stabilization allows for higher learning rates, reduces sensitivity to initialization, and acts as a mild regularizer, significantly accelerating training. Dropout is incorrect because it is a regularization technique that randomly deactivates neurons during training to prevent overfitting, rather than normalizing layer inputs. Learning Rate Decay is incorrect because it gradually reduces the learning rate over time to help the model converge to a local minimum, but does not standardize layer inputs. Gradient Clipping is incorrect because it caps gradients at a maximum threshold to prevent exploding gradients, rather than standardizing activations within layers.