During the training of a deep neural network, the distribution of inputs to each internal layer changes constantly as the weights of the preceding layers are updated. To stabilize and accelerate the training process, you decide to implement Batch Normalization. How does Batch Normalization achieve this goal?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Think of it like this: imagine trying to coordinate a group dance, but the floor beneath everyone is constantly shifting and sliding. You'd spend all your energy just trying to stand upright instead of learning the moves! That's exactly what happens to the deep layers of a neural network. As the early layers update their weights, the range of inputs flowing into the deep layers changes wildly. This is called internal covariate shift. Batch normalization acts like a stabilizer. It steps in at each layer and normalizes the inputs for each mini-batch so they have a consistent mean and variance. This keeps the data stable, which means you can use higher learning rates and your network trains much, much faster. Plus, it has a nice side effect of acting like a mild regularizer. Pretty cool, right?
Full explanation below image
Full Explanation
Batch Normalization (BatchNorm) is a technique designed to address the challenges of training deep neural networks. As a network trains, the parameters of the prior layers change, which shifts the distribution of inputs to the subsequent layers. This phenomenon is known as internal covariate shift, and it forces layers to constantly adapt to new distributions, slowing down convergence. Batch Normalization resolves this by standardizing the inputs to each activation function within a mini-batch. For a given mini-batch, BatchNorm calculates the mean and variance of the activations, subtracts the mean, and divides by the standard deviation. It then applies two learnable parameters—gamma (scale) and beta (shift)—to ensure the network can still represent the identity transform if needed. This stabilization allows for much higher learning rates, reduces sensitivity to weight initialization, and speeds up the overall training process. Dropout, not Batch Normalization, randomly deactivates connections during training to prevent overfitting. BatchNorm does not replace non-linear activations; it is typically applied right before (or sometimes after) the non-linear activation function. BatchNorm does not inject random noise into the gradient updates, although the batch-level statistics do introduce a small amount of noise that acts as a regularizer, this is not its primary goal or mechanism. Therefore, the correct answer is that Batch Normalization standardizes inputs to a layer to stabilize and accelerate training.