Which activation function is most widely used in the hidden layers of deep neural networks because its linear, non-saturating nature for positive inputs helps mitigate the vanishing gradient problem?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Alright, if you're building a deep neural network, you've got to choose activation functions for your hidden layers. Sigmoid and Tanh were popular back in the day, but they have a massive flaw: they saturate. When inputs get very large or very small, the curve flattens out, and the gradient drops to near zero. That means your network stops learning. ReLU solves this beautifully. For any positive input, the slope is a constant 1. This keeps the gradient active and flowing all the way back to the initial layers, allowing us to train deep networks without them locking up. Stick with ReLU or its variants for hidden layers, and you'll do just fine on the exam.
Full explanation below image
Full Explanation
The choice of activation function in hidden layers determines how well a deep neural network can propagate error gradients during backpropagation. The Rectified Linear Unit (ReLU) activation function, which outputs $f(x) = \max(0, x)$, is widely used because it addresses the vanishing gradient problem that plagues alternative activation functions.
Sigmoid and Hyperbolic Tangent (Tanh) activation functions are 'saturating' functions. As their input values move away from zero, their outputs flatten (saturate) toward their asymptotic limits (0 and 1 for Sigmoid; -1 and 1 for Tanh). In these saturated regions, the derivatives of these functions approach zero. During backpropagation, when these small derivatives are multiplied through the layers, the overall gradient decreases exponentially. This results in the vanishing gradient problem, preventing the weights of early layers from being updated. ReLU prevents this saturation for all positive inputs ($x > 0$) because its derivative is constant at 1. Consequently, gradients can flow back through deep networks without scaling down, allowing for effective training of many layers.
Let's analyze the incorrect options: - Option A (Softmax) is used in the output layer of multi-class classification networks to convert raw scores (logits) into a probability distribution that sums to 1. It is not suitable for hidden layers. - Option B (Sigmoid) and Option D (Tanh) are saturating activation functions that suffer from the vanishing gradient problem, making them poorly suited for the hidden layers of deep networks.