Which activation function is commonly used in deep networks specifically to help mitigate the vanishing gradient problem, compared to sigmoid or tanh?
Select an answer to reveal the explanation.
Short Explanation
Sigmoid and tanh both have a nasty habit: for large positive or large negative inputs, they flatten out almost completely, and when they flatten, their derivative shrinks toward zero. Stack a bunch of these layers together, and those tiny derivatives multiply together during backpropagation, and the gradient reaching early layers can shrink to almost nothing, that's vanishing gradients. ReLU largely dodges this for positive inputs because its gradient is a constant 1, no saturation, no shrinking, which is exactly why it became the default choice for deep networks, making it the correct answer. Softmax isn't a general hidden-layer activation, it's specifically used to produce a probability distribution at an output layer for classification, not throughout hidden layers. A linear activation in every hidden layer would collapse the whole network into one big linear function, losing the non-linearity deep learning depends on, and it doesn't inherently fix vanishing gradients either. And just shrinking the learning rate on sigmoid doesn't address the root cause, saturating derivatives, it would only slow training down further.
Full Explanation
The vanishing gradient problem arises because certain activation functions, notably sigmoid and tanh, saturate for large positive or large negative input values, meaning their output flattens out and their derivative approaches zero in those saturated regions. During backpropagation, the chain rule multiplies these small derivatives together across many layers, and in a sufficiently deep network this repeated multiplication can shrink the gradient signal reaching the earliest layers to a value so small that those layers effectively stop learning. ReLU (Rectified Linear Unit), defined as the maximum of zero and the input value, was widely adopted specifically because it does not saturate for positive inputs: its derivative is a constant 1 for any positive input, meaning gradients can flow backward through many layers without shrinking due to saturation in that region, which substantially alleviates the vanishing gradient problem and was a major factor in enabling the training of much deeper networks. (ReLU does have its own known drawback, the 'dying ReLU' problem for consistently negative inputs, which variants like Leaky ReLU or Parametric ReLU are designed to address, but its core advantage against vanishing gradients for positive activations is well established.)
The first distractor, softmax applied at every hidden layer, misunderstands softmax's role; softmax is specifically used to convert a vector of raw scores into a probability distribution that sums to 1, and it is applied at an output layer for multi-class classification, not as a general-purpose hidden-layer activation, and it does not address the vanishing gradient problem in hidden layers.
The second distractor, a linear activation function used throughout every hidden layer, is problematic because stacking linear activations collapses the entire network into an equivalent single linear transformation, regardless of depth, eliminating the network's ability to model non-linear relationships; this defeats the purpose of using multiple hidden layers altogether and is not a solution to vanishing gradients, it is a different problem entirely (loss of representational power).
The third distractor, sigmoid with a much smaller learning rate, does not address the root cause of vanishing gradients, which is the saturating shape of the sigmoid function itself; reducing the learning rate changes how large each weight update step is, but it does nothing to prevent sigmoid's derivative from approaching zero in its saturated regions, so gradients would still vanish through many layers, just with even slower training on top of it.
Memory aid: sigmoid and tanh 'flatten out and go quiet' at the extremes, choking off gradient flow in deep networks; ReLU stays 'loud and constant' for positive inputs, which is why it became the standard default activation for hidden layers in deep architectures.