A researcher is choosing between a sigmoid and a tanh activation function for a hidden layer. In which situation would tanh typically be preferred over sigmoid?
Select an answer to reveal the explanation.
Short Explanation
Tanh and sigmoid look similar — both are S-shaped — but tanh squashes values into a range of -1 to 1 instead of 0 to 1, which means its output is zero-centered. Here's why that matters: when your activations hover around zero on average, gradients tend to move more consistently during training instead of always pushing weights in the same direction, so optimization tends to converge faster and smoother. That's the correct pick. Needing outputs strictly between 0 and 1 actually favors sigmoid, not tanh. Neither activation eliminates vanishing gradients — both still saturate at their extremes, they just do it in different ranges. And a full probability distribution over multiple classes is softmax's job, not tanh's or sigmoid's.
Full Explanation
Tanh (hyperbolic tangent) and sigmoid are both S-shaped (sigmoidal) activation functions, but they differ in output range: sigmoid maps inputs to the range (0, 1), while tanh maps inputs to the range (-1, 1). Because tanh's output is centered around zero, its use as a hidden-layer activation can produce activations whose average is close to zero, which in turn tends to keep the gradients flowing through the network more balanced during backpropagation. This zero-centering is helpful because when all activations feeding into the next layer are positive (as with sigmoid), the gradient updates to that layer's weights are constrained to move in a correlated direction, which can slow convergence in gradient-based optimization; tanh's zero-centered range mitigates that specific inefficiency, which is why tanh is often the default second choice for hidden layers when sigmoid's saturation properties are still wanted.
The first distractor is incorrect because a strict requirement that outputs lie only between 0 and 1 is exactly what favors sigmoid, not tanh; sigmoid's range is naturally suited to representing values like probabilities or gating signals bounded in [0, 1], such as the gates in an LSTM cell.
The second distractor is incorrect because neither tanh nor sigmoid eliminates the vanishing gradient problem. Both functions saturate (flatten out) for large positive or large negative inputs, and in that saturated regime the derivative approaches zero for both activations, contributing to vanishing gradients in deep networks. This is one of the key reasons ReLU-family activations became popular for deep networks.
The third distractor is incorrect because outputting a proper probability distribution across multiple mutually exclusive classes is the role of the softmax function, which normalizes a vector of scores so they sum to 1; tanh and sigmoid operate elementwise and do not enforce that a set of outputs sums to 1.
Memory aid: sigmoid ranges 0 to 1 (think 'probability-like'), tanh ranges -1 to 1 (think 'zero-centered sigmoid'), and both still saturate at their tails, so neither one is immune from vanishing gradients on its own.