What is a distinguishing property of the tanh (hyperbolic tangent) activation function compared to sigmoid?
Select an answer to reveal the explanation.
Short Explanation
The big selling point of tanh compared to sigmoid is that it's zero-centered. Sigmoid squashes everything into a range between 0 and 1, meaning its output is always positive, which can cause awkward zig-zagging during gradient updates. Tanh instead squashes values into a range between negative 1 and positive 1, so its outputs can be negative, positive, or right around zero, which tends to make gradient-based learning more well-behaved. That's answer B. Tanh doesn't output only integers between 0 and 10, that's not how any standard activation function works, they output continuous real values. Setting all negative inputs to exactly zero describes ReLU, a different activation function with a different shape. And there's no restriction limiting tanh to only the output layer, it's used often in hidden layers, particularly in older architectures and certain RNN variants. Zero-centered output in the range -1 to 1 is the defining trait here.
Full Explanation
The hyperbolic tangent (tanh) activation function maps any real-valued input to an output in the range (-1, 1), and it is zero-centered, meaning its output distribution is symmetric around zero rather than being strictly positive. This is a key distinction from the sigmoid activation function, which maps inputs to the range (0, 1) and is therefore never negative. The zero-centering property of tanh is practically significant because when all outputs of a layer are strictly positive (as with sigmoid), the gradients for the weights in the following layer tend to all point in the same direction during a given update, which can lead to inefficient, zig-zagging convergence during gradient descent. Tanh's symmetric output around zero helps mitigate this issue, which is part of why it was historically preferred over sigmoid in many hidden-layer contexts, including early recurrent neural network designs, even though both functions share a similar S-shaped curve and both can suffer from vanishing gradients for very large-magnitude inputs.
The first distractor is incorrect because tanh, like virtually all standard activation functions used in neural networks, produces continuous real-valued outputs, not discrete integers, and its range is (-1, 1), not (0, 10). The second distractor describes the ReLU (Rectified Linear Unit) activation function, which outputs zero for any negative input and passes positive inputs through unchanged; this is a distinctly different shape and behavior from tanh's smooth, bounded, symmetric curve, and the two functions address different tradeoffs (ReLU avoids saturation for positive inputs and is computationally cheaper, while tanh is smooth and zero-centered but can saturate at both extremes). The third distractor is incorrect because tanh is commonly used as a hidden-layer activation function, particularly in traditional RNN cells and certain gating mechanisms within LSTMs, and is not restricted to output layers; output-layer activation choice instead typically depends on the task (for example, softmax for multi-class classification or a linear activation for regression).
A helpful memory aid: sigmoid squashes into (0, 1) and is always positive, tanh squashes into (-1, 1) and is zero-centered, and ReLU doesn't squash the positive side at all, it simply zeroes out the negative side — three related but distinctly shaped activation functions worth keeping straight.