A team wants an activation function that squashes any real-valued input into an output strictly between -1 and 1, with an output of 0 when the input is 0. Which activation function fits this description?
Select an answer to reveal the explanation.
Short Explanation
You're looking for the activation that's centered on zero and squeezes everything into the range -1 to 1 — that's tanh, answer D. Feed it 0, get 0 out; feed it a huge positive number, it creeps toward 1; feed it a huge negative number, it creeps toward -1. ReLU is a trap here because it's the default workhorse in hidden layers, but it just clips negative inputs to 0 and passes positive inputs straight through unbounded — no -1-to-1 squashing at all. Softmax is for turning a whole vector of raw scores into a set of probabilities that sum to 1 across multiple classes, not for squashing a single value into -1 to 1. And sigmoid is the close cousin that trips people up: it does squash into a bounded range, but that range is 0 to 1, not -1 to 1, and it outputs 0.5 (not 0) when the input is 0.
Full Explanation
The hyperbolic tangent function, tanh(x) = (e^x - e^-x) / (e^x + e^-x), maps any real-valued input to an output in the open interval (-1, 1). It is zero-centered, meaning tanh(0) = 0, and it is an odd function, so tanh(-x) = -tanh(x). As x grows large and positive, tanh(x) approaches 1; as x grows large and negative, it approaches -1. This zero-centered, symmetric, bounded behavior is exactly what the question describes, making tanh the correct answer.
ReLU (Rectified Linear Unit), defined as max(0, x), is incorrect because it is unbounded on the positive side (it simply passes positive inputs through unchanged) and produces exactly 0 for any non-positive input rather than smoothly squashing values into a symmetric bounded range. ReLU is popular in hidden layers of deep networks because it avoids certain vanishing-gradient issues and is computationally cheap, but it does not match the -1-to-1 squashing behavior described.
Softmax is incorrect because it is not an elementwise squashing function for a single scalar; it takes a vector of raw scores (logits) and converts them jointly into a probability distribution where all outputs are non-negative and sum to 1. It is typically used in the output layer for multi-class classification, not as a general-purpose hidden-layer activation for bounding individual values into -1 to 1.
Sigmoid, defined as 1/(1+e^-x), is the closest distractor because it is also a smooth, bounded, S-shaped squashing function, but its output range is (0, 1), not (-1, 1), and sigmoid(0) = 0.5, not 0. This lack of zero-centering can slow convergence in hidden layers, which is one reason tanh is sometimes preferred over sigmoid for hidden units, even though sigmoid remains standard for binary classification output layers.
Memory aid: 'tanh is sigmoid's zero-centered, symmetric cousin, stretched to run from -1 to 1 instead of 0 to 1.'