When designing a deep neural network, you must choose the appropriate activation functions for your hidden layers. If you select the Hyperbolic Tangent (tanh) activation function, which of the following describes a key mathematical and operational characteristic of this function?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal: activation functions are the secret sauce that introduces non-linearity into our networks. Without them, your complex neural network is just a giant, glorified linear regression model that can't learn complex patterns. Now, you might have heard of the Sigmoid function, which squeezes everything between 0 and 1. But Sigmoid has a big drawback—it's not zero-centered. That means all the outputs going to the next layer are positive, which can make the gradient updates during backpropagation zig-zag and slow down training. Enter the Tanh function. Think of Tanh as Sigmoid's cooler cousin. It works similarly, but it squeezes the outputs into a range between -1 and 1. Because it goes negative and positive, the average output is right around zero. This zero-centered nature makes the optimization process much smoother and faster. Keep in mind, though, that Tanh still suffers from the 'vanishing gradient' problem if your inputs get too large or too small, but it's a solid choice that outperformed Sigmoid for hidden layers back in the day!
Full explanation below image
![A plot showing the mathematical curve of the Tanh activation function. The vertical y-axis should range from -1 to 1, and the horizontal x-axis should range from -5 to 5. The curve should pass through the origin (0,0), show saturation at y = 1 for large positive x, and saturation at y = -1 for large negative x. Annotations should point to the y-axis range [-1, 1] and highlight 'Zero-Centered Output'.](/media/blockchain-council/caid/CAID-q114-infographic.webp)
Full Explanation
Activation functions are mathematical formulas applied to the output of a neural network's node. Their primary purpose is to introduce non-linear properties to the network, allowing it to learn complex, non-linear relationships in data.
The Hyperbolic Tangent activation function, commonly referred to as tanh, is defined mathematically as: $$\tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}$$
Key characteristics and operational details of the tanh function include: 1. Output Range (-1 to 1): Unlike the sigmoid activation function, which outputs values in the range [0, 1], tanh maps real-valued inputs to a range between -1 and 1. 2. Zero-Centered Outputs: Because its output range is symmetric around zero, the mean of the outputs from a tanh layer is close to 0. This zero-centered property is highly beneficial for training. When outputs are zero-centered, the gradients during backpropagation are not systematically biased in one direction (all positive or all negative), which helps the optimization algorithm (like Gradient Descent) converge faster. 3. Vanishing Gradient Trap: Although tanh generally performs better than sigmoid in hidden layers, it still suffers from the vanishing gradient problem. For very large positive or negative input values, the function saturates (flattens out), causing its derivative to approach zero. This halts weight updates in early layers during backpropagation.
Let's evaluate the incorrect options: - It maps inputs from 0 to infinity: This describes the Rectified Linear Unit (ReLU) function, which outputs 0 for negative inputs and $x$ for positive inputs, rather than mapping to [-1, 1]. - It directly prevents overfitting: Activation functions do not act as direct regularization techniques (like Dropout or L2 normalization), though they affect training dynamics. - It is used in output layers for multiclass classification: This is the role of the Softmax activation function, which normalizes outputs into a probability distribution summing to 1. Tanh is unsuitable for multiclass output because it outputs negative values.