When designing a neural network to perform multi-class classification (where an input must be assigned to one of several mutually exclusive categories), which activation function should be applied to the final output layer?
Select an answer to reveal the explanation.
Short Explanation and Infographic
If you're setting up a network to classify images of animals—say, dogs, cats, and birds—you want your output to be a nice, neat probability distribution. You want it to tell you something like: "Hey, there's an 80% chance this is a dog, a 15% chance it's a cat, and a 5% chance it's a bird." To get that, you need the Softmax activation function on your output layer. Softmax takes those raw numbers from your network's final layer and squashes them so they all fall between 0 and 1, and they all add up to exactly 1 (or 100%). If you used Sigmoid, you'd get separate probabilities that don't add up to 1, and if you used ReLU, you'd just get raw numbers. Trust me, Softmax is what you want for multi-class classification!
Full explanation below image
![A diagram showing the final layer of a neural network processing 3 logits (e.g., [2.0, 1.0, 0.1]) passing through a Softmax block, which outputs a probability distribution vector (e.g., [0.7, 0.2, 0.1]) summing to 1.0, labeled with class names (e.g., Dog, Cat, Bird).](/media/blockchain-council/caid/CAID-q097-infographic.webp)
Full Explanation
In multi-class classification, the goal is to classify an input into one of $N$ mutually exclusive classes. The output layer of the neural network typically produces raw, unnormalized real-valued scores (known as logits) for each class. To interpret these logits as probabilities, we apply the Softmax activation function.
Mathematically, the Softmax function is defined for an output vector $z$ as: $$\text{Softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{N} e^{z_j}}$$
This function exponentiates each output score and normalizes it by dividing by the sum of all exponentiated scores. This ensures two properties required for a probability distribution: 1. Every output probability lies strictly in the range $[0, 1]$. 2. The sum of all output probabilities across all classes is exactly equal to $1.0$.
Let's look at why the other options are inappropriate for a multi-class classification output layer: - Option A (ReLU): The Rectified Linear Unit outputs $\max(0, x)$. It is unbounded on the positive side, meaning outputs do not represent probabilities and cannot be directly used to compute multi-class cross-entropy loss. It is primarily used in hidden layers. - Option C (Sigmoid): The Sigmoid function, $\sigma(x) = \frac{1}{1 + e^{-x}}$, maps inputs to the range $[0, 1]$ independently. It is the correct choice for binary classification (two classes) or multi-label classification (where an input can belong to multiple categories simultaneously), but not for multi-class classification because the outputs do not sum to $1$. - Option D (Tanh): The Hyperbolic Tangent function maps inputs to the range $[-1, 1]$. Since probabilities cannot be negative, it is unsuitable for output layer classification probabilities.