You are building a network with a softmax output layer that must choose among ten mutually exclusive digit classes (0 through 9). Which loss function is the most appropriate choice for training this model?
Select an answer to reveal the explanation.
Short Explanation
Here's the deal: whenever you've got softmax probabilities on one side and a one-hot true label on the other, categorical cross-entropy is the natural partner — it directly measures how far off your predicted probability distribution is from the true class, and its gradient behaves beautifully with softmax during backprop. So that's your answer. Mean squared error and mean absolute error are built for continuous-valued regression targets, not probability distributions over categories — using them here gives you weak, mismatched gradients and slower, worse training. Hinge loss is the classic SVM loss for margin-based binary or one-vs-rest classification, not the standard pairing for softmax multi-class networks. When you see softmax plus multiple mutually exclusive classes, think categorical cross-entropy every time.
Full Explanation
Categorical cross-entropy is the correct choice because it is mathematically designed to compare a predicted probability distribution (produced by a softmax layer) against a true distribution, typically represented as a one-hot encoded label vector. It computes the negative log-likelihood of the correct class under the model's predicted probabilities, penalizing confident wrong predictions heavily and rewarding confident correct predictions. When paired with softmax, the combined gradient of the loss with respect to the pre-activation logits simplifies elegantly to (predicted probability minus true label), which makes optimization efficient and stable. This is precisely why categorical cross-entropy is the standard loss for multi-class, mutually exclusive classification problems such as digit recognition.
Mean squared error is incorrect for this task because it is designed for continuous regression targets, where the goal is to minimize the squared distance between predicted and actual numeric values. Applying MSE to probability outputs produces much smaller gradients when predictions are very wrong (because squaring small probability differences yields tiny numbers), which slows learning and does not appropriately penalize misclassification the way a probability-based loss does.
Hinge loss is incorrect because it is the loss function associated with maximum-margin classifiers like Support Vector Machines, and it is formulated around margin violations for scores rather than calibrated probability distributions. While hinge loss can be adapted for multi-class settings, it is not the conventional or most appropriate choice when the network already outputs softmax probabilities, since it does not exploit the probabilistic interpretation the way cross-entropy does.
Mean absolute error, like MSE, is designed for continuous-valued regression tasks and measures the average absolute difference between predicted and true numeric values. It is not designed to handle categorical probability distributions and again produces suboptimal gradient behavior compared to cross-entropy in a classification setting.
Memory aid: softmax and categorical cross-entropy are a matched pair — one produces probabilities, the other measures probability error.