Why do deep networks rely on nonlinear activation functions between layers rather than purely linear transforms?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's where it gets interesting for you. If every layer is just a linear transform, stacking ten of them is still one big linear transform—expensive and pointless. Nonlinear activations (ReLU and friends) bend the signal so the network can learn curves, corners, and weird decision shapes. Think of it like cooking: only stirring (linear) never browns the food—you need heat (nonlinearity). Trap: folks think "nonlinear" means the graph is a straight line—nope. And they're not output-only or shallow-only. Pay attention: depth + nonlinearity = real power. Land it: no curves in the activation, no complex functions learned.
Full explanation below image
Full Explanation
A linear layer computes an affine function of its inputs: weighted sums plus optional bias. Composing affine functions yields another affine function. Therefore, a multilayer network built only from linear maps has the same representational power as a single linear layer, regardless of depth—extra layers add parameters and compute without adding true expressive power. Nonlinear activation functions—ReLU, sigmoid, tanh, GELU, Swish, and others—break that collapse. After each linear transform, a pointwise nonlinearity allows the overall mapping from inputs to outputs to approximate highly complex functions, which is the theoretical and practical foundation of deep learning and hierarchical feature learning.
Saying a nonlinear activation "always outputs a straight line" confuses terminology: a straight-line relationship is linear by definition. Nonlinear means the output is not strictly proportional (in an affine sense) to the input across the domain; ReLU, for example, is piecewise linear but overall nonlinear because of the kink at zero that changes the slope. Nonlinearities are used throughout hidden layers so intermediate representations can separate classes in warped feature spaces; the output layer may use a task-specific function (softmax, sigmoid, or linear) in addition to those hidden nonlinearities. The claim that nonlinear activations only suit shallow nets is backwards: universal approximation results and modern practice show that depth plus nonlinearity is what unlocks hierarchical features such as edges, parts, and objects in vision or syntax and semantics in language.
Training implications include gradient flow (vanishing gradients with saturated sigmoids historically pushed adoption of ReLU variants), careful initialization schemes, and normalization layers that keep pre-activations in healthy ranges so nonlinear regions remain useful. Dying ReLU and alternative activations are practical tuning concerns. Memory aid: "Linear stack = one line; add nonlinear bends = deep expressivity." Exam items that ask what "nonlinear activation" means are testing this expressivity argument, not layer-count limits or output-only placement.