In a multilayer neural network, why are nonlinear activation functions applied after (or as part of) each neuron's weighted sum?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Check this out: if every layer only does multiply-and-add, the whole deep net is just one big linear equation — not very useful for real data. Activation functions are the secret sauce that bends the signal so you can learn curves, XOR-ish patterns, edges, language quirks — the messy stuff. Think of it like a highway of straight roads only: you never turn. Activations let you take the off-ramps. Exam trap: people confuse activations with the loss ("how wrong am I?"), with the weights themselves, or with input normalization. Those are different gears in the pipeline. Remember: activations = nonlinearity so depth actually matters.
Full explanation below image
Full Explanation
In a feedforward neural network, each unit typically computes a weighted sum of inputs (plus bias) and then applies an activation function such as ReLU, sigmoid, tanh, or GELU. The weighted sum alone is a linear operation. A composition of purely linear maps is still linear, so a deep stack without nonlinearities is mathematically equivalent to a single linear layer—incapable of representing most real-world mappings such as XOR-like decision regions, image hierarchies, or linguistic structure.
Nonlinear activations break that collapse. They enable the network to approximate highly nonlinear functions when combined with sufficient width, depth, and training. Choice of activation also affects gradient flow (for example ReLU versus saturating sigmoids), sparsity of activations, and training stability. Modern practice favors activations that mitigate vanishing gradients in deep stacks while remaining simple to compute.
The loss function is a separate component: it compares model outputs to targets and drives backpropagation. Weights are learnable parameters optimized by gradient descent or variants; the activation is usually a fixed function applied to the pre-activation, not the place where parameters are stored. Input normalization (min-max scaling, z-score, and similar steps) happens outside the core layer math as data preprocessing. Batch and layer normalization are related internal normalizers, yet they still are not substitutes for the activation’s role in introducing nonlinearity.
Underlying principle: depth only buys expressive power when nonlinearities separate successive linear transforms. Best practice includes choosing activations appropriate to architecture (ReLU-family for many CNNs, GELU or SiLU in many transformers), monitoring dead units, and not confusing preprocessing or loss design with the activation itself. Memory aid: linear layers plus nonlinear activations equal an expressive model; linear layers only equal fancy linear regression. On the exam, if the question asks what lets deep nets learn complex patterns, answer nonlinear activation—not loss, not weight assignment, not input scaling.