In a fully connected neural network layer, each neuron computes a weighted sum of its inputs before applying an activation function. What role does the bias term play in this computation?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Think of the bias as a knob that lets a neuron fire even when all its inputs are zero. Each neuron computes weights-times-inputs and sums them up — that's the matrix multiplication part — and then it adds its own bias constant on top, before the activation function sees it. That bias doesn't care what the inputs are; it just shifts the whole function left or right, giving the network more flexibility to fit the data. Scaling inputs by a learned factor is what the weights already do, so that's not bias's job. Normalizing to zero mean/unit variance is batch normalization, a totally separate technique. And randomly zeroing neurons is dropout, also unrelated. Bias is simply the 'plus a constant' term.
Full explanation below image
Full Explanation
In a fully connected layer, each neuron's pre-activation value is computed as z = (w · x) + b, where w is the neuron's weight vector, x is the input vector, and b is the bias — a single learned scalar constant specific to that neuron. The bias is added after the weighted sum and before the activation function is applied, and unlike the weights, it does not multiply or interact with the inputs at all; it is a fixed offset that shifts the entire weighted-sum output up or down regardless of what the input values are. This gives the neuron the flexibility to activate (or not) even when all inputs are zero, and more generally lets the network fit functions whose decision boundary or output doesn't have to pass through the origin — geometrically, in a single neuron with one input, the bias corresponds to the y-intercept of the line, while the weight corresponds to its slope. Without bias terms, every neuron's output would be forced to equal zero whenever the input is zero, severely constraining what the network can represent. 'Scales the input features by a learned multiplicative factor' describes the role of the weights, not the bias; weights multiply against inputs, while bias is strictly additive and input-independent, so conflating the two ignores the fundamental additive-versus-multiplicative distinction. 'Normalizes the activations to have zero mean and unit variance' describes batch normalization (or layer normalization), a separate architectural technique that recenters and rescales activations using batch statistics (and its own learned scale/shift parameters), which is conceptually and mechanically distinct from a per-neuron bias constant. 'Randomly zeroes out a fraction of neurons during training' describes dropout, a regularization technique for reducing overfitting by stochastically disabling neurons during training; it has nothing to do with the deterministic bias term that exists in every forward pass. A useful memory aid: weights decide how much each input matters (multiplicative, input-dependent), while bias decides the neuron's baseline output when there's no meaningful signal at all (additive, input-independent) — together they define the familiar 'wx + b' linear form underlying every layer before activation.