How does the ReLU (Rectified Linear Unit) activation behave?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Check this out—ReLU is almost embarrassingly simple, and that's why everybody uses it. Positive in, same number out. Negative in, zero out. Think of it like a one-way valve: flow goes through when the signal is positive, and it shuts off on the negative side. That little kink is enough nonlinearity to stack deep nets that actually learn useful features. Exam trap: folks call ReLU "linear" because the right half looks like a straight line—but the bend at zero makes the whole function nonlinear. Don't confuse it with sigmoid's smooth 0–1 curve or softmax's probability vector. I remember when everyone lived on sigmoids and training crawled; ReLU sped things up big time. Land this: max(0, x). That's the whole story.
Full explanation below image
Full Explanation
The rectified linear unit defines an elementwise nonlinearity f(x) = max(0, x). For any positive pre-activation the function behaves like the identity, passing the value unchanged; for non-positive inputs it returns zero. Because the slope is not constant everywhere, ReLU is nonlinear as a mapping even though each half-line is linear. That property lets multilayer compositions approximate a rich class of functions, which pure linear stacks cannot do beyond a single affine transform. Without some form of nonlinearity between layers, stacking dense transforms would collapse into a single affine map no matter how deep the stack appears on paper.
ReLU became a default hidden-unit activation in many deep models because its gradient is either zero or one in the non-kink regions, reducing the saturation problems common with sigmoid and tanh when activations fall into flat tails. Faster, more stable gradient flow often translates into quicker convergence and deeper practical networks. Variants such as Leaky ReLU, parametric ReLU, and GELU soften the hard zero region to mitigate "dying ReLU" units that stay inactive if they always receive negative inputs, but the core exam definition remains the max(0, x) rectifier. Knowing that definition lets you recognize ReLU in diagrams, code, and multiple-choice stems without confusing it with output-layer normalizers.
Confusing ReLU with sigmoid is a frequent error. Sigmoid squeezes all reals into (0, 1) with a smooth logistic curve and saturates for large |x|. Softmax is not an elementwise activation in the same sense; it normalizes a vector of scores into a categorical probability distribution and is typically applied at multi-class outputs. Calling ReLU purely linear ignores the critical discontinuity in derivative at zero and the zeroing of negatives that provide nonlinearity. Those three confusions—sigmoid curve, probability vector, pure linearity—are exactly the distractors this item targets.
In practice, ReLU is applied after affine transforms inside dense, convolutional, and many residual blocks. Understanding its shape helps diagnose dead units, choose initializations, and know when a smooth alternative might help. For assessments, state the piecewise definition clearly, note that it is nonlinear despite a linear positive regime, and separate it from probability-producing and saturating activations. If you remember only one formula from activation-function questions, make it ReLU equals max of zero and x.