A machine learning engineer chooses ReLU as the activation function for the hidden layers of a new network. Which property of ReLU is she relying on?
Select an answer to reveal the explanation.
Short Explanation
ReLU's whole trick is dead simple: if the input is negative, output zero; if it's positive, pass it straight through unchanged. That's the second option, and it's correct. Because a chunk of neurons end up outputting exactly zero for any given input, the network ends up with sparse activations, which tends to make learning more efficient and helps sidestep some of the vanishing-gradient pain that older activations like sigmoid caused. Don't confuse it with sigmoid, which squashes everything into 0 to 1 for probability-style outputs — that's a different function entirely. ReLU also isn't a pure identity function, since it clips negatives to zero rather than passing everything through. And no activation function 'guarantees' gradients never shrink — ReLU helps a lot in the positive region, but it can still suffer from dying neurons when inputs stay negative.
Full Explanation
ReLU, short for Rectified Linear Unit, is defined as f(x) = max(0, x): it outputs zero whenever the input is negative and outputs the input unchanged whenever the input is positive or zero. This simple piecewise behavior gives ReLU two practically important properties: it introduces sparsity, since any neuron receiving a negative pre-activation value contributes exactly zero to the forward pass, and it avoids the vanishing-gradient problem in the positive region, since the gradient there is a constant 1 rather than a small fraction like the derivatives produced by sigmoid or tanh in their saturated regions. This makes the second option correct. The first option, describing ReLU as squashing inputs into a 0-to-1 range for probability interpretation, actually describes the sigmoid function, which is typically reserved for binary classification outputs, not ReLU, which has no upper bound and can output arbitrarily large positive values. The third option, describing ReLU as a pure identity function for every input, is incorrect because ReLU explicitly clips all negative inputs to zero; a true identity function would pass negative values through unchanged, which ReLU does not do, and this clipping is precisely what creates its sparsity property. The fourth option, claiming ReLU guarantees gradients never shrink no matter how many layers are stacked, overstates ReLU's benefit; while ReLU's constant gradient of 1 for positive inputs helps mitigate vanishing gradients compared to saturating activations, ReLU units can still suffer from the 'dying ReLU' problem, where a neuron's input stays permanently negative and its gradient becomes permanently zero, so gradients can indeed be lost, just via a different mechanism than saturation. A useful memory aid is that ReLU acts like a one-way valve: anything positive flows through freely and at full strength, while anything negative is simply shut off, and it is this shutting-off behavior across many neurons that produces the sparse, efficient representations ReLU is prized for in deep hidden layers.