A grocery chain trains a deep network to score whether a shopper will redeem a coupon, and the hidden layers use a sigmoid. After several layers, updates to the earliest weights shrink to almost nothing. What is the sigmoid doing to those updates?
Select an answer to reveal the explanation.
Short Explanation
Think of a sigmoid as a volume knob that will not travel outside a narrow band. When your signal sits on the flat ends, nudging an early weight barely changes what comes out, so the update fades. Do not call that an explosion; the slope is too small, not too steep.
Full Explanation
A logistic sigmoid maps each hidden input onto a narrow open interval between zero and one. On the flat tails the derivative is close to zero, so a small change in an early weight barely changes the output. Stack several of those sigmoids and each layer multiplies the backward signal by another near-zero slope, which is why the earliest coupon weights receive a faded update. Hard-zeroing every activation overstates the function: outputs may sit near zero, but a sigmoid does not switch the layer off as a binary gate or delete weights. Unbounded growth with steeper slopes is the exploding-gradient pattern, and a bounded sigmoid with flat tails cannot produce it. Replacing the loss with a count of missed coupons would change the objective, not the shape of the hidden units, and would not single out the early layers. Exam caveat: one sigmoid on a two-class output is a normal final layer and is not evidence that hidden gradients have vanished; the fade shows up when saturating sigmoids are stacked. Operational check: during a few training steps, compare the first-layer gradient size with how many hidden units sit near the tails, and confirm heavier saturation lines up with smaller early updates.