A researcher stacks 150 convolutional layers and finds training accuracy actually gets worse than with a shallower 50-layer version. A colleague suggests adding skip connections, as in a ResNet architecture. Why would this help?
Select an answer to reveal the explanation.
Short Explanation
That degradation problem, where a deeper plain network trains worse than a shallower one, is exactly what residual connections were invented to fix. A skip connection adds the input of a block directly to its output, so the layers only need to learn a residual, the difference from identity, rather than the whole transformation from scratch. This gives gradients a direct shortcut back to earlier layers during backprop, so they don't fade away, and it means a block can easily learn to do nothing if that's the best option, instead of struggling to approximate an identity mapping through weights. It's not dropout, that's the random-zeroing option. It's not weight sharing, and it's not the batch-norm option either, those solve different problems entirely.
Full Explanation
Residual, or skip, connections address the degradation problem observed in very deep plain networks, where adding more layers beyond a certain depth causes both training and test accuracy to worsen, a phenomenon distinct from overfitting since even training error increases. A residual block computes its output as F(x) + x, where F(x) is the transformation learned by the stacked layers and x is the block's original input, added via an identity shortcut. This architecture provides two key benefits. First, during backpropagation, gradients can flow directly through the identity shortcut back to earlier layers, largely avoiding the vanishing-gradient attenuation that would otherwise occur when gradients must pass through many sequential nonlinear transformations. Second, if the optimal mapping for a block is close to the identity function, the network only needs to drive F(x) toward zero, which is a much easier optimization target than learning an exact identity mapping through a stack of nonlinear layers with arbitrary weight initialization. This lets networks scale to very substantial depths, as in ResNet-50, ResNet-101, and beyond, while continuing to improve accuracy as depth increases.
The dropout distractor is incorrect because dropout is a regularization technique that randomly deactivates a subset of neurons during each training step specifically to reduce overfitting and improve generalization; it operates independently of network depth and does not address gradient flow or the degradation problem. The weight-sharing distractor is incorrect because reducing trainable parameters via shared weights describes techniques like convolutional parameter sharing or tied weights in autoencoders, not the mechanism of residual connections, which actually add parameter-free identity paths rather than reducing existing parameters. The normalization distractor is incorrect because standardizing activations to zero mean and unit variance describes batch normalization or layer normalization, a separate architectural technique that stabilizes and accelerates training by controlling the distribution of layer inputs, often used alongside residual connections but conceptually and mechanically distinct from them.
The key memory aid: residual connections let a network 'skip the hard part when it isn't needed,' turning the learning problem from 'learn the whole mapping' into 'learn just the adjustment,' which is why they enable reliable training of networks hundreds of layers deep.