A gradient boosting model is trained with SHRINKAGE=0.01 (low learning rate) and NTREES=500. A colleague trains the same model with SHRINKAGE=0.1 and NTREES=50. Assuming the same training dataset, what is the expected difference in model behavior?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal — b is correct because smaller learning rates (SHRINKAGE=) cause the boosting algorithm to take smaller steps in the gradient direction per tree, requiring more trees to reach the same training loss. However, smaller steps tend to produce smoother, more stable solutions that generalize better.
Full explanation below image
Full Explanation
B is correct because smaller learning rates (SHRINKAGE=) cause the boosting algorithm to take smaller steps in the gradient direction per tree, requiring more trees to reach the same training loss. However, smaller steps tend to produce smoother, more stable solutions that generalize better. Larger learning rates converge faster but may overshoot optimal solutions or overfit if insufficient trees are used. C is wrong; the product of learning rate and tree count does not determine equivalence — the interaction is non-linear. A is wrong; given enough trees, the low learning rate model can achieve very low training error. D is wrong; lower complexity does not guarantee better performance.