A retail grocer grows one decision tree until every training leaf is pure, including leaves that hold a single receipt. Training accuracy is far above holdout accuracy. What did that very deep tree do?
Select an answer to reveal the explanation.
Short Explanation
A tree grown until every leaf is a single receipt has memorized the training baskets the way a student memorizes answer keys. You should expect a shiny training score and a weak holdout score. The trap is calling that depth underfitting, or pretending one tree already averaged a forest.
Full Explanation
A decision tree that keeps splitting until every training leaf is pure, including leaves with a single receipt, has enough flexibility to memorize the training rows. It can isolate one-off baskets and noise so the training accuracy looks excellent while the same splits fail on new receipts. The gap between training accuracy and holdout accuracy is the usual symptom of that memorization. Calling the tree underfit reverses the diagnosis. Underfitting would show a simple tree that misses structure on both the training rows and the holdout, not pure single-row leaves with a much better training score. Averaging many trees is what a forest or a bagged ensemble does to reduce variance. This grocer grew one tree, so no average of trees occurred. A single diagonal boundary is not what ordinary axis-aligned splits draw. Those splits cut one feature at a time, and a very deep tree of them can stair-step a pattern while still memorizing rows. Exam caveat: depth is a capacity knob, and purity on the training set is not evidence the leaves will stay pure on new receipts. Operational check: compare training and holdout accuracy, and inspect whether many leaves contain only one training receipt.