Compared with smaller models, what is a well-known disadvantage of training a very large, deep neural network?
Select an answer to reveal the explanation.
Short Explanation
Let me show you how this works in the real world. Stacking dozens of layers can unlock wild representation power—but your GPU bill and coffee budget go up with it. Think of it like a skyscraper: more floors, more elevators, more power, more time to build. Exam trap: answers that claim deep nets magically need less data, are easier to explain, or never overfit. Nope. Capacity is a double-edged sword. When your boss walks in Friday and says “just make it deeper,” ask what cluster and timeline they just approved. Takeaway: the classic downside is compute cost and training time—remember that for the test.
Full Explanation
Large, deep neural networks achieve strong performance on vision, language, and multimodal tasks by composing many nonlinear layers and millions or billions of parameters. That capacity comes with practical costs. Each training step performs more floating-point operations; memory for activations and optimizer states grows; distributed training and longer schedules become necessary. Hence a central disadvantage is high computational demand and extended training time, plus energy and infrastructure cost. Organizations must budget GPUs or TPUs, engineering for distributed training, and longer iteration cycles.
The distractors invert well-known trade-offs. Deep models usually require large datasets or transfer learning rather than less data; claiming they need less labeled data reverses the usual sample-complexity story. Interpretability generally worsens relative to linear models or shallow decision trees; post-hoc explanation tools only partially close the gap. Overfitting risk typically rises with capacity unless countered by regularization, data augmentation, dropout, weight decay, early stopping, or massive pretraining corpora. Depth is not a free lunch on any of those axes.
On exams and in production planning, pair “deeper or wider” with questions about FLOPs, latency, batch size limits, and carbon or cost budgets—not with claims of automatic data efficiency or transparency. Smaller models may be preferable when latency, edge deployment, or auditability dominate. Distillation and pruning can recover some efficiency after a large model is trained, but they do not erase the initial training cost.
Underlying principle: model capacity buys expressiveness that you pay for in compute, data discipline, and often opacity. Best practice is to justify scale with ablations, monitor generalization carefully, and choose the smallest model that meets quality and latency targets. Memory aid: capacity buys power; the classic disadvantage of very large deep nets is compute and training time—not magical data thrift, easy interpretability, or inherent resistance to overfitting.