What is the benefit of using a mini-batch size greater than one, compared to updating weights after every single training example?
Select an answer to reveal the explanation.
Short Explanation
One example at a time gives you a gradient estimate that's basically a coin flip's worth of noise — it's just one data point's opinion on which direction to move. Average the gradient over a batch of, say, 32 or 64 examples instead, and that noise starts canceling out, giving you a much steadier, more trustworthy sense of which direction actually reduces the loss. That's answer B, and it's the whole reason mini-batches exist. It doesn't remove backpropagation — you still need backprop to compute gradients no matter what batch size you use. It absolutely doesn't guarantee no overfitting — overfitting is governed by model capacity, data size, regularization, and training duration, not batch size. And you still need a loss function no matter how big or small your batch is — batch size changes how many examples' worth of loss you average per step, it doesn't remove the loss function itself.
Full Explanation
Using a mini-batch of more than one training example to compute each gradient update averages the individual per-example gradients together, which reduces the variance of the resulting gradient estimate compared to using a single example. A single-example gradient reflects only the idiosyncrasies of that one data point and can point in a direction that doesn't represent the broader trend in the data, causing the parameter updates to be noisy and erratic. Averaging over a mini-batch smooths out this per-example noise, producing an update direction that more reliably reflects the true gradient of the loss over the underlying data distribution, while still being far cheaper to compute than a full-batch gradient over the entire dataset.
The first distractor is incorrect because backpropagation — the algorithm used to compute gradients of the loss with respect to every parameter via the chain rule — is required regardless of batch size; whether you process one example or many at a time, backpropagation is how those gradients get computed. The second distractor is incorrect because overfitting is governed by factors such as model capacity relative to dataset size, regularization strength, and training duration, not by mini-batch size directly; while batch size can have secondary effects on generalization (very large batches are sometimes associated with sharper minima and slightly worse generalization in some studies), it does not guarantee immunity to overfitting. The third distractor is incorrect because a loss function must still be defined and computed for every mini-batch (as the average or sum of per-example losses within that batch) — batch size does not remove the need to choose an appropriate loss function for the task.
In practice, mini-batch size represents a tunable tradeoff: very small batches (close to 1) give noisy but frequent updates and can help escape sharp local minima due to that noise, while very large batches give smoother, more stable gradient estimates but require more memory and yield fewer updates per epoch. Commonly used batch sizes such as 32, 64, 128, or 256 are chosen empirically to balance gradient stability, memory constraints, hardware parallelism (GPU utilization), and training speed.