A batch of text sequences fed into an RNN has varying lengths — some 10 words, some 40 words. Since a batch tensor requires a single, uniform shape, what technique is used to make all sequences in the batch the same length?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Neural nets want their batches shaped like a nice uniform rectangle, but real sentences obviously don't come in matching lengths. Padding solves that mismatch: pick a target length (often the longest sequence in the batch, or some fixed max), and fill the shorter sequences out with a special padding token, usually zeros, until every sequence in the batch reaches that same length. Now the whole batch stacks cleanly into one tensor. That's why 'padding' is correct. Dropout randomly zeroes out neurons during training for regularization — completely different job. Batch normalization rescales activations to stabilize training, again unrelated to sequence length. And one-hot encoding turns categories into binary vectors; it has nothing to do with making sequences the same length, though it's sometimes used elsewhere in text preprocessing.
Full explanation below image
Full Explanation
Padding is the standard technique used to handle variable-length sequences when they need to be combined into a single, fixed-shape batch tensor for processing by an RNN, LSTM, GRU, or similar sequence model. Since deep learning frameworks require tensors within a batch to share a consistent shape, sequences shorter than a chosen target length (commonly the length of the longest sequence in the batch, or a fixed maximum length set for the whole dataset) are extended by appending (or sometimes prepending) a special padding value, typically 0 or a designated padding token/index, until they reach that target length. This allows sequences of different original lengths to be stacked into one uniformly shaped batch tensor that the model can process efficiently in parallel. To prevent the padded positions from influencing the model's learning or output — since they carry no real information — practitioners commonly pair padding with a mask (an auxiliary tensor marking which positions are real data versus padding) so that loss computations, attention mechanisms, or recurrent computations can ignore or down-weight the padded entries appropriately.
Dropout is a regularization technique that randomly sets a fraction of neuron activations to zero during each training step, to prevent the network from over-relying on specific neurons and reduce overfitting. This is unrelated to reconciling sequence length differences; dropout operates on activations within a layer, not on adjusting input sequence lengths.
Batch normalization is a technique that normalizes layer activations across a batch to have roughly zero mean and unit variance (with learnable scale and shift parameters), which stabilizes and speeds up training. It addresses the internal distribution of activations during training, not the structural problem of sequences having different lengths.
One-hot encoding is a technique for representing categorical values (such as words in a vocabulary, in some simpler text-processing pipelines) as binary vectors. While one-hot encoding might be applied to represent individual tokens within a sequence, it does nothing to reconcile differing sequence lengths across a batch — a batch of one-hot-encoded sequences of different lengths would still need padding (or another length-handling technique) before it could be stacked into a uniform tensor.
Padding, together with masking, is foundational to efficient batch processing of sequential data (text, time series, audio) in nearly every deep learning framework, and understanding it is essential for correctly preparing variable-length data for RNNs, LSTMs, GRUs, and even Transformer-based models.