Since a Transformer processes all tokens in a sequence simultaneously rather than one at a time, why is positional encoding added to the input embeddings?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the trade-off Transformers made: by ditching recurrence and processing every token at once, they gained speed and parallelism, but they lost any built-in sense of word order. Self-attention alone treats a sequence almost like a bag of tokens, it has no idea which one came first. Positional encoding patches that gap by adding a unique, position-dependent signal to each token's embedding, so the model can tell "this word is first" from "this word is fifth." It's not about shrinking parameter count, it's not gradient normalization, and it definitely doesn't replace attention, positional encoding and attention work together, each solving a different problem.
Full explanation below image
Full Explanation
Positional encoding exists to solve a specific structural gap in the Transformer architecture: because self-attention computes relationships between all tokens in parallel, with no inherent notion of sequence order (unlike an RNN, which processes tokens one at a time and naturally encodes order through its sequential structure), the model has no way to distinguish token order purely from the attention mechanism itself. Positional encoding addresses this by adding a fixed or learned vector, typically based on sine and cosine functions of varying frequencies in the original Transformer paper, to each token's embedding before it enters the attention layers. This encodes each token's absolute (and, through the relative differences between encodings, relative) position in the sequence, giving the model the information it needs to reason about word order, proximity, and sequence structure even though attention itself is order-agnostic.
The parameter-reduction distractor is wrong because positional encodings are typically either fixed, non-learned sinusoidal functions or a small learned embedding table; either way, their purpose is not to reduce the parameter count of the attention layers, and they have no direct effect on the size of the query, key, or value projection matrices used in self-attention. The gradient-normalization distractor is wrong because positional encoding is unrelated to controlling gradient magnitudes; that concern is instead addressed by mechanisms like layer normalization, residual connections, and careful weight initialization, all of which are separate components of the Transformer architecture. The distractor claiming it replaces attention is wrong because positional encoding and self-attention serve complementary, not competing, purposes: attention computes contextual relevance between tokens, while positional encoding supplies the order information attention lacks on its own; removing attention and keeping only positional encoding would eliminate the model's ability to weigh contextual relationships altogether.
A useful memory aid: think of positional encoding as stamping a timestamp or seat number onto each token before it joins the conversation, so that even though all tokens speak (attend to each other) at once, each one still knows where it sits in line. Without this, a Transformer would process "the dog bit the man" and "the man bit the dog" as if they carried identical positional information, losing an essential piece of meaning.