What Is Positional Encoding?

Positional Encoding Explainer

Positional encoding supplies information about order to a model. Self-attention compares tokens by their content, but by itself it does not inherently know whether one token came first, last, or a specific distance from another token.

The original transformer added fixed sine and cosine patterns to token embeddings. Other systems learn position vectors, encode relative distances, or rotate representation components in ways that make position affect attention scores.

Position information helps models distinguish sequences containing the same tokens in different orders. It also influences whether a model can handle contexts longer than those used in training, because some schemes extrapolate more gracefully than others.

There is no single positional method for every architecture. Designers consider context length, memory cost, attention pattern, training distribution, and serving requirements. Evaluation should test both familiar lengths and the longer or shifted sequences the application expects.

Positional encoding supplies order information to models whose attention operations do not inherently know token positions.

Key terms include token order, sequence position, embedding, absolute position, relative position, sinusoidal encoding, rotary encoding, context length and attention.

Absolute methods identify a token’s place in the sequence, while relative methods emphasize distances or relationships between token positions.

The position signal combines with content representations before or during attention calculations.

A model trained on shorter sequences may not automatically use much longer contexts well, even if the software accepts them.

Positional design, training examples, attention behavior, and evaluation length all affect performance beyond the familiar range.

Explore more "Explainers"

Discover additional explainers across politics, science, business, technology, and other fields. Each explainer breaks down a complex idea into clear, everyday language—helping you better understand how major concepts, systems, and debates shape the world around us.