Attention and Positional Information
A word’s meaning depends on the words around it, so a program reading text has to let each word borrow meaning from its neighbours. A token is one unit of text the model reads, usually a word or a piece of one, and a token’s context is the other tokens it can see. Attention is the step inside a language model that does the borrowing. For every token it scores how relevant each other token is, turns the scores into weights that add to one, and replaces the token’s representation with a weighted average of what those other tokens carry. The weights are computed from the tokens themselves, so the same word reads different neighbours in different sentences, as the figure shows for “bank” in a sentence about a river and a sentence about money.
Attention has one blind spot that the second half of this note repairs. The weighted average treats its inputs as an unordered set, so on its own it cannot tell “dog bites man” from “man bites dog”. Position information has to be added, and the three common ways to add it are adding a position vector to each token, rotating the vectors by an angle that grows with position, and subtracting a penalty that grows with distance. Each choice decides how a model behaves when it meets a text longer than any it was trained on.

Every later section changes one of three things in the same pipeline. Sections 2 to 5 build the score and the average, sections 6 to 9 decide what position information reaches the score, and sections 10 and 11 count what the table of scores costs.
1. Mixing information across positions
Section titled “1. Mixing information across positions”A sentence has to be read as a whole, since a word’s meaning often depends on a word several places away. In “the animal did not cross the street because it was tired”, the word “it” means the animal, and in “because it was wide” it means the street. A neural network is a function built from many learned numbers, its parameters, adjusted during training so its outputs improve. Inside a language model every token is held as a vector, a list of a few thousand numbers, and a layer’s job is to update each token’s vector using the others.
The simplest fix is a fixed window, where each word averages the vectors of its two neighbours with equal weight. That fails twice. The useful word may sit outside the window, and inside the window “the” gets the same weight as “river” although only “river” changes what “bank” means. Attention replaces the fixed window with weights the model computes for each pair from the content of the two tokens, over every position it is allowed to see. The Transformer paper built a whole architecture from this operation, and every model in this series is a stack of such layers.
The opening figure shows the effect on toy vectors with three features meaning money, water and grammar. The same query from “bank” puts 0.36 of its weight on “river” in one sentence and 0.31 on “cash” in the other, so the averaged vector leans toward water in the first case and money in the second. A real model learns its features and uses many such reads in parallel, but the mechanism is the same.
2. Queries keys and values
Section titled “2. Queries keys and values”Attention gives each token three roles, which is easiest to see with a library search. You type a question, each book has a catalogue entry describing it, and when an entry matches you read the book’s contents. The query is the question token asks, the key is the catalogue entry token shows, and the value is the content token hands over. Each is made by a projection, multiplying the token vector by a learned matrix, a rectangular table of numbers: , and . Multiplying a row vector by a matrix means each output number is the sum of the vector’s entries times one column of the matrix.
The dot product multiplies matching entries and adds them, so it is large when the two vectors point the same way. Here is the score of position for position , is the head width, the number of entries in each query and key, and section 3 explains the division by . Softmax is the middle formula: raise to each score and divide by the sum, which makes every attention weight positive, keeps the scores’ order, and makes the weights in a row add to 1. The output is the weighted average of the values, and just runs over every position may read.
A complete calculation on three tokens
Section titled “A complete calculation on three tokens”Take the phrase “the river bank” with two numbers per token so every step fits on paper. The token vectors are , and . The projection matrices have rows , and . These are chosen teaching numbers, while a trained model learns its matrices.
- Project every token. The queries are the , river and bank . The keys are the , river and bank . The values are the , river and bank . For example bank’s query is .
- Score bank’s query against every key. Against the: . Against river: . Against bank itself: .
- Divide by , the square root of the head width 2, giving , and .
- Exponentiate: , and , which add to .
- Divide by that sum. The weights are 0.0743 for the, 0.6200 for river and 0.3057 for bank, and they add to 1.
- Average the values: .
The new vector for bank is mostly river’s value , so after this layer bank carries information about the word beside it. The query, key and value come from three separate matrices because the three roles differ. River should be easy to find when bank asks about water, which is the key’s job, and what it hands over can be a different summary, which is the value’s job. The animation replays these steps.
3. Scaling by the square root of head width
Section titled “3. Scaling by the square root of head width”A dot product adds up one product per entry, so wider vectors give bigger sums for the same kind of agreement. Think of judging a match by summing points across survey questions: a 64-question survey produces bigger totals and bigger gaps between candidates than a 4-question survey, even when the candidates are equally different. Variance measures how spread out a quantity is around its average, as the average squared distance from it, and the standard deviation is its square root. If each entry of a query and a key has average 0 and variance 1, and the entries are independent, each product also has variance 1, and the sum of of them has variance and standard deviation .
Width changes how peaked the softmax gets. Take two keys whose scores differ by one standard deviation. At width 4 the gap is 2, and the winner gets of the weight. At width 64 the gap is 8 and the winner gets 0.9997, so attention has become a hard pick of one key. At the 7B model’s width of 128 the gap is 11.3 and the winner gets 0.99999. Dividing every score by brings the gap back to 1, and the winner gets 0.731 at any width.
The peaked case hurts training, which adjusts each parameter in the direction that reduces the error, a direction found from the gradient, the rate at which the error changes as each number changes. For two keys the gradient through softmax is proportional to , where is the winner’s weight. That is 0.197 at and 0.0003 at , so a saturated softmax passes back almost no signal and the scores stop learning. The figure repeats the experiment with 16 keys. Without scaling the largest weight climbs from 0.42 at width 4 to 0.89 at width 128, while with scaling it stays near 0.25. The argument assumes unit-variance entries, which trained models only roughly have, so some models add a learned temperature or normalize queries and keys instead. Architecture works the gradient version of this argument further.

4. The causal mask
Section titled “4. The causal mask”A model that writes text one token at a time must not peek at the word it is about to write. A student practicing dictation with the answer sheet face up learns nothing, and a model trained the same way learns to copy instead of predict. Autoregressive generation means producing text one token at a time, each chosen from everything before it, and the language model of Language Modeling and Next Token Prediction is trained to predict every next token of a training text in parallel. A causal mask makes that parallel training honest: position may read positions and nothing later.
The mask entry is added to the score before softmax. A score of has exponential , so a future position gets weight exactly zero and the remaining weights still add to 1, with meaning the normalization runs over in one row. Arranged as a grid with one row per query position, the allowed cells form a lower triangle.
Apply it to section 2’s phrase. The row for “the” may read only itself, so its weight is 1 and its output is its own value . The row for “river” reads the and river with scores and , so its weights are 0.3302 and 0.6698 and its output is . The row for “bank” sits last, so it reads everything and keeps its section 2 output .
A future perturbation test checks that a mask works: change a later token and confirm that no earlier output moves. Changing bank’s value from to moves bank’s own output by 2.14 summed over its two numbers, and moves the and river by exactly 0. With the mask removed the same edit moves the’s output by 1.99 and river’s by 2.81, which is how a leak through a broken mask would show up. A suspiciously low training loss is often the first symptom.

5. Multi head attention
Section titled “5. Multi head attention”One set of weights per token can only answer one question at a time, while a word usually needs several answered. For “it” in a long sentence, one question is which noun it refers to and another is what verb comes next. Multi-head attention runs attention operations side by side, each called a head, with its own query, key and value projections, so each head can learn its own notion of relevance.
The heads split the model width instead of adding to it. A token vector of width is projected to queries, keys and values of width , and each is cut into slices of width . In a toy with and , a query becomes head 1’s query and head 2’s query , and each head runs section 2’s calculation on its own slice. In the 7B model, splits into heads of . The 32 head outputs of width 128 are placed side by side, giving width 4,096 again, and an output projection mixes them so information found by one head can reach every coordinate.
flowchart TB X["token vector, width 4,096"] --> P["three projections<br/>W_Q W_K W_V<br/>each 4,096 by 4,096"] P --> H1["head 1: width 128"] P --> H2["head 2: width 128"] P --> H3["... heads 3 to 32"] H1 --> C["concatenate 32 outputs<br/>back to width 4,096"] H2 --> C H3 --> C C --> O["output projection W_O,<br/>4,096 by 4,096"]The parameter count follows from the four square matrices. Each of is numbers, so one layer’s attention holds parameters, and Llama 2 uses no bias terms that would add to it. Across layers that is 2,147,483,648, about 32% of the model’s 6.74 billion. The count does not depend on , since 32 heads of 128 and one head of 4,096 use the same four matrices cut differently. The head count changes what the layer can represent, with more heads giving more independent patterns of narrower width. The Transformer Block and Model Anatomy places this layer inside the full block.
6. Permutation equivariance
Section titled “6. Permutation equivariance”Attention, as built so far, has no idea where any token sits. Shuffle a deck of cards, deal them, and a dealer who only looks at faces will treat each card the same whatever its place in the deck. Permutation equivariance means that reordering the inputs reorders the outputs in exactly the same way and changes nothing else. Without a mask, attention has this property, because each output is a weighted sum over a set, and every score depends only on the two vectors involved.
Check it with section 2’s numbers and no mask. Swap the first two tokens to get “river the bank”. Bank’s query is still and it meets the same three keys, now in a different order, so its weights are still 0.6200 on river, 0.0743 on the and 0.3057 on itself, and its output is still . The outputs for the and river swap places along with their inputs, and none of the numbers change. The projections and the later per-token layers treat every position identically too, so the whole stack inherits the property.
So “dog bites man” and “man bites dog” contain the same three tokens and would give every token the same output vector, though the two sentences mean different things. The causal mask breaks the symmetry slightly, since position 5 sees five tokens and position 2 sees two, and some decoder models trained without any position scheme do learn rough positions from that, as NoPE reports. Most models still add position information explicitly, by one of the schemes in sections 7 to 9. Positional Encoding and Long Context draws the same failure on real attention scores.
7. Absolute and sinusoidal positions
Section titled “7. Absolute and sinusoidal positions”The most direct repair is to stamp each token with its seat number. A position encoding is any signal that lets the computation depend on where a token sits, and an absolute position embedding does it by adding a vector to the token vector, so the layer receives . The simplest version is a learned position table, one trained row per position. GPT-2 has 1,024 rows, which also makes 1,024 a hard limit, since position 1,025 has no row to look up.
The original Transformer used a fixed sinusoidal encoding instead, built from sine and cosine waves of many speeds. Coordinates come in pairs, and pair turns at frequency :
Here is the position, numbers the pair from 0, and is the vector width. The fast pairs change a lot between neighbours and the slow pairs barely move across thousands of positions, so together they read like the hands of a clock. Take , so the two frequencies are and . Position 1 gets and position 2 gets .
The dot product of two such vectors depends only on the gap between positions. By the identity , each pair contributes . For positions 1 and 2 that is , the same as for positions 101 and 102. The figure plots this similarity at width 128, where it falls from 1 at offset 0 to 0.97 at offset 1 and 0.71 at offset 8, with the same curve starting from position 100 or 1,000.

The relative structure gets blurred once the position vector is added to content. The query becomes , so a score mixes content with content, content with position, and position with position, and the model must learn to separate them. Relative position schemes put the offset into the score directly, and the next two sections are the two most used.
8. Rotary position embedding
Section titled “8. Rotary position embedding”Think of two clock hands that both start at their own angle and both advance by the same amount each hour. Whatever the hour, the angle between them stays the same. Rotary position embedding, or RoPE from the RoFormer paper, uses that fact. It splits each query and key into pairs of coordinates and rotates each pair by an angle proportional to the token’s position, for the query at position and for the key at position . A rotation turns a two-number vector by angle and keeps its length.
Here and are one coordinate pair of the query and key before rotation, and is the angle added per position. The left side is the score after both rotations, and the right side contains positions only through the offset , so the score sees how far apart the tokens are and never where they sit. Rotating both vectors by the same extra angle leaves the angle between them, and so the dot product, unchanged.
A rotation worked to numbers
Section titled “A rotation worked to numbers”Use , a raw query pair and a raw key pair . Before rotation the score is .
- Query at position 2 turns by 90° to . Key at position 1 turns by 45° to . The score is .
- Move both two positions later, to 4 and 3. The query becomes and the key , which are new directions, and the score is again.
- At positions 6 and 5 the score is 2.121 once more, because the offset is still 1.
- Move only the key back to position 4. The offset becomes 2, the angle between the vectors opens from 75° to 120°, and the score falls to . At positions 6 and 6 the rotations cancel and the score returns to 7.
A real head rotates many pairs at different speeds. Llama 2 uses RoPE with head width 128, so 64 pairs, and pair turns by radians per position. The fastest pair turns 1 radian per token and repeats every 6.3 tokens, while the slowest turns 0.000115 radians per token and repeats only every 54,410 tokens. Fast pairs separate neighbours and slow pairs separate distant tokens, the same clock design as section 7. RoPE rotates only queries and keys, so the values carry content alone, and it is applied in every layer instead of once at the input.
9. ALiBi and length extrapolation
Section titled “9. ALiBi and length extrapolation”A simpler idea is to tell each head “trust nearby words more” directly. A person skimming a long thread gives a reply right above more weight than one twenty messages back, and does so the same way whether the thread has 50 messages or 500. ALiBi, attention with linear biases from Press and colleagues, adds no position vector at all. It subtracts a penalty proportional to distance from each score, , where is the head’s slope. Different heads get different slopes, a geometric series for 8 heads, so some heads stay local and some look far.
Work the slopes through. A head with slope subtracts 2.5 from a key five tokens back, which multiplies that key’s unnormalized weight by . A head with slope subtracts 0.625 at the same distance, a factor of 0.54. The penalty is a simple rule defined at every distance, so a sequence longer than training needs nothing new. The paper reports a model trained on 1,024 tokens that matched, at 2,048 tokens, a sinusoidal model trained on 2,048.

Length extrapolation is a model’s ability to work on inputs longer than its trained context length , and most schemes are poor at it. A learned table has no rows past its end. Sinusoidal and rotary encodings still produce numbers, but the slow pairs reach angles training never showed. With rotary position embedding (RoPE) in the 7B model, the slowest pair has turned at most radians by the end of training, and at position 16,384 it reaches 1.89 radians. The attention layers have never seen keys rotated that far, so scores become erratic. Perplexity, the exponential of the average per-token prediction loss defined in Language Modeling and Next Token Prediction, typically shoots up within a few hundred tokens past .
Position interpolation, from Chen and colleagues, avoids the unseen angles by squeezing positions. To run at 16,384 tokens, multiply every position by , so position 16,000 rotates like position 4,000, a range the model knows. Neighbours now differ by a quarter step, so the model needs a short fine-tuning run, meaning more training on the new setting, and the paper reports about 1,000 steps. ALiBi extrapolates more gracefully but its far heads still attend weakly to distant tokens, so most current open models use RoPE with some form of interpolation. Positional Encoding and Long Context and Document Level LLMs cover the extension methods in depth.
10. Quadratic cost and the key value cache
Section titled “10. Quadratic cost and the key value cache”Attention compares every token with every other token, so a text twice as long has four times as many pairs. A round of introductions at a party with 10 guests takes 45 handshakes, and with 20 guests it takes 190. For tokens one head computes scores, or under the causal mask, which is still on the order of . At that is 16,777,216 scores per head per layer.
Storing the full score table for one layer of the 7B model takes heads scores bytes GiB, where a GiB is bytes. At 32,768 tokens it takes times as much, 64 GiB per layer. FlashAttention computes the same result in small tiles without ever storing the table, which Efficient Attention Mechanisms explains, but the arithmetic stays quadratic.
The arithmetic shows up when the model writes a new token. A FLOP is one floating-point arithmetic operation, and a multiply followed by an add counts as two. The new token’s query is compared with every earlier key and averages every earlier value, costing FLOPs each, so per layer. The layer’s weights cost a fixed million FLOPs per token. At 4,096 tokens attention adds 67 million, 17% of the weight work. At 32,768 it adds 537 million, 133%, and the two are equal at about 24,705 tokens.

Memory grows too, because the model keeps every earlier key and value instead of recomputing them for each new token. This store is the key value cache. Per token it holds bytes, where 2 counts keys and values, layers, key value heads, and bytes. That is 524,288 bytes, or 0.5 MiB per token, where a MiB is bytes. A 4,096-token conversation needs 2 GiB and a 32,768-token one needs 16 GiB, more than the model’s 12.55 GiB of weights. Inference and Decoding shows how this cache, more than arithmetic, limits generation speed.
11. Head sharing and design selection
Section titled “11. Head sharing and design selection”The cache in section 10 stores one key and one value for every head, but the heads do not need private copies. A study group can share one set of library books and still take different notes from them. Multi-query attention, or MQA from Shazeer, keeps all the query heads but gives them one shared key head and value head, so . Grouped-query attention, or GQA from Ainslie and colleagues, sits between: query heads are split into groups that each share one key value head. With the 7B shapes and the cache drops from 0.5 MiB to 128 KiB per token, so 512 MiB at 4,096 tokens. With MQA it is 16 KiB per token. Llama 2 70B uses 8 key value heads, while the 7B model keeps full multi-head attention. The score arithmetic does not shrink, because every query head still reads every cached key, and Efficient Attention Mechanisms works through the quality tradeoff.
The choices in this note reduce to a few decisions a model designer actually makes.
| Situation | Choice | Reason |
|---|---|---|
| New decoder model trained from scratch | Scaled dot-product attention, causal mask, RoPE | Relative positions and a well-tested route to longer context |
| Need to serve contexts 4 times longer than | RoPE with position interpolation and a short fine-tune | Keeps every rotation angle inside the trained range |
| Must handle longer inputs with no further training | ALiBi or a similar distance bias | The penalty is defined at any distance |
| Generation memory dominated by the cache | GQA with about 8 key value heads | Cuts cache size by with small quality loss |
| Training loss suspiciously low | Future perturbation test on the mask | A leak lets the model copy the answer |
| Attention arithmetic dominates at long context | FlashAttention or sparse attention, see 14a | The pair count grows with |