Transformers and Attention from Scratch
To predict the next word, a language model has to pull information from earlier words: in "the animal was tired, so it lay down", the vector for "it" needs to know about "animal". Attention is the layer that does that. This page follows a piece of text from raw characters to token vectors, through attention and the rest of a transformer block, to the key-value cache that keeps generation fast, with every number computed live.
- Explain how a tokenizer turns text into IDs, and learn a small BPE vocabulary by hand.
- Compute one attention step from queries, keys and values, including the √dk scaling.
- Apply a causal mask correctly, and spot the two common ways to get it wrong.
- Explain how rotary position embeddings make attention depend on the distance between tokens.
- Size a key-value cache in bytes and line up its causal mask with absolute positions.
01Text becomes a sequence of vectors
A model can't read characters. It needs numbers, and it needs a fixed, finite set of pieces to assign them to. The tokenizer does that job: it splits text into tokens, the pieces in the model's vocabulary, and replaces each with an integer ID. A token can be a whole word, a fragment of one, a single character or a byte sequence.
Each ID then picks one row from a learned embedding table: a list of numbers that training arranges so that related tokens get similar vectors. The Embeddings page shows how such vectors are learned. From here on, the text is a sequence of vectors, one per token.
Where do the pieces come from? BPE (byte pair encoding) is one common way to learn them from data. In Sennrich et al.'s character-based version, each word starts as single characters plus an end-of-word marker. Count every pair of neighbouring symbols across the training words, weighted by how often each word appears. Merge the most frequent pair into one new symbol, then count again and repeat. The list of merges, in order, becomes part of the tokenizer.[1]
Worked example
Training words: low, low, lower, newest, widest. They start as l o w </w> (twice), l o w e r </w>, n e w e s t </w> and w i d e s t </w>: 28 symbols in all.
The pairs (l, o) and (o, w) each appear 3 times, more than any other pair. A tie-break rule picks (l, o), so every "l o" becomes "lo", leaving 25 symbols. Recount: (lo, w) now appears 3 times, so the second merge makes "low". After two merges "low" is a single token and "lower" is low e r </w>.
The widget shows training. Tokenizing new text later doesn't relearn anything: it applies the stored merges, in their learned order, to the new words.
The integer ID itself carries no meaning. Token 80 isn't "twice" token 40; the IDs are just row numbers. Meaning lives in the embedding vectors, and later layers adjust those vectors using the surrounding text, so the same token can end up with different hidden representations (the vectors the model carries for each position) in different sentences.
Tokenization is part of the model's contract. Feed a model IDs from a different vocabulary or merge table and it reads the wrong embedding rows. Context limits are counted in tokens, not characters, and the same number of characters can take very different numbers of tokens depending on the language and content, so a character budget doesn't guarantee a token budget.
02Attention: each position gathers from the others
In "the animal was tired, so it lay down", the vector for "it" needs information from "animal", several tokens back. A dense layer can't move it there: it transforms each position's vector on its own. is the layer that moves information between positions, and it decides how much to take from each position based on the input itself.
Every position produces three vectors from its hidden representation, each through its own learned projection (a matrix multiply):
- A query: what this position is looking for.
- A key: what this position offers to be matched against.
- A value: the content this position hands over if it's chosen.
For one position, take the dot product of its query with every key. A high score means a good match. A softmax turns the row of scores into weights that are positive and sum to 1, and the output is the weighted average of the values.[2]
Worked example
Three tokens, "the cat sat", with 2-number vectors. The query for "sat" is (2, 0). The keys are the = (0, 1), cat = (2, 0), sat = (1, 1), and the values are the = (0, 0), cat = (10, 0), sat = (0, 10).
Scores: (2, 0) · each key = 0, 4, 2. Divide by √2 (the key width is 2): 0, 2.83, 1.41.
Softmax: weights 0.045, 0.768, 0.187. Output: 0.768 × (10, 0) + 0.187 × (0, 10) = (7.68, 1.87), mostly the value of "cat".
Done for all positions at once, with the vectors stacked as rows of matrices, that's the whole operation:
A = softmax(QKᵀ / √dk)
Y = AV
In words: project every token into a query, a key and a value. Score every query against every key, scale the scores down by √dk, and softmax each row into weights. Each output row is the weighted average of the value rows. With N tokens, A is an N×N grid.
Why divide by √dk? A dot product adds up dk products, so it grows with the vector width. If the entries were independent random numbers with mean 0 and variance 1, the dot product would have variance dk. Scores that large push the softmax to put nearly all its weight on one key, and the other keys then get almost no gradient to learn from. Dividing by √dk brings the spread back to about 1.[2] Real learned vectors don't follow those assumptions exactly; the argument explains the size of the factor.
Worked example
dk = 64, so random dot products have a standard deviation of √64 = 8, and a score gap of 16 is unremarkable. A softmax over (16, 0) gives the first key 0.9999999. Divided by 8 the scores are (2, 0), and the softmax gives 0.88 and 0.12.
The names query, key and value describe each vector's role in the calculation. Training doesn't have to give them any meaning a person would recognize. Attention weights aren't probabilities that something is true. They're the coefficients of a weighted average: a large weight just means that position's value counts for more in this output.
Multi-head attention runs several of these mixtures side by side, each with its own projections and a smaller width, then concatenates their outputs and applies one more projection. The total width can stay the same as with a single head. Different heads can learn different relations, but nothing in the architecture guarantees a head a fixed human-readable job such as "the grammar head".[2]
03Hide the future before the softmax
A language model trains on ordinary text by predicting each next token from the ones before it, and it makes all those predictions in one pass. If position 2 could attend to position 3, it could read the answer instead of predicting it. prevents that: each position may attend to itself and to earlier positions, never later ones.[2]
The exclusion has to happen in the scores, before the softmax. Set each forbidden score to −∞; its exponential is 0, so it gets no weight and the allowed weights still sum to 1. Two tempting shortcuts are wrong. Setting a forbidden score to 0 still gives it weight, because exp(0) = 1. Zeroing weights after the softmax leaves the remaining weights summing to less than 1.
Worked example
A query's scores against tokens A, B and the future token C are 1.0, 2.0 and 3.0.
Mask C with −∞: the softmax over (1.0, 2.0) gives A 0.27 and B 0.73, and C gets 0.
Replace C's score with 0 instead: the softmax over (1.0, 2.0, 0) gives 0.245, 0.665 and 0.09, so 9% of the output still comes from the future token.
A position may attend to itself because the targets are shifted by one. For the input A, B, C the targets are B, C, D: the position holding B is asked to predict C, so reading A and B is fair. Padding adds a second rule. When short sequences are padded to a common length, a separate mask marks the padding positions, and it's combined with the causal rule so padding gets no weight either.
In training, every position is computed in parallel under the mask. Generation is different: the next token isn't known until the model has produced the previous one, so autoregressive decoding still runs one token at a time.[2] Softmax over a row with every score masked is undefined, so an implementation that can produce such a row needs an explicit rule for it.
04Telling the model where each token is
"dog bites man" and "man bites dog" contain the same tokens. Attention as described so far can't tell them apart: shuffle the input rows of an unmasked attention layer and the output rows come back shuffled the same way, with the same values. That property is called being equivariant to reordering, and it means word order has to be put in some other way.
The original transformer adds a position vector to each token's embedding. Other designs put position inside the attention calculation itself.[2] The causal mask also imposes an order, since each position sees a different prefix, but that limits what each position can see; it isn't an explicit position signal.
(RoPE) is one way to do the second. It splits each query and key into pairs of coordinates and rotates each pair by an angle proportional to the token's position, θ radians per position for that pair. Rotation never changes a vector's length. What changes is the angle between a query and a key, and that angle depends only on how far apart the two tokens are.[3]
Worked example
Query = key = (1, 0), θ = 0.4 radians per position.
Positions 3 and 5: the query turns by 1.2 radians, the key by 2.0. They end up 0.8 radians apart, so the dot product is cos 0.8 = 0.697.
Positions 13 and 15: turns of 5.2 and 6.0, still 0.8 apart, still 0.697. Positions 3 and 7: 1.6 radians apart, and the dot product drops to −0.029.
The equation says the same thing for any query q at position m and key k at position n, with R(angle) a 2D rotation.
In words: rotating the query by its position and the key by its position gives the same score as leaving the query alone and rotating the key by the gap between them. Only n − m survives, so the score depends on relative position. A full head uses many coordinate pairs, each with its own θ.
Why the identity holds
Rotations compose by adding angles, and the transpose of a rotation is the rotation by the negative angle. So R(mθ)ᵀR(nθ) = R(−mθ)R(nθ) = R((n−m)θ), and (R(mθ)q)ᵀ(R(nθ)k) = qᵀR(mθ)ᵀR(nθ)k = qᵀR((n−m)θ)k. The widget computes both sides separately and shows they agree to rounding error.[3]
RoPE doesn't give a trained model unlimited context. The formula accepts any position, but a model can behave poorly at positions far beyond the lengths it was trained on. A cache also has to use the same position numbering as the original pass: restarting positions at zero for a cached query changes its relation to every stored key.
In a full model the vectors being rotated already depend on their context. The identity is about rotating fixed vectors; it doesn't claim that moving words around leaves those vectors unchanged.
05Attention is one part of a transformer block
Attention moves information between positions, but it only averages values. A transformer block follows it with a position-wise feed-forward network: a small two-layer network that processes each position's vector on its own, using the same weights at every position. Attention gathers; the feed-forward network computes on what was gathered.[2]
Each of those two sublayers is wrapped in a residual addition: the block adds the sublayer's output to its input rather than replacing the input, so the original signal always has a direct path through the stack.
The block also keeps the numbers in a steady range. As vectors pass through dozens of layers, their overall size can drift. Layer normalization rescales each token's vector on its own: subtract the vector's mean, divide by its standard deviation (the square root of its variance), then apply a learned per-feature gain and bias. Each token gets its own statistics. Unlike batch normalization in training, it never borrows statistics from other examples in the batch.[4]
Worked example
One token's vector: (1, 2, 3, 5). Mean 2.75. Deviations −1.75, −0.75, 0.25, 2.25; their squares average to a variance of 2.1875, a standard deviation of 1.479.
Normalized: (−1.18, −0.51, 0.17, 1.52), mean 0 and variance 1. Add 10 to every entry, or multiply every entry by 3, and the normalized vector comes out the same, apart from a tiny effect of the ε below.
outputi = gaini × normalizedi + biasi
In words: centre the token's features on zero and scale them to unit variance, then let a learned gain and bias stretch and shift each feature. The variance averages squared deviations over the feature count. Because of the gain and bias, which form a learned affine step, the final output doesn't have to keep mean 0 or variance 1.
The ε makes the scale invariance slightly inexact: multiplying the input by a constant cancels almost completely when the variance is much bigger than ε, but not perfectly. Adding a constant to every feature cancels exactly in the centring step.
Where the normalization sits matters. The original transformer uses post-LN, LayerNorm(x + F(x)): add the sublayer's output, then normalize. A pre-LN block uses x + F(LayerNorm(x)): normalize only the sublayer's input, and let the residual path skip the normalization. They compute different functions with different gradient paths, which Xiong et al. study in detail, so weights trained with one can't be loaded into the other.[5]
06Reuse keys and values during generation
A chatbot writing a 500-token reply generates one token at a time. Done naively, each new token reruns the whole model over the entire prompt and everything generated so far, recomputing hundreds of vectors that haven't changed. The stores those vectors instead.[6]
It works because of the causal mask. A token never attends to later tokens, so appending a new token doesn't change any earlier token's hidden states, keys or values, as long as positions keep the same numbering. For each new token: compute its query, key and value; append the key and value to the cache; attend over everything in the cache. This matches a full recomputation only when the weights are fixed and inference-time dropout (random zeroing used during training) is switched off.[6]
Worked example
Generate up to 7 tokens. Without a cache, step t reprocesses all t tokens: 1 + 2 + … + 7 = 28 token projections in total. With a cache, each step projects only the new token: 7 in total. The widget's readout counts the same thing.
The cache saves recomputation. It doesn't shrink attention: the new query still reads every cached key and value, so each step's attention work still grows with the length of the context. Processing the prompt, called prefill, can handle all the known tokens at once; the decoding steps after it commonly handle one new token per sequence.
The cache costs memory, and the amount is easy to compute. For an ordinary dense cache, the payload is:
In words: every layer stores one key and one value (the factor 2) per head, per token, per sequence, each head width numbers long. Allocator overhead and metadata come on top.
Worked example
A model with 32 layers, 8 KV heads of width 128 and 16-bit entries: 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 128 KiB, per token. An 8,192-token conversation needs 1 GiB of cache. With 32 KV heads instead of 8, it needs 4 GiB.
Grouped-query attention gets that reduction by letting several query heads share one key/value head. It changes the model, so the model has to be trained or adapted that way; you can't reinterpret the cache of an ordinary multi-head model as a grouped one.[7]
The causal mask also needs care with a cache. A query in a cached chunk has an absolute position. When a 100-token prompt (positions 0 to 99) is cached and one new token arrives, that token is row 0 of a one-row chunk but sits at absolute position 100, so its mask must allow keys 0 through 100. A small triangular mask built for the chunk alone allows only key 0, the first cached key, and the code still runs. The code below makes the offset explicit.[6]
Full attention over N tokens scores every pair, so prefill does work that grows with N². Storing those scores as an N×N matrix is a choice, not a requirement: for N = 8,192 that's 67 million entries per head. FlashAttention computes exact attention in tiles and never writes the full matrix to the GPU's main memory, which cuts memory traffic. The equation and the number of score pairs stay the same; only the storage and scheduling of the work change.[8]
07A causal attention kernel with a cache offset
These complete programs implement one attention head on 2-number queries, keys and values. They apply the causal mask, with a cache offset, before the softmax, and they subtract the largest allowed score before exponentiating, so exp never sees a large positive number. Each program then checks three things: the first row, which may see only one key, returns that key's value; a cached query matches the full computation; and changing a future value leaves an earlier row unchanged.
#include <algorithm>
#include <array>
#include <cassert>
#include <cmath>
#include <vector>
using Vector = std::array<double,2>;
using Sequence = std::vector<Vector>;
Sequence attention(const Sequence& queries, const Sequence& keys,
const Sequence& values, int queryOffset = 0) {
assert(keys.size() == values.size() && !keys.empty());
Sequence output(queries.size(), Vector{0,0});
for (int row = 0; row < static_cast<int>(queries.size()); ++row) {
// queryOffset is the number of cached tokens before this query chunk.
const int lastKey = row + queryOffset;
assert(lastKey >= 0 && lastKey < static_cast<int>(keys.size()));
std::vector<double> scores(lastKey+1);
for (int column = 0; column <= lastKey; ++column)
scores[column] = (queries[row][0]*keys[column][0]
+ queries[row][1]*keys[column][1])/std::sqrt(2.0);
const double maximum = *std::max_element(scores.begin(), scores.end());
double denominator = 0;
for (double& score : scores) {
score = std::exp(score-maximum); // Prevent large positive exponentials.
denominator += score;
}
for (int column = 0; column <= lastKey; ++column)
for (int feature = 0; feature < 2; ++feature)
output[row][feature] += scores[column]/denominator * values[column][feature];
}
return output;
}
int main() {
const Sequence queries{{1,0},{0,1},{1,1}};
Sequence values{{2,0},{0,3},{-1,1}};
const auto full = attention(queries,queries,values);
assert(std::abs(full[0][0]-2.0) < 1e-12); // First token has one legal key.
const auto cached = attention(Sequence{queries.back()},queries,values,2);
for (int feature = 0; feature < 2; ++feature)
assert(std::abs(full.back()[feature]-cached[0][feature]) < 1e-12);
values.back() = {100,100};
const auto changed = attention(queries,queries,values);
assert(full[1] == changed[1]); // Future values cannot influence row one.
}
type Vector = [f64;2];
fn attention(queries: &[Vector], keys: &[Vector], values: &[Vector],
query_offset: usize) -> Vec<Vector> {
assert!(keys.len() == values.len() && !keys.is_empty());
let mut output = vec![[0.0;2];queries.len()];
for row in 0..queries.len() {
// query_offset is the number of cached tokens before this query chunk.
let last_key = row + query_offset;
assert!(last_key < keys.len());
let mut scores = Vec::with_capacity(last_key+1);
for key in &keys[..=last_key] {
scores.push((queries[row][0]*key[0]+queries[row][1]*key[1])/2.0_f64.sqrt());
}
let maximum = scores.iter().copied().fold(f64::NEG_INFINITY,f64::max);
let mut denominator = 0.0;
for score in &mut scores {
*score = (*score-maximum).exp(); // Prevent large positive exponentials.
denominator += *score;
}
for column in 0..=last_key { for feature in 0..2 {
output[row][feature] += scores[column]/denominator * values[column][feature];
}}
}
output
}
fn main() {
let queries = [[1.0,0.0],[0.0,1.0],[1.0,1.0]];
let mut values = [[2.0,0.0],[0.0,3.0],[-1.0,1.0]];
let full = attention(&queries,&queries,&values,0);
assert!((full[0][0]-2.0).abs() < 1e-12); // First token has one legal key.
let cached = attention(&[queries[2]],&queries,&values,2);
for feature in 0..2 {
assert!((full[2][feature]-cached[0][feature]).abs() < 1e-12);
}
values[2] = [100.0,100.0];
let changed = attention(&queries,&queries,&values,0);
assert_eq!(full[1],changed[1]); // Future values cannot influence row one.
}
This reference kernel leaves out learned projection matrices, multiple heads, position encoding, padding masks, batching and fast matrix multiplication. It defines the mixing step and nothing more. The caller must pass finite vectors whose scores stay finite, and every row needs at least one allowed key: subtracting the maximum can't rescue a dot product that has already overflowed. A complete decoder also needs a token embedding, a stack of blocks, and an output projection onto the vocabulary, trained to predict the shifted targets.
When you build the block stack, keep tensor shapes and the normalization order exactly as the reference model has them. Apply the position encoding to new queries and keys using their absolute positions, store keys in the same form the reference pass uses (after rotation, if that's where it stores them), and compare the whole decoded output against an uncached run. A cached implementation can return plausible numbers while attending to the wrong tokens.
08Check meaning as well as tensor shapes
Attention weights describe one internal mixing step. A confident-looking map doesn't show that the output is factually right, that the model's uncertainty is calibrated, or why the final model picked a token.
Test a kernel with small, lopsided vectors. If all the keys are equal, every weight is the same, and a transposed matrix or a broken mask gives the same answer as a correct one. Change a forbidden future value and assert that earlier outputs don't move. Compare cached and uncached results for several prompt lengths, for chunks of more than one token, and for nonzero position offsets.
Check the mask convention every time you call a library. One API treats true as "allowed" and another treats it as "masked", so a correct boolean array with the opposite meaning silently computes a different model. Read the library's contract and check one case by hand.
Correct arithmetic isn't the same as useful predictions. These widgets compute exact attention on small arrays whose values were never trained on language. Building the layer gets the numbers right; useful predictions need suitable data, a training objective and an evaluation on the tasks you care about.
09What's next
The capstone, Train a Language Model, puts this block to work: a character-level GPT trained in the browser with every gradient written by hand, from the data pipeline through sampling. After it, Reinforcement Learning changes the source of the learning signal from a target token to a reward, which is what alignment training needs.
10Sources
The cited equations define the algorithms. Widget readouts report the displayed toy arrays and settings; they are not hardware benchmarks.
- Rico Sennrich, Barry Haddow, Alexandra Birch, 2016. Neural Machine Translation of Rare Words with Subword Units. Frequency-based subword merges and a word-end marker.
- Ashish Vaswani et al., 2017. Attention Is All You Need. Scaled dot-product attention, multiple heads, causal masks, and the original transformer block.
- Jianlin Su et al., 2021. RoFormer: Enhanced Transformer with Rotary Position Embedding. Query/key rotations and relative-position dot products.
- Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton, 2016. Layer Normalization. Feature normalization and learned gain and bias.
- Ruibin Xiong et al., 2020. On Layer Normalization in the Transformer Architecture. The distinction between pre-LN and post-LN blocks.
- Hugging Face Transformers contributors, cache documentation. How caching works. Key/value reuse, cache dimensions, and attention-mask alignment.
- Joshua Ainslie et al., 2023. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. Groups of query heads sharing key/value heads.
- Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré, 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. Tiled exact attention without materializing a full score matrix.