1 min lesson

From one head to a block

Explain the practical point behind "Multi-head attention runs several of these in parallel on lower-dimensional slices, then concatenates and projects back."

From one head to a blockmulti-head, residuals, norm placement, FFN

Multi-head attention runs several of these in parallel on lower-dimensional slices, then concatenates and projects back. Splitting the representation lets different heads specialize, one tracking syntax, another long-range references, without growing the total compute.

Learn more

Full explanation

One Transformer Block, Layer by Layer

ONE TRANSFORMER BLOCK, LAYER BY LAYER

Token + position: Tokens are embedded, then position is injected - RoPE rotates Q/K by a position-dependent angle so attention depends on relative offset, which is why long agent sessions extrapolate past the trained length.

diagram: stack

Each sublayer adds to a residual stream that runs straight through the block.

The block is: x = x + attn(norm(x)) then x = x + ffn(norm(x)). Post-norm wraps the norm around the residual sum instead, which was the original design and is harder to train deep without careful warmup. If you say "pre-norm" and can explain why (gradient stability at depth), that reads as someone who has actually trained these.

From one head to a block

Step 1 of 3

From one head to a blockmulti-head, residuals, norm placement, FFN

One Transformer Block, Layer by Layer