The Transformer

core · 14 available lessons

The Transformer

Assemble the architecture behind modern LLMs, beginning with self-attention and its query, key, and value operations.

1. "Attention Is All You Need" in context
2. Self-attention from first principles: Q, K, V
3. Scaled dot-product attention & the √d_k
4. Causal masking & why order matters
5. Multi-head attention
6. Positional encoding I: sinusoidal & learned
7. Positional encoding II: RoPE
8. Positional encoding III: ALiBi & relative bias
9. The feed-forward block & where knowledge lives
10. Residuals, pre-norm vs post-norm
11. The full block, assembled
12. Encoder-only, decoder-only, encoder–decoder
13. Build a GPT from scratch, annotated
14. Reading real weights: what a trained model looks like