llmdeepdive
PT

Type to search

    Learning track

    The Transformer

    Assemble the architecture behind modern LLMs, beginning with self-attention and its query, key, and value operations.

    05

    core · 17 available lessons

    The Transformer

    Assemble the architecture behind modern LLMs, beginning with self-attention and its query, key, and value operations.

    1. 05.01"Attention Is All You Need" in context
    2. 05.02Self-attention from first principles: Q, K, V
    3. 05.03Scaled dot-product attention & the √d_k
    4. 05.04Causal masking & why order matters
    5. 05.05Multi-head attention
    6. 05.06Positional encoding I: sinusoidal & learned
    7. 05.07Positional encoding II: RoPE
    8. 05.08Positional encoding III: ALiBi & relative bias
    9. 05.09The feed-forward block & where knowledge lives
    10. 05.10Residuals, pre-norm vs post-norm
    11. 05.11The classic block — and Qwen's two block types
    12. 05.12Transformer families, from encoder-only to hybrid
    13. 05.13Build a GPT from scratch: the uniform baseline
    14. 05.14Reading real weights: opening Qwen3.8-27B's checkpoint
    15. 05.15Gated DeltaNet: the mixer in 48 of 64 layers
    16. 05.16Hybrid layouts: three DeltaNet, one attention
    17. 05.17How images join the sequence: the vision encoder