llmdeepdive

Type to search

    Curriculum

    Learning tracks

    A connected path from first principles to modern language-model systems.

    foundations · 8 available lessons

    Orientation & Foundations

    Build an accurate mental model of language models and the mathematics needed for everything that follows.

    1. What is a language model, really?
    2. A 60-second history: n-grams to the GPT-5 era
    3. Probability you actually need
    4. Vectors, matrices, and dot products
    5. Matrix multiplication as transformation
    6. Derivatives and gradients, geometrically
    7. Python, NumPy, and PyTorch in 20 minutes
    8. How to read an ML paper

    foundations · 7 available lessons

    From Text to Tensors

    See how text becomes tokens, vectors, and numerical structures a neural network can process.

    1. Why computers can't read
    2. Tokenization I: characters, words, and subwords
    3. Tokenization II: BPE, step by step
    4. Tokenization III: SentencePiece, byte-level, and why token counts bite
    5. Embeddings: meaning as geometry
    6. word2vec, GloVe, and the analogy trick
    7. Entropy, perplexity, and what 'good' means

    foundations · 9 available lessons

    Neural Network Fundamentals

    Learn how neural networks represent functions, measure error, propagate gradients, and improve through optimization.

    1. The perceptron
    2. Multilayer perceptrons and non-linearity
    3. Activation functions: ReLU, GELU, and SwiGLU
    4. Loss functions and cross-entropy
    5. Backpropagation, visualized
    6. Gradient descent and the loss landscape
    7. Optimizers: SGD to Momentum to Adam to AdamW
    8. Initialization, normalization, and residuals
    9. Overfitting, regularization, and the bitter lesson

    core · 5 available lessons

    Sequence Models

    Follow the path from recurrent state to encoder-decoder models and attention, the immediate ancestors of the Transformer.

    1. Modelling sequences: the setup
    2. RNNs and the vanishing gradient
    3. LSTM and GRU: gates that remember
    4. Seq2seq, encoder-decoder, and the bottleneck
    5. Bahdanau attention: the idea that changed everything

    core · 14 available lessons

    The Transformer

    Assemble the architecture behind modern LLMs, beginning with self-attention and its query, key, and value operations.

    1. "Attention Is All You Need" in context
    2. Self-attention from first principles: Q, K, V
    3. Scaled dot-product attention & the √d_k
    4. Causal masking & why order matters
    5. Multi-head attention
    6. Positional encoding I: sinusoidal & learned
    7. Positional encoding II: RoPE
    8. Positional encoding III: ALiBi & relative bias
    9. The feed-forward block & where knowledge lives
    10. Residuals, pre-norm vs post-norm
    11. The full block, assembled
    12. Encoder-only, decoder-only, encoder–decoder
    13. Build a GPT from scratch, annotated
    14. Reading real weights: what a trained model looks like

    advanced · 13 available lessons

    Pretraining at Scale

    Design the objectives, data, optimization, parallelism, and cost model behind a serious language-model pretraining run.

    1. Objectives: causal LM, MLM, span corruption, FIM
    2. Data: sourcing, filtering, dedup, decontamination
    3. Training your own tokenizer
    4. Scaling laws I: Kaplan
    5. Scaling laws II: Chinchilla & compute-optimality
    6. Scaling laws III: inference-aware & over-training
    7. Data parallelism, ZeRO & FSDP
    8. Tensor, pipeline & sequence parallelism
    9. Mixed precision: fp16, bf16, fp8
    10. Gradient checkpointing & memory maths
    11. Learning-rate schedules & warmup
    12. When training goes wrong: spikes, divergence, NaNs
    13. What a real pretraining run costs

    advanced · 11 available lessons

    Post-training & Alignment

    Turn pretrained predictors into useful assistants through instruction tuning, preference learning, reinforcement learning, and distillation.

    1. Why base models aren't assistants
    2. Supervised fine-tuning & instruction data
    3. LoRA, QLoRA & PEFT
    4. Reward models & human preference data
    5. RLHF with PPO
    6. DPO: skipping the reward model
    7. ORPO, KTO, SimPO & the alignment zoo
    8. GRPO & RL on verifiable rewards
    9. Reasoning models: test-time compute & long CoT
    10. Constitutional AI & RLAIF
    11. Distillation: making small models punch up

    advanced · 12 available lessons

    Inference & Efficiency

    Understand how trained models generate text and how decoding choices trade off diversity, coherence, latency, and cost.

    1. What actually happens when you hit send
    2. The KV cache
    3. Prefill vs decode: two different machines
    4. Sampling: temperature, top-k, top-p, and min-p
    5. Beam search, speculative decoding & Medusa
    6. MQA, GQA & MLA
    7. FlashAttention & IO-awareness
    8. PagedAttention & continuous batching
    9. Quantization I: int8, int4, the basics
    10. Quantization II: GPTQ, AWQ, GGUF, QAT
    11. Serving stacks: vLLM, SGLang, TensorRT-LLM
    12. Running models locally