llmdeepdive
PT

Type to search

    Curriculum

    Learning tracks

    A connected path from first principles to modern language-model systems.

    01

    foundations · 8 available lessons

    Orientation & Foundations

    Build an accurate mental model of language models and the mathematics needed for everything that follows.

    Open track
    1. 01.01What is a language model, really?
    2. 01.02A 60-second history: n-grams to the GPT-5 era
    3. 01.03Probability you actually need
    4. 01.04Vectors, matrices, and dot products
    5. 01.05Matrix multiplication as transformation
    6. 01.06Derivatives and gradients, geometrically
    7. 01.07Python, NumPy, and PyTorch in 20 minutes
    8. 01.08How to read an ML paper
    02

    foundations · 7 available lessons

    From Text to Tensors

    See how text becomes tokens, vectors, and numerical structures a neural network can process.

    Open track
    1. 02.01Why computers can't read
    2. 02.02Tokenization I: characters, words, and subwords
    3. 02.03Tokenization II: BPE, step by step
    4. 02.04Tokenization III: SentencePiece, byte-level, and why token counts bite
    5. 02.05Embeddings: meaning as geometry
    6. 02.06word2vec, GloVe, and the analogy trick
    7. 02.07Entropy, perplexity, and what 'good' means
    03

    foundations · 9 available lessons

    Neural Network Fundamentals

    Learn how neural networks represent functions, measure error, propagate gradients, and improve through optimization.

    Open track
    1. 03.01The perceptron
    2. 03.02Multilayer perceptrons and non-linearity
    3. 03.03Activation functions: ReLU, GELU, and SwiGLU
    4. 03.04Loss functions and cross-entropy
    5. 03.05Backpropagation, visualized
    6. 03.06Gradient descent and the loss landscape
    7. 03.07Optimizers: SGD to Momentum to Adam to AdamW
    8. 03.08Initialization, normalization, and residuals
    9. 03.09Overfitting, regularization, and the bitter lesson
    04

    core · 6 available lessons

    Sequence Models

    Follow the path from recurrent state to encoder-decoder models and attention, the immediate ancestors of the Transformer.

    Open track
    1. 04.01Modelling sequences: the setup
    2. 04.02RNNs, fixed-size state, and the vanishing gradient
    3. 04.03LSTM and GRU: gates that remember
    4. 04.04Seq2seq, encoder-decoder, and the bottleneck
    5. 04.05Bahdanau attention: the idea that changed everything
    6. 04.06Linear attention and fast weights
    05

    core · 17 available lessons

    The Transformer

    Assemble the architecture behind modern LLMs, beginning with self-attention and its query, key, and value operations.

    Open track
    1. 05.01"Attention Is All You Need" in context
    2. 05.02Self-attention from first principles: Q, K, V
    3. 05.03Scaled dot-product attention & the √d_k
    4. 05.04Causal masking & why order matters
    5. 05.05Multi-head attention
    6. 05.06Positional encoding I: sinusoidal & learned
    7. 05.07Positional encoding II: RoPE
    8. 05.08Positional encoding III: ALiBi & relative bias
    9. 05.09The feed-forward block & where knowledge lives
    10. 05.10Residuals, pre-norm vs post-norm
    11. 05.11The classic block — and Qwen's two block types
    12. 05.12Transformer families, from encoder-only to hybrid
    13. 05.13Build a GPT from scratch: the uniform baseline
    14. 05.14Reading real weights: opening Qwen3.8-27B's checkpoint
    15. 05.15Gated DeltaNet: the mixer in 48 of 64 layers
    16. 05.16Hybrid layouts: three DeltaNet, one attention
    17. 05.17How images join the sequence: the vision encoder
    06

    advanced · 13 available lessons

    Pretraining at Scale

    Design the objectives, data, optimization, parallelism, and cost model behind a serious language-model pretraining run.

    Open track
    1. 06.01Objectives: causal LM, MLM, span corruption, FIM
    2. 06.02Data: sourcing, filtering, dedup, decontamination
    3. 06.03Training your own tokenizer
    4. 06.04Scaling laws I: Kaplan
    5. 06.05Scaling laws II: Chinchilla & compute-optimality
    6. 06.06Scaling laws III: inference-aware & over-training
    7. 06.07Data parallelism, ZeRO & FSDP
    8. 06.08Tensor, pipeline & sequence parallelism
    9. 06.09Mixed precision: fp16, bf16, fp8
    10. 06.10Gradient checkpointing & memory maths
    11. 06.11Learning-rate schedules & warmup
    12. 06.12When training goes wrong: spikes, divergence, NaNs
    13. 06.13What a real pretraining run costs
    07

    advanced · 11 available lessons

    Post-training & Alignment

    Turn pretrained predictors into useful assistants through instruction tuning, preference learning, reinforcement learning, and distillation.

    Open track
    1. 07.01Why base models aren't assistants
    2. 07.02Supervised fine-tuning & instruction data
    3. 07.03LoRA, QLoRA & PEFT
    4. 07.04Reward models & human preference data
    5. 07.05RLHF with PPO
    6. 07.06DPO: skipping the reward model
    7. 07.07ORPO, KTO, SimPO & the alignment zoo
    8. 07.08GRPO & RL on verifiable rewards
    9. 07.09Reasoning models: test-time compute & long CoT
    10. 07.10Constitutional AI & RLAIF
    11. 07.11Distillation: making small models punch up
    08

    advanced · 13 available lessons

    Inference & Efficiency

    Understand how trained models generate text and how decoding choices trade off diversity, coherence, latency, and cost.

    Open track
    1. 08.01What actually happens when you hit send
    2. 08.02The KV cache
    3. 08.03Prefill vs decode: two different machines
    4. 08.04Sampling: temperature, top-k, top-p, and min-p
    5. 08.05Beam search, speculative decoding & Medusa
    6. 08.06MQA, GQA & MLA
    7. 08.07FlashAttention & IO-awareness
    8. 08.08PagedAttention & continuous batching
    9. 08.09Quantization I: int8, int4, the basics
    10. 08.10Quantization II: GPTQ, AWQ, GGUF, QAT
    11. 08.11Serving stacks: vLLM, SGLang, TensorRT-LLM
    12. 08.12Running models locally
    13. 08.13Choosing and judging a community quant
    09

    advanced · 12 available lessons

    Building & Serving Stacks

    Master the software that actually runs Qwen3.8-27B — PyTorch, Transformers, vLLM, SGLang, TensorRT-LLM, llama.cpp, Ollama, MLX, and Modular MAX — and learn which one each situation calls for.

    Open track
    1. 09.01PyTorch: the reference runtime
    2. 09.02The Hugging Face Transformers ecosystem
    3. 09.03torch.compile and CUDA graphs
    4. 09.04vLLM
    5. 09.05SGLang
    6. 09.06TensorRT-LLM
    7. 09.07llama.cpp and GGUF
    8. 09.08Ollama and local runners
    9. 09.09MLX
    10. 09.10Modular MAX and Mojo
    11. 09.11Serving in the cloud
    12. 09.12Choosing your stack
    10

    frontier · 10 available lessons

    Hardware & Infrastructure

    Understand the silicon beneath Qwen3.8-27B — NVIDIA, AMD, Apple, Qualcomm, and Cerebras — and the bandwidth, roofline, and cost arithmetic that decides what runs where, and for how much.

    Open track
    1. 10.01Memory bandwidth vs FLOPs
    2. 10.02The roofline model
    3. 10.03KV cache sizing and memory budgets
    4. 10.04NVIDIA, CUDA, Hopper and Blackwell
    5. 10.05AMD, ROCm and the MI3xx line
    6. 10.06Apple silicon and unified memory
    7. 10.07Qualcomm Hexagon and the edge
    8. 10.08Cerebras and wafer scale
    9. 10.09Interconnects and clusters
    10. 10.10Cost per token