Pretraining at Scale

advanced · 13 available lessons

Pretraining at Scale

Design the objectives, data, optimization, parallelism, and cost model behind a serious language-model pretraining run.

1. Objectives: causal LM, MLM, span corruption, FIM
2. Data: sourcing, filtering, dedup, decontamination
3. Training your own tokenizer
4. Scaling laws I: Kaplan
5. Scaling laws II: Chinchilla & compute-optimality
6. Scaling laws III: inference-aware & over-training
7. Data parallelism, ZeRO & FSDP
8. Tensor, pipeline & sequence parallelism
9. Mixed precision: fp16, bf16, fp8
10. Gradient checkpointing & memory maths
11. Learning-rate schedules & warmup
12. When training goes wrong: spikes, divergence, NaNs
13. What a real pretraining run costs