llmdeepdive
PT

Type to search

    Learning track

    Sequence Models

    Follow the path from recurrent state to encoder-decoder models and attention, the immediate ancestors of the Transformer.

    04

    core · 6 available lessons

    Sequence Models

    Follow the path from recurrent state to encoder-decoder models and attention, the immediate ancestors of the Transformer.

    1. 04.01Modelling sequences: the setup
    2. 04.02RNNs, fixed-size state, and the vanishing gradient
    3. 04.03LSTM and GRU: gates that remember
    4. 04.04Seq2seq, encoder-decoder, and the bottleneck
    5. 04.05Bahdanau attention: the idea that changed everything
    6. 04.06Linear attention and fast weights