llmdeepdive
PT

Type to search

    The anatomy explorer · Interactive 3D specimen

    Anatomy
    of an LLM

    Rotate an exploded transformer block. Select any marker to examine its maths, failure modes, and place in the stack.

    Drag to rotate · Arrow keys move between markers

    Diagram of a transformer block, from embeddings up to the sampler
    1. OutputLM head
    2. ComputeFeed-forward
    3. RoutingSelf-attention
    4. InputEmbeddings

    3D SPECIMEN · CLICK A DOT TO EXPLORE

    Input encoding

    Tokenizer

    Text into integers

    Splits text into subword units and maps each to an integer. Every downstream cost — context length, price per call, throughput — is denominated in these units.

    Key facts

    Parameters
    None (a lookup table, not learned weights)
    Cost
    Negligible at inference
    Introduced
    BPE: Sennrich et al., 2016
    Variants
    BPE · SentencePiece · byte-level BPE · Unigram
    View lesson

    The maths

    Parameters: None (a lookup table, not learned weights). Dominant cost: Negligible at inference.

    Across architectures

    The main alternatives are BPE · SentencePiece · byte-level BPE · Unigram.

    A token's journey

    Text into integers. Splits text into subword units and maps each to an integer. Every downstream cost — context length, price per call, throughput — is denominated in these units.

    Failure modes

    If this component is constrained or misconfigured, the model loses capacity for input encoding.

    Where it sits

    Historical marker: BPE: Sennrich et al., 2016. Variants in use: BPE · SentencePiece · byte-level BPE · Unigram.