Reading real weights: what a trained model looks like

1

Concept

A trained model is not stored as prose or a visible graph of concepts. A checkpoint is usually a mapping from parameter names to multidimensional arrays. The architecture code supplies the operations; the checkpoint supplies the learned numbers. Reading weights begins by reconstructing that contract exactly.

List each tensor’s name, shape, dtype, and device. A token embedding might have shape $V\times d_{model}$. A combined query–key–value projection may have shape $3d_{model}\times d_{model}$, although frameworks disagree about input-first versus output-first layout. An FFN expansion reveals $d_{ff}$; normalization vectors reveal model width; repeated numeric prefixes reveal layer count. These are reliable structural clues.

Names are conventions, not standards. One repository may store attn.c_attn.weight; another may split q_proj, k_proj, and v_proj. Some fuse gated FFN projections. Some omit linear biases. Some tie the language-model head to the token embedding, meaning two logical components reference one parameter. Counting named tensors without understanding sharing can double-count or miss parameters.

The tokenizer and configuration are checkpoint components too. Vocabulary size must match embedding rows. Head count must agree with projection reshaping. RoPE frequency, pairing, and scaling must match. LayerNorm versus RMSNorm changes parameter meaning. A tensor can fit a destination shape while still being transposed or assigned to the wrong convention, producing fluent-looking failure that is harder to detect than a loader error.

Basic statistics are useful diagnostics. Check for NaNs and infinities, then calculate minimum, maximum, mean, standard deviation, and norm. Plot histograms per tensor and compare layers. Quantized checkpoints need scale and zero-point metadata; interpreting packed integer codes as ordinary weights gives nonsense. Outliers may be expected and important, especially under quantization, so “looks Gaussian” is not a correctness test.

Embedding rows can be compared with cosine similarity, but neighbors reflect the trained representation and tokenizer. A row may mix syntax, frequency, morphology, and semantic associations. Projection columns and FFN neurons are likewise basis-dependent: rotating an internal space can preserve model behavior while changing individual coordinates. A large weight is not automatically an important feature.

To move from structure to function, collect activations on controlled inputs. Ask which inputs activate a feature, what direction a component writes into the residual stream, and how downstream logits change. Then intervene: ablate a head, patch an activation from one run into another, or project out a direction. Correlation suggests a hypothesis; a reproducible causal effect strengthens it.

Histograms help spot training and conversion problems. A layer with all zeros may be missing. A norm vector at an unexpected scale may reveal a bad dtype cast. Q, K, and V distributions that differ radically from a known-good conversion can expose a slicing error. The strongest comparison uses the same input through the source and converted implementations and checks intermediate activations layer by layer.

Real checkpoints can be sharded across files. An index maps parameter names to shards so a loader need not open everything at once. Distributed-training checkpoints may store optimizer moments and partitions rather than a consolidated inference model. Those files can be much larger than parameter weights because training state includes gradients, master weights, and optimizer buffers.

The responsible conclusion is modest but powerful. Tensor metadata reveals architecture; statistics reveal health and scale; activations reveal associations; interventions reveal causal contribution. No single plot reveals “where the model thinks.” Reading weights is experimental model archaeology guided by the block equations you now understand.