LoRA, QLoRA & PEFT

01 · Concept

Concept

Start with the number that ends the conversation about full fine-tuning. Qwen3.8-27B is 54 GB of bf16 weights. Gradients are another copy of the same size. AdamW carries roughly 12 to 16 bytes of state per parameter, which for 27B parameters is between 324 and 432 GB. Add them and full fine-tuning wants something in the neighbourhood of 430 to 540 GB of accelerator memory before a single activation is stored. That is a multi-node job. Meanwhile the machine you actually have is a single 24 GB or 48 GB card. Parameter-efficient fine-tuning (PEFT) exists to close that gap, and the arithmetic of how it closes it is the substance of this lesson.

Low-Rank Adaptation, or LoRA, changes how a linear layer is represented during fine-tuning. Let the frozen pretrained weight be $W_0\in\mathbb{R}^{d_{out}\times d_{in}}$. Instead of updating all of $W_0$, LoRA learns

with $A\in\mathbb{R}^{r\times d_{in}}$ and $B\in\mathbb{R}^{d_{out}\times r}$, and rank $r$ far below either dimension. The forward pass adds the adapter’s contribution to the original projection, scaled by an alpha parameter divided by the rank. The low-rank constraint is an inductive bias, not a theorem: useful adaptation may live in a small subspace, and rank, placement, and data decide whether it does.

Placement is where this model diverges sharply from the recipe most tutorials hand you. The 2021 LoRA paper adapted the query and value projections of a uniform transformer, and that default has been copied ever since. Apply it to Qwen3.8-27B and here is what happens. The model has 64 layers arranged as sixteen repetitions of three Gated DeltaNet blocks followed by one Gated Attention block. Only 16 of the 64 layers contain the query and value projections your config file names. The other 48 — three quarters of the depth — receive no adapter at all. You will train, the loss will fall, and you will conclude that LoRA underperforms on this model. It did not; your target list missed most of it.

The correct target list follows the architecture. In the 16 attention layers, the projections feeding 24 query heads and 4 key/value heads at head dimension 256, plus the output projection. In the 48 Gated DeltaNet layers, the projections producing its 16 query/key heads and 48 value heads at head dimension 128, plus its output projection; the short convolution with kernel width 4 and the gating parameters are usually left frozen, since they are tiny and numerically delicate. And in all 64 layers, the gated feed-forward matrices that map the 5120-wide residual stream out to 17,408 and back — roughly 17B parameters live there, more than half the model. Read the tensor names out of the safetensors index rather than guessing them, and check the shapes while you are there: the attention projections in this model are deliberately non-square, and a config-derived guess that “everything is 5120 wide” will not survive contact with the checkpoint.

Now the memory arithmetic, worked step by step. Take a rank of 16, and consider one matrix that reads and writes the 5120-wide residual stream. Its adapter costs $r(d_{in}+d_{out}) = 16 \times (5120 + 5120) \approx 164{,}000$ parameters — call it 0.16M. Adapt four projections in each of the 64 layers and you have 256 adapted matrices, so about 42M trainable parameters. (Real widths vary per module, so treat this as illustrative sizing, not a spec.) In bf16 those weights are about 84 MB. AdamW state at 12 to 16 bytes per parameter adds roughly 0.5 to 0.7 GB. Set that beside the 324 to 432 GB the same optimizer would demand for full fine-tuning: three orders of magnitude, from the same optimizer, because it now has three orders of magnitude fewer parameters to track.

That solves optimizer memory, but not the frozen backbone, which must still be resident for forward and backward passes. QLoRA stores that backbone in a low-bit representation and dequantizes blocks into a higher compute dtype as operations run. Pure 4-bit codes set a 13.5 GB floor for a rounded 27B parameters, but actual QLoRA residency depends on the backend’s quantization metadata, buffers, activations, adapters, and optimizer state; a GGUF inference-file size is not a training-memory measurement. The QLoRA paper’s normal-float format, double quantization, and paged optimizers all reduce this budget. Measure the exact training stack at the intended sequence length and micro-batch.

LoRA also buys deployment flexibility. One base model can serve several small adapters, and an adapter can be merged into the base weights for inference when the numerical formats allow it — which they often do not after 4-bit quantization, so a QLoRA adapter merged back into a 4-bit base is a lossy operation that deserves its own evaluation. Unmerged adapters let a server switch behaviours per request, at the cost of batching complexity.

PEFT is a wider family than LoRA. Adapter layers insert small modules, prompt and prefix tuning learn continuous vectors injected into the context or the layers, selective training unfreezes a chosen subset. Each cuts a different seam, and runtime support, latency, multi-tenant switching, and the behaviour being learned decide which seam fits.

None of this repairs bad instruction data, prevents forgetting, or guarantees parity with full fine-tuning. A small adapter can memorize a narrow dataset or degrade general behaviour just as a full run can. The practical mental model is a frozen foundation plus a learned steering surface: LoRA expresses the surface as a low-rank update, QLoRA makes the foundation cheap enough to hold while you learn it, and the engineering win is separating shared pretrained capability from compact, versioned, task-specific change.