LoRA, QLoRA & PEFT

1

Concept

Full fine-tuning updates every trainable parameter in a model. That gives the optimizer maximum freedom, but it also requires gradients and optimizer state across an enormous checkpoint. Parameter-efficient fine-tuning (PEFT) asks a sharper question: can a task-specific behavior be expressed through a much smaller set of trainable parameters while the pretrained backbone remains frozen?

Low-Rank Adaptation, or LoRA, answers by changing how a linear layer is represented during fine-tuning. Let the frozen pretrained weight be $W_0\in\mathbb{R}^{d_{out}\times d_{in}}$. Instead of updating all of $W_0$, LoRA learns

with $A\in\mathbb{R}^{r\times d_{in}}$ and $B\in\mathbb{R}^{d_{out}\times r}$. The chosen rank $r$ is much smaller than the input and output dimensions. The forward pass adds the adapter contribution to the original projection. A scale, commonly expressed using an alpha parameter and rank, controls its magnitude.

The low-rank constraint is an inductive bias: useful adaptation may lie in a smaller subspace than the full matrix permits. It is not a theorem that every task update is low-rank. Rank, placement, and data determine capacity. Adapters are often attached to attention projections and sometimes feed-forward layers. Targeting more modules or raising rank increases flexibility and trainable state, but also increases memory and may overfit.

LoRA saves more than checkpoint space. Frozen base weights need no optimizer moments, and gradients are retained only for adapter parameters. A deployment can store one base model plus several small adapters. The adapters can also be merged into the base weights for inference when the numerical formats permit it. Unmerged adapters are useful when a server must switch tasks, though batching requests with different adapters introduces operational complexity.

QLoRA combines LoRA with a quantized frozen backbone. The base weights are stored in a low-bit representation to reduce memory, then dequantized into a suitable compute type as operations run. The adapter parameters remain trainable at higher precision. QLoRA’s 2023 paper introduced specific techniques including a normal-float format, double quantization, and paged optimizers. The essential boundary is important: quantized storage does not mean gradients through the entire base model, and it does not mean every arithmetic operation uses low-bit integers.

PEFT is a broader family. Adapter layers, prompt tuning, prefix tuning, and selective parameter training alter different seams. Prompt-like methods learn continuous vectors inserted into the model’s context or layers. Adapters add small modules. LoRA modifies linear transformations through a factored delta. The best choice depends on runtime support, latency, multi-tenant switching, and the behavior being learned.

PEFT does not correct poor instruction data, prevent forgetting automatically, or guarantee parity with full fine-tuning. A small adapter can memorize a narrow dataset or degrade general behavior. Conversely, full fine-tuning can be unnecessary and harder to operate. Compare methods under the same data, evaluation set, and compute assumptions.

The practical mental model is a frozen foundation plus a learned steering surface. LoRA expresses that surface as a low-rank matrix update. QLoRA makes the foundation cheaper to hold during training by quantizing it. The engineering win comes from separating shared pretrained capability from compact, versioned, task-specific changes.