What does LoRA change in the model, and when is parameter-efficient tuning the right choice?

The mechanism

Full fine-tuning updates every weight in the network. LoRA leaves them all frozen and instead learns a small additive correction to selected weight matrices. For a frozen matrix W of shape d × k, it trains two much smaller matrices — A of shape r × k and B of shape d × r, with r far below both d and k — and the layer behaves as if its weights were W + (α/r)·BA. Only A and B receive gradients. B is initialised at zero so the adapted model starts out numerically identical to the base, which is why training is stable from the first step.

The parameter count for that matrix drops from d × k to r × (d + k). For a 4096 × 4096 projection at rank 16 that is roughly 131,000 trainable parameters instead of about 16.8 million, and the saving compounds across every adapted layer.

The headline saving is usually quoted as parameters, but the reason it fits on hardware that full fine-tuning does not is optimiser state. An adaptive optimiser keeps a couple of momentum tensors per trainable parameter, and gradients are needed for those parameters too, so the memory attached to each trainable weight is several times its own size. Freezing the base removes all of that for 99-plus per cent of the network; you still hold the frozen weights for the forward pass, but you no longer hold their gradients or their optimiser moments. QLoRA extends the same idea by holding the frozen base in a low-bit quantised form and training the adapters in higher precision on top, which is what brings large models within reach of a single accelerator.

What it changes and what it does not

The adapter changes the linear maps inside chosen layers. It does not change the tokeniser, the vocabulary, the architecture, or the model's pretrained knowledge, and it does not add capacity in any meaningful sense. Everything the adapted model can do, it does with the base model's representations, nudged.

That framing predicts what adapters are good at. Behaviour, register, output discipline, task framing, the shape of a well-formed answer in your domain: these are all mappings from a representation the model already computes to a preferred output, and a low-rank correction expresses them efficiently. A few hundred to a few thousand consistent examples can move them convincingly.

It also predicts what adapters are bad at, which is the part interviewers probe. A low-rank update over a small dataset is a poor mechanism for installing facts. There is very little room in it for new content, so the model absorbs the style of your material rather than its substance, and the outcome is confident invention in the correct house voice. Pushing harder makes it worse rather than better: raise the rank and the epochs until the training set really is memorised, and you get verbatim recall of those examples together with visible degradation elsewhere, because the correction is now large enough to disturb general behaviour. Knowledge that must be current or attributable belongs in retrieval, and saying so is the strongest single sentence in an answer to this question.

Rank is the capacity dial

Rank controls how expressive the correction can be, and both directions have a failure mode. Too low and the adapter cannot represent the distinction you are teaching, so the loss plateaus early and outputs stay stubbornly base-like. Too high and you have enough parameters to memorise a small dataset, so validation quality falls away from training quality and the general capability you were relying on starts to erode — while the memory and storage advantages that motivated the approach shrink towards those of a full fine-tune.

The scaling factor α/r interacts with this: it sets how strongly the learned correction is applied, and because r appears in the denominator, raising the rank without adjusting α changes the effective step size as well as the capacity. That is why rank and learning rate should be swept together rather than one at a time.

Where to place the adapters is the other lever. The original work applied them to attention projections; common practice since has been to adapt all the linear layers, including the feed-forward projections, which costs more parameters and generally learns more. If a run underperforms, widening the set of adapted modules is usually a better next move than raising the rank on a narrow set.

Serving, and the coupling nobody documents

At inference you can either keep the adapter separate, computing the extra low-rank product alongside the frozen layer, or fold it into the base by computing W + (α/r)·BA once and storing the result. Merging removes any per-token overhead, so the adapted model runs at exactly base-model speed — a genuine advantage over adapter designs that insert extra layers into the forward path.

Keeping them separate buys something different and often more valuable: one copy of the base model in memory serving many adapters, selected per request. For a product with per-tenant or per-task adaptations that is the difference between one deployment and thirty, and it is a strong argument for the approach beyond training economics.

The coupling to watch is that an adapter is meaningful only against the exact base checkpoint it was trained on. The weights it corrects are that checkpoint's weights, so applying it to a different version — even a minor revision of the same model family — produces behaviour nobody has evaluated, and it will not error. Pin the base checkpoint alongside the adapter as one versioned artefact, and treat a base-model change as a retraining trigger rather than a substitution.