How do you decide between prompting, retrieval and fine-tuning?
5 min readUpdated 2026-08-15
0
Answer
Short answer
Ask what is missing.If the answer changes when your documents change, you need retrieval; if the model knows enough but will not behave, format or specialise the way you need, you fine-tune.
Was this answer helpful?
Likely follow-ups
→You fine-tune to teach a model your product's API, and it invents plausible endpoints. What went wrong?
→What would make you fine-tune a small model to replace a large prompted one?
→How many examples would you want before attempting a fine-tune, and how would you decide?
→Your fine-tuned model needs to know about a policy that changed yesterday. What is your plan?
Ask what is missing before you ask which technique
The three options are not competing implementations of one thing. They address different deficits, and the useful first move is to characterise the deficit precisely.
If the model lacks information — your product's current pricing, this customer's order history, a policy document written after training finished — the deficit is knowledge, and knowledge belongs in the context at request time. The test that settles it in one sentence: would the correct answer be different if your documents changed? If yes, you need retrieval, because whatever you bake into weights is a snapshot and the snapshot is wrong the next time somebody edits a page.
If the model has the knowledge but will not do the thing — it writes in the wrong register, ignores your output conventions under pressure, fails to use the domain's idiom, or performs a narrow classification less consistently than you need — the deficit is behaviour, and behaviour is what fine-tuning genuinely moves. Style, format, task framing, calibration of when to refuse, the shape of a good answer in your domain: these are mappings from input to output, which is exactly what supervised examples teach.
Loading diagram…
The branch worth dwelling on is the middle one: obscure but stable facts feel like a fine-tuning case and usually are not, because retrieval still gives you citations and a way to correct an error without retraining.
Why fine-tuning is a poor way to add knowledge
This is the mistake to be able to explain, because it is the one most teams make and it fails in a way that looks like success at first.
Fine-tuning on documents adjusts weights so that the training text becomes more likely. What that teaches, on a modest dataset, is the surface form of statements in your domain — the vocabulary, the cadence, the shape of an answer about your product. It does not install a lookup table. The model becomes fluent in the register of your documentation while remaining unable to reliably retrieve any particular fact from it, and the result is confident invention: endpoints that sound exactly like your endpoints and do not exist, prices in the right format and the wrong amount. That is worse than not knowing, because it defeats the reader's ability to tell.
Two further properties finish the argument. Facts learned in weights cannot be attributed, so you lose citations and with them the user's ability to verify. And they cannot be updated without another training run, so a policy change becomes a retraining project rather than an edit to a document. Retrieval gives you both for free.
Prompting first, always, and not out of laziness
Start with a serious prompt on a strong model, not a token attempt. The iteration loop is minutes, the cost is nothing, and the outcome tells you which of the other two techniques you need — because the way a good prompt fails is diagnostic. If it gets the format right and the facts wrong, that is retrieval. If it has the facts and keeps drifting out of the required form, that is fine-tuning. If it fails on both, do the retrieval work first, since a fine-tune evaluated against a broken context pipeline teaches the model to compensate for a bug you are about to fix.
This is also the honest answer to why so many fine-tuning projects are abandoned: the prompt was never properly written, and once it is, the gap closes. An interviewer is listening for whether you would spend a week on a training pipeline to fix something a schema and three examples would have fixed.
Where fine-tuning earns its keep
The strongest case is often not quality at all, it is economics. A prompt that needs 4,000 tokens of instructions and examples to get a large model to perform a narrow task can frequently be replaced by a small model fine-tuned on a few thousand examples of that task, with a prompt of a few hundred tokens. You trade a training pipeline for lower per-request cost and lower latency, and at high volume that trade is decisive. Classification, extraction, routing, and normalising messy input into a fixed schema are the shapes where this works best, because the task is narrow, the output space is small, and labelled examples can be harvested from the large model's own outputs after review.
The other solid case is behaviour that resists instruction. Some conventions are easier to demonstrate a thousand times than to describe: a house style for clinical notes, the tone a regulated business is required to use, the specific way your analysts want a finding phrased. Instructions for these grow long, brittle and mutually contradictory, and examples do not.
Understand what you are signing up for. Fine-tuning is a dataset obligation more than a compute one: curated, deduplicated, consistently labelled examples, a held-out set kept genuinely separate, and an evaluation harness that can show both the intended gain and any capability lost elsewhere. It also couples you to a base model that has a retirement date, which means the pipeline has to be re-runnable rather than a one-off artefact somebody produced in a notebook.
They compose
The framing as a choice is itself a mild trap. Production systems commonly do all three: retrieval supplies the facts and the citations, a fine-tune fixes the register and the output discipline so the prompt no longer has to argue for them, and the prompt carries the task and the guardrails. Ordering the work by cost of iteration — prompt, then retrieval, then adaptation — gets you there while keeping each change independently measurable, which is the part that matters when you have to explain a regression.