Freshness settles the knowledge question
The general framework, knowledge goes to retrieval and behaviour goes to adaptation, is covered in prompting versus retrieval versus fine-tuning. A support assistant is the case where that framework stops being a judgement call, because the knowledge base has a change rate, and the change rate decides it.
Suppose twenty articles change in a typical week. With retrieval, each edit reaches users as soon as the index picks it up, which is minutes if ingestion is incremental. With fine-tuning, each edit waits for the next training run, then an evaluation, then a deployment. Even a weekly retraining cadence means a customer can be told last week's return window for up to a week, and nobody can say which answers came from which version of the article.
Withdrawal is the stronger argument. When legal pulls a policy, you need the assistant to stop stating it today. In a retrieval system you delete or flag the document and it stops appearing. In a fine-tuned model the policy lives in the weights, and there is no operation that removes one fact. You retrain without it and hope the old phrasing does not persist. A support assistant that cannot forget a withdrawn refund policy is a liability, not a quality problem.
Citations matter more in support than almost anywhere else. A human agent reviewing an escalated chat, or a customer disputing an answer, needs to see which article the assistant relied on. Retrieval gives you that link per answer. Weights give you nothing to point at.
Decide from failed conversations, not from the architecture diagram
The useful first step is an audit. Take a few hundred recent conversations where the assistant was rated badly or escalated, and label each failure by what was missing. A small table makes the decision visible.
| Failure seen in the transcript | What was missing | Where the fix belongs |
|---|
| Quoted an old shipping fee | Current fact | Retrieval: freshness, versioning |
| Answered for the wrong plan tier | Customer context | Retrieval filters and account lookup |
| Correct facts, cold and long-winded | House tone | Prompt first, then fine-tuning |
| Kept answering when it should hand over | Escalation judgement | Prompt rules, then fine-tuning on labelled cases |
| Reply broke the ticketing macro format | Output discipline | Structured output, then fine-tuning |
If the first two rows dominate, you have retrieval and data problems, and no training run will fix them. The audit is what lets you say that with evidence in the interview rather than as a slogan.
The last three rows are where fine-tuning is a genuine option. Escalation judgement is the clearest case. It is a narrow classification your best human agents make consistently, it is hard to describe completely in instructions, and you can label examples of it. Tone is similar: a few thousand reviewed replies demonstrate a house voice better than a page of style rules.
Old tickets teach old answers
The obvious training data for a support model is the archive of resolved tickets, and this is where a strong answer separates itself from an adequate one. Those transcripts contain the answers that were correct when each ticket was closed. Fine-tuning on them teaches the model the tone you want and, silently, every fee, limit and policy that has since changed. You have reintroduced stale knowledge through the behaviour channel.
So a behaviour dataset needs the facts stripped or neutralised. Train on the decision and the form, not the content: whether to escalate, how to acknowledge a complaint, how to structure a step-by-step fix, with the factual payload supplied by retrieval at training time exactly as it will be at inference. If the training examples include the retrieved passages, the model learns to use context rather than to remember answers.
There is a second risk worth naming. A model fine-tuned heavily on past answers can start to prefer its own memory over a retrieved passage that contradicts it. Test for this directly by giving it passages that disagree with the archive and checking which one it follows.
Often both, measured separately
The architecture that usually wins is retrieval for every fact, with an optional fine-tune on top for escalation, tone or format. The prompt still carries the task and the guardrails. The order matters: fix retrieval first, because a model tuned against a broken context pipeline learns to compensate for a bug you are about to fix.
Evaluation has to keep the two apart. Build a knowledge slice, questions with a known source article and answer, scored on retrieval recall and answer correctness. Build a behaviour slice, conversations labelled for whether escalation was right and whether the reply met the house format. A retrieval change should move the first slice and leave the second alone, and a fine-tune should do the reverse. Proving that a fine-tune helped covers the held-out discipline in detail.
Cost belongs in the answer too, stated honestly. Fine-tuning is mainly a dataset and evaluation obligation, and it ties you to a base model with its own retirement date. Retrieval has running costs in ingestion, indexing and longer prompts. Neither is free, so the audit is also what tells you whether the behaviour gap is large enough to justify the second system.
Put every fact the knowledge base can change behind retrieval, and fine-tune only on decisions and form. Training on old tickets quietly turns last year's answers into behaviour.
© 2026 Preptima. Originally published at preptima.com (opens in a new tab).