When to fine-tune rather than prompt or retrieve, LoRA and parameter-efficient methods, dataset curation, and evaluating whether it helped.
3 questions in Generative AI & Large Language Models.
Ask what is missing. If the answer changes when your documents change, you need retrieval; if the model knows enough but will not behave, format or specialise the way you need, you fine-tune. Prompting comes first either way because it costs an afternoon and tells you which of the other two you actually need.
Compare it against a genuinely strong prompted baseline on a held-out set drawn from real traffic, using the task metric your product cares about, and run a broad regression set alongside to catch capability lost elsewhere. Falling training loss is evidence that training worked, not that the product improved.
LoRA freezes the base weights and trains a small low-rank update added to chosen projections, so you train a fraction of the parameters and can swap adapters per use case. It reshapes behaviour and format readily, installs facts poorly, and rank is the capacity dial with overfitting beyond it.