Do you use one large model, or a smaller model with retrieval? How do you decide?

Ask what the task is short of

The choice only looks like a single question. It is two: does the model lack knowledge, or does it lack reasoning capacity? A model cannot know your pricing rules, your incident history or last quarter's contract amendments, and no amount of scale fixes that, because the information was never in the weights. Conversely, a task that requires holding a long chain of dependent constraints — reconciling a specification against an implementation, planning a multi-step migration, working through an argument where an early mistake invalidates everything downstream — is limited by capability, and no amount of retrieval fixes that.

Most product features that people frame as "which model" are the first kind. Support answering, policy lookup, document question-answering, internal search with synthesis: all of them are asking the system to be right about facts it must be shown. For those, the interesting variable is not the model.

For grounded work, retrieval quality dominates

When the answer must come from your corpus, the ceiling on quality is whether the passage containing the answer is in the context window at all. If it is absent, every model fails, and the larger one fails more expensively and often more fluently, which is worse. If it is present and unambiguous, a mid-sized instruction-following model will usually extract and present it correctly. That is why swapping to a bigger model is the most common wasted week in this area: it produces a small, real improvement in phrasing and instruction adherence, which reads as progress, while the retrieval failures that generate the actual complaints are untouched.

The diagnostic is cheap and you should insist on running it before the model debate. Take the failing cases and check by hand whether the supporting passage was retrieved. Sort the failures into retrieval misses, cases where the passage was retrieved but buried among a dozen irrelevant ones, and cases where the passage was there and clean and the model still got it wrong. Only the third bucket is an argument about model choice. In practice the first two dominate, and the work that follows is chunking, hybrid keyword-plus-vector retrieval, reranking, metadata filters and fixing the documents themselves, none of which is glamorous and all of which moves the number.

There is a real counter-case worth conceding, because an interviewer will probe for it. Synthesis across many retrieved documents, where the answer exists in no single passage and must be assembled while noticing that two sources disagree, does scale with capability. So does following a long, conditional instruction set about how to format and qualify the answer. If your failures are concentrated there, the larger model is the right purchase.

What routing actually costs

Having established that most requests do not need the expensive model, the obvious move is to route: classify the request, send the easy ones to the small model, escalate the hard ones. The saving is genuine and can be large, because request difficulty in most products is heavily skewed towards easy. State the cost honestly, though, because that is what separates a candidate who has run this from one who has read about it.

You now have two generation paths, so every evaluation, every prompt change and every regression check is duplicated, and a prompt tuned for the small model is not automatically correct for the large one. You have a classifier, which is itself a model with its own accuracy, its own failure modes and its own latency added to the front of every request. Misroutes are asymmetric: sending an easy request to the expensive model wastes money quietly, while sending a hard request to the cheap model produces a wrong answer that looks like a normal answer. And your monitoring must now be segmented by path, because an aggregate quality metric will hide a small model quietly degrading on 80% of traffic behind a large model still performing on the rest.

A cascade is often the better shape than a classifier, because it decides on evidence instead of prediction: try the cheap path, check the result against a cheap verifier — did it cite a retrieved passage, did it produce valid structured output, did it decline — and escalate on failure. You pay two calls on the escalated fraction, which is the honest price of not needing to predict difficulty in advance. It also gives you a naturally instrumented escalation rate, which is exactly the signal you want alerting on.

Deciding it properly

Build the eval set first, from real traffic rather than invented questions, with the cases you currently get wrong over-represented. Then run the same set through the candidate configurations: small model with the current retriever, small model with an improved retriever, large model with the current retriever. The comparison is usually decisive and frequently unflattering to the assumption you walked in with.

Price the options per thousand requests rather than per million tokens, using your own measured input and output lengths, because a system that retrieves fifteen chunks per call has an input cost that dwarfs its output cost and a large model amplifies exactly that. Then decide with both numbers in front of you, and write down the quality delta you were willing to buy and what you paid for it, so that when someone proposes the swap again next quarter the answer is a measurement rather than an opinion.

The failure mode to avoid naming last: choosing the largest model because it makes the prototype work, and never learning which part of the system was carrying the result. That decision is cheap to make and expensive to unwind, because by the time the bill matters, the prompts, the evals and the product's expectations have all been built on top of it.