What belongs in the system prompt, and what belongs in the user turn?
4 min readUpdated 2026-08-15
0
Answer
Short answer
The system prompt holds what is identical on every request - role, policy, output contract, tool schemas, examples. Anything varying by user or request, retrieved documents included, belongs in the turns below, to keep the cached prefix stable and untrusted text away from system authority.
Was this answer helpful?
Likely follow-ups
→Your tenant-specific policy differs per customer. Where does it go, and what does that cost you?
→How do you version a system prompt so a rollback is possible mid-incident?
→A retrieved document contains the sentence "ignore your previous instructions". What in your design stops that from working?
→Two features share most of a system prompt. Do you factor out the common part, and what breaks if you do?
The instinct is to put important things in the system prompt and incidental things in the user turn. That is the wrong axis. The question to ask of each piece of text is whether it is byte-for-byte identical on every request. Role and voice, the output contract, refusal and escalation policy, tool descriptions, and few-shot examples all qualify: they belong to the feature, not to the caller. The user's name, their locale, their entitlements, the current date, retrieved documents, and the request itself do not qualify, however important they are.
That criterion is worth stating in an interview because it produces the right layout for two independent reasons at once — one about cost and one about trust — and candidates usually know one of them.
Prefix caching is exact, so one varying token near the top is expensive
Provider-side prompt caching works by matching a prefix of the token sequence. If the first several thousand tokens of this request are identical to a recent one, the stored intermediate state for those tokens is reused rather than recomputed, which is why cached input is both cheaper and faster than fresh input. The match is on the prefix, and it is exact.
Interpolating Today is 2026-07-28 or You are helping Priya, on the Enterprise plan into the top of the system prompt therefore does not cost you a few tokens of cache. It costs you all of it, on every request, because the divergence happens before the shared body. Injecting a timestamp with a seconds component is the purest form of this bug: the prefix is unique every single call, the cache never hits, and nothing in your logs looks wrong other than the bill.
The fix is ordering, not deletion. Put the invariant block first, then whatever varies least, then conversation history, then the retrieved context and the current request last. Conversation history sits happily in the middle because it only ever grows at the end, so each new turn extends a prefix that was already cached rather than displacing it.
Loading diagram…
Everything left of the boundary can be reused; the interesting thing to notice is that history sits immediately right of it, so a design that summarises and rewrites history in place gives up the cheap append and pays for the whole tail again.
The system prompt confers standing, which is why untrusted text must stay out of it
Models are post-trained to treat the system message as higher-priority guidance than the messages beneath it. That is a useful property and it has a mirror image: text placed there inherits that standing. A support-bot design that pastes the customer's last email into the system prompt so the model "takes it seriously" has told the model that an arbitrary sender's words carry the same weight as your own policy. If that email says to disregard the refund rules, you have built the injection in yourself rather than merely failing to defend against it.
So untrusted content goes below, wrapped in a clear delimiter, labelled with its provenance, and accompanied by an instruction in the system prompt saying that material inside such blocks is data to be summarised, quoted or cited and is never to be followed as instruction. Retrieved documents are untrusted in exactly this sense even when they come from your own corpus, because someone put them there and that someone may not be you.
Be direct about the limit of all this. The system prompt is not a privilege boundary. There is no memory protection between the messages; they are one token sequence, and priority is a learned tendency rather than an enforced rule. Layout reduces the attack surface and makes the model's job easier, but the controls that actually hold are outside the prompt: least-privilege tools, authorisation checked in your own code against the real caller's identity, and output filtering on anything that leaves the system. A candidate who claims correct message roles make injection impossible has said something false, and it is the kind of false claim an interviewer is specifically listening for.
Practical consequences worth naming
Per-tenant policy is the case where the two forces genuinely conflict: it is invariant per tenant and variant across tenants. The honest answer is that you segment the cache by tenant and accept a lower hit rate, or you keep the shared instructions in the common prefix and express the tenant's differences as compact structured parameters lower down. Which one wins depends on how many tenants share the traffic.
The other consequence is operational. Because the system prompt is stable, it is an artefact you can version, diff, review and roll back independently of a deployment, and it should be. The per-request material cannot be versioned like that, which is another way of saying that anything you would want to change without a release should be up top.