Generative AI and LLM fundamentals

Predicting one token at a time. The model takes the tokens so far, runs them through a stack of transformer layers, and produces a probability distribution over its entire vocabulary for what comes next; a sampler picks one token from that distribution, appends it, and the whole forward pass runs again. There is no plan and no draft being revised. Everything that looks like reasoning is the consequence of each token being conditioned on all the tokens before it, including the ones the model itself just produced.

That single fact explains most behaviour candidates find surprising. A model cannot know how its sentence will end when it starts, it cannot revise a claim it has already emitted without saying so out loud, and asking it to think before answering genuinely changes the answer, because the intermediate tokens become part of the condition for the final ones.

Temperature rescales the distribution before sampling. Low values sharpen it so the highest-probability token dominates; high values flatten it so unlikely tokens get a real chance. It does not make the model more or less capable, and it does not make it more or less truthful — a confidently wrong distribution stays wrong at temperature zero, it simply becomes reliably wrong instead of variably so.

The practical reading is that temperature is a determinism and diversity control, not a quality control. Extraction, classification and anything you intend to parse want it near zero. Drafting and ideation, where you will look at several attempts and pick one, want it higher. Turning it down because the model is hallucinating is the wrong lever, and the fact that it sometimes appears to work is worth understanding: it only ever suppresses the tail.

Because temperature zero removes sampling randomness, not every source of nondeterminism. Floating-point addition is not associative, so the order in which a GPU reduces partial sums changes the last bits of a logit, and two logits that differ in the last bits can swap rank. Batching makes this concrete: your request is executed alongside whatever else arrived at the same moment, and the kernel chosen and the reduction order can differ between runs.

Beyond arithmetic, the provider's model is not frozen unless you pin a version, and mixture-of-experts routing adds its own batch-dependent behaviour. So treat temperature zero as "as repeatable as this stack gets", and if a test needs an exact string, assert on the parsed structure rather than on the bytes.

A logprob is the log probability the model assigned to a token it emitted, and where a provider exposes them they give you the one honest confidence signal available without a second call. The classic use is classification: ask for a single-token label and read the probability mass on each candidate, which lets you set an abstention threshold rather than accepting whatever came out.

They are also a cheap detector for the places a model was unsure — a sudden dip across a span of a generated answer is a reasonable trigger for review. What they are not is a truth signal. The model can be confidently wrong, so a high probability tells you the model was unsurprised by its own output, not that the output corresponds to anything real.

Four stages, and knowing which one you are waiting on is most of latency work.

sequenceDiagram
    participant C as Client
    participant G as Gateway
    participant S as Scheduler
    participant M as Model
    C->>G: request with prompt
    G->>S: tokenised prompt
    S->>M: batched prefill
    M-->>C: first token
    M-->>C: streamed tokens

Look at where the batch appears. Tokenisation and routing are cheap and roughly constant. The prefill stage processes your whole prompt in parallel and builds the attention cache for it, and its cost scales with prompt length — this is the part that grows when you paste ten pages of context. Then decode runs one token at a time, each step reusing the cache, so output tokens arrive at a fairly steady rate that depends on how many other requests share the device.

The consequence is that a long prompt delays the first token, while a long answer extends the tail. They are separate problems with separate fixes: trimming context or caching a shared prefix helps time to first token, and only asking for less output, or streaming so the reader starts sooner, helps the tail. Candidates who report one number for "latency" usually cannot say which of the two they measured.

Because it collapses every text task into one objective. An encoder-decoder split suits translation, where input and output are distinct sequences, but it needs task-specific heads and paired data. A decoder-only stack trained purely to predict the next token learns from any text at all, and then classification, extraction, summarisation and dialogue are all expressible as "continue this". That uniformity is what made scale pay: the same architecture and the same objective absorb more data and more parameters without redesign.

The cost is that the model has no separate representation of the input. Your instructions and the document you pasted occupy the same sequence, which is the structural reason prompt injection is hard to fix rather than a bug in any one product.

A base model is the raw next-token predictor. Give it a question and a plausible continuation is a list of similar questions, because that is what documents containing questions tend to look like. An instruction-tuned model has been further trained on instruction and response pairs, usually followed by preference tuning, so that the highest-probability continuation of a request is an attempt to satisfy it.

The distinction matters in interviews because it separates capability from behaviour. Base models are the better starting point for further fine-tuning on a narrow task, since you are not fighting an existing response style. Anything user-facing wants the tuned variant, and the tuning is also where refusal behaviour and format habits come from.

A token is a subword unit from a fixed vocabulary produced by byte-pair encoding or a similar scheme, where frequent words become single tokens and rare ones split into pieces. It matters because tokens are the unit of everything: the context window is counted in them, the bill is counted in them, and generation speed is tokens per second.

Two practical consequences. First, character or word counts are the wrong estimate — the common rule of thumb of roughly four characters of English per token breaks badly on code, JSON, non-Latin scripts and long identifiers, which can approach one token per character. Second, the tokeniser differs between model families, so a prompt that fits one model's window may not fit another's, and cost comparisons between providers must be made on their own tokenisers.

Treat it as a fixed byte budget you are dividing, because prompt and completion share it.

Window                       128,000 tokens
  system prompt + policies       1,200
  tool and schema definitions    2,400
  conversation history          12,000   <- grows every turn
  retrieved passages            18,000   <- 12 chunks x ~1,500
  reserved for the answer        4,000   <- must be reserved, not hoped for
  ------------------------------------
  used                          37,600
  headroom                      90,400

The line that causes incidents is the reservation. If you fill the window with context and leave no room, generation is truncated mid-sentence, and with structured output that means unparseable JSON rather than a short answer — a failure that looks like a model defect and is arithmetic.

The line that causes cost surprises is history. It grows every turn while everything else stays flat, so a conversation that was cheap at turn three is several times the price at turn thirty, and the growth is silent. Either summarise older turns into a running digest, or drop them on a policy you can state.

Retrieval is the line to question first when quality is poor. Twelve chunks is not obviously better than four: the middle of a long context is where models attend least reliably, so more passages can lower answer quality while raising both latency and cost.

You get an error from the provider, or silent truncation, depending on the API and the client library — and which of those two you get is worth knowing before it happens in production. Truncation is the more dangerous outcome because the request appears to succeed while the model never saw the end of your prompt, which is typically where the actual instruction sits.

The failure is also load-dependent in a way that hides it in testing. A prompt built from retrieved documents and a growing conversation crosses the limit only for the users with the longest histories or the largest documents, so it arrives as sporadic complaints rather than a clean break. Count tokens before sending rather than reacting to the error.

No, for three reasons that survive every increase in window size. Cost and latency scale with what you actually send, so pasting a whole corpus each turn is expensive and slow even where it fits. Attention is not uniform across a long context, so a fact buried in the middle of a very long prompt is less reliably used than the same fact retrieved and placed near the instruction. And a corpus grows without bound while a window does not.

There is a real shift, though, and pretending otherwise dates you: with large windows, chunking can be coarser and a whole document can often be included rather than four fragments of it, which removes a class of chunk-boundary defects.

Providers can cache the computed attention state for a prefix of your prompt, so a second request beginning with the identical prefix skips prefill for that part — charged at a lower rate and returning the first token sooner. The unit is an exact prefix match, which is what dictates prompt layout.

GOOD                              BAD
  [ system prompt      ]  cached    [ system prompt        ]
  [ tool schemas       ]  cached    [ user name and time   ] <- varies
  [ policy examples    ]  cached    [ tool schemas         ] <- now uncacheable
  [ retrieved context  ]            [ policy examples      ] <- now uncacheable
  [ user turn          ]            [ user turn            ]

Everything static goes first, everything variable goes last. Putting a timestamp, a user id or a session token near the top invalidates the entire prefix behind it, which is how teams end up with a cache hit rate near zero while believing caching is enabled.

It pays where a large fixed preamble is reused across many requests — a long system prompt, a big tool catalogue, a document being asked about repeatedly. It does nothing for one-shot requests with unique prompts, and it is not a correctness mechanism: expiry is a provider policy, so behaviour must not depend on a hit.

Because the work is different. Input tokens are processed in one parallel prefill pass over the whole prompt, which uses the device efficiently. Output tokens are produced one at a time, and each one requires a full forward pass through the model, reading the entire weight set from memory to produce a single token. That makes decode memory-bandwidth bound and poorly parallelised per request, so it costs more per token — providers price it accordingly, typically several times the input rate.

The design consequence is that verbosity is the expensive failure mode. Asking for a structured answer with named fields, capping the response, and not asking for restatements of the question are all cost controls, and they usually improve the output as well.

Gradually, and in a shape worth naming: models attend most reliably to the beginning and the end of a long input and least reliably to the middle. So as a prompt grows, you do not get an error — you get answers that quietly ignore a paragraph that was definitely present, most often one sitting in the middle.

There is a second effect once the prompt contains many similar passages: the model has to pick which of several plausibly relevant chunks to ground on, and near duplicates make that choice worse rather than safer. Both degradations are invisible without an eval set, because each individual answer looks fluent and reasonable. This is why "we increased the number of retrieved chunks" needs a measurement attached rather than an assumption of improvement.

Anything stable across requests: the role, the output contract, the policies, the tone, and the tool catalogue. Anything varying per request belongs in the user turn or in a separate context block. The split is partly semantic — instruction- tuned models weight system instructions more heavily and are more resistant to overriding them — and partly operational, because a stable prefix is what prompt caching keys on.

The mistake worth naming is putting per-user data in the system prompt because it feels authoritative. It destroys cacheability, and it teaches the model that untrusted content carries system authority, which is exactly the confusion injection exploits.

Enough to pin the format and the edge cases, which is usually two to five, and adding more has a poor return once the format is unambiguous. Examples are a demonstration of shape rather than a teaching set: they cost tokens on every request, and if a task needs dozens of them to work you are describing a fine-tuning job, not a prompt.

What matters more than the count is coverage of the awkward cases. Include an example of the ambiguous input and the correct refusal or abstention, because without one the model will invent a confident answer for those, and include a negative example only if it is genuinely instructive. Examples that all look alike also bias the output towards their surface features, so vary length and phrasing.

The contract is a schema plus explicit handling for what the model cannot know.

{
  "type": "object",
  "additionalProperties": false,
  "required": ["decision", "confidence", "evidence"],
  "properties": {
    "decision": { "enum": ["approve", "decline", "refer"] },
    "confidence": { "type": "number", "minimum": 0, "maximum": 1 },
    "evidence":  { "type": "array", "items": { "type": "string" }, "maxItems": 3 },
    "missing":   { "type": "array", "items": { "type": "string" } }
  }
}

Three details do the work. The enum removes the entire class of near-miss values — no "Approved", no "approve with conditions", no free-text explanation where a label was wanted. additionalProperties: false stops the model appending a helpful extra field that your parser then ignores or rejects. And the missing array gives the model somewhere to put "I did not have the income figure", which is the alternative to inventing one.

The field to be careful with is confidence. A model asked for a number will produce one, and it is not a calibrated probability; it is a fluent guess. Treat it as an ordinal hint at best, and if a real threshold matters, derive it from logprobs or from an outer evaluation rather than from a self-report.

Where a provider supports schema-constrained decoding, pass the schema to the API rather than describing it in prose, because then invalid output is impossible at the sampler level rather than merely discouraged.

Because it is a request, not a constraint, and the model satisfies it most of the time — which is the worst failure rate to design around.

what you asked for            what arrives sometimes
{"total": 42}                 Sure! Here is the JSON:
                              ```json
                              {"total": 42}
                              ```
                              Let me know if you need anything else.

The wrapper text, the code fence, a trailing comma, a stray apology, or a truncated object when the window ran out are all common. A regex that hunts for the first brace works until an answer legitimately contains braces in a string.

So the fix is layered rather than verbal. Use schema-constrained decoding where the provider offers it, since then the sampler can only emit tokens the grammar allows. Where it does not exist, parse defensively, validate against the schema rather than trusting field presence, and retry once with the validation error appended — a retry that quotes the specific violation succeeds far more often than a blind repeat. And reserve output tokens, because a truncated response is unparseable for reasons no prompt wording can fix.

Letting a model produce intermediate tokens before its answer measurably improves tasks with real multi-step structure — arithmetic, constraint satisfaction, careful extraction from a messy document — because each subsequent token is conditioned on the work so far. On simple lookup or classification it adds nothing.

The costs are concrete. Those tokens are billed and they delay the answer, and on tasks where reasoning is not needed the extra length can hurt by giving the model room to talk itself into a wrong commitment. There is also an interpretability trap: the stated reasoning is not a faithful log of the computation, so it is a poor audit artefact even when the answer is right. Where the format matters, separate the thinking from the result so you can drop the former before parsing.

Assume the prompt is fitted to the model it was written against, because it is. Instructions that compensate for a specific quirk — pleading for JSON, banning a phrasing the old model overused, an example that fixed one failure — are the parts most likely to become dead weight or actively harmful on a new version.

The defence is to hold the contract and the evals rather than the wording. Keep the output schema outside the prompt where the API can enforce it, keep a versioned eval set that runs against any candidate model, and pin the model version in production so an upgrade is a deliberate act with a measurement rather than a Tuesday surprise. When you do upgrade, delete the compensating instructions and see what breaks instead of carrying them forward untested.

When the task has genuinely separable stages with different failure modes. Extraction, then validation, then generation is three prompts because each wants a different temperature, a different schema and a different retry policy — and because a failure in a chain tells you which stage broke, while a failure in one enormous prompt tells you only that the output was wrong.

The counter-pressure is real, though: every extra call adds latency, cost and its own failure surface. So split on the boundary where you would want a different measurement, not on every conceptual step. Three purposeful calls beat one unreadable prompt and also beat eleven micro-calls with a step budget nobody can reason about.

Structured output constrains the shape of the model's answer. Tool calling is a protocol: you describe callable functions with schemas, the model emits a request to call one with arguments, your code executes it, and the result goes back into the conversation for the model to continue from.

{
  "tool_calls": [
    {
      "name": "get_account_balance",
      "arguments": { "account_id": "AC-4471", "as_of": "2026-07-28" }
    }
  ]
}

The critical point is what the model has and has not done. It has chosen a function and produced arguments; it has not executed anything, and it has no way to know whether the arguments are valid, permitted, or destructive. Every authorisation check, every validation, and every rate limit lives in your executor, and treating the model's argument object as trusted input is how tool calling turns into a vulnerability.

The second point is that the model routinely gets arguments subtly wrong — an invented id, a date in the wrong shape, an enum value that does not exist. So validate against the schema, return a specific error as the tool result rather than throwing, and let the model correct itself. A tool that returns "account not found" recovers; one that raises an exception ends the turn.

Four things, in ascending cost. Rewrite the prompt, because most "the model cannot do this" turns out to be an underspecified instruction or a missing output contract. Add examples, which fixes format and edge-case behaviour cheaply. Retrieve, if the gap is knowledge the model was never trained on — that is the correct fix for facts, and fine-tuning is the wrong one. Then decompose the task into stages, which fixes the class of failure where one call is being asked to do three jobs.

Fine-tuning comes after those because it is the only option that adds a training pipeline, a dataset to maintain, an artefact to version and a model to redeploy whenever the base moves. Reaching for it first is the most expensive way to discover the prompt was the problem.

It freezes the base weights and trains small low-rank matrices alongside them, so what you store and serve is a set of adapters measured in megabytes rather than a full copy of the model.

base weight W        frozen, shared by every tenant
adapter  A x B       trained, rank r  - r much smaller than the weight dimension
effective weight     W + scale * A x B

rank      capacity to change behaviour     overfitting risk     size
low       small                            low                  tiny
high      larger                           higher               larger

Three consequences follow from that shape. The base model stays intact, so you can serve many adapters against one loaded base and swap them per request or per tenant, which is what makes per-customer tuning affordable at all. Training is cheaper in memory because gradients only flow through the adapters, which is what puts it within reach of a single GPU. And rank is the main capacity dial: too low and the adapter cannot express the behaviour you want, too high and you have rebuilt full fine-tuning's cost and overfitting risk without its ceiling.

What it does not change is knowledge coverage. An adapter reshapes behaviour, style, format adherence and task framing far more readily than it inserts new facts, and facts it does absorb cannot be updated without retraining. QLoRA is the same idea with the frozen base held in quantised form to cut memory further, trading a little fidelity for fitting a larger model on smaller hardware.

When the requirement is behavioural rather than informational, and repetitive enough to amortise. A rigid house format that prompting keeps drifting from, a classification task with thousands of labelled examples where you want a small cheap model to match a large one, a domain register the base model does not naturally produce, or latency and cost targets that only a smaller tuned model can hit — these are the cases where it earns its keep.

The reverse test is just as useful. If the answer changes when your documents change, you want retrieval. If the answer is right but formatted wrong, try the schema first. If you have fewer than a few hundred good examples, you have a prompt problem with a training-shaped budget attached.

Enough examples to teach the pattern, consistency in the thing you are teaching, and separation from anything you will evaluate on. Consistency is the part teams underestimate: two annotators disagreeing about the same input teaches the model to average them, which shows up as hedged, mid-way outputs rather than as a visible error.

It also needs the hard cases, including the ones where the right answer is a refusal or an abstention, or the tuned model will confidently answer them. Hold out a test split before training, not after, and keep it stable across runs so successive attempts are comparable. And record the provenance of every example, because a dataset you cannot explain is one you cannot defend when the model's output is questioned.

Supervised fine-tuning shows the model correct outputs and trains it to reproduce them. Preference tuning shows it pairs — this response is better than that one — and optimises for the ranking, whether through a reward model and reinforcement learning or through a direct method such as DPO that skips the separate reward model.

The difference matters when the target is a quality nobody can write down. Nobody can author the ideal helpful, non-evasive, appropriately hedged answer for every input, but people can reliably say which of two answers is better. That is why the instruction-following behaviour of production models comes from this stage. It also explains characteristic side effects: optimise against judges who prefer longer, more confident answers and you get longer, more confident answers.

Training hard on a narrow task degrades capabilities you did not test. A model tuned to emit terse classification labels can lose the ability to explain itself, follow unrelated instructions, or refuse appropriately, because nothing in the training signal preserved those behaviours.

You notice it only if you kept a broad regression set from before the tune, which is why one is worth building before the first training run rather than after the first incident. Mitigations are lower learning rates, fewer epochs, adapter-based methods that leave the base intact, and mixing a slice of general instruction data into the training set. The failure is quiet: the tuned metric improves, ships, and the regression surfaces weeks later as unrelated complaints.

By comparing it against the prompted baseline on a held-out set you fixed before training, on the metric that matters to the feature rather than on training loss. Loss going down tells you optimisation worked, not that the product improved, and those two come apart routinely.

The comparison has to be honest in two ways. The baseline must be a genuine effort — a well-written prompt with examples, not a deliberately weak one — because beating a strawman is the commonest way a fine-tune is justified. And the evaluation must include the broad regression set as well as the target task, so an improvement of four points on the narrow metric that cost you refusal behaviour is visible as the trade it is.

Production traffic and real failures, not from imagination. The cheapest good starting point is the log of actual user requests, sampled to cover the shapes you see, plus every complaint and every incident turned into a case. Cases invented at a desk cluster around what the author already thought of, which is precisely the part the system already handles.

Each case needs an input, a statement of what a correct answer must contain, and a note of why it is in the set. That last field is what stops an eval set rotting: in six months, nobody remembers whether a case was added because of a bug or as an example of desired tone, and without knowing, nobody dares change the expectation. Keep it small enough to run on every change and grow it from failures.

You use a model to score another model's output against a rubric. It is the only way to evaluate open-ended generation at any volume, and it is systematically biased in ways you can measure and partly correct.

bias                what it does                        mitigation
position            prefers the first option shown      run both orders, average
verbosity           prefers longer answers              cap length, score per criterion
self-preference     prefers its own family's style      judge with a different family
sycophancy          agrees with an assertion made       never state your expectation
                    in the judge prompt                 in the judge prompt
leniency            scores everything 4 out of 5        force a rubric with anchors

The two structural fixes matter more than the individual ones. Pairwise comparison — is A or B better — is far more reliable than absolute scoring, because a model is better at ranking than at calibrating a number. And a rubric with concrete anchors for each level beats a bare "rate helpfulness from one to five", for the same reason a hiring rubric with behavioural anchors beats a gut score.

Then validate the judge itself. Have humans score a couple of hundred cases, measure agreement with the judge, and only trust the automated scores in the regions where they agree. A judge nobody has checked against human judgement is a number generator, and teams routinely ship against one for months.

The point of a scorecard is that quality decomposes into criteria which fail independently.

criterion        how it is measured                     gate
groundedness     every claim traceable to source        hard fail below 0.98
coverage         key points present, judged pairwise    regression below baseline
length           within 10 percent of target            warn
format           parses against schema                  hard fail below 1.00
refusals         declines when source insufficient      hard fail below 0.95
cost             tokens per request                     warn above baseline x 1.2
latency p95      end to end                             hard fail above 4s

Two things make this usable rather than decorative. Each criterion has an independent gate, so a change that improves coverage while breaking groundedness cannot ship on an averaged score — averaging is how a fluent, subtly unfaithful summariser passes. And format and refusal are deterministic checks rather than judged ones, which means most of the suite runs without a judge at all and is therefore fast, free and stable.

The row candidates leave out is refusals. A summariser that cheerfully summarises an empty or irrelevant document is the single most common production embarrassment in this feature class, and it is invisible unless you deliberately include cases where the correct behaviour is to decline.

Not as a single rate, because there are two different failures. One is ungroundedness: the answer asserts something the provided source does not support, which is checkable — decompose the answer into claims and test each against the retrieved passages, by exact attribution where the format allows it and by a judge where it does not. The other is factual error against the world, which needs ground truth you have to supply, so it can only be measured on a curated set.

Report them separately. A retrieval-grounded assistant is judged on groundedness, because that is the property you can enforce; a general question-answering feature is judged against known answers. Conflating them produces a number that moves for reasons nobody can attribute, and it hides the useful diagnosis, which is whether the model invented something or faithfully repeated a bad retrieval.

The same shape as a test suite, with the prompt as the artefact under test.

prompts/summarise/v7.md          the prompt, versioned in the repo
evals/summarise/cases.jsonl      120 cases, each with input and expectations
evals/summarise/report.json      last run - pass rate, per-criterion scores, cost

on change to either file:
  run all cases against the pinned model
  compare per-criterion against the stored baseline
  block the merge on any hard-gate failure or a regression over 2 points

The mechanism that makes this work is pinning the model, because otherwise a regression is unattributable — you cannot tell your prompt edit from a provider's silent update. Pin in CI, and run a separate scheduled job against the floating version so provider drift shows up as its own alert rather than as noise in everyone's pull requests.

Sampling variance is the other detail. At temperature zero most cases are stable, but a handful sit near a decision boundary and flip between runs. Run those several times and score the majority rather than chasing a flaky suite, and treat a case that flips constantly as a signal about the prompt rather than about the harness.

Because they measure what you thought to include. Real traffic contains inputs nobody imagined, in proportions nobody predicted, and the distribution moves as users learn what the feature can do. An offline suite at ninety-five percent tells you the known cases pass.

So the offline suite is a gate, and production measurement is the truth. That means logging inputs and outputs with consent and retention rules, sampling for human review on a schedule, and instrumenting the implicit signals — retries, abandonment, edits to the generated text, escalation to a human, thumbs down. Those signals are noisy individually and reliable in aggregate, and their job is to feed new cases back into the offline set.

At two levels, because a correct outcome can hide a terrible path. Outcome evaluation asks whether the final state is right — the ticket updated, the answer correct, the file written. Trajectory evaluation asks what it did on the way: how many steps, which tools, how many failed calls and retries, whether it looped, and whether any step had a side effect that should have required approval.

Outcome alone is insufficient because two agents with identical success rates can differ tenfold in cost and risk. Trajectory alone is insufficient because a tidy path to the wrong answer is still wrong. Both need per-run cost and step counts attached, and a step budget in the harness, or a single pathological run can spend a day's budget while the aggregate success rate looks unchanged.

It is the interval between the request and the first streamed token, and it is dominated by prefill — processing your entire prompt — plus queueing behind other requests. Perceived speed tracks it because a reader waiting at a blank screen cannot distinguish a slow model from a broken one, whereas text appearing immediately buys patience for a long answer.

The two numbers to hold separately are time to first token and inter-token latency. A feature can be fast to start and slow to finish, or the reverse, and the fixes do not overlap: shortening the prompt, caching a shared prefix and reducing queueing address the first, while a smaller or quantised model, less output and speculative decoding address the second.

Batching runs several requests through the device together, which raises throughput substantially because the weights are read once for the whole batch rather than per request. It raises individual latency, because your request waits to be scheduled and then shares the device.

Continuous batching is the refinement that makes this practical for interactive serving: instead of waiting for every sequence in a batch to finish, the scheduler admits new requests into free slots as others complete, so a short request is not held hostage by a long one. The trade-off remains a dial rather than a solved problem, which is why interactive and bulk traffic usually want separate pools with different batching policies rather than one queue serving both.

During decoding, each new token attends to every previous token, so the key and value tensors for the prompt and the generated text so far are cached rather than recomputed. That cache lives in GPU memory alongside the weights, and it grows with sequence length and with the number of concurrent requests.

GPU memory
  model weights        fixed        loaded once
  KV cache             variable     grows per request, per token
  activations          transient

concurrency = memory available minus weights, divided by KV cache per sequence

so:  longer contexts  -> fewer concurrent requests on the same hardware
     more concurrency -> shorter contexts, more memory, or paged attention

The planning consequence is that context length and concurrency trade against each other on fixed hardware, which is not obvious from a model card. Doubling the context you allow can halve how many users a device serves, so a decision that looks like a product improvement is a capacity decision.

This is also why paged attention and similar techniques matter: allocating the cache in fixed blocks rather than one contiguous reservation per request removes the waste from reserving for the worst case, and raises achievable concurrency without new hardware.

Storing and computing weights at lower precision — eight-bit or four-bit rather than sixteen — shrinks memory and raises throughput, letting a larger model fit on smaller hardware or a given model serve more users. The cost is fidelity, and it is unevenly distributed: aggregate benchmark scores often move little while specific capabilities degrade, typically the fragile ones such as long arithmetic, exact formatting and multilingual edge cases.

So it is an empirical decision, not a table lookup. Run your own eval set against the quantised variant rather than trusting a published average, and pay particular attention to structured-output adherence, which is where degradation most often shows up as a production defect rather than as a lower score.

Whenever the task is grounded rather than open-ended, which covers most enterprise work. If the answer must come from your documents, retrieval quality dominates and the model's job is to read and phrase rather than to know — and a small model is usually adequate at that while being cheaper, faster and easier to run under a latency budget.

The larger model earns its cost on genuinely hard reasoning, on ambiguous instructions, on long multi-step tool use, and on tasks where breadth of world knowledge is the point. The useful discipline is to route rather than to pick once: establish the cheap path as the default and escalate on a measurable trigger, which is a decision you can defend with numbers.

By classifying the request and sending it down the cheapest path that will satisfy it, with an escalation route for the cases that fail.

flowchart TD
    R[request] --> C{cheap model<br/>confident}
    C -->|yes| A[answer]
    C -->|no| L[large model]
    L --> V{validates<br/>against schema}
    V -->|yes| A
    V -->|no| H[human review]

The interesting edge is the second gate rather than the first. Escalating on low confidence is easy to build and easy to fool, since a small model can be confidently wrong; validating the escalated answer against a checkable contract is what stops the router silently degrading quality to save money.

Routing pays when traffic is genuinely mixed, and it costs you a second system to evaluate and monitor — two models, two prompt sets, and a classifier whose own errors are now part of your quality story. Do it when the cost gap is large and the traffic mix is stable; do not do it to shave a few percent.

Several things that are fine in a single response. You cannot validate a schema until the output is complete, so streaming structured data means either buffering it and losing the benefit, or parsing incrementally and handling partial states. You cannot run a moderation pass over the whole answer before the user sees the beginning of it. Errors arrive mid-stream, after the client has already rendered half an answer, so the client needs a way to retract or mark it. And retries are no longer transparent.

Streaming is still right for anything a human reads, because it transforms perceived latency. The rule is to stream prose and buffer contracts: text for a person streams, JSON for a parser does not.

Instructions smuggled into content the model reads rather than typed by the user. A web page, a PDF, an email or a calendar invite contains text addressed to the model, and because instructions and data occupy the same token sequence, the model has no reliable way to tell your policy from the document's demand.

flowchart LR
    A[attacker page] --> B[retrieval]
    B --> C[model context]
    U[user request] --> C
    C --> D{tool call<br/>chosen}
    D --> E[send email tool]
    E --> F[attacker inbox]

The edge to look at is where the attacker's text and the user's request merge into one context, because everything after that point is the model acting on both with equal authority. The damage is done by the tool, not by the text: a model with no capabilities can be manipulated into saying something wrong, while a model that can send mail, write to a repository or call an internal API can be manipulated into doing something wrong.

That is why the defences that work are architectural rather than textual. Give the model the narrowest possible tool set, require approval for anything irreversible or outbound, validate and authorise every tool call in your executor against the real user's permissions rather than the model's intent, and constrain egress so a tool cannot reach an arbitrary destination. Instructing the model to ignore instructions in documents helps at the margin and fails under a determined attempt.

Because it is the same kind of system with the same weakness, applied to the same ambiguous input. A classifier that flags injection attempts raises the cost of an attack and does not close it: attacks can be paraphrased, encoded, split across documents, or expressed in another language, and the guardrail has no more access to intent than the model it protects.

There is also a base-rate problem. Injection attempts are rare against most corpora, so a detector with a plausible false-positive rate blocks a great deal of legitimate content to catch very little, and teams respond by loosening the threshold until it stops catching anything. Use guardrails as one layer among capability limits, approval gates and egress control, and never as the reason it is safe to give an agent write access.

By deciding what it may do before deciding what it should do. Each tool gets the narrowest signature that satisfies the use case, so a lookup tool cannot write, a write tool takes an explicit target rather than an arbitrary path, and a query tool runs a parameterised statement rather than accepting SQL.

Then authorisation runs on the real user's identity in your executor, not on the model's assertion, and anything with an irreversible or outbound effect requires a human confirmation step. The useful mental model is that the model is an untrusted client of your API: you would not accept arbitrary requests from a browser because it claimed to be acting for an administrator, and a model deserves exactly the same suspicion.

An attacker persuades the model to place secret data where the attacker can read it, using a capability you deliberately gave it. The classic shapes are a URL fetched with the data in its query string, an image reference whose host logs the request, an outbound email or webhook, and a comment written into a shared document.

It is stopped at the boundary rather than in the prompt. Restrict which hosts a fetch tool may reach, strip or refuse to render remote references in model output, require approval for outbound messages, and log every tool call with its arguments so an attempt is visible after the fact. Notice that markdown image rendering is a capability: if your client fetches an image URL the model produced, the model has a data channel nobody wrote a tool for.

Decide what may cross the provider boundary and enforce it before the call, not after. That means redacting or tokenising identifiers where the task does not need them, and knowing your provider's terms on retention and training use rather than assuming.

Then treat traces as the copy of user data they are. A trace useful for debugging contains the prompt, the retrieved documents and the answer, which is often more sensitive than your application database and rarely has the same retention rules, access controls or deletion path. Sample rather than logging everything, redact on the way in, set a retention period you can justify, and make sure a deletion request reaches the trace store — because a subject access request that misses your observability stack is a gap you will have to explain.

A jailbreak is the user trying to make the model violate its own policies — roleplay framings, hypotheticals, encoded requests. Injection is a third party making the model act against the user's interests using content the model reads. The distinction matters because the threat models differ: a jailbreak is a policy-compliance problem where the user is the adversary, and injection is a security problem where the user is the victim.

They need different responses. Jailbreak resistance is largely the provider's tuning plus your output filtering and refusal policy. Injection defence is your architecture — capabilities, authorisation and egress. Teams that conflate them buy a content filter and believe they have addressed both.

You are, if you put it in front of users. That is not a legal opinion so much as the operating assumption every regulator and every customer applies: the feature speaks with your name on it, and "the model produced that" is not a defence anyone accepts.

Practically, accountability means three things being in place before launch. A stated scope for what the feature does and refuses to do, so a wrong answer outside that scope is a caught case rather than a surprise. Logging good enough to reconstruct what was retrieved and produced for a specific request. And a route for a human to intervene and correct — because the first serious complaint is answered by what you can show and what you can fix, not by the model card.

Because it is code that determines behaviour, and treating it as configuration edited in a console removes every property you rely on elsewhere. In the repository it gets review, history, a diff you can read during an incident, and a deploy process; in a database field edited at will it gets none of those, and the answer to "what changed at four o'clock" is nobody knows.

Versioning also makes evaluation meaningful. A stored score belongs to a specific prompt against a specific pinned model, and neither number means anything if either side can move without a version bump. The practical arrangement is prompts as files, evals alongside them, and the model version pinned in the same place.

Enough to reconstruct the request and to attribute both cost and failure.

request_id            correlates with your application logs
prompt_version        which artefact produced this
model + version       pinned identifier, not just the family name
input_tokens          split by cached and uncached prefix
output_tokens         the expensive half
retrieved_doc_ids     with scores, so a bad answer is diagnosable
tool_calls            name, arguments, result status, duration
latency               time to first token and total, separately
outcome               parsed, schema-failed, refused, retried, error
user_feedback         attached later, by request_id

The two fields teams omit and then need are the retrieved document ids and the prompt version. Without the first, a wrong answer cannot be split into a retrieval failure and a generation failure, which is the only useful first question. Without the second, a quality change cannot be attributed to the prompt edit that caused it.

Note also that this record is sensitive by construction, so retention and access belong in the design rather than being added after the first audit.

By running the new model against your eval set first, then against real traffic in a limited way, and comparing on the metrics you already track rather than on impressions. Shadow mode is the cheapest first step: send a copy of production requests to the candidate, compare outputs offline, and pay for two calls on a sampled slice rather than risking user-visible change.

Then ramp a percentage of live traffic with the quality, cost and latency dashboards split by variant, and hold the ramp long enough for the slower signals — retries, escalations, complaints — to accumulate. The rollback path must be a configuration change rather than a deploy, because the moment you need it is the moment you do not want to be shipping.

Input tokens, output tokens, and everything you paid to build the input.

assume, for a retrieval-backed answer:
  system prompt and tools     3,000 tokens   cached after the first call
  retrieved passages          6,000 tokens   uncached, varies per request
  conversation history        2,000 tokens   grows with the turn count
  answer                        600 tokens   priced several times the input rate

so per request:  ~11,000 input tokens, 600 output tokens
plus:            1 embedding call for the query
plus:            1 reranker pass over 50 candidates
plus:            the retry rate, which multiplies everything above

The lines that surprise people are the last two. A reranker or a second validation call can cost more than the generation it protects, and a fifteen percent retry rate is a fifteen percent surcharge on the whole stack, not on the failures.

The other omission is the embedding and index cost, which is capital rather than per-request: re-embedding a corpus after a model change is a one-off bill that belongs in the same budget conversation.

You migrate on their timetable, which is why the model version is pinned and the eval set exists. The work is not swapping a string: prompts contain instructions fitted to the old model's quirks, few-shot examples tuned to its formatting habits, and latency and cost assumptions baked into product decisions.

So treat it as a change with a harness. Run the eval set against the candidate, delete the compensating instructions rather than carrying them forward, re-measure cost and latency because both will have moved, and shadow the new model on real traffic before switching. The teams that find this painless are the ones who kept the evals; the teams that find it a crisis are the ones whose prompt was the only specification.

On the things that are cheap to measure and specific enough to act on: schema validation failures, error and timeout rates, p95 time to first token, refusal rate, retry rate, and cost per request against a baseline. Each of those has an obvious first action when it moves.

What not to alert on is an averaged quality score. It moves for many reasons, it lags, and it produces the alert nobody can act on at three in the morning. Quality belongs in a dashboard reviewed on a schedule, with a sampled human review feeding new cases into the eval set. Alert on the mechanics; inspect the quality deliberately.

At the retrieval, before anything else. In a grounded feature the great majority of production wrongness is the model faithfully answering from the wrong context: the passage was never retrieved, or it was retrieved and outranked, or it was in the prompt and stale. That is checkable per request if you logged the retrieved document ids, which is why that field is the one worth insisting on.

stage                      what proves it            verdict if it fails
in the index at all        lookup by document id     ingestion or freshness
in the top k for the       replay the query          retrieval or ranking
  query as phrased
survived into the prompt   the assembled prompt      truncation or ordering
answer matches passage     read them side by side    generation

A strong answer then walks the stages in order and names what would prove each one. Was the supporting document in the index at all, and at what version. Did retrieval return it in the top k for that query phrasing. Did it survive into the prompt after truncation and reordering. Does the answer contradict the passage that was supplied, which is the only case that is genuinely a generation failure. And it separates the two distributions that make demos misleading: demo questions are the ones the author already knew worked, while production questions arrive in phrasings and edge cases nobody wrote down.

A weak answer reaches for the model. It proposes a bigger model, a lower temperature, or a sterner instruction not to make things up, all of which are plausible and none of which are diagnoses — and it treats "twice a day" as a rate to be argued down rather than as a set of specific failures each with a traceable cause. The tell is whether the candidate asks for the failing examples. Two wrong answers a day is roughly sixty cases a month, which is an eval set, and the first job is to turn them into one.