What do you instrument and alert on for an LLM feature running in production?

The unit of observation is the request, with versions attached

Start with a span per model call, and make it a child of the application request so you can see the whole chain rather than the model in isolation. On it record the model identifier including its version, the prompt version, the input and output token counts, cached input tokens if the provider reports them, computed cost, latency split into time-to-first-token and total, the finish reason, and any retry attempts with their errors. For a retrieval feature, add the identifiers and scores of the documents retrieved; for an agent, the tool calls with their outcomes.

The version fields are what make everything else usable. Without them, every metric is an average over an unknown mixture of prompt variants and model versions, and the first question anyone asks during an incident — did this start when we changed something — is unanswerable. With them, every chart can be split by arm, which turns a canary from an act of faith into a measurement.

Finish reason and retries deserve specific mention because they are cheap and frequently ignored. A rising rate of length-limited completions means answers are being truncated mid-sentence, which users experience as the feature being broken while every dashboard stays green. Retries after rate limiting show up as latency, not as errors, so unless you count them separately you will diagnose a provider throttle as a slow model.

Cost, at the resolution where you can act

Aggregate spend tells you there is a problem after the invoice. Attribute cost per request, and then roll it up by feature, by prompt version, by customer or tenant, and by model. The interesting statistic is cost per successful outcome — per resolved ticket, per accepted suggestion — because that is the number that justifies the feature and the only one that reveals when a system is spending more to achieve the same thing.

Watch input tokens as a first-class series, separately from output tokens. Input growth is the usual cause of a cost surprise, and it creeps: a retriever returning more chunks, conversation history no longer truncated, a system prompt that gained three paragraphs. Track prompt cache hit rate alongside it, because a prompt edit that moves a volatile line above the stable block can take the hit rate to zero and multiply cost with no other visible symptom. And put a per-tenant ceiling somewhere, since a single automated client looping on your endpoint is a financial incident that no quality metric will show.

Quality signals you can count

The temptation is to grade every response and average the grades. Resist it, and build instead from signals that are countable per request, because those have the statistical properties an alert needs.

Output validation is the strongest of them and it is free: if you constrain output to a schema, the rate of responses failing to parse or failing your business-rule checks is a direct quality measure with no judgement involved. Refusal rate is next, detected either from a classifier or from the provider's own stop reasons, and it is bidirectional — a spike means the model has become skittish about legitimate requests, a collapse to zero means your safety behaviour has stopped working. Add tool-call error rates and, for agents, steps per task and the rate of hitting the step limit, which is what a stuck loop looks like from outside.

For groundedness, sample rather than grade everything. Take a fraction of responses, run a judge that checks whether each claim is supported by the retrieved context, and treat the resulting rate as a trend line reviewed daily rather than a page. Retrieval has its own diagnostics that are cheaper than judging the output: the score of the top retrieved chunk, the number of results above threshold, and the rate at which retrieval returns nothing useful. A slow rise in empty-retrieval rate predicts a rise in hallucination, and it is a plain counter.

Layer human review on top at low volume. A standing weekly review of a stratified sample, weighted towards low-confidence and high-cost requests, catches the failures that no automated check was written for, and it is the mechanism by which new automated checks get invented.

What users tell you without being asked

Implicit signals are the most honest quality data available, because they cost the user nothing to produce. The specifics depend on the product, but the pattern is consistent: regeneration and retry, editing the output before using it, copying part of it rather than all of it, abandoning the response part-way, escalating to a human, rephrasing the same question immediately. Thumbs-up and thumbs-down widgets are worth having and are heavily biased towards the annoyed, so use them as a source of examples to read rather than as a rate to trust.

Instrument these as events joined to the trace by request identifier. That join is what lets you pull the twenty worst-rated conversations of the week with their full prompts and retrieved context, which is where most real improvements come from.

Why an averaged quality score makes a terrible alert

Suppose you compute a nightly mean quality score of 4.2 out of 5 across sampled traffic. Now a change breaks one request category that is 5% of volume, completely. The mean moves by roughly a sixth of a point — inside the noise from sampling and from judge variance, indistinguishable from yesterday's fluctuation. The metric that summarises everything is precisely the metric least able to detect anything specific, and averaging also destroys the shape you need: a bimodal distribution of excellent and useless answers has the same mean as uniformly mediocre ones and a completely different remedy.

Alert instead on countable rates with a clear denominator, and segment them, because the segment is where the signal lives. Validation failure rate per prompt version. Refusal rate per request category. Empty-retrieval rate per document source. Tool-error rate per tool. Cost per request per tenant. Those pages when 5% of traffic breaks, because in that 5% the rate goes to a number nowhere near baseline. Reserve the sampled quality score for a weekly trend review and for comparing canary arms, where you have a controlled comparison and time to look at a distribution rather than a threshold.

Two structural rules keep this honest. Every alert needs a named response, or it will be muted within a month; if nobody knows what to do when refusal rate doubles, the alert is decoration. And the traces behind an alert must be reachable in one click, because the entire value of this instrumentation is that a rate tells you something is wrong and the trace tells you what.