Five months after a machine failure, an engineer asks you for the high-rate data around it. Was your retention policy right?

Retention is a bet about which questions get asked

A retention policy is not a storage decision, it is a decision about which future questions you are prepared to answer. Nobody writes it that way, which is why the conversation in the interview usually reveals whether a candidate has ever been on the receiving end of the request in the question.

The request has a shape that recurs. Something failed. Weeks or months later a root cause investigation, a warranty claim, a supplier dispute or a customer complaint reaches back for evidence, and what it needs is not the trend that a dashboard showed but the detail underneath it: what the current did in the two seconds before the drive faulted, whether the temperature spiked or merely rose, whether the pressure oscillated. Those are the facts a mean over five minutes cannot carry, and a policy that rolled up to five-minute means after thirty days has not compressed that evidence, it has destroyed it.

So the honest framing of the answer is that you cannot keep everything at full rate forever and you should not try, and therefore the design task is to be deliberate about what you are choosing to make unanswerable. A candidate who can state that sentence and then name three questions their policy has knowingly given up is doing better than one who quotes a tiering scheme.

Resolution is lost twice, and the first loss is invisible

There are two independent places detail disappears, and candidates almost always discuss only the second.

The first is at the collector, before anything is stored. Industrial data acquisition rarely records every sample. A collector configured with a deadband reports a value only when it has moved by more than a tolerance since the last reported value, which is exception reporting, and it is why a historian holding a slow process signal can be far smaller than the sample rate would suggest. Archive-side compression goes further, keeping a point only where the signal departs from a straight line drawn through recent points by more than a tolerance, so a steadily ramping value costs almost nothing to store.

Both are sensible and both are lossy in a specific direction: they discard small, fast movements. A deadband tuned for a temperature that drifts over hours will erase the ripple that indicates a control loop hunting, and a linear-trend compression will smooth away a brief excursion that never got large enough to breach the tolerance. That loss happened at acquisition time. No retention policy can recover it, and no query will tell you it occurred, because the stored series looks perfectly clean. Knowing which tags have deadbands, and what those settings imply about the faults you could detect, is part of knowing what your data means.

There is a second-order consequence worth naming. Compressed series are unevenly spaced, so a value at a given instant is an interpolation rather than a measurement, and computing an average by treating stored points as equally weighted samples silently over-weights the periods when the signal was moving. Time-weighted aggregation is the correct operation on a compressed historian series, and it is a common source of numbers that do not tie up between two tools.

Downsample to extremes and counts, not to a mean

The rollup, when it comes, should preserve the shape of what happened rather than its central tendency. For each bucket keep the minimum, the maximum, the count of samples, the sum, and where it matters the first and last values. That set costs a handful of numbers per bucket per tag, and it answers a strikingly wide range of later questions: whether a limit was ever exceeded, how long the machine was in a state, what the excursion reached, and whether the sensor was reporting at all.

Contrast that with a mean, which answers only what the level was on average and actively conceals the two things investigations most often need, namely the peak and the gap. A five-minute mean of a current signal will hide a two-second stall entirely, and it will look identical whether the sensor delivered three hundred samples or three.

The count is the field most often omitted and the one that saves you. A bucket with a count far below what the sample rate implies is either an outage, a network problem or a deadband doing more work than you expected, and without the count all three are invisible in the rollup. Retaining a per-tag, per-interval count is the cheapest data-quality instrument available, and it is what lets you answer honestly, five months later, whether the data you are handing over is complete.

Where the tag drives a control decision or a quality claim, the state changes matter more than any statistic. For discrete signals, store transitions with their timestamps rather than sampling a boolean, because a machine that stopped for eleven seconds is a fact and a boolean sampled every thirty seconds is a rumour.

Let the event promote the data

The technique that resolves the question properly is to decouple retention from age. Instead of one policy per tag applied uniformly over time, keep a short window of full-rate data everywhere and let significant events promote a slice of it to long retention.

Mechanically this lives at the edge. The gateway or collector holds a rolling buffer of the last few minutes of full-rate samples for the tags around a machine. When a trip, an alarm, a quality reject or a threshold crossing occurs, the buffer contents for a window before and after the event are copied out and marked for long retention, tagged with the event that caused them. The plant then has full-rate evidence for the moments that mattered and rollups for the ninety-nine per cent of the time that nothing happened, and the volume of promoted data is bounded by the event rate rather than by the sample rate.

This is the only affordable answer for high-frequency signals. A vibration waveform captured at tens of kilohertz for a few seconds is a large object, and capturing one per machine per event, plus a scheduled reference capture so you have a healthy baseline to compare against, is entirely practical where capturing continuously is not. The derived features you compute from each waveform are small and can be retained indefinitely, so the trend survives even when the waveform is aged out.

Two implementation details make or break it. The event has to be able to reach the edge quickly enough that the buffer still holds the pre-event window, which argues for detecting the trigger locally rather than in the cloud. And the promotion has to be idempotent and traceable, so that a replayed trigger does not duplicate the slice and so that anyone reading the archive can see why that window exists.

Do the arithmetic before you have the argument

The retention conversation is usually lost because it is conducted in adjectives. Bring numbers derived from the tag census instead, and the discussion changes character.

Take a plant with five hundred tags collected at one sample per second. That is five hundred points per second, which is forty-three million points per day. At an order of eight bytes per point once compressed, that is a few hundred megabytes per day and something in the region of a hundred gigabytes per year for the raw tier. Those are assumptions, and stating them as assumptions is the point: the exercise shows that a year of raw data for a modest plant is not the expensive thing anyone feared, and that the cost problem lives elsewhere, in high-frequency waveforms, in image data from inspection, and in replication of all of it into three analytics platforms.

Which is also the answer to a candidate who says keep everything. Storage is often affordable and the cost of undisciplined retention is not primarily storage. It is that a very large, unindexed archive with no catalogue of what each tag means, no record of which periods are complete, and no owner is not evidence. Five months later somebody will still fail to answer the question, not because the data was deleted but because nobody can determine which of four similarly named tags is the one the engineer means, or whether the gap in March was an outage or a shutdown. The metadata is the retention policy's other half, and it is the half that decays without anyone noticing.

A retention policy is a list of questions you have agreed not to answer. Write that list down, show it to the reliability engineer who will one day ask, and let them tell you which items on it are unacceptable.