Design the ingestion path for sensor telemetry from a few hundred machines. Which decisions matter most?

Put a number on it first

Any design conversation that starts with a product name is guessing. Start by sizing, out loud, from assumptions you state so the interviewer can correct them. Take 300 machines with around 200 tags each: that is 60,000 series. Sampled once a second, that is 60,000 points per second, which is 5.2 billion points a day. At a compressed twenty bytes or so per point you are looking at roughly a hundred gigabytes a day, thirty-something terabytes a year, before replicas.

Those numbers immediately drive the rest of the design. A hundred gigabytes a day is not a problem for a purpose-built time-series store and is a serious problem for a relational table with a b-tree index per column. It also means retaining raw data for five years, which someone will ask for, is a tens-of-terabytes commitment that has to be argued rather than assumed. And it tells you that the ingestion path needs batching: 60,000 individual writes per second is a very different system from 600 batches of a hundred.

State the assumptions you are least sure of, because they are where the estimate is fragile. Tag counts per machine vary wildly, and one vision system or power-quality meter can produce more data than the other 299 machines combined.

Cardinality is what actually kills the system

Volume is manageable; cardinality is the thing that takes a working system down six months after launch. In a time-series database the unit of storage and indexing is the series, identified by its name plus the full set of label values attached to it. Every distinct combination is a separate series with its own index entry and its own in-memory state, so cardinality is a product, not a sum.

The mistake is nearly always well-intentioned. Someone wants to filter readings by batch, so batch identifier becomes a label. Now every new batch creates a fresh copy of every series on that line, and cardinality grows without bound for as long as the plant keeps producing. Serial number, work order, operator, shift instance and firmware version all do the same thing. Anything whose distinct-value count grows with time does not belong in series identity.

The discipline is that labels describe the thing being measured — site, line, machine, component, tag name, unit — and never the event being produced. Contextual identifiers go in a separate relational or columnar table keyed by machine and time interval, and you join when you query. That join is more work at read time and it is the reason the system still runs in year three. If your store distinguishes indexed tags from unindexed fields, unbounded-but-useful values can go in a field, which costs storage rather than index.

Sampling: on interval, on change, and the deadband

There are two collection regimes and they produce differently shaped data. Fixed-interval sampling gives you a regular series that is easy to reason about and mostly redundant, since a tank temperature at one-second resolution repeats itself. Change-based collection with a deadband emits a point only when the value moves more than a configured amount, which is how historians typically work and is far cheaper on a slow-moving signal.

The consequence engineers get wrong is that a change-based series cannot be averaged by averaging its samples. If a value sat at 20 for fifty-five minutes and at 80 for five, the mean of the two recorded samples is 50 and the true hourly mean is 25. You need a time-weighted average, treating each sample as holding until the next one. Related: a long gap in a change-based series means "unchanged", whereas in an interval series it means "missing", and the same gap-filling logic cannot serve both. So the series has to carry which regime produced it, or you will apply the wrong maths to half your tags.

Pick the regime per signal rather than globally. Anything you intend to compute frequency content from — vibration, current signature — needs fixed-interval sampling at a rate set by what you want to see, and a deadband destroys it. Slow process values are the opposite.

Downsampling and retention as one decision

Downsampling and retention are the same conversation, because the retention you can afford depends on what you are willing to reduce. A workable shape is raw resolution for weeks, a one-minute rollup for months, and a five- or fifteen-minute rollup for years, with the rollups written by a continuous aggregate rather than a batch job you have to babysit.

The important detail is what the rollup keeps. Store only the mean and you have destroyed exactly the information anyone downstream wants: a two-second pressure spike vanishes into a one-minute average, and it was the spike that mattered. Keep min, max, count, sum and last for each window, and where you can afford it a representative percentile. Count is the one people forget, and it is what lets you distinguish "flat because nothing happened" from "flat because the collector was down", and what makes a mean of means correct when you roll up again.

Retention also has non-technical inputs. Regulated production may require the process record supporting a released batch to be retrievable for years, and that record is not the same set of tags as your analytics history. Separate them: a small, precisely specified, long-retention set for compliance, and a larger aggressively-reduced set for engineering.

The hour after the link comes back

Every design handles the outage. Fewer handle the recovery, and the recovery is where these systems break.

Buffering at the edge is straightforward in outline: the collector writes to local durable storage and forwards from it, so a network drop costs latency rather than data. Three properties make it actually work. The buffer must be bounded with an explicit policy for what happens when it fills — drop oldest, drop by tag priority, or stop accepting — chosen deliberately, because the default of growing until the disk is full takes the edge device down and loses everything including the current data. Timestamps must be applied at the source, so a value from 04:12 is stored as 04:12 rather than as its arrival time. And the buffer must be sized against a stated worst case: at 200 tags per machine and change-based collection, work out the bytes per hour and multiply by the longest outage you intend to survive, then say what happens beyond that.

Then the drain. When connectivity returns, every edge device on the site has hours of backlog and tries to deliver it at once, into a pipeline sized for steady state. The burst overwhelms ingestion, or it succeeds and floods a downstream consumer that is watching for anomalies and now sees hours of history arrive as new events. So the drain is rate-limited to some multiple of the normal rate, live data is prioritised over backlog rather than the buffer being drained in strict order, and backfill is tagged so consumers can distinguish replay from real time. The whole path must also be idempotent on machine, tag and source timestamp, because at-least-once delivery plus a retried drain guarantees duplicates.

One more thing the outage exposes: writing hours-old points into a store that has already compacted those time ranges is expensive in a way that writing current data is not, and on some engines it is much more expensive. Test the recovery path under load before you rely on it, because it is the path you will only ever exercise on a bad day.