You need to flag anomalies and nobody has labelled a single one. How do you build it?

Anomaly is defined against a model of normal

Without labels there is no such thing as an anomaly in the abstract. You choose a description of ordinary behaviour, fit it to the bulk of your data, and score each record by how badly it fits. Every method below is a different answer to what "fits" means, and stating that assumption is what separates a candidate who has done this from one who has read a library index.

The first question to ask is whether you can assume the training data is clean. If you have a period you believe contains no anomalies, you are doing novelty detection: fit on the clean period, score new data against it. If the training data itself contains an unknown fraction of anomalies, you are doing outlier detection, and the method has to be robust to that contamination because the anomalies are shaping the model of normal you are fitting.

Distance and density

The simplest useful framing is that anomalies live in sparse regions. A k-nearest-neighbour distance score does this directly: the distance to a point's kth nearest neighbour is small in a crowd and large on the fringe. Local Outlier Factor refines it by comparing a point's local density to the density of its neighbours, which matters when the data has one tight cluster and one diffuse one. A single global distance cut-off would flag the whole diffuse cluster; a local comparison does not.

Both inherit the problems of distance in high dimensions. As you add features, distances between all pairs of points concentrate, and the contrast between near and far that the method depends on erodes. They also require a distance function that treats your features sensibly, which means scaling and careful handling of categoricals. A Gaussian mixture or a kernel density estimate makes the density model explicit and gives you a likelihood you can threshold, at the price of assuming a functional form.

Isolation forest

Isolation forest inverts the framing. It builds many trees by repeatedly picking a random feature and a random split value, and records how many splits it takes to isolate each point. Points in sparse regions get cut off in very few splits; points in dense regions need many. The average path length across the ensemble becomes the score.

The consequence is that it never estimates a density, so it does not degrade the way distance methods do as dimensionality rises, and it scales well because each tree is built on a subsample. Its weakness is the axis-parallel splitting: a correlation structure that is diagonal in feature space is awkward for it, and it can carve out low-score regions where no data lives, so genuinely odd combinations of otherwise ordinary values sometimes score tamely.

Reconstruction error

If the data has structure worth compressing, train something to compress and restore it and treat the failure to restore as the score. PCA gives you the linear version: project onto the leading components, project back, and measure the residual. An autoencoder gives you the non-linear version. Either way the model has learned the manifold the ordinary data lies on, and a record that cannot be rebuilt from that manifold is off it.

This is the most natural choice for images, sequences and high-dimensional sensor data, and it has a real diagnostic advantage: the per-feature residual tells an investigator which part of the record was unexplainable, which is far more actionable than a scalar. The trap is that a sufficiently large autoencoder trained on contaminated data will learn to reconstruct the anomalies too, and your scores flatten out. Keep the bottleneck genuinely narrow, and train on the cleanest period you have.

The threshold comes from the queue, not from the scores

Everything above produces a score. Turning a score into an alert needs a cut-off, and the reflex answers are all wrong for the same reason. Three standard deviations assumes a Gaussian your score distribution is not. The top one per cent assumes anomalies occur at a fixed rate, which is exactly the thing you do not know. A visible gap in the score histogram is usually not there.

Work backwards from operations instead. Ask how many alerts a day the team can genuinely investigate — say forty — and set the threshold to the score that produces roughly forty. That number is defensible, it is stable, it survives the score distribution drifting because you re-derive it, and it forces the conversation about what the system is worth. If forty alerts a day is unusable, you have learned something before building anything: the answer is to raise the bar and accept lower recall, or to group alerts by entity so one misbehaving account is one alert rather than nine hundred.

The threshold decision is also how you escape having no labels. Every investigated alert is a label. Log the verdict, and within weeks you have a precision estimate at your operating point, a growing supervised dataset, and the beginnings of an honest evaluation of the detector you shipped on faith.

The modelling choice is the easy part and the methods differ less in practice than their documentation suggests. What determines whether the system is used is the threshold, and the only credible source for it is the number of investigations the organisation can actually perform.