One of your features is a postcode with about 40,000 distinct values. How do you encode it?

Count the rows per level first

The encoding follows from arithmetic you can do before writing any code. Suppose 200,000 training rows and 40,000 distinct postcodes. The mean is five rows per level, and because postcode frequency is heavily skewed the median is lower still, with a long tail of levels appearing exactly once.

That single number settles most of the argument. One-hot encoding would produce a 40,000-column matrix in which the average column has five non-zero entries, so almost every coefficient is estimated from five observations. It also makes tree-based models behave badly, because each binary column offers only a trivially unbalanced split and the ensemble has to spend many splits to express what one categorical split could. Sparse matrices make the memory tolerable; they do not make the estimates meaningful.

Frequency encoding, which is safe and often enough

Replace each level with how often it occurs, as a raw count or a proportion. This costs one column, uses no label information whatsoever, and therefore cannot leak the target. It is a real signal in many business problems, because "rare postcode" and "common postcode" often correspond to something meaningful such as a sparsely populated area or a mistyped entry. It is cheap to compute at serving time from a small lookup, and an unseen level maps naturally to a count of zero. The limitation is obvious: two postcodes with identical counts become indistinguishable, so all the level-specific signal is thrown away.

Smoothed target encoding, and the arithmetic that makes it work

Target encoding replaces each level with the mean outcome for that level. It is powerful because it compresses 40,000 levels into one numeric column that carries the label-related ordering directly, which is precisely why it needs handling.

The immediate problem is the tail. A postcode with three rows and two positives has a raw mean of 0.667, against a global positive rate of say 0.05. That 0.667 is almost entirely noise, and a tree will happily split on it. The fix is to shrink each level's mean towards the global prior in proportion to how little data supports it:

encoded(level) = (sum_of_y + m * prior) / (count + m)

with m a smoothing weight expressed in pseudo-observations. Taking m = 20 and prior = 0.05 for the three-row level above:

(2 + 20 x 0.05) / (3 + 20)  =  (2 + 1) / 23  =  0.130

The raw 0.667 becomes 0.130, still above the prior of 0.05 because two positives out of three is some evidence, but no longer a confident claim. Now contrast a level with 800 rows and 40 positives:

(40 + 1) / (800 + 20)  =  41 / 820  =  0.050

which is essentially the empirical mean, because 800 observations swamp 20 pseudo-observations. The same formula therefore behaves differently at each end of the frequency distribution, which is exactly what you want, and m is the one hyperparameter it introduces.

The fold discipline is a correctness requirement

Target encoding uses the label, so computing it on all the training rows and then cross-validating puts each row's own outcome into its own feature. The encoded value for a level is contaminated by the very rows you are about to score, the model discovers it can read the label through that column, and validation reports a number that has no relationship to production.

Fitting the encoding on the training split only, then applying it to the validation split, is better and still not sufficient. Within the training split every row contributed to the mean it now receives, so the model overfits the encoding itself, and the validation score is depressed rather than inflated in a way that misleads you about the feature's value. The correct construction is out-of-fold: for each inner fold, compute level means from the other folds and assign them to this one. That is what scikit-learn's TargetEncoder does internally, and it is why the encoder belongs inside a pipeline rather than in a notebook cell above the split.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import TargetEncoder
from sklearn.model_selection import cross_val_score

# Inside the pipeline, so the encoding is refitted per fold and never
# sees the labels of the rows it is encoding.
pipe = make_pipeline(TargetEncoder(smooth="auto"), model)
scores = cross_val_score(pipe, X, y, cv=5)

Two further precautions belong in the same breath. Any unseen level at serving time must resolve to the prior, not to a null and not to a crash, and that behaviour has to be tested rather than assumed. Where the data is time-ordered, the encoding must be built only from periods preceding each row.

Hashing, native handling, and knowing what you gave up

Hashing maps each level into one of a fixed number of buckets, say 1,024, with no dictionary to maintain and no unseen-level problem at all. You accept collisions, which mix unrelated levels together, and you lose all interpretability. It is the right choice when cardinality is unbounded and growing, such as user-agent strings or URLs.

The mainstream gradient-boosting libraries also accept categorical columns directly. LightGBM partitions levels into two groups at each split, which is far more efficient than one-hot but can overfit on rare levels. CatBoost was designed around this problem and computes ordered target statistics, using only the rows preceding each observation in a permutation, which is the out-of-fold idea built into the training loop.

Whatever you choose, name what the encoding discarded. A postcode is a location, and reducing it to one number ignores that. Latitude and longitude, distance to the nearest branch, an area-level deprivation index, or an aggregation to the outward code all give the model real structure and far more rows behind each value. A candidate who reaches for that instead of tuning an encoder is answering a better question than the one asked.

Target encoding is the strongest option and the one that will quietly hand the model its own labels. Smooth every level towards the prior, fit it out of fold inside the pipeline, and define what happens to a level you have never seen.