When does dimensionality reduction help, and when does it hide the problem?

What PCA is optimising, and what that costs you

Principal component analysis finds the orthogonal directions along which your data varies most, and lets you keep the first few. The compression is optimal in a precise sense: for a given number of dimensions, no other linear projection retains more variance. Two things follow from that sentence and both are usually skipped.

The first is that the criterion is variance, and variance is not relevance. Nothing in the fit knows about your target, so a direction with small spread but a clean relationship to the label can be dropped in favour of a high-variance direction that is pure measurement noise. On a problem where the discriminative signal lives in a narrow feature, PCA will throw it away and the loss will look like the model underfitting.

The second is that the projection is linear. PCA can only rotate and drop axes, so structure that is curved in the original space is not recovered, merely flattened. Because the criterion is variance measured in whatever units the features arrived in, standardising first is not optional: leave one feature in seconds and another as a proportion, and the seconds feature becomes the first component by arithmetic rather than by importance.

Where it genuinely earns its place

Reduction helps most when redundancy is real rather than hoped for. A hundred sensor channels on one machine, or a wide set of near-duplicate financial ratios, carry far fewer independent quantities than columns, and collapsing them stabilises a linear model whose coefficients were previously being split arbitrarily between collinear inputs.

It also helps when the downstream method needs it. Distance-based clustering and k-nearest-neighbour methods degrade as dimensionality rises because pairwise distances concentrate, so projecting to a few dozen dimensions first often improves the clustering rather than merely speeding it up. Approximate nearest-neighbour indexes have the same property and a direct cost argument alongside it, because index size and query latency both scale with the vector width you store.

The mundane case is cost. If a 1,536-dimensional embedding at half the width retrieves nearly as well, halving it halves your memory and your index. That is a measurable engineering trade, and it is the one case where you can evaluate the reduction end to end without arguing about anything.

Whichever the motivation, the transform must be fitted inside the cross-validation loop. Fitting PCA on the full dataset before splitting lets validation rows influence the axes the training rows are projected onto, which is leakage, and it is the more insidious kind because the resulting score is only slightly and unreproducibly optimistic.

Visualisation is not evidence

t-SNE and UMAP produce two-dimensional pictures that look like discoveries. Both optimise the preservation of local neighbourhood structure, and both do so at the explicit cost of global geometry, which means the properties you want to read off the picture are the properties it does not preserve.

Distances between the visible blobs are not meaningful, so you cannot say two groups are similar because they sit close together. Cluster sizes are not meaningful, because the algorithms expand sparse regions and compress dense ones. Apparent separation depends on hyperparameters, and t-SNE's perplexity in particular will produce different numbers of apparent clusters on the same data. Most damaging of all, run on genuinely unstructured data both will still return something that looks like clusters, because separating points is what the objective rewards.

So the rule to state in an interview is that these plots are for generating hypotheses and never for settling them. If a UMAP picture suggests four populations, the next step is to test the claim in the original feature space: cluster there, check the grouping is stable under resampling, and check the groups differ on a variable you did not embed. A decision defended by a projection is a decision defended by a hyperparameter.

The interpretability you are trading away

Replacing features with components means every explanation now runs through a linear combination of everything. A coefficient on the third principal component is not something you can put in front of a credit-risk reviewer, an auditor or a clinician, and any regulatory requirement to state the reason for a decision becomes materially harder to satisfy.

The operational cost is quieter and bites later. Your model's inputs are now derived from a fitted transform, so the transform is an artefact that must be versioned and served with the model, and a drift in one raw column redistributes itself across all the components at once. Monitoring which input moved becomes an inference rather than a lookup, and debugging a bad prediction means un-projecting to find out what the model saw.

That cost is why reduction should not be the first response to a wide dataset. If the problem is collinearity, L2 regularisation handles it while keeping the original columns. If the problem is too many features, a gradient-boosted model is largely indifferent to irrelevant ones, and feature selection keeps names you can talk about. Reach for extraction when redundancy is the actual finding, not because the feature count feels uncomfortable.

PCA optimises variance and neighbour embeddings optimise local structure. Neither optimises the thing you are about to use the output to justify, so state which property you needed preserved before you reduce anything.