When is a neural network the wrong choice?

Tabular data with modest row counts

For a table of numeric and categorical columns with thousands to low millions of rows, a gradient-boosted tree ensemble is the model to beat, and beating it with a neural network takes real effort for a margin that is usually not worth having. The reasons are structural rather than accidental.

Trees split on one feature at a time at a learned threshold, which is exactly the shape of most tabular relationships: a rule that changes at an age, a limit, a tenure. A network has to approximate that step with smooth functions and needs data to do it. Trees are indifferent to feature scaling, handle skewed distributions and outliers without transformation, and are largely unaffected by irrelevant columns, because a useless feature simply never gets chosen for a split. A network needs every one of those handled by you: scaling, encoding, imputation, and a decision about how to represent high-cardinality categoricals. Missing values are native to most boosting implementations and are a design question in a network.

The effort asymmetry is the part to say out loud in an interview. Boosting gives you a strong result from defaults in an afternoon, with three hyperparameters worth tuning. The network needs an architecture, an embedding scheme, a normalisation strategy, a learning-rate schedule and an early-stopping rule before it is competitive, and each of those is a place to be wrong. When both land in the same accuracy band, the one that took a day is the better engineering decision, and the one that took three weeks has consumed the budget you needed for feature work that would have moved the metric more than either model.

Deep learning earns its place where the input has structure a network can exploit and features cannot easily be hand-built: images, audio, long text, sequences with long-range dependencies, graphs. If your input is a spreadsheet, that argument is not available to you.

When the explanation has to be exact

There is a real difference between explaining a model and inspecting one, and it decides some projects.

A logistic regression on a modest feature set gives you a coefficient per feature. That is the model, not a summary of it, so the statement "this application was declined because the debt-to-income ratio contributed this much" is arithmetic anyone can reproduce. A shallow decision tree or a scorecard gives you the same property as a readable path. Under credit, insurance or clinical review, where you may have to state the reason for an individual adverse decision and defend it later, that reproducibility is the requirement.

Attribution methods applied to a network give you something weaker: an approximation of local behaviour, computed by a method with its own assumptions, which can disagree between methods on the same prediction. That is genuinely useful for debugging and it is not the same as a stated reason. It also fails a practical test that comes up constantly — whether the model behaves monotonically in a feature that everyone agrees should be monotone. A linear model with a signed coefficient guarantees it. Boosting libraries can constrain it. A network can be persuaded towards it and cannot be made to promise it, and "our model sometimes improves your score when your income falls" is not a sentence you want to defend.

Hard latency and memory budgets on CPU

The last case is arithmetic rather than judgement, so it is the easiest to make concrete. Suppose the serving budget for the model is five milliseconds inside a request that must complete in fifty, on the CPU nodes you already run, with no accelerator and no batching because requests arrive one at a time.

A linear model is a dot product and it fits in that budget with room to spare. A boosted ensemble of a few hundred shallow trees is a few hundred short branch sequences and generally fits comfortably. A network deep enough to have been worth building is a chain of dense matrix multiplications, and on a single CPU core without a batch to amortise the overheads it may not fit. GPU throughput numbers do not transfer: the accelerator's advantage is parallelism across a batch, and serving one request at a time is the case where it helps least.

Memory follows the same logic. A model that has to run inside a mobile application, on an embedded controller, or in a serverless function with a cold start you are billed for, is competing against a size limit, and the parameter count is the whole argument. Quantisation and distillation exist and genuinely help, but they are additional projects with their own accuracy cost, and choosing a model that already fits avoids both.

The failure to guard against is the reverse one

The mistake worth naming is not reaching for a network too eagerly. It is refusing to re-examine the decision. All three arguments above have conditions attached, and the conditions change.

If your row count grows into the tens of millions, if a raw text or image column appears alongside the tabular ones, if a genuine sequence structure emerges, or if you gain the ability to fine-tune a pretrained model rather than train from scratch, the balance moves. A pretrained model in particular breaks the small-data argument, because the network is no longer learning representations from your data alone — which is why an embedding from a pretrained text model fed as features into a boosted tree is often the best of both, and is worth volunteering as the answer to a hybrid case.

State the conditions when you give the recommendation. A candidate who says "trees here, and I would revisit it if we get text columns or pass ten million rows" is demonstrating the judgement being tested. One who says networks are overkill for tabular data and stops has given a correct answer for the wrong reason and will give the same answer when it is wrong.