Ask who needs it and what they will do with it
"Explainable" is not a property, it is a relationship between a model and an audience, so the first move is to find out which audience and for what purpose. Four are common and they want incompatible things.
A customer who has been declined wants a small number of specific, true, actionable reasons about their own case — a local explanation, in their language, that does not enumerate the model's internals. A regulator or an internal model-risk function wants to understand the model's behaviour in general: which factors drive it, whether it behaves monotonically where monotonicity is expected, whether it relies on anything impermissible — a global account. An operator triaging a queue wants to know why this item was flagged so they can check it quickly, which is local and needs to be fast enough to render in a list. A data scientist debugging wants whatever is most diagnostic and has no constraints on rigour or presentation.
The audience determines nearly everything downstream, and skipping it is what produces the answer that names SHAP and stops. For a customer-facing statement of reasons, latency, wording and legal defensibility dominate; for a supervisory review, stability and documentation dominate and per-request cost is irrelevant.
Global and local are different problems
Global explanation characterises the model as a whole. Permutation importance tells you which features the model depends on, measured by how much performance degrades when a feature is scrambled. Partial dependence and accumulated local effects show the shape of the relationship between a feature and the output across the population, which is how you check that a factor moves the prediction in the direction the business expects. A surrogate — a shallow tree or a linear model fitted to reproduce the complex model's predictions — gives a readable global summary whose value is entirely contingent on its fidelity, so report how well it agrees with what it stands in for. A surrogate with weak agreement is not a simplified explanation, it is a different model with a misleading name.
Local explanation accounts for one prediction. Shapley-value attributions distribute the difference between this prediction and a baseline expectation across the features, with the appealing property that the attributions sum to that difference, which makes them presentable as a decomposition. LIME fits a simple local model to perturbations around the point of interest. Counterfactual explanations answer a different and often more useful question — what would need to change for the decision to be different — which is what a declined applicant is usually actually asking, and which is directly actionable in a way an attribution is not.
The two do not substitute for each other. A stack of local explanations is not a global account, because it says nothing about the regions of input space you did not sample. A global summary tells an individual nothing about their own case.
What attribution methods assume, and where they mislead
Attribution methods are useful and they are not oracles. State the limits, because a reviewer who knows them will be testing whether you do.
The baseline is a choice with consequences. A Shapley-style attribution measures the contribution of each feature relative to a reference distribution, so the same prediction explained against the training-set average and against a segment average yields different attributions, both correct with respect to their own baseline. If the explanation is going to a customer, the baseline is effectively part of the claim you are making, and it needs to be chosen and documented rather than defaulted.
Correlated features break the clean story. When two features carry overlapping information, credit can be distributed between them in more than one defensible way, and the answer depends on whether you condition on the correlation structure or intervene on features independently. The interventional treatment assigns credit to features the model uses; the conditional treatment can assign credit to a correlated feature the model never reads. Neither is wrong, they answer different questions, and the difference is invisible in the output. This is the main reason two implementations can disagree about which feature drove a decision.
Exactness and cost also vary. Attributions for tree ensembles can be computed exactly and quickly by algorithms exploiting the tree structure; the model-agnostic version is a sampling approximation, so it carries variance and two runs on one input may disagree. Before showing an explanation to anyone outside the team, test its stability: re-run it, perturb the input trivially, confirm the top reasons do not reorder. Nothing raises an error when they do, which is exactly why it must be tested.
Finally, an attribution says nothing about causation in the world. A high attribution to a feature means the model's output is sensitive to it, not that changing it would change the outcome. Presenting an attribution as advice — "reduce this and you will be approved" — makes a causal claim the model cannot support, and a counterfactual explanation over the model has exactly the same limitation.
The honest position
A post-hoc explanation is a second model of the first. It approximates the behaviour of the original well enough to be useful and it is not the mechanism by which the original arrived at its answer. The original arrived at its answer through several hundred boosted trees or a few hundred million weights, and no faithful rendering of that in one sentence exists. Saying this plainly is the strongest thing available in this answer, and candidates avoid it because it sounds like a concession.
It matters practically for two reasons. First, an explanation can be plausible and consistently wrong in the same direction, which means it can conceal the very thing the requirement exists to expose — a framework will happily produce a respectable-looking reason for a decision driven by a feature nobody intended the model to use. Second, if the requirement is regulatory, an approximation may not discharge it.
Where a rule obliges you to state the principal reasons for an adverse decision, or where Article 22 of the General Data Protection Regulation governs a solely automated decision with significant effects, the reasons you give are a representation about how the decision was made. Whether an approximation is an adequate basis for that representation is a legal judgement, needing compliance's agreement in writing rather than an engineer's assumption.
Which is why an interpretable model is a serious option
If explainability is a genuine hard requirement rather than a preference, building a model whose behaviour is directly readable removes the entire class of problem, and it should be evaluated properly rather than dismissed as a compromise.
The candidates are stronger than their reputation. A regularised linear or logistic model with a small, deliberately chosen feature set gives exact per-feature contributions. A scorecard, still standard in consumer credit, is a linear model over binned features and produces a decision that can be reproduced by hand. A generalised additive model allows a non-linear shape per feature while keeping contributions separable and plottable. A gradient-boosted model with monotonic constraints on the features where direction is required — income should not reduce a creditworthiness score — is not fully interpretable but is checkably consistent, which is often what a reviewer is really testing for.
Frame the decision as a measured trade rather than a philosophical one. Train both on the same data, compare them on the same evaluation set, and put the actual gap in front of the business alongside the cost of the alternative: an approximate explanation, a stability test suite, an argument with a reviewer, and residual legal exposure. Often the gap is small enough that the interpretable model wins on total cost, and where it is not, you have a documented reason for the complex one — far more defensible than never having checked.
An explanation has an audience and a purpose, and a post-hoc attribution is a model of your model rather than its reasoning. Where the requirement is genuinely binding, measure what an interpretable model would cost you before deciding an approximation will do.
© 2026 Preptima. Originally published at preptima.com (opens in a new tab).