Model Interpretability — SHAP and LIME
Explain any individual prediction. Global feature importance, local SHAP explanations, LIME for black-box models, and presenting model decisions to regulators.
Your loan rejection model has AUC = 0.94. The customer calls asking why their loan was rejected. "The model said so" is not a legal answer.
The Equal Credit Opportunity Act requires lenders to give applicants specific, explainable reasons when a credit decision is adverse. The SEC requires explanation of algorithmic trading decisions. Healthcare regulations require that diagnostic AI justify its conclusions. The EU AI Act mandates explanations for high-risk AI systems. Interpretability is not optional in regulated industries — it is a legal requirement.
But even outside regulation, interpretability matters for trust. A data scientist at Brex who cannot explain why the model rejected a specific applicant cannot debug the model when it makes systematic errors. Cannot detect bias. Cannot improve it. The model is a black box that produces outputs nobody understands — including the people responsible for it.
This module covers two complementary techniques. SHAP (SHapley Additive exPlanations) computes the exact contribution of each feature to each prediction using game theory — it is mathematically rigorous and model-agnostic. LIME (Local Interpretable Model-agnostic Explanations) fits a simple interpretable model in the local neighbourhood of a prediction — faster and more flexible but less rigorous. Together they cover the full range of interpretability needs in production.
A cricket team wins a match. How much credit does each player deserve? You cannot just look at the final score — you need to figure out each player's contribution. SHAP uses Shapley values from cooperative game theory: simulate all possible team subsets, measure how much the score changes when each player joins. Average across all subsets. That average is each player's fair credit.
SHAP does the same for model features. Simulate all possible feature subsets, measure how much the prediction changes when each feature is added. Average across all subsets. That average is each feature's fair contribution to this specific prediction.
Built-in, permutation, and SHAP — what each measures and when each misleads
Before SHAP, there were two common approaches to feature importance. Both have significant limitations that SHAP fixes. Understanding why they fail makes SHAP's value obvious.
Counts how many times each feature is used to split nodes, weighted by the improvement in the split criterion. Available in sklearn, XGBoost, LightGBM via .feature_importances_.
Heavily biased toward high-cardinality features. A feature with 1,000 unique values will be used in more splits than a binary feature even if both have equal predictive power. Tells you about the model structure, not about the data.
Quick sanity check only. Never use for regulatory reporting.
For each feature, randomly shuffle its values and measure how much the model performance drops. A large drop = the feature is important. No drop = the model ignores it. Available via sklearn.inspection.permutation_importance.
When two features are correlated (e.g. income and loan_amount), shuffling one breaks the correlation — the model appears to rely less on each than it actually does. Correlated features share importance between them rather than reflecting true individual contributions.
Better than split-based for final model analysis. But misleads on correlated features.
Computes the exact marginal contribution of each feature to each individual prediction using Shapley values from cooperative game theory. Mathematically proven to be the only attribution method satisfying four key fairness axioms.
Slower than built-in importance. KernelExplainer is very slow on large datasets. TreeExplainer is fast but tree-model-only.
Use for all production reporting, regulatory compliance, and debugging. The gold standard.
SHAP values — from global importance to individual explanations
SHAP computes two levels of explanation simultaneously. Global SHAP importance(mean |SHAP| across all predictions) tells you which features matter most for the model overall — comparable to feature importance but more reliable. Local SHAP valuesexplain one specific prediction — which features pushed this particular applicant's default probability up or down, and by how much.
Shapley proved in 1951 that there exists exactly one attribution satisfying all four axioms. SHAP implements that attribution. No other feature importance method satisfies all four simultaneously.
Three SHAP explainers — which one to use for which model
SHAP has different explainers optimised for different model types. TreeExplainer is exact and fast for tree models. LinearExplainer is exact for linear models. KernelExplainer works for any model but is slow. DeepExplainer works for neural networks.
LIME — explain any prediction by fitting a local simple model
LIME takes a fundamentally different approach to SHAP. Instead of computing exact Shapley values, it asks: what simple model (linear regression or decision tree) best approximates the complex model's behaviour in the immediate neighbourhood of this prediction? That simple model's coefficients are the explanation.
LIME generates synthetic samples near the prediction point, gets the complex model's predictions for all of them, then fits a weighted linear model where samples closer to the original point are weighted more heavily. The linear model's coefficients tell you which features pushed the prediction up or down locally.
Production explanation pipeline — Brex loan rejection letters
At Brex, when a loan application is rejected the system must generate a plain-English explanation that satisfies ECOA guidelines. The explanation must name the specific factors that led to rejection, not just say "algorithmic decision." Here is the complete pipeline.
Every common interpretability mistake — explained and fixed
Five things people get wrong about model interpretability
SHAP measures how much a feature moved the model's prediction — a statistical attribution within the model, not a causal claim about the world. If city_tier is correlated with the true driver of default risk (say, local economic conditions the model never sees directly), city_tier can receive a large SHAP value even though changing an applicant's city_tier alone, in reality, would not change their true default risk. SHAP answers "what did the model rely on," which is essential for debugging and compliance — but "what the model relied on" and "what actually causes the outcome" are different questions, and only the second one needs a causal method (e.g. a randomised experiment or a causal graph) to answer.
A local explanation is only valid in the immediate neighbourhood of that one applicant's feature values — tree-based and other non-linear models routinely have interaction effects where a feature's contribution flips sign in a different region of the input space. High employment_yrs might reduce default risk for applicants with strong credit scores but barely matter for applicants with weak ones. Generalising from one applicant's SHAP breakdown to "this is how the model treats employment_yrs" is exactly the mistake this module's global-vs-local distinction exists to prevent — global patterns require aggregating many local explanations (mean |SHAP| across the dataset), not extrapolating from a single one.
For many structured, tabular problems — exactly the loan-default and credit-scoring examples used throughout this module — well-regularised logistic regression, shallow trees, or generalised additive models often perform within a percentage point or two of a tuned gradient boosting model. The accuracy gap that people attribute to "black box models are just better" is frequently attributable to something else entirely: better feature engineering, more training data, or a genuinely non-linear relationship that a slightly more expressive interpretable model (a GAM, not just plain linear regression) could also capture. The tradeoff is real in some domains (image and text especially), but it is not a law of nature — it should be measured per problem, not assumed.
"Black box" describes how the model computes its output, not whether any insight into that computation is available. SHAP's TreeExplainer gives exact, not approximate, per-prediction attributions for XGBoost and LightGBM despite them being ensembles of hundreds of trees no human could read directly. KernelExplainer and LIME extend this to SVMs and neural networks by fitting a local surrogate model. Even for deep networks, attention weights and DeepExplainer provide partial windows into what the model is weighting. None of these give you the same transparency as reading a 5-line decision rule — but "harder to interpret directly" and "impossible to interpret at all" are very different claims, and this entire module is built on tools that only exist because the second claim is false.
The right explanation depends entirely on who is asking. A regulator enforcing ECOA needs specific, individually adverse reason codes tied to this applicant's rejection — exactly the top-3-risk-factors output this module's production pipeline generates. A data scientist debugging a systematic error needs global SHAP importance and interaction analysis to find patterns across thousands of predictions. A rejected applicant needs one or two plain-English sentences, not a table of SHAP values in log-odds units. Building "the" explanation and handing it to all three audiences unchanged satisfies none of them well — production interpretability systems typically generate multiple views from the same underlying SHAP values, not one universal report.
Model interpretability — 5 questions interviewers actually ask
A good answer skips the game theory entirely: "For any prediction the model makes, SHAP tells you exactly how much each piece of information pushed that prediction up or down, and the contributions add up to the full difference between this prediction and our average prediction. Think of it like a hiring committee scoring a candidate — SHAP tells you how many points each factor added or subtracted, so you can see precisely why this candidate scored the way they did, not just the final number." The key thing to convey is that SHAP explanations are exact and additive, not a rough approximation — that's why they hold up under regulatory scrutiny.
Permutation importance shuffles one feature at a time and measures the performance drop — but if two features are correlated (income and income_monthly, say), shuffling just one of them barely hurts performance because the model can still lean on the other, unshuffled, correlated feature. Both features end up looking individually unimportant even though together they matter a great deal — the importance gets split between them rather than measured per feature. SHAP does not eliminate this issue entirely — it still has to divide credit between correlated features in some principled way — but it does so using Shapley values' symmetry axiom, which guarantees two equally-contributing features get identical, consistent credit rather than the somewhat arbitrary split permutation importance produces. It's a more principled answer to the same underlying difficulty, not a magic fix for correlation itself.
First, compute the local SHAP explanation for that specific applicant using TreeExplainer (assuming a tree-based model), which gives the exact contribution of each feature to their individual predicted default probability. Rank those contributions by magnitude and translate the top 3 into plain-English reason codes using a feature description mapping — "existing monthly debt payments are high relative to income" rather than "existing_emis SHAP = +0.14." This satisfies ECOA's requirement for specific, individualised adverse-action reasons rather than a vague "the algorithm decided." I'd also keep the underlying SHAP values and the model's expected_value (base rate) on file in case the audit needs to verify the numbers behind the plain-English reasons.
LIME is the right choice when SHAP's exact explainers don't apply well to your model type — image classifiers, text models, or any model where KernelExplainer would be prohibitively slow — because LIME's local-surrogate approach is model-agnostic and often faster in those cases, and its rule-based output ("income under 25000") is intuitive for non-technical audiences. The tradeoff is that LIME is stochastic: it generates random synthetic samples in the neighbourhood of the prediction, so two runs on the same prediction can give meaningfully different explanations, especially in high-dimensional or sparse feature spaces. For anything where consistency matters — regulatory reporting, repeatable audits — that instability is a real cost, and SHAP's determinism (for tree and linear models) makes it the safer default whenever it's available.
No — feature importance and SHAP both measure the feature's association with the model's output, which is not the same as the feature causing the real-world outcome. A feature can be highly predictive because it's correlated with the true cause without being the cause itself. To actually establish causality you need a different toolkit entirely: a randomised experiment (change the feature, hold everything else fixed, measure the real outcome), a causal graph with domain assumptions made explicit so you can reason about confounders, or quasi-experimental methods like instrumental variables or difference-in-differences when a true experiment isn't feasible. SHAP is the right tool for "what is the model doing" — it was never designed to answer "what does this variable do in reality," and treating it as if it does is a common and serious mistake, especially in regulated and policy-relevant applications.
The Evaluation section is complete. Section 7 — Deep Learning — begins next.
You have now completed the full Model Evaluation section: evaluation metrics, calibration, ROC curves, cross-validation, hyperparameter tuning, and interpretability. You can build, evaluate, tune, calibrate, and explain any classical ML model.
Section 7 — Deep Learning — begins with neural networks. Everything changes: instead of hand-crafted features, the model learns its own representations. Instead of gradient boosting on tabular data, you train multi-layer networks on images, sequences, and text. Module 40 builds a neural network from scratch — forward pass, backpropagation, and gradient descent — before introducing PyTorch.
Forward pass, backpropagation, and gradient descent — built from NumPy before touching PyTorch. The foundation every deep learning framework is built on.
🎯 Key Takeaways
- ✓Interpretability is a legal requirement in regulated industries — ECOA, the Fed, and EU AI Act all mandate that algorithmic decisions be explainable. "The model said so" is not acceptable. SHAP and LIME provide the explanation infrastructure.
- ✓Three types of feature importance, in order of reliability: built-in split-based (biased toward high-cardinality features), permutation importance (misleads on correlated features), and SHAP (mathematically proven correct — the only attribution satisfying all four fairness axioms). Always prefer SHAP for production reporting.
- ✓SHAP computes two levels simultaneously: global importance (mean |SHAP| across all predictions — reliable feature ranking) and local importance (individual SHAP values per prediction — which features drove this specific outcome and by how much).
- ✓Choose the right SHAP explainer: TreeExplainer for XGBoost/LightGBM/RF (fast, exact), LinearExplainer for logistic/linear regression (fastest, exact), KernelExplainer for any model including SVM and neural nets (slow, approximate). Always pass the underlying model, not a Pipeline wrapper.
- ✓LIME fits a local linear model in the neighbourhood of each prediction. It is faster than KernelSHAP for non-tree models and produces intuitive rule-based explanations. But it is stochastic — different runs give different results. Use a fixed random_state and high num_samples. Prefer SHAP when determinism matters.
- ✓For production loan or credit decisions: pre-compute the SHAP explainer once at startup, store it, and reuse it for all requests. Generate explanations in plain English using a feature description dictionary that maps technical feature names to human-readable phrases. Always report the top 3 risk factors — more than 3 overwhelms the applicant.
Discussion
0Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.