Ensembles — Stacking and Blending
Combine diverse models with out-of-fold stacking and a held-out blending set, while keeping evaluation leakage-free.
A team of imperfect models can make a stronger prediction
Before this lesson, you should know train/validation/test splits, cross-validation, classification probabilities, and how tree ensembles such as Random Forest and gradient boosting behave. You should also be able to compare a model against a simple baseline. The goal is to build two ways of combining independently trained models, then understand when the extra complexity is worth operating.
- Explain why stacking uses out-of-fold predictions.
- Build and compare a stack and a blend without using the final test set for model selection.
- Assess whether diversity improves the operating decision enough to justify added complexity.
Imagine a delivery platform estimating whether an order will arrive late. A regularised linear model may capture broad trends; a tree model may capture interactions between distance, rain and time of day; a nearest-neighbour model may recognise unusual local patterns. If their errors are not identical, a second-stage model can learn how to combine their predictions. This is an ensemble of different model families, not just more trees in one forest.
| Method | How combination weights are learned | Data used for the combiner |
|---|---|---|
| Soft voting | Fixed average or explicit weights | No separate combiner training |
| Stacking | A meta-model learns from out-of-fold predictions | Cross-validation predictions from training rows |
| Blending | A meta-model learns from predictions on a holdout subset | A dedicated blending split |
Measure whether the extra models add value
First lock a test set away. Use only the development data to choose base learners, tune them and train the combiner. A good comparison includes a simple model, each base model by itself, and the final ensemble. On a fraud task, report precision-recall behaviour at an operating threshold as well as a threshold-independent summary; accuracy alone can conceal a model that misses almost every rare case.
The data here is synthetic so the example is runnable without downloads; in a real project, replace it with a properly split dataset. Keep the final test rows out of every preprocessing fit, feature decision, model choice and stacking step. If you repeatedly inspect test scores and revise the model, that set has become validation data.
Train the meta-model on predictions it did not train on
A tempting but incorrect workflow is to fit each base model on all development rows and then train the meta-model on those same rows' predictions. Those predictions are in-sample: a flexible learner may look unusually confident on rows it has already seen. The meta-model then learns from an easier problem than it will face at inference.
Stacking avoids that mismatch with out-of-fold (OOF) predictions. For each fold, fit each base learner on the other folds and predict the held-out fold. Concatenate those held-out predictions into a new feature matrix. Every row in that matrix was predicted by a model that did not train on that row. Fit the final estimator on this matrix; then refit base estimators on all development data for production predictions.
A simpler alternative that spends a holdout split
Blending reserves part of development data for the meta-model. Fit base models on the base-training portion, predict the blend portion, then fit the combiner on those predictions and labels. It is easier to explain and inspect than folds, but base models see fewer examples and the combiner can overfit if the blend set is small. Never use the final test set as the blending set.
The example uses a small blend holdout only to demonstrate the data flow. In cross-validation or production, fit all data-dependent preprocessing inside each base-model pipeline. Keep feature columns, class ordering, calibration and missing-value handling identical between blend-time and inference-time predictions.
A better ranking is not automatically a better decision
Stacking can improve ranking while producing poorly calibrated probabilities. If a downstream system interprets 0.8 as an 80% event probability, evaluate calibration on data not used to fit the stack. Choose a decision threshold based on the cost of false positives and false negatives, capacity limits, and any policy constraints. Recheck that threshold when prevalence or costs change.
In a delivery operation, the ensemble could trigger a proactive customer message or a courier reassignment. Those actions have different costs. A single global threshold may be wrong if service levels vary by region, and a model score should not be turned into an automatic punitive action without appropriate review.
| Question | Evidence to inspect | Decision |
|---|---|---|
| Is the ensemble useful? | Out-of-fold or untouched-test metrics vs. each base model | Keep only if lift is stable and practically meaningful |
| Is the gain robust? | Variation across folds, time periods and important groups | Prefer stable gains to one lucky split |
| Does it fit operations? | Latency, memory, feature availability, failure modes | Account for serving and monitoring costs |
Four ways a stack can look better than it is
Build the comparison before you ship
- Train a logistic-regression baseline and two different tree-based models on a stratified split. Record PR-AUC and ROC-AUC for each.
- Fit a StackingClassifier with out-of-fold predictions. Compare it with the strongest base model using the same untouched test set.
- Implement blending with a development-only holdout, then describe the trade-off in training-data size and meta-model variance.
- Inspect false positives and false negatives by a business-relevant segment. Decide which error is more costly and propose a threshold-selection method.
- Measure inference latency and model size for base models versus the stack. Write a short keep-or-remove recommendation based on evidence.
Key takeaways: stacking learns a combiner from out-of-fold predictions; blending uses a dedicated holdout; both require leakage-safe boundaries; diversity and stable gains matter more than model count; evaluate threshold, calibration, fairness, latency and maintenance—not just a headline metric.
Move from supervised prediction to discovering structure when your data has no labels.
🎯 Key Takeaways
- ✓A stack trains a meta-model on out-of-fold predictions so the combiner does not learn from in-sample overconfidence.
- ✓Blending is conceptually simple but reserves data for the combiner and can be unstable when that holdout is small.
- ✓Use a final test set only after every preprocessing, model, and threshold decision is fixed.
- ✓Ensemble gains depend on competent models with complementary errors; more models do not guarantee improvement.
- ✓Assess ranking, calibration, business thresholds, group performance, latency and operational fallbacks together.
HELP US IMPROVE
How was this lesson?
Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.
Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.
Discussion
0Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.
