Imbalanced Data — Class Weights, Resampling and SMOTE
Handle rare classes without leakage: compare class weighting, random resampling and SMOTE using suitable metrics and thresholds.
When a rare class matters, accuracy can reward failure
Before this lesson, understand confusion matrices, precision, recall, ROC-AUC, PR-AUC and train/validation/test splits. Imbalance means one class appears much less often than another. It is common in fraud, equipment failures and medical screening, but the ratio alone does not determine the right method. The costs of missed positives and unnecessary investigations matter just as much.
- Establish a naturally distributed baseline using precision-recall metrics.
- Compare weighting, random sampling and SMOTE without leakage.
- Set a decision threshold from validation evidence and explain its operational trade-offs.
Imagine 10,000 payment attempts with 80 confirmed fraudulent transactions. A classifier that always says “legitimate” has 99.2% accuracy and catches no fraud. Start with a simple baseline, inspect the confusion matrix and choose a metric and threshold that reflect the operation. Then compare training-time interventions—class weights and resampling—under the same evaluation protocol.
| Approach | Changes | What it does not solve |
|---|---|---|
| Class weights | Penalises mistakes on selected classes more during fitting | Does not create more information or choose a business threshold |
| Random over/undersampling | Changes class frequencies in the training fold | Can duplicate examples or discard useful majority information |
| SMOTE | Interpolates synthetic minority feature vectors | Does not guarantee realistic samples or fix label noise |
Split first, then inspect the failure costs
Make a stratified random split for independent, identically distributed rows; for time-dependent events use a chronological split instead. Stratification approximately preserves the class ratio across folds but does not prevent temporal or entity leakage. Any imputation, scaling, feature selection or sampling must be learned from training data only.
For rare positives, PR-AUC (often reported as average precision) makes precision and recall explicit and can be more informative than ROC-AUC. Its baseline depends on the prevalence, so report prevalence beside it. Also report operating-point precision, recall, false-positive count and the review volume your team can handle.
Make the learner pay more attention to costly mistakes
Many estimators accept class weights. With class_weight="balanced", scikit-learn computes weights inversely related to class frequencies in the training data. The loss assigns a larger penalty to errors on rare classes, encouraging a different decision boundary. Weighting changes the learning objective; it does not duplicate rows or change the evaluation distribution.
Weighting is a strong first experiment because it avoids synthetic feature vectors. It can still change calibration, increase false positives and make the model sensitive to a few minority examples. Compare the default model, weighted model and a reasonable threshold adjustment; select the threshold on validation data using a stated cost or capacity constraint. Do not select the threshold on the final test set.
Change the training fold, never the test population
Random oversampling repeats minority examples, which can help an estimator see them more often but may overfit duplicates. Random undersampling removes majority examples, reducing compute and shifting the class ratio but discarding potentially useful variation. Resampling should happen inside each training fold, after the split. Applying it before cross-validation lets near-duplicate or synthetic information leak into validation folds.
imbalanced-learn package, imported as imblearn. Install it in your project environment if needed. Its pipeline is designed to apply samplers during fitting and skip resampling when predicting.Interpolate minority neighbours—with domain checks
SMOTE selects a minority example, finds nearby minority examples in feature space, and creates a synthetic point along a line between neighbours. This can soften overfitting from exact duplicates, but it assumes that interpolation in the selected feature representation creates plausible examples. That assumption often fails for raw categorical codes, timestamps, text IDs, constrained financial quantities and structured sequences.
Scaling before SMOTE is important because nearest-neighbour calculations are distance-based. The imbalanced-learn pipeline ensures the scaler and sampler are fit separately within each cross-validation training fold. For mixed numeric and categorical data, use a method designed for that representation, such as a categorical-aware sampler, and confirm the generated combinations are valid for the domain.
Treat sampling as a hypothesis, not a guaranteed upgrade
Compare a baseline, class-weighted model, random sampler and SMOTE using identical development folds and metrics. Use a final naturally distributed test set once after model and threshold selection. Record fold variation: a method that wins on average by a tiny amount but varies dramatically may be less reliable than a stable baseline. Check calibration separately if predicted probabilities inform downstream risk or cost decisions.
| Check | Question to answer | Evidence |
|---|---|---|
| Recall and precision | How many positives are caught and how many alerts are correct? | Threshold-specific confusion matrix and PR curve |
| Workload | Can reviewers or downstream services handle the alert count? | Alerts per time period and operational capacity |
| Calibration | Does a score correspond to an observed event rate? | Reliability plot and calibration metrics on natural prevalence |
| Segments | Does performance vary across regions, products or user groups? | Slice-level metrics, sample counts and uncertainty |
On temporal or grouped data, use time-aware or group-aware splits. A transaction from the same account, patient or machine in both train and validation can make evaluation optimistic even if class proportions are balanced. Resampling cannot repair a bad split or a leaking feature.
Keep every intervention inside the correct boundary
- Measure the positive-class prevalence and majority-only baseline on a held-out split.
- Compare class_weight, random oversampling and SMOTE in the same stratified cross-validation folds.
- Plot precision and recall as the threshold changes. Choose a policy based on a minimum recall, false-positive budget or explicit cost.
- Review false positives and missed positives by a meaningful segment and identify any data leakage or proxy concerns.
- Write a short model card entry describing the sampling method, evaluation prevalence, intended use and known limits.
Use robust splits to estimate how much your model and sampling choices vary across data.
🎯 Key Takeaways
- ✓A rare positive class makes accuracy misleading; state prevalence and choose metrics around the cost of errors.
- ✓Class weighting changes the training loss, while resampling changes training-fold examples; neither changes the real test population.
- ✓Use samplers inside the cross-validation pipeline to prevent leakage.
- ✓SMOTE interpolates minority neighbours and is only suitable when that feature-space interpolation is meaningful.
- ✓Select thresholds and assess calibration using natural-prevalence validation data; report alert burden and group-level behavior.
HELP US IMPROVE
How was this lesson?
Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.
Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.
Discussion
0Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.
