Train / Validation / Test Split

Think of preparing for a competitive exam like GATE or CAT. Your textbook problems are the training set — you practice on these, make mistakes, learn from them. Practice mock tests are the validation set — you use your score to decide which topics to study more. The actual exam on exam day is the test set — you sit it exactly once, at the very end, to get your true performance.

If you kept using the exam paper to decide what to study, your exam score would look great — but you would have cheated yourself out of knowing your real ability. Same with the test set in ML.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error

np.random.seed(42)
n = 5000

# DoorDash delivery dataset
distance   = np.abs(np.random.normal(4.0, 2.0, n)).clip(0.5, 15)
traffic    = np.random.randint(1, 11, n).astype(float)
prep       = np.abs(np.random.normal(15, 5, n)).clip(5, 35)
value      = np.abs(np.random.normal(350, 150, n)).clip(50, 1200)
delivery   = (8.6 + 7.3*distance + 0.8*prep + 1.5*traffic
              + np.random.normal(0, 4, n)).clip(10, 120)

X = np.column_stack([distance, traffic, prep, value])
y = delivery

# ── Two-step split: first carve out test, then split remainder ────────
# Step 1: hold out 15% as test set — never touch until the very end
X_trainval, X_test, y_trainval, y_test = train_test_split(
    X, y,
    test_size=0.15,
    random_state=42,
    shuffle=True,        # shuffle before splitting (default True)
)

# Step 2: split remaining 85% into train (82%) and validation (18%)
# 0.18 of 0.85 ≈ 15% of total
X_train, X_val, y_train, y_val = train_test_split(
    X_trainval, y_trainval,
    test_size=0.18,
    random_state=42,
)

total = len(X)
print(f"Total dataset:     {total:,} samples (100%)")
print(f"Training set:      {len(X_train):,} samples ({len(X_train)/total*100:.0f}%)")
print(f"Validation set:    {len(X_val):,}  samples ({len(X_val)/total*100:.0f}%)")
print(f"Test set:          {len(X_test):,}  samples ({len(X_test)/total*100:.0f}%)")
print(f"Sum check:         {len(X_train)+len(X_val)+len(X_test):,} ✓")

# ── How to use the three splits correctly ─────────────────────────────
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge

scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)    # fit ONLY on train
X_val_sc   = scaler.transform(X_val)          # transform with train stats
X_test_sc  = scaler.transform(X_test)         # transform with train stats

# Step 1: Try different hyperparameters — use VALIDATION score to choose
print("
Hyperparameter tuning using validation set:")
best_alpha, best_val_mae = None, float('inf')
for alpha in [0.001, 0.01, 0.1, 1.0, 10.0, 100.0]:
    model = Ridge(alpha=alpha)
    model.fit(X_train_sc, y_train)
    val_mae = mean_absolute_error(y_val, model.predict(X_val_sc))
    print(f"  alpha={alpha:<8}: val MAE = {val_mae:.4f}")
    if val_mae < best_val_mae:
        best_val_mae, best_alpha = val_mae, alpha

print(f"
Best alpha: {best_alpha}  (val MAE = {best_val_mae:.4f})")

# Step 2: Retrain on train+validation with best hyperparameter
X_trainval_sc = scaler.fit_transform(X_trainval)   # refit on all non-test data
X_test_sc2    = scaler.transform(X_test)

final_model = Ridge(alpha=best_alpha)
final_model.fit(X_trainval_sc, y_trainval)

# Step 3: Evaluate on test set — ONCE, ONLY NOW, NEVER AGAIN
test_mae = mean_absolute_error(y_test, final_model.predict(X_test_sc2))
print(f"
Final test MAE (honest evaluation): {test_mae:.4f} min")
print("This is the number you report. It was computed exactly once.")

import numpy as np
from sklearn.model_selection import train_test_split, StratifiedKFold
from sklearn.preprocessing import StandardScaler

np.random.seed(42)
n = 3000

distance  = np.abs(np.random.normal(4.0, 2.0, n)).clip(0.5, 15)
traffic   = np.random.randint(1, 11, n).astype(float)
prep      = np.abs(np.random.normal(15, 5, n)).clip(5, 35)
delivery  = (8.6 + 7.3*distance + 0.8*prep + 1.5*traffic
             + np.random.normal(0, 4, n)).clip(10, 120)

X = np.column_stack([distance, traffic, prep])
y_binary = (delivery > 45).astype(int)   # classification: late or not

print(f"Class distribution in full dataset:")
print(f"  On-time (0): {(y_binary==0).sum()} ({(y_binary==0).mean()*100:.1f}%)")
print(f"  Late    (1): {(y_binary==1).sum()} ({(y_binary==1).mean()*100:.1f}%)")

# ── Random split — may produce imbalanced sets ─────────────────────────
X_tr, X_te, y_tr, y_te = train_test_split(
    X, y_binary, test_size=0.2, random_state=42
)
print(f"
Random split — late % in test: {y_te.mean()*100:.1f}%")

# ── Stratified split — preserves class balance ─────────────────────────
X_tr_s, X_te_s, y_tr_s, y_te_s = train_test_split(
    X, y_binary,
    test_size=0.2,
    random_state=42,
    stratify=y_binary,    # ← this is the only addition
)
print(f"Stratified split — late % in test: {y_te_s.mean()*100:.1f}%  ← matches original")

# ── Three-way stratified split ────────────────────────────────────────
X_trainval, X_test, y_trainval, y_test = train_test_split(
    X, y_binary, test_size=0.15, random_state=42, stratify=y_binary
)
X_train, X_val, y_train, y_val = train_test_split(
    X_trainval, y_trainval, test_size=0.18, random_state=42, stratify=y_trainval
)

print(f"
Three-way stratified split:")
for name, y_split in [('Train', y_train), ('Val', y_val), ('Test', y_test)]:
    print(f"  {name:<6}: {len(y_split):4d} samples  {y_split.mean()*100:.1f}% late")

# ── Stratified KFold — for cross-validation on imbalanced data ─────────
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

print("
StratifiedKFold — late % per fold:")
for fold, (tr_idx, va_idx) in enumerate(skf.split(X, y_binary), 1):
    fold_late_pct = y_binary[va_idx].mean() * 100
    print(f"  Fold {fold}: {fold_late_pct:.1f}% late  ← consistent across folds")

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_absolute_error

np.random.seed(42)
n = 2000
X = np.random.randn(n, 5)
y = X[:, 0] * 3 + X[:, 1] * 2 + np.random.randn(n) * 2

# ── Demonstrate leakage from preprocessing before split ───────────────

# WRONG: fit scaler on ALL data, then split
scaler_leaky = StandardScaler()
X_scaled_leaky = scaler_leaky.fit_transform(X)    # uses test set stats!
X_tr_l, X_te_l, y_tr, y_te = train_test_split(X_scaled_leaky, y, test_size=0.2, random_state=42)
model_leaky = Ridge().fit(X_tr_l, y_tr)
mae_leaky   = mean_absolute_error(y_te, model_leaky.predict(X_te_l))

# CORRECT: split first, then fit scaler on train only
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler_correct = StandardScaler()
X_tr_c = scaler_correct.fit_transform(X_train)    # fit on train only
X_te_c = scaler_correct.transform(X_test)         # apply stored stats to test
model_correct = Ridge().fit(X_tr_c, y_train)
mae_correct   = mean_absolute_error(y_test, model_correct.predict(X_te_c))

print("Effect of preprocessing leakage:")
print(f"  Leaky (scaler fit on full data):  MAE = {mae_leaky:.4f}  ← optimistic lie")
print(f"  Correct (scaler fit on train):    MAE = {mae_correct:.4f}  ← honest estimate")
print(f"  Difference: {mae_leaky - mae_correct:.4f} min  ← how much the leakage flatters the model")

# ── Check for duplicate rows across splits ────────────────────────────
# In a real dataset you would use a business key like order_id
train_indices = set(range(len(X_train)))   # simulate with indices
test_indices  = set(range(len(X_train), len(X_train) + len(X_test)))
overlap       = train_indices & test_indices
print(f"
Duplicate check: {len(overlap)} overlapping samples (should be 0)")
assert len(overlap) == 0, "Train and test sets overlap — data leakage!"
print("✓ No overlap confirmed")

# ── Verify stratification ─────────────────────────────────────────────
y_binary = (y > y.mean()).astype(int)
_, _, y_tr_b, y_te_b = train_test_split(
    X, y_binary, test_size=0.2, random_state=42, stratify=y_binary
)
print(f"
Stratification check:")
print(f"  Original positive rate: {y_binary.mean():.3f}")
print(f"  Train positive rate:    {y_tr_b.mean():.3f}")
print(f"  Test positive rate:     {y_te_b.mean():.3f}")

For any dataset ordered by time, your split must respect chronological order. Training data must come entirely before validation data. Validation data must come entirely before test data. There must be no temporal overlap between any two splits.

This simulates what actually happens in production: your model was trained on historical data and is now predicting future events it has never seen.

import numpy as np
import pandas as pd
from sklearn.model_selection import TimeSeriesSplit, train_test_split
from sklearn.linear_model import Ridge
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_absolute_error

np.random.seed(42)

# ── Instacart daily order volume — time-series ────────────────────────────
dates    = pd.date_range('2024-01-01', periods=365, freq='D')
trend    = np.linspace(1000, 1500, 365)           # growing business
seasonal = 200 * np.sin(2 * np.pi * np.arange(365) / 7)  # weekly cycle
noise    = np.random.normal(0, 50, 365)
orders   = (trend + seasonal + noise).clip(500)

# Features: day of week, month, trend proxy, lag features
df = pd.DataFrame({
    'date':        dates,
    'orders':      orders,
    'day_of_week': dates.dayofweek,
    'month':       dates.month,
    'day_of_year': dates.dayofyear,
})
# Lag feature: yesterday's orders (must shift to avoid leakage)
df['orders_lag1'] = df['orders'].shift(1)
df['orders_lag7'] = df['orders'].shift(7)
df = df.dropna().reset_index(drop=True)

X = df[['day_of_week', 'month', 'day_of_year', 'orders_lag1', 'orders_lag7']].values
y = df['orders'].values
dates_clean = df['date'].values

# ── WRONG: random split on time-series ────────────────────────────────
X_tr_r, X_te_r, y_tr_r, y_te_r = train_test_split(X, y, test_size=0.2, random_state=42)
sc_r = StandardScaler()
model_random = Ridge().fit(sc_r.fit_transform(X_tr_r), y_tr_r)
mae_random   = mean_absolute_error(y_te_r, model_random.predict(sc_r.transform(X_te_r)))

# ── CORRECT: chronological split ──────────────────────────────────────
split_idx    = int(len(X) * 0.8)
X_train_ts   = X[:split_idx]
X_test_ts    = X[split_idx:]
y_train_ts   = y[:split_idx]
y_test_ts    = y[split_idx:]

sc_ts = StandardScaler()
model_ts   = Ridge().fit(sc_ts.fit_transform(X_train_ts), y_train_ts)
mae_ts     = mean_absolute_error(y_test_ts, model_ts.predict(sc_ts.transform(X_test_ts)))

print("Time-series split comparison:")
print(f"  Random split MAE:       {mae_random:.1f} orders  ← optimistically wrong")
print(f"  Chronological MAE:      {mae_ts:.1f} orders  ← honest estimate")
print(f"  Training period:        {dates_clean[0]} → {dates_clean[split_idx-1]}")
print(f"  Test period:            {dates_clean[split_idx]} → {dates_clean[-1]}")

# ── TimeSeriesSplit — walk-forward cross-validation ───────────────────
# Each fold: train on past, validate on immediate future
# Simulates exactly what happens in production
tscv = TimeSeriesSplit(n_splits=5, gap=0)

print(f"
TimeSeriesSplit walk-forward cross-validation:")
print(f"{'Fold':<6} {'Train size':<13} {'Val size':<11} {'Val MAE'}")
print("─" * 42)

for fold, (tr_idx, va_idx) in enumerate(tscv.split(X), 1):
    X_tr, X_va = X[tr_idx], X[va_idx]
    y_tr, y_va = y[tr_idx], y[va_idx]

    sc   = StandardScaler()
    m    = Ridge().fit(sc.fit_transform(X_tr), y_tr)
    mae  = mean_absolute_error(y_va, m.predict(sc.transform(X_va)))
    print(f"  {fold}     {len(X_tr):<13} {len(X_va):<11} {mae:.1f}")

# The MAE increases in later folds — the model struggles to predict
# further into the future. This is realistic. Random CV would hide this.

import numpy as np
from sklearn.model_selection import (
    train_test_split, KFold, StratifiedKFold,
    cross_val_score, cross_validate,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_absolute_error

np.random.seed(42)
n = 1000
X = np.random.randn(n, 4)
y = X[:, 0] * 3 + X[:, 1] * 2 + np.random.randn(n) * 2

pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('model',  Ridge(alpha=1.0)),
])

# ── Holdout split ─────────────────────────────────────────────────────
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)
pipeline.fit(X_tr, y_tr)
holdout_mae = mean_absolute_error(y_te, pipeline.predict(X_te))
print(f"Holdout MAE: {holdout_mae:.4f}")

# ── K-Fold cross-validation ────────────────────────────────────────────
cv_scores = cross_val_score(
    pipeline, X, y,
    cv=5,
    scoring='neg_mean_absolute_error',
    n_jobs=-1,
)
print(f"5-Fold CV MAE: {-cv_scores.mean():.4f} ± {cv_scores.std():.4f}")
print(f"Per-fold:      {(-cv_scores).round(3)}")

# ── cross_validate — more metrics at once ─────────────────────────────
cv_results = cross_validate(
    pipeline, X, y,
    cv=5,
    scoring=['neg_mean_absolute_error', 'r2'],
    return_train_score=True,
    n_jobs=-1,
)
print(f"
cross_validate results:")
print(f"  Train MAE: {-cv_results['train_neg_mean_absolute_error'].mean():.4f}")
print(f"  Val MAE:   {-cv_results['test_neg_mean_absolute_error'].mean():.4f}")
print(f"  Val R²:    {cv_results['test_r2'].mean():.4f}")

# Large gap between train and val MAE = overfitting
gap = (-cv_results['train_neg_mean_absolute_error'].mean() -
       (-cv_results['test_neg_mean_absolute_error'].mean()))
if abs(gap) > 1.0:
    print(f"  ⚠ Train/val gap = {gap:.3f} — possible overfitting")

import numpy as np
from sklearn.model_selection import train_test_split

def smart_split(X, y, dataset_size_hint: str = 'auto',
                random_state: int = 42, stratify=None):
    """
    Automatically choose split strategy based on dataset size.
    Returns X_train, X_val, X_test (or X_train, X_test for small datasets).
    """
    n = len(X)
    strat = stratify

    if n < 1000 or dataset_size_hint == 'small':
        print(f"  Dataset too small ({n} rows) — use cross-validation, no holdout")
        # Return 80/20 for basic use, but recommend CV
        return train_test_split(X, y, test_size=0.2,
                                random_state=random_state, stratify=strat)

    elif n < 10_000 or dataset_size_hint == 'medium':
        # 70/15/15
        test_size = 0.15
        val_size  = 0.15 / 0.85   # fraction of remaining after test split
        X_tv, X_test, y_tv, y_test = train_test_split(
            X, y, test_size=test_size, random_state=random_state,
            stratify=strat,
        )
        strat2 = y_tv if strat is not None else None
        X_train, X_val, y_train, y_val = train_test_split(
            X_tv, y_tv, test_size=val_size, random_state=random_state,
            stratify=strat2,
        )
        print(f"  Medium dataset ({n} rows): 70/15/15 split")

    else:
        # Large: 80/10/10
        test_size = 0.10
        val_size  = 0.10 / 0.90
        X_tv, X_test, y_tv, y_test = train_test_split(
            X, y, test_size=test_size, random_state=random_state,
            stratify=strat,
        )
        strat2 = y_tv if strat is not None else None
        X_train, X_val, y_train, y_val = train_test_split(
            X_tv, y_tv, test_size=val_size, random_state=random_state,
            stratify=strat2,
        )
        print(f"  Large dataset ({n} rows): 80/10/10 split")

    print(f"  Train: {len(X_train)}  Val: {len(X_val)}  Test: {len(X_test)}")
    return X_train, X_val, X_test, y_train, y_val, y_test

# Test it
for n in [500, 5_000, 50_000]:
    X_demo = np.random.randn(n, 4)
    y_demo = np.random.randn(n)
    print(f"
Dataset size: {n}")
    smart_split(X_demo, y_demo)

import numpy as np
import pandas as pd
from sklearn.model_selection import GroupShuffleSplit, GroupKFold

np.random.seed(42)

# ── Group leakage — many orders per customer ───────────────────────────
n_customers = 400
orders_per_customer = np.random.randint(3, 15, n_customers)
customer_id = np.repeat(np.arange(n_customers), orders_per_customer)
n = len(customer_id)

distance = np.abs(np.random.normal(4.0, 2.0, n)).clip(0.5, 15)
delivery = (8.6 + 7.3 * distance + np.random.normal(0, 4, n)).clip(10, 120)
X = distance.reshape(-1, 1)
y = delivery

# WRONG: random split ignores that the same customer appears many times
from sklearn.model_selection import train_test_split
X_tr_wrong, X_te_wrong, y_tr_wrong, y_te_wrong, cust_tr, cust_te = train_test_split(
    X, y, customer_id, test_size=0.2, random_state=42
)
overlap_customers = set(cust_tr) & set(cust_te)
print(f"Random split — customers appearing in BOTH train and test: {len(overlap_customers)}")

# CORRECT: GroupShuffleSplit keeps every row for a given customer on one side
gss = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_idx, test_idx = next(gss.split(X, y, groups=customer_id))
cust_tr_g, cust_te_g = customer_id[train_idx], customer_id[test_idx]
overlap_g = set(cust_tr_g) & set(cust_te_g)
print(f"GroupShuffleSplit — customers appearing in BOTH sets: {len(overlap_g)}")

# GroupKFold — the cross-validation equivalent, for hyperparameter tuning
gkf = GroupKFold(n_splits=5)
print("
GroupKFold — no customer ever appears in both train and val for any fold:")
for fold, (tr_idx, va_idx) in enumerate(gkf.split(X, y, groups=customer_id), 1):
    fold_overlap = set(customer_id[tr_idx]) & set(customer_id[va_idx])
    print(f"  Fold {fold}: overlap = {len(fold_overlap)} customers")

# ── Splitting after feature engineering — the subtler leak ─────────────
dates = pd.date_range('2024-01-01', periods=200, freq='D')
orders = pd.Series(np.random.normal(1000, 100, 200), index=dates)

# WRONG: rolling average computed BEFORE the train/test boundary is chosen
# A row near the boundary can "see" a window that reaches into test days
rolling_wrong = orders.rolling(window=14).mean()
split_point = dates[160]
train_feature_wrong = rolling_wrong[:split_point]   # looks fine, but...
# the LAST few rows of this training feature were computed using a window
# that included days beyond split_point if the split happens later than here

# CORRECT: compute the split boundary FIRST, then compute rolling
# features using only data available up to and including that boundary
train_orders = orders[:split_point]
test_orders  = orders[split_point:]
rolling_train_correct = train_orders.rolling(window=14).mean()
# Test-period rolling features are computed by extending the window
# using only train + already-elapsed test data — never future test data
print(f"
Train period: {train_orders.index[0].date()} to {train_orders.index[-1].date()}")
print(f"Test period:  {test_orders.index[0].date()} to {test_orders.index[-1].date()}")

Random splitting assumes every row is independent and exchangeable with every other row — swap any two rows between train and test and nothing about the underlying problem changes. That assumption breaks for time-ordered data, where a row's position in time matters, and it breaks for grouped data, where multiple rows share an underlying entity like a customer or patient. In both cases a random split can leak information across the train/test boundary even though nothing in the code looks wrong. "Split randomly" is the right default only once you have checked that your rows really are independent of each other.

A single split gives you one sample of "how well did this model do on unseen data," and that sample has its own randomness — a slightly different split could easily rank two similar models the other way around, especially on a small dataset. Using that one score to choose between several hyperparameter settings or model families is effectively tuning to the noise in that particular split, not to genuine generalisation ability. Cross-validation, which averages performance across several different splits, gives a materially more reliable basis for model selection than any single holdout can.

Hyperparameter tuning is its most common use, but far from its only one. Validation performance is also how you decide when to stop training a neural network (early stopping), how you compare entirely different model families against each other, and how you catch overfitting during development before it ever reaches the test set. Treating it purely as a tuning knob undersells how much of routine model development — architecture choices, feature additions, preprocessing changes — is actually validated against this set, not the training set and not the sealed test set.

A good test score is consistent with a correct split, but it is equally consistent with several kinds of leakage that make the score look better than it should. Preprocessing fit on the full dataset, a target-encoded feature computed before cross-validation, or the same customer appearing in both train and test can all inflate test performance without producing any error or warning. A high test score tells you the evaluation is optimistic or accurate — it cannot by itself tell you which one, which is exactly why auditing the pipeline for leakage has to happen regardless of how good the numbers look.

Every leakage example in this module — a scaler fit before splitting, a rolling feature computed across the split boundary, a customer's rows scattered across train and test — happens without anyone including the label anywhere in X. Leakage is usually a statistical dependency introduced by the order operations happen in, not a deliberate shortcut. That is exactly why it is dangerous: it survives a visual code review that only checks "is the target column present in the features," and requires actually tracing when each preprocessing step was fit relative to when the split happened.

A random split scatters test samples throughout the timeline, which means some training rows come from dates after some test rows — the model ends up learning from the future to predict the past, a situation that can never occur at actual prediction time. I would instead use a chronological split, where every training row comes strictly before every validation and test row, or TimeSeriesSplit for cross-validation, which performs walk-forward validation: train on an expanding window of the past, validate on the period immediately following it, and repeat. This simulates exactly what the model faces in production — predicting a future it has never seen.

Group leakage happens when a single entity contributes multiple rows to the dataset and a random split scatters that entity's rows across both train and test. The model can then partially recognise the entity from rows it already saw in training, which inflates test performance without the model having learned anything that generalises to a genuinely new entity. I would use GroupShuffleSplit or GroupKFold, passing the entity ID as the groups parameter, which guarantees every row belonging to the same entity ends up entirely in one split or entirely in the other, never split across both.

The test set's entire value is that it gives an honest, unbiased estimate of production performance — but that only holds if no decision during development was ever influenced by its score. The moment you check the test score and use it to choose a hyperparameter, pick a model, or decide whether to keep a feature, the test set has effectively become a second validation set, and every subsequent "improvement" is partly tuned to that specific holdout rather than to the underlying problem. Repeated peeking accumulates: each individual check feels harmless, but across many experiments it silently overfits your final reported number to that one test set.

I would trace the order of operations from raw data to final metric. First, I would check whether any preprocessing — scaling, imputation, encoding, feature selection — was fit before the train/test split rather than after. Second, I would check whether any feature was engineered using the full dataset's chronology or full-dataset aggregates before the split existed, such as a rolling window or a groupby computed pre-split. Third, I would check whether the data has a natural grouping (customers, sessions, devices) and whether the split respected it. Finally I would check the project history for any hyperparameter decision that was made by directly consulting the test score rather than a validation score.

It mostly comes down to dataset size. Under a few thousand rows, a single holdout test set is too small to trust — I would use cross-validation for tuning and evaluation, since it uses the data far more efficiently. In the tens of thousands of rows range, a three-way 70/15/15 split becomes reliable enough to use directly. Above roughly a hundred thousand rows, I would shrink the validation and test proportions further, since even five or ten percent of a very large dataset is statistically enough to give a stable estimate, and it leaves more data available for training. Time-series or grouped data changes the method, not this general sizing logic.