Cross-Validation and the Bias-Variance Tradeoff

You want to measure your average commute time to work. Measuring it once on a Monday gives you one number — but was Monday typical? What if there was unusual traffic? Measure it every day for 3 weeks and take the mean and standard deviation. The mean is your reliable estimate. The std tells you how much it varies. One measurement is a point estimate. Many measurements give you a distribution.

Cross-validation is measuring model performance on 5 or 10 different "days" — different random subsets of the data — and averaging. The result is a reliable estimate of how the model performs on data it has not seen, not a number that got lucky on one split.

import numpy as np
from sklearn.model_selection import (KFold, StratifiedKFold, cross_val_score,
                                      cross_validate, LeaveOneOut)
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import warnings
warnings.filterwarnings('ignore')

np.random.seed(42)
n = 3000

# Brex loan default dataset
income        = np.abs(np.random.normal(50_000, 30_000, n)).clip(8_000, 500_000)
existing_emis = np.abs(np.random.normal(8_000, 6_000, n)).clip(0, 80_000)
credit_score  = np.abs(np.random.normal(680, 80, n)).clip(300, 900)
employment_yr = np.abs(np.random.normal(4, 3, n)).clip(0, 30)
loan_amount   = np.abs(np.random.normal(200_000, 150_000, n)).clip(10_000, 2_000_000)
default_score = (
    -(credit_score-680)/80*0.40 + (existing_emis/income)*0.30
    + (loan_amount/income/12)*0.20 - employment_yr/30*0.10
    + np.random.randn(n)*0.15
)
y = (default_score > 0.3).astype(int)
X = np.column_stack([income, existing_emis, credit_score, employment_yr, loan_amount])

print(f"Dataset: {n} samples, {y.mean()*100:.1f}% default rate")

# ── Always use Pipeline so scaler is fit inside each fold ─────────────
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('model',  GradientBoostingClassifier(
        n_estimators=100, learning_rate=0.1,
        max_depth=3, random_state=42,
    )),
])

# ── Standard 5-fold CV ─────────────────────────────────────────────────
kf     = KFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(pipeline, X, y, cv=kf, scoring='roc_auc', n_jobs=-1)

print(f"
5-fold CV AUC scores: {scores.round(4)}")
print(f"  Mean:  {scores.mean():.4f}")
print(f"  Std:   {scores.std():.4f}")
print(f"  95% CI: [{scores.mean()-1.96*scores.std():.4f}, "
      f"{scores.mean()+1.96*scores.std():.4f}]")

# ── Stratified K-Fold — preserves class balance per fold ──────────────
skf    = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
s_scores = cross_val_score(pipeline, X, y, cv=skf, scoring='roc_auc', n_jobs=-1)

print(f"
Stratified 5-fold CV AUC: {s_scores.mean():.4f} ± {s_scores.std():.4f}")
print(f"  Per-fold positive rates:")
for fold, (tr_idx, te_idx) in enumerate(skf.split(X, y), 1):
    rate = y[te_idx].mean()
    print(f"    Fold {fold}: {rate*100:.1f}% default  ← consistent across folds")

# ── cross_validate — get train AND test scores ─────────────────────────
cv_results = cross_validate(
    pipeline, X, y,
    cv=StratifiedKFold(5, shuffle=True, random_state=42),
    scoring='roc_auc',
    return_train_score=True,
    n_jobs=-1,
)
train_mean = cv_results['train_score'].mean()
val_mean   = cv_results['test_score'].mean()
gap        = train_mean - val_mean

print(f"
cross_validate (train + val scores):")
print(f"  Train AUC: {train_mean:.4f}")
print(f"  Val AUC:   {val_mean:.4f}")
print(f"  Gap:       {gap:.4f}  {'← possible overfitting' if gap > 0.05 else '← healthy'}")

import numpy as np
from sklearn.model_selection import cross_validate, StratifiedKFold
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import warnings
warnings.filterwarnings('ignore')

np.random.seed(42)
n = 2000

income        = np.abs(np.random.normal(50_000, 30_000, n)).clip(8_000, 500_000)
existing_emis = np.abs(np.random.normal(8_000, 6_000, n)).clip(0, 80_000)
credit_score  = np.abs(np.random.normal(680, 80, n)).clip(300, 900)
employment_yr = np.abs(np.random.normal(4, 3, n)).clip(0, 30)
loan_amount   = np.abs(np.random.normal(200_000,150_000,n)).clip(10_000,2_000_000)
default_score = (
    -(credit_score-680)/80*0.40 + (existing_emis/income)*0.30
    + (loan_amount/income/12)*0.20 - employment_yr/30*0.10
    + np.random.randn(n)*0.15
)
y = (default_score > 0.3).astype(int)
X = np.column_stack([income,existing_emis,credit_score,employment_yr,loan_amount])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

models = {
    'Logistic (high bias)':         Pipeline([('sc', StandardScaler()),
                                    ('m', LogisticRegression(C=0.0001, max_iter=1000))]),
    'Decision Tree depth=1 (bias)': Pipeline([('sc', StandardScaler()),
                                    ('m', DecisionTreeClassifier(max_depth=1))]),
    'Decision Tree depth=20 (var)': Pipeline([('sc', StandardScaler()),
                                    ('m', DecisionTreeClassifier(max_depth=20))]),
    'GBM overfit (high var)':       Pipeline([('sc', StandardScaler()),
                                    ('m', GradientBoostingClassifier(
                                        n_estimators=500, max_depth=8,
                                        learning_rate=0.5, random_state=42))]),
    'GBM balanced':                 Pipeline([('sc', StandardScaler()),
                                    ('m', GradientBoostingClassifier(
                                        n_estimators=200, max_depth=3,
                                        learning_rate=0.1, subsample=0.8,
                                        random_state=42))]),
}

print(f"{'Model':<30} {'Train AUC':>10} {'Val AUC':>10} {'Std':>7} {'Gap':>8} {'Diagnosis'}")
print("─" * 88)

for name, model in models.items():
    res     = cross_validate(model, X, y, cv=cv, scoring='roc_auc',
                              return_train_score=True, n_jobs=-1)
    tr_mean = res['train_score'].mean()
    va_mean = res['test_score'].mean()
    va_std  = res['test_score'].std()
    gap     = tr_mean - va_mean

    if va_mean < 0.75 and gap < 0.05:
        diagnosis = 'High bias'
    elif gap > 0.08:
        diagnosis = 'High variance'
    elif va_std > 0.04:
        diagnosis = 'Unstable (high var)'
    else:
        diagnosis = '✓ Good balance'

    print(f"  {name:<28}  {tr_mean:>10.4f}  {va_mean:>10.4f}  {va_std:>7.4f}  {gap:>7.4f}  {diagnosis}")

import numpy as np
import pandas as pd
from sklearn.model_selection import (
    StratifiedKFold, RepeatedStratifiedKFold,
    TimeSeriesSplit, GroupKFold, cross_val_score,
)
from sklearn.ensemble import GradientBoostingClassifier, GradientBoostingRegressor
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import warnings
warnings.filterwarnings('ignore')

np.random.seed(42)
n = 2000

income        = np.abs(np.random.normal(50_000, 30_000, n)).clip(8_000, 500_000)
existing_emis = np.abs(np.random.normal(8_000, 6_000, n)).clip(0, 80_000)
credit_score  = np.abs(np.random.normal(680, 80, n)).clip(300, 900)
employment_yr = np.abs(np.random.normal(4, 3, n)).clip(0, 30)
loan_amount   = np.abs(np.random.normal(200_000,150_000,n)).clip(10_000,2_000_000)
default_score = (
    -(credit_score-680)/80*0.40 + (existing_emis/income)*0.30
    + (loan_amount/income/12)*0.20 - employment_yr/30*0.10
    + np.random.randn(n)*0.15
)
y = (default_score > 0.3).astype(int)
X = np.column_stack([income,existing_emis,credit_score,employment_yr,loan_amount])
customer_ids = np.repeat(np.arange(400), 5)   # 400 customers, 5 loans each

pipeline = Pipeline([
    ('sc', StandardScaler()),
    ('m',  GradientBoostingClassifier(n_estimators=100, learning_rate=0.1,
                                       max_depth=3, random_state=42)),
])

# ── Standard StratifiedKFold ───────────────────────────────────────────
skf     = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
skf_auc = cross_val_score(pipeline, X, y, cv=skf, scoring='roc_auc', n_jobs=-1)
print(f"StratifiedKFold (5):          {skf_auc.mean():.4f} ± {skf_auc.std():.4f}")

# ── RepeatedStratifiedKFold — more stable std estimate ────────────────
rskf     = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=42)
rskf_auc = cross_val_score(pipeline, X, y, cv=rskf, scoring='roc_auc', n_jobs=-1)
print(f"RepeatedStratKFold (5×10):    {rskf_auc.mean():.4f} ± {rskf_auc.std():.4f}")
print(f"  50 scores — much more reliable std estimate")

# ── GroupKFold — customers must not split across folds ─────────────────
gkf     = GroupKFold(n_splits=5)
gkf_auc = cross_val_score(pipeline, X, y, cv=gkf,
                            groups=customer_ids, scoring='roc_auc', n_jobs=-1)
print(f"
GroupKFold (no customer leakage): {gkf_auc.mean():.4f} ± {gkf_auc.std():.4f}")
print(f"  Each fold: all loans from a customer in same split")
print(f"  Difference from StratKFold: {skf_auc.mean()-gkf_auc.mean():+.4f}")
print(f"  Positive gap = identity leakage was inflating StratKFold score")

# ── TimeSeriesSplit — for sequential data ─────────────────────────────
tscv = TimeSeriesSplit(n_splits=5)
ts_auc = cross_val_score(pipeline, X, y, cv=tscv, scoring='roc_auc', n_jobs=-1)
print(f"
TimeSeriesSplit:              {ts_auc.mean():.4f} ± {ts_auc.std():.4f}")
print(f"  Per-fold sizes (train→val):")
for fold, (tr, te) in enumerate(tscv.split(X), 1):
    print(f"    Fold {fold}: {len(tr)} train → {len(te)} val")

Because both models are evaluated on the same folds, their scores are paired. Model A's fold-1 score and Model B's fold-1 score both came from the exact same test samples. A paired t-test on the k differences tests whether the mean difference is significantly different from zero — i.e. whether one model is genuinely better.

import numpy as np
from scipy import stats
from sklearn.model_selection import cross_val_score, RepeatedStratifiedKFold
from sklearn.ensemble import (GradientBoostingClassifier, RandomForestClassifier)
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import warnings
warnings.filterwarnings('ignore')

np.random.seed(42)
n = 2000

income        = np.abs(np.random.normal(50_000,30_000,n)).clip(8_000,500_000)
existing_emis = np.abs(np.random.normal(8_000,6_000,n)).clip(0,80_000)
credit_score  = np.abs(np.random.normal(680,80,n)).clip(300,900)
employment_yr = np.abs(np.random.normal(4,3,n)).clip(0,30)
loan_amount   = np.abs(np.random.normal(200_000,150_000,n)).clip(10_000,2_000_000)
default_score = (
    -(credit_score-680)/80*0.40 + (existing_emis/income)*0.30
    + (loan_amount/income/12)*0.20 - employment_yr/30*0.10
    + np.random.randn(n)*0.15
)
y = (default_score > 0.3).astype(int)
X = np.column_stack([income,existing_emis,credit_score,employment_yr,loan_amount])

# Use RepeatedStratKFold for more reliable comparison
cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=5, random_state=42)

candidates = {
    'LogisticReg':       Pipeline([('sc', StandardScaler()),
                          ('m', LogisticRegression(max_iter=1000, random_state=42))]),
    'RandomForest':      Pipeline([('sc', StandardScaler()),
                          ('m', RandomForestClassifier(n_estimators=100,
                                                        random_state=42, n_jobs=-1))]),
    'GradientBoosting':  Pipeline([('sc', StandardScaler()),
                          ('m', GradientBoostingClassifier(n_estimators=200,
                                  learning_rate=0.1, max_depth=3, random_state=42))]),
}

all_scores = {}
print(f"{'Model':<20} {'Mean AUC':>10} {'Std':>8} {'95% CI'}")
print("─" * 60)

for name, model in candidates.items():
    scores = cross_val_score(model, X, y, cv=cv,
                              scoring='roc_auc', n_jobs=-1)
    all_scores[name] = scores
    ci_lo = scores.mean() - 1.96 * scores.std()
    ci_hi = scores.mean() + 1.96 * scores.std()
    print(f"  {name:<18}  {scores.mean():>10.4f}  {scores.std():>8.4f}  "
          f"[{ci_lo:.4f}, {ci_hi:.4f}]")

# ── Paired t-test: is GBM significantly better than RandomForest? ──────
best   = 'GradientBoosting'
second = 'RandomForest'

scores_a = all_scores[best]
scores_b = all_scores[second]
diff     = scores_a - scores_b

t_stat, p_value = stats.ttest_rel(scores_a, scores_b)

print(f"
Paired t-test: {best} vs {second}")
print(f"  Mean difference: {diff.mean():+.4f}")
print(f"  t-statistic:     {t_stat:.4f}")
print(f"  p-value:         {p_value:.4f}")

if p_value < 0.05:
    winner = best if diff.mean() > 0 else second
    print(f"  → p < 0.05: {winner} is SIGNIFICANTLY better (95% confidence)")
else:
    print(f"  → p ≥ 0.05: difference is NOT statistically significant")
    print(f"  → Choose the simpler model: {second}")

# ── Practical decision rule ────────────────────────────────────────────
print(f"
Practical decision rule:")
print(f"  If p < 0.05 AND mean diff > 0.01: choose complex model")
print(f"  If p ≥ 0.05 OR mean diff < 0.01: choose simpler model")
print(f"  Statistical significance alone is not enough —")
print(f"  a 0.001 AUC difference may be significant but not practically meaningful")

import numpy as np
from sklearn.model_selection import (cross_val_score, GridSearchCV,
                                      StratifiedKFold, cross_validate)
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import warnings
warnings.filterwarnings('ignore')

np.random.seed(42)
n = 1500

income        = np.abs(np.random.normal(50_000,30_000,n)).clip(8_000,500_000)
existing_emis = np.abs(np.random.normal(8_000,6_000,n)).clip(0,80_000)
credit_score  = np.abs(np.random.normal(680,80,n)).clip(300,900)
employment_yr = np.abs(np.random.normal(4,3,n)).clip(0,30)
loan_amount   = np.abs(np.random.normal(200_000,150_000,n)).clip(10_000,2_000_000)
default_score = (
    -(credit_score-680)/80*0.40 + (existing_emis/income)*0.30
    + (loan_amount/income/12)*0.20 - employment_yr/30*0.10
    + np.random.randn(n)*0.15
)
y = (default_score > 0.3).astype(int)
X = np.column_stack([income,existing_emis,credit_score,employment_yr,loan_amount])

# ── Non-nested CV — optimistically biased ─────────────────────────────
pipeline  = Pipeline([('sc', StandardScaler()),
                       ('m', GradientBoostingClassifier(random_state=42))])
param_grid = {
    'm__n_estimators': [100, 200],
    'm__max_depth':    [3, 4],
    'm__learning_rate':[0.05, 0.1],
}

inner_cv  = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)
outer_cv  = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

# Tune on full data then evaluate on same folds — biased
grid_search_biased = GridSearchCV(pipeline, param_grid, cv=inner_cv,
                                   scoring='roc_auc', n_jobs=-1)
grid_search_biased.fit(X, y)
best_pipe = grid_search_biased.best_estimator_

# Evaluating best_pipe on same outer folds is biased
biased_scores = cross_val_score(best_pipe, X, y, cv=outer_cv,
                                 scoring='roc_auc', n_jobs=-1)

# ── Nested CV — unbiased ──────────────────────────────────────────────
# Outer loop: evaluation folds
# Inner loop: hyperparameter search (sees only outer training data)
nested_grid = GridSearchCV(pipeline, param_grid, cv=inner_cv,
                            scoring='roc_auc', n_jobs=-1)
nested_scores = cross_val_score(nested_grid, X, y, cv=outer_cv,
                                 scoring='roc_auc', n_jobs=-1)

print("Non-nested (biased) CV AUC:   "
      f"{biased_scores.mean():.4f} ± {biased_scores.std():.4f}")
print("Nested (unbiased) CV AUC:     "
      f"{nested_scores.mean():.4f} ± {nested_scores.std():.4f}")
gap = biased_scores.mean() - nested_scores.mean()
print(f"Optimism bias:                {gap:+.4f}  ← biased estimate inflated by this much")

print("
Key rule:")
print("  Use nested CV when BOTH tuning and evaluating on same dataset.")
print("  Use simple CV when evaluating a fixed (pre-tuned) model.")
print("  Nested CV is slower (k_outer × k_inner × n_param_combos) but honest.")

The feature days_since_last_support_ticket was computed as today's date minus the date of the customer's last ticket — where "today" meant the date the feature pipeline happened to run, not the historical labelling date each training row actually belonged to. For a row labelled six months ago, the feature was silently computed using six months of hindsight the model would never have at real prediction time.

The fix was to compute every time-based feature relative to a stored snapshot_date column carried alongside each row, matching the moment that row's label was actually decided, and to assert in the pipeline that no feature ever references a date later than its own row's snapshot. Once fixed, CV AUC dropped to a far more believable 0.81 — a worse-looking number that was, for the first time, an honest one.

import pandas as pd

# ── The leakage assertion — catch this class of bug before CV ever runs ──
def build_time_feature(df, ticket_date_col, snapshot_date_col):
    """
    Compute days-since-last-ticket relative to each row's OWN snapshot
    date, never relative to "today". This is what makes the feature
    valid to compute identically in training and in production.
    """
    days_since = (df[snapshot_date_col] - df[ticket_date_col]).dt.days

    # The assertion that would have caught the original bug immediately:
    # a time-based feature must never look past its own row's snapshot.
    assert (df[ticket_date_col] <= df[snapshot_date_col]).all(), (
        "Leakage: a support ticket date falls after its row's snapshot date"
    )
    return days_since

data = pd.DataFrame({
    'ticket_date':   pd.to_datetime(['2025-01-10', '2025-06-01']),
    'snapshot_date': pd.to_datetime(['2025-01-15', '2025-05-20']),
})

try:
    data['days_since_last_ticket'] = build_time_feature(
        data, 'ticket_date', 'snapshot_date'
    )
except AssertionError as e:
    print(f"Caught before CV even ran: {e}")

print("
Rule of thumb: a cross-validation score that looks too good for")
print("the problem is a reason to audit the feature pipeline, not a")
print("reason to celebrate -- CV can only be honest about data that was")
print("already honest before it got there.")

CV only protects you if every data-dependent step happens inside the fold loop. If you call scaler.fit_transform(X_all) or run feature selection on the entire dataset before handing folds to cross_val_score, every fold's "held-out" test data already influenced the transformation that was applied to it — the leakage happened before CV ever saw the data, and CV has no way to detect or undo it. Cross-validation is a splitting strategy, not a leakage firewall; the firewall is wrapping every fitted transformer (scaler, PCA, feature selector, imputer) inside a Pipeline so it refits fresh on each fold's training portion only.

CV estimates performance on data drawn from the same distribution as your training set, evaluated at the same point in time. It cannot detect distribution shift that hasn't happened yet — new fraud patterns, a changed customer base, a macroeconomic shift — nor can it catch leakage that is baked identically into both the training and test folds (like a feature computed using information that would not exist at prediction time in production, e.g. a "final account balance" column). A CV score is an honest estimate of past-and-present performance under stated assumptions, never a forward-looking guarantee. Production monitoring exists precisely because CV cannot see the future.

On an imbalanced classification problem, plain KFold splits by row count only — with a 1.5% positive rate and 5 folds, random chance alone can put noticeably different numbers of positives in each fold, occasionally leaving a fold with almost none. That produces AUC or F1 scores that swing wildly between folds not because the model is unstable, but because the folds themselves have inconsistent class balance — inflating the variance of your estimate for a reason that has nothing to do with model quality. StratifiedKFold fixes this by preserving the overall class ratio in every fold. For any classification task with meaningful imbalance, it should be the default, not an occasional upgrade.

Disabling shuffle keeps each fold as a contiguous chronological block, but standard KFold still rotates which block is "test" — meaning folds 2 through 5 are trained on data that includes chunks from later in time than the test block itself. Any fold where the test block sits earlier than some of the training blocks lets the model learn from the future to predict the past, which is exactly the leakage you're trying to avoid. The only fix is a forward-chaining split — TimeSeriesSplit or a manual walk-forward scheme — where every training set consists exclusively of data that occurred strictly before its corresponding test set, every single fold.

Increasing k reduces bias in the performance estimate — each training set is closer in size to the full dataset — but it does not straightforwardly reduce variance, and often increases it in practice: with k=n (Leave-One-Out), the n training sets overlap almost completely, so the n resulting models are highly correlated with each other, and for unstable models a single influential outlier can swing many of those near-identical folds in the same direction. LOOCV is also n times more expensive to compute than 5-fold CV, for an estimate whose variance properties are not clearly superior. In practice 5-fold or 10-fold CV, or RepeatedStratifiedKFold for extra stability, is the better default — LOOCV is reserved for genuinely tiny datasets where every training sample is too precious to exclude even one at a time in a normal k-fold scheme.

The dataset is split into k equal folds. In each of k rounds, one fold is held out as the test set and the model is trained from scratch on the remaining k−1 folds, then scored on the held-out fold. After k rounds, every sample has served as test data exactly once, giving k independent scores instead of one. A single train/test split is essentially a sample of size one from the distribution of possible splits — its score depends heavily on which particular samples the random seed happened to place in the test set. Averaging k scores gives a mean that is far less sensitive to that randomness, and the standard deviation across folds tells you how much to trust that mean — something a single split literally cannot report.

Any transformer that is *fit* on data — a StandardScaler learning a mean and standard deviation, a feature selector learning which columns correlate with the target, a PCA learning principal components — is learning statistics from that data. If you fit it on the full dataset before splitting into folds, every fold's "held-out" test portion already contributed to those statistics, so the test score is contaminated by information it should never have seen — this is data leakage, and it makes CV scores optimistically biased relative to true held-out performance. The fix is to wrap the scaler/selector/model together in an sklearn Pipeline and pass the whole pipeline to cross_val_score — sklearn then refits every transformer from scratch on each fold's training data alone.

StratifiedKFold is the default for any classification task, especially with class imbalance — it keeps the same positive/negative ratio in every fold instead of letting it vary by chance. GroupKFold is for data with a natural grouping where samples from the same entity must never be split across train and test — all of one customer's transactions, all of one patient's visits — otherwise the model can partially memorise entity-specific patterns and the score is inflated by identity leakage rather than genuine generalisation. TimeSeriesSplit is mandatory for any chronologically ordered data — it only ever trains on the past and tests on the immediate future, which prevents the model from learning from information that would not have existed yet at prediction time.

No — CV only estimates performance on data that looks like your training set, evaluated as if the world stays the same. It can't detect problems that are baked identically into every fold, like a feature that leaks future information in a way that also exists in production data at training time. And it says nothing about distribution shift that happens after deployment — new fraud tactics, seasonal changes in customer behaviour, a change in the sensor hardware feeding your features. A high CV score means "this model generalises well to unseen data drawn from the same distribution as my training set, right now" — which is exactly why production monitoring, not just a good offline CV number, is required before and after shipping a model.

No, and this is a common misconception. LOOCV does reduce bias, since each training set uses n−1 of n samples — about as close to the full dataset as you can get. But the n resulting models are trained on nearly identical data and are therefore highly correlated with each other, so the variance of the overall estimate isn't necessarily better than k-fold, and for models sensitive to individual data points a single outlier can distort many of the n folds simultaneously in a correlated way. It's also n times more expensive to run than 5-fold CV. In practice, 5- or 10-fold (or repeated k-fold for extra stability) is the standard choice, and LOOCV is reserved for genuinely small datasets — a few hundred samples or fewer — where standard k-fold would leave too little training data per fold.