Calibration — Are Your Probabilities Trustworthy?

A weather forecaster who says "70% chance of rain" is well-calibrated if it actually rains on about 70% of the days they make that prediction. A poorly-calibrated forecaster might say "70%" but it only rains 30% of those days — their confidence is systematically too high. You still trust their ranking (70% means more likely than 40%) but you cannot trust the actual number.

ML models are forecasters. ROC-AUC measures whether their rankings are correct. Calibration measures whether their actual numbers are correct. Both matter — but only calibration lets you use the probabilities for quantitative business decisions.

import numpy as np
from sklearn.calibration import calibration_curve
from sklearn.ensemble import GradientBoostingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import warnings
warnings.filterwarnings('ignore')

np.random.seed(42)
n = 8_000

# Stripe credit scoring dataset
income        = np.abs(np.random.normal(50_000, 30_000, n)).clip(8_000, 500_000)
existing_emis = np.abs(np.random.normal(8_000, 6_000, n)).clip(0, 80_000)
credit_score  = np.abs(np.random.normal(680, 80, n)).clip(300, 900)
employment_yr = np.abs(np.random.normal(4, 3, n)).clip(0, 30)
loan_amount   = np.abs(np.random.normal(200_000, 150_000, n)).clip(10_000, 2_000_000)

default_score = (
    -(credit_score - 680) / 80 * 0.40
    + (existing_emis / income) * 0.30
    + (loan_amount / income / 12) * 0.20
    - employment_yr / 30 * 0.10
    + np.random.randn(n) * 0.15
)
y = (default_score > 0.3).astype(int)
X = np.column_stack([income, existing_emis, credit_score,
                     employment_yr, loan_amount])

X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)
sc      = StandardScaler()
X_tr_sc = sc.fit_transform(X_tr)
X_te_sc = sc.transform(X_te)

# ── Train three models ─────────────────────────────────────────────────
models = {
    'LogisticRegression': LogisticRegression(max_iter=1000, random_state=42),
    'RandomForest':       RandomForestClassifier(n_estimators=100, random_state=42),
    'GradientBoosting':   GradientBoostingClassifier(n_estimators=200,
                            learning_rate=0.1, max_depth=3, random_state=42),
}

print("Calibration check per model (reliability diagram data):")
print(f"{'Model':<22} {'Bin':<8} {'Predicted':>10} {'Actual':>10} {'Gap':>8}")
print("─" * 62)

for name, model in models.items():
    model.fit(X_tr_sc, y_tr)
    y_prob = model.predict_proba(X_te_sc)[:, 1]

    # calibration_curve: bin predicted probs, compare to actual fraction
    fraction_pos, mean_pred = calibration_curve(
        y_te, y_prob, n_bins=10, strategy='uniform'
    )
    print(f"
  {name}:")
    for fp, mp in zip(fraction_pos, mean_pred):
        gap = fp - mp
        bar = '▸' * int(abs(gap) * 40)
        direction = '+' if gap > 0 else '-'
        print(f"    pred={mp:.2f}  actual={fp:.2f}  gap={direction}{abs(gap):.3f}  {bar}")

    # Mean calibration error
    mce = np.mean(np.abs(fraction_pos - mean_pred))
    print(f"  Mean calibration error: {mce:.4f}")

The Brier score can be decomposed into three components: calibration (how far are the predicted probs from the actual rates?), resolution (does the model separate positives from negatives?), and uncertainty (inherent noise in the problem). Good models minimise calibration error while maximising resolution.

import numpy as np
from sklearn.metrics import brier_score_loss
from sklearn.calibration import calibration_curve, CalibratedClassifierCV
from sklearn.ensemble import GradientBoostingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import warnings
warnings.filterwarnings('ignore')

np.random.seed(42)
n = 8_000

income        = np.abs(np.random.normal(50_000, 30_000, n)).clip(8_000, 500_000)
existing_emis = np.abs(np.random.normal(8_000, 6_000, n)).clip(0, 80_000)
credit_score  = np.abs(np.random.normal(680, 80, n)).clip(300, 900)
employment_yr = np.abs(np.random.normal(4, 3, n)).clip(0, 30)
loan_amount   = np.abs(np.random.normal(200_000, 150_000, n)).clip(10_000, 2_000_000)
default_score = (
    -(credit_score-680)/80*0.40 + (existing_emis/income)*0.30
    + (loan_amount/income/12)*0.20 - employment_yr/30*0.10
    + np.random.randn(n)*0.15
)
y = (default_score > 0.3).astype(int)
X = np.column_stack([income, existing_emis, credit_score, employment_yr, loan_amount])

X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)
sc      = StandardScaler()
X_tr_sc = sc.fit_transform(X_tr)
X_te_sc = sc.transform(X_te)

# ── Baseline Brier score ───────────────────────────────────────────────
base_rate         = y_te.mean()
baseline_brier    = base_rate * (1 - base_rate)
always_zero_brier = brier_score_loss(y_te, np.zeros(len(y_te)))
always_base_brier = brier_score_loss(y_te, np.full(len(y_te), base_rate))

print(f"Baseline context:")
print(f"  Default rate:               {base_rate:.3f}  ({base_rate*100:.1f}%)")
print(f"  Baseline Brier (predict mean): {always_base_brier:.4f}")
print(f"  Brier (always predict 0):   {always_zero_brier:.4f}")
print()

# ── Compare models ─────────────────────────────────────────────────────
models = {
    'LogisticRegression': LogisticRegression(max_iter=1000, random_state=42),
    'RandomForest':       RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1),
    'GradientBoosting':   GradientBoostingClassifier(n_estimators=200,
                            learning_rate=0.1, max_depth=3, random_state=42),
}

from sklearn.metrics import roc_auc_score
print(f"{'Model':<22} {'Brier':>8} {'vs baseline':>12} {'ROC-AUC':>9}")
print("─" * 56)

for name, model in models.items():
    model.fit(X_tr_sc, y_tr)
    y_prob = model.predict_proba(X_te_sc)[:, 1]
    brier  = brier_score_loss(y_te, y_prob)
    auc    = roc_auc_score(y_te, y_prob)
    improv = (always_base_brier - brier) / always_base_brier * 100
    print(f"  {name:<20}  {brier:>8.4f}  {improv:>+10.1f}%  {auc:>9.4f}")

# ── Brier skill score — normalised Brier ─────────────────────────────
# BSS = 1 − Brier / Brier_baseline
# BSS = 1.0: perfect. BSS = 0: no better than baseline. BSS < 0: worse.
print("
Brier Skill Score (BSS = 1 − Brier/Baseline):")
for name, model in models.items():
    y_prob = model.predict_proba(X_te_sc)[:, 1]
    brier  = brier_score_loss(y_te, y_prob)
    bss    = 1 - brier / always_base_brier
    bar    = '█' * int(bss * 40)
    print(f"  {name:<20}: {bar:<40} {bss:.4f}")

import numpy as np
from sklearn.calibration import CalibratedClassifierCV, calibration_curve
from sklearn.ensemble import GradientBoostingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import brier_score_loss
import warnings
warnings.filterwarnings('ignore')

np.random.seed(42)
n = 10_000

income        = np.abs(np.random.normal(50_000, 30_000, n)).clip(8_000, 500_000)
existing_emis = np.abs(np.random.normal(8_000, 6_000, n)).clip(0, 80_000)
credit_score  = np.abs(np.random.normal(680, 80, n)).clip(300, 900)
employment_yr = np.abs(np.random.normal(4, 3, n)).clip(0, 30)
loan_amount   = np.abs(np.random.normal(200_000, 150_000, n)).clip(10_000, 2_000_000)
default_score = (
    -(credit_score-680)/80*0.40 + (existing_emis/income)*0.30
    + (loan_amount/income/12)*0.20 - employment_yr/30*0.10
    + np.random.randn(n)*0.15
)
y = (default_score > 0.3).astype(int)
X = np.column_stack([income, existing_emis, credit_score, employment_yr, loan_amount])

# Three-way split: train / calibrate / test
X_temp, X_te, y_temp, y_te = train_test_split(X, y, test_size=0.2,
                                                stratify=y, random_state=42)
X_tr, X_cal, y_tr, y_cal   = train_test_split(X_temp, y_temp, test_size=0.25,
                                                stratify=y_temp, random_state=42)
sc      = StandardScaler()
X_tr_sc  = sc.fit_transform(X_tr)
X_cal_sc = sc.transform(X_cal)
X_te_sc  = sc.transform(X_te)

print(f"Train: {len(X_tr):,}  Calibration: {len(X_cal):,}  Test: {len(X_te):,}")

# ── Calibrate GradientBoosting (typically overconfident) ──────────────
base_gb = GradientBoostingClassifier(n_estimators=200, learning_rate=0.1,
                                      max_depth=3, random_state=42)
base_gb.fit(X_tr_sc, y_tr)

# Method 1: CalibratedClassifierCV with prefit=True
# Use when you want to calibrate on a separate held-out set
from sklearn.calibration import CalibratedClassifierCV

calib_sigmoid  = CalibratedClassifierCV(base_gb, cv='prefit', method='sigmoid')
calib_isotonic = CalibratedClassifierCV(base_gb, cv='prefit', method='isotonic')

calib_sigmoid.fit(X_cal_sc, y_cal)
calib_isotonic.fit(X_cal_sc, y_cal)

# Method 2: CalibratedClassifierCV with cv=5
# Calibration happens inside 5-fold CV — no need for separate calibration set
calib_cv = CalibratedClassifierCV(
    GradientBoostingClassifier(n_estimators=200, learning_rate=0.1,
                                max_depth=3, random_state=42),
    cv=5, method='isotonic',
)
calib_cv.fit(X_tr_sc, y_tr)   # fits on all training data with internal CV

# ── Compare Brier scores ───────────────────────────────────────────────
models_eval = {
    'Uncalibrated GB':        base_gb,
    'Platt Scaling (sigmoid)': calib_sigmoid,
    'Isotonic Regression':    calib_isotonic,
    'CV Calibration (k=5)':   calib_cv,
    'Logistic (reference)':   LogisticRegression(max_iter=1000, random_state=42),
}

print(f"
{'Model':<28} {'Brier':>8} {'MCE':>8}  {'Calibration quality'}")
print("─" * 70)

lr_ref = LogisticRegression(max_iter=1000, random_state=42)
lr_ref.fit(X_tr_sc, y_tr)
models_eval['Logistic (reference)'] = lr_ref

for name, model in models_eval.items():
    y_prob = model.predict_proba(X_te_sc)[:, 1]
    brier  = brier_score_loss(y_te, y_prob)

    frac, mean_pred = calibration_curve(y_te, y_prob, n_bins=10)
    mce = np.mean(np.abs(frac - mean_pred))

    quality = '●●●' if mce < 0.03 else '●●○' if mce < 0.06 else '●○○'
    print(f"  {name:<26}  {brier:>8.4f}  {mce:>6.4f}  {quality}")

# ── Practical calibration check on specific probability bins ───────────
print("
Calibration at key probability thresholds (isotonic-calibrated GBM):")
y_prob_calib = calib_isotonic.predict_proba(X_te_sc)[:, 1]
for low, high in [(0.0,0.1),(0.1,0.3),(0.3,0.5),(0.5,0.7),(0.7,0.9),(0.9,1.0)]:
    mask   = (y_prob_calib >= low) & (y_prob_calib < high)
    if mask.sum() > 10:
        actual = y_te[mask].mean()
        midpt  = (low + high) / 2
        gap    = actual - midpt
        print(f"  Predicted {low:.1f}–{high:.1f}: actual={actual:.3f}  "
              f"mid={midpt:.2f}  gap={gap:+.3f}  n={mask.sum()}")

import numpy as np
import pandas as pd
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.calibration import CalibratedClassifierCV, calibration_curve
from sklearn.model_selection import train_test_split, StratifiedKFold
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import brier_score_loss, roc_auc_score
from sklearn.pipeline import Pipeline
import joblib, warnings
warnings.filterwarnings('ignore')

np.random.seed(42)
n = 20_000

df = pd.DataFrame({
    'annual_income':   np.abs(np.random.normal(50_000, 30_000, n)).clip(8_000, 500_000),
    'existing_emis':   np.abs(np.random.normal(8_000, 6_000, n)).clip(0, 80_000),
    'credit_score':    np.abs(np.random.normal(680, 80, n)).clip(300, 900).astype(int),
    'employment_yrs':  np.abs(np.random.normal(4, 3, n)).clip(0, 30).round(1),
    'loan_amount':     np.abs(np.random.normal(200_000,150_000,n)).clip(10_000, 2_000_000),
    'n_existing_loans':np.random.randint(0, 6, n).astype(float),
    'age':             np.abs(np.random.normal(35, 10, n)).clip(21, 70).astype(int),
})
default_score = (
    -(df['credit_score']-680)/80*0.35
    + (df['existing_emis']/df['annual_income'])*0.30
    + (df['loan_amount']/df['annual_income']/12)*0.20
    - df['employment_yrs']/30*0.10
    + df['n_existing_loans']*0.03
    + np.random.randn(n)*0.12
)
y = (default_score > 0.3).astype(int)

feat_cols = list(df.columns)
X_tr, X_te, y_tr, y_te = train_test_split(
    df, y, test_size=0.15, stratify=y, random_state=42
)

# ── Pipeline: scale → GBM → isotonic calibration ──────────────────────
# Using cv=5: calibration is done inside cross-validation
# This is the most robust approach — no need to carve out a calibration set
base_model = GradientBoostingClassifier(
    n_estimators=300, learning_rate=0.05, max_depth=4,
    subsample=0.8, min_samples_leaf=10, random_state=42,
)
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('model',  CalibratedClassifierCV(base_model, cv=5, method='isotonic')),
])
pipeline.fit(X_tr[feat_cols], y_tr)

# ── Evaluation ─────────────────────────────────────────────────────────
y_prob = pipeline.predict_proba(X_te[feat_cols])[:, 1]
brier  = brier_score_loss(y_te, y_prob)
auc    = roc_auc_score(y_te, y_prob)

frac, mean_pred = calibration_curve(y_te, y_prob, n_bins=10)
mce = np.mean(np.abs(frac - mean_pred))

print(f"Production model metrics:")
print(f"  ROC-AUC:                {auc:.4f}")
print(f"  Brier score:            {brier:.4f}")
print(f"  Mean calibration error: {mce:.4f}")

# ── Expected calibration error (ECE) — weighted by bin size ───────────
# ECE is the standard calibration metric in ML literature
# Weighted by fraction of samples in each bin — penalises large bins more
frac_all, pred_all = calibration_curve(y_te, y_prob, n_bins=10, strategy='uniform')
bin_sizes = []
bins = np.linspace(0, 1, 11)
for i in range(len(bins)-1):
    mask = (y_prob >= bins[i]) & (y_prob < bins[i+1])
    bin_sizes.append(mask.sum())
bin_sizes = np.array(bin_sizes[:len(frac_all)])
weights = bin_sizes / bin_sizes.sum()
ece = np.sum(weights * np.abs(frac_all - pred_all))
print(f"  ECE (weighted):         {ece:.4f}")

# ── Score new applicants ───────────────────────────────────────────────
new_applications = pd.DataFrame([
    {
        'annual_income': 85_000, 'existing_emis': 12_000,
        'credit_score': 720, 'employment_yrs': 6,
        'loan_amount': 500_000, 'n_existing_loans': 1, 'age': 32,
    },
    {
        'annual_income': 25_000, 'existing_emis': 18_000,
        'credit_score': 580, 'employment_yrs': 1,
        'loan_amount': 300_000, 'n_existing_loans': 3, 'age': 26,
    },
    {
        'annual_income': 150_000, 'existing_emis': 5_000,
        'credit_score': 800, 'employment_yrs': 15,
        'loan_amount': 1_000_000, 'n_existing_loans': 0, 'age': 45,
    },
])

probas   = pipeline.predict_proba(new_applications[feat_cols])[:, 1]
for i, (_, row) in enumerate(new_applications.iterrows()):
    p      = probas[i]
    risk   = 'HIGH' if p > 0.4 else 'MEDIUM' if p > 0.2 else 'LOW'
    action = 'REJECT' if p > 0.4 else 'REVIEW' if p > 0.2 else 'APPROVE'
    print(f"
Applicant {i+1}: income=${row['annual_income']:,.0f}  "
          f"credit={row['credit_score']}  loan=${row['loan_amount']:,.0f}")
    print(f"  P(default) = {p:.3f}  →  {risk} risk  →  {action}")

# ── Save ─────────────────────────────────────────────────────────────
joblib.dump(pipeline, '/tmp/stripe_credit_calibrated.pkl')
print("
Calibrated pipeline saved.")

import numpy as np
from sklearn.metrics import brier_score_loss
from sklearn.calibration import calibration_curve

np.random.seed(42)

# ── Simulate monthly calibration monitoring ───────────────────────────
# Model trained in January, deployed for 6 months
# Default rate increases over time (economic stress)

def simulate_month(month: int, base_default_rate: float, n: int = 2000):
    """Simulate production data for a given month."""
    # Default rate increases 0.5% per month (economic stress scenario)
    actual_default_rate = base_default_rate + month * 0.005

    # Model was trained on Jan data — scores become increasingly optimistic
    # as the distribution shifts
    model_scores = np.random.beta(2, 5, n)   # model outputs skewed low
    # Actual labels: more defaults than model expects by month 6
    actual_labels = (np.random.random(n) < (
        model_scores * (1 + month * 0.08)   # drift factor
    )).astype(int)
    return model_scores, actual_labels

print("Monthly calibration monitoring:")
print(f"{'Month':<8} {'Actual default%':>16} {'Predicted avg%':>15} {'Brier':>8} {'MCE':>8} {'Alert'}")
print("─" * 68)

base_default = 0.12   # 12% default rate at training time
alert_thresh = 0.05   # alert if MCE > 5%

for month in range(7):
    scores, labels = simulate_month(month, base_default)

    brier    = brier_score_loss(labels, scores)
    actual_r = labels.mean()
    pred_avg = scores.mean()

    frac, pred = calibration_curve(labels, scores, n_bins=5)
    mce = np.mean(np.abs(frac - pred)) if len(frac) > 0 else 0

    alert = '⚠ RECALIBRATE' if mce > alert_thresh else '✓ OK'
    month_name = ['Jan (train)', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul'][month]
    print(f"  {month_name:<8}  {actual_r*100:>13.1f}%  {pred_avg*100:>12.1f}%  "
          f"{brier:>8.4f}  {mce:>6.4f}  {alert}")

print("
Key insight:")
print("  Brier score and MCE both increase as distribution shifts")
print("  Set automated alerts: if MCE > 0.05 → trigger recalibration")
print("  Recalibration: refit CalibratedClassifierCV on recent labelled data")
print("  Does not require retraining the full model — just the calibrator")

# ── Efficient recalibration — no full retrain needed ──────────────────
# When calibration drifts: collect recent labelled data → refit calibrator
print("
Recalibration workflow:")
print("  1. Collect last 30 days of transactions with known outcomes")
print("  2. Get base model scores: scores = base_model.predict_proba(X_recent)[:,1]")
print("  3. Fit new isotonic calibrator on (scores, y_recent)")
print("  4. Replace old calibrator — base model unchanged")
print("  5. Monitor ECE weekly — recalibrate monthly or on drift alert")

Accuracy and ROC-AUC measure discrimination — can the model rank positives above negatives? Calibration measures something orthogonal — do the actual probability numbers mean what they say? A Random Forest with AUC = 0.93 routinely has predictions of 0.9 that are actually right only 70% of the time, because RF probabilities are vote fractions across trees, not maximum-likelihood estimates trained to match observed frequencies. You can have excellent ranking and badly wrong probability values at the same time — they are two separate properties, and one gives you no information about the other.

This is the same error in reverse. A model that always predicts the base rate — say 0.12 for every single applicant, regardless of their features — is perfectly calibrated: among all predictions of 0.12, exactly 12% are positive, because that's everyone. Its reliability diagram is a single dot sitting exactly on the diagonal. But its Brier score equals base_rate × (1 − base_rate), no better than guessing, and it has zero discriminative power — it cannot tell a safe applicant from a risky one. Good calibration is necessary for trustworthy probabilities but says nothing about whether the model has learned anything useful. You need both calibration and discrimination, and they must be checked independently.

More flexible only helps if you have enough data to estimate that flexibility reliably. Isotonic regression fits an arbitrary non-decreasing step function with potentially thousands of steps — with a calibration set under roughly 1,000–5,000 samples, it overfits to the exact scores it saw and generalises poorly to new data. Platt scaling fits only 2 parameters (a sigmoid's slope and intercept), so it is far more data-efficient and safer on small calibration sets, even though it can only correct sigmoid-shaped miscalibration. The right choice is a function of calibration-set size and the shape of the miscalibration, not a blanket "more flexible wins."

If the base model has already seen and fit to those labels, its scores on that data are artificially confident — the calibrator then learns a mapping between "scores this model happens to nail" and the true labels, which does not hold on unseen data. This produces a calibrator that looks flawless on paper and is measurably worse than no calibration at all in production. The fix is a genuinely held-out calibration set (a three-way train/calibrate/test split) or CalibratedClassifierCV(cv=5), which handles the held-out split internally via cross-validation — never cv='prefit' with training data.

It matters enormously when the probability's numeric value drives a downstream calculation — credit risk pricing, expected-loss reserves, medical treatment probabilities, insurance premiums — because someone is doing arithmetic with that exact number. It matters far less for search ranking, recommendation systems, or "top-k" fraud triage, where only the relative order of scores determines the outcome and the absolute value of 0.73 vs 0.68 is never read by a human or used in a formula. Spending calibration effort on a pure ranking system is often wasted engineering time that would be better spent improving discrimination (AUC) instead.

Calibration means that among all the times a model predicts probability p, roughly p fraction of those cases are actually positive — if it says 80% a hundred times, about 80 of them should be true. Accuracy and AUC only measure whether the model separates positives from negatives correctly in relative terms, which is a completely different property from whether the absolute probability values are meaningful. A model can rank perfectly — every fraud case scored above every legit one — while its actual numbers are systematically too extreme or too conservative, because nothing in a typical training objective (like accuracy) forces the raw output to match observed frequencies.

Two complementary tools. First, a reliability diagram: bin predictions by predicted probability, plot the actual fraction of positives per bin against the predicted mean per bin, and look for deviation from the diagonal — an S-curve below it means overconfidence, above it means underconfidence. Second, a single summary number: the Brier score, the mean squared error between predicted probabilities and actual labels, compared against the baseline of always predicting the base rate. I'd also compute Expected Calibration Error (ECE), which weights the reliability-diagram gap by how many samples fall in each bin, so it isn't skewed by a handful of underpopulated bins.

It comes down to calibration-set size and the shape of the miscalibration. Platt scaling fits a 2-parameter sigmoid, so it's robust on small calibration sets (a few hundred to a couple thousand samples) but can only correct sigmoid-shaped miscalibration — it's the right default for models like SVMs or gradient boosting that tend to be overconfident in a roughly sigmoidal way. Isotonic regression fits an arbitrary monotonic step function, so it can correct any monotonic miscalibration pattern, including the underconfident S-curve typical of Random Forests, but it needs 5,000+ calibration samples to avoid overfitting the step function to noise. When in doubt, CalibratedClassifierCV(cv=5) with either method sidesteps a lot of this by calibrating inside cross-validation.

Because the base model's outputs on its own training data are optimistically biased — it has already fit those exact labels, so its scores on that set are more confident and more "correct-looking" than its scores will be on genuinely new data. A calibrator fit on that inflated relationship learns a mapping that undoes calibration rather than fixing it — it looks perfect on the training set and is measurably worse than doing nothing once deployed. The fix is a strict three-way split — train the base model on one partition, fit the calibrator on a second partition the base model never saw, and evaluate Brier score and the reliability diagram only on a third, held-out test partition.

Calibration matters enormously in credit risk scoring: if a model says a loan applicant has a 15% default probability, that number often feeds directly into an interest-rate or expected-loss formula, so if the true rate is actually 40%, the business systematically underprices risk and loses money — the absolute value of the number is load-bearing. It matters far less in a search or recommendation ranking system, where the product only ever surfaces the top-k highest-scoring items — swapping 0.91 and 0.88 for two items changes nothing about the user experience as long as their relative order stays correct. I'd invest calibration effort proportional to how directly the raw probability number is consumed downstream, not spend it uniformly across every model.