ROC Curve and AUC — Threshold-Independent Evaluation

Imagine 100 Brex loan applicants — 10 will default, 90 will not. You line them up ordered by your model's default score, highest first. The ROC curve asks: as you walk down the line and draw a threshold between each pair of adjacent applicants, what fraction of the 10 defaulters have you caught so far (TPR), and what fraction of the 90 non-defaulters have you incorrectly included (FPR)?

If your model is perfect, all 10 defaulters appear at the top of the list before any non-defaulter. TPR reaches 1.0 while FPR is still 0.0 — a curve that hugs the top-left corner. AUC = 1.0. If your model is random, defaulters and non-defaulters are scattered randomly — TPR and FPR increase at the same rate. AUC = 0.5.

import numpy as np
from sklearn.metrics import roc_curve, auc

# ── Manual ROC curve construction — every step visible ─────────────────
y_true  = np.array([1, 1, 0, 1, 0, 1, 0, 0, 0, 0])
y_score = np.array([0.95, 0.88, 0.81, 0.74, 0.68,
                    0.55, 0.42, 0.31, 0.20, 0.09])

n_pos = y_true.sum()        # 4 actual fraud cases
n_neg = (1 - y_true).sum()  # 6 actual legit cases

print("Building ROC curve manually:")
print(f"{'Threshold':>12} {'TP':>5} {'FP':>5} {'TPR':>8} {'FPR':>8} {'Point'}")
print("─" * 58)

# Sort by score descending — walk from high threshold to low
order       = np.argsort(y_score)[::-1]
y_sorted    = y_true[order]
score_sorted = y_score[order]

roc_points = [(0.0, 0.0)]   # start at origin
tp, fp = 0, 0
for i, (label, score) in enumerate(zip(y_sorted, score_sorted)):
    if label == 1:
        tp += 1
    else:
        fp += 1
    tpr = tp / n_pos
    fpr = fp / n_neg
    roc_points.append((fpr, tpr))
    print(f"  t≥{score:.2f}     {tp:>5} {fp:>5} {tpr:>8.3f} {fpr:>8.3f}  ({fpr:.2f}, {tpr:.2f})")

# ── AUC using the trapezoid rule ───────────────────────────────────────
fprs = [p[0] for p in roc_points]
tprs = [p[1] for p in roc_points]

# Trapezoid rule: sum of (width × average height) for each step
manual_auc = sum(
    (fprs[i+1] - fprs[i]) * (tprs[i+1] + tprs[i]) / 2
    for i in range(len(fprs) - 1)
)
print(f"
Manual AUC (trapezoid rule): {manual_auc:.4f}")

# ── sklearn verification ───────────────────────────────────────────────
fpr_sk, tpr_sk, thresholds_sk = roc_curve(y_true, y_score)
auc_sk = auc(fpr_sk, tpr_sk)
print(f"sklearn AUC:                 {auc_sk:.4f}  ← matches")

A concordant pair is a (positive, negative) pair where the model correctly scores the positive higher. A discordant pair is one where the negative scores higher. AUC equals the fraction of all possible positive-negative pairs that are concordant. This is the Mann-Whitney U statistic — a non-parametric test that predates ROC analysis by decades.

import numpy as np
from sklearn.metrics import roc_auc_score

np.random.seed(42)
n_pos, n_neg = 200, 800   # Brex loan dataset: 20% default rate

# Simulate model scores
pos_scores = np.random.beta(5, 2, n_pos)   # defaulters score higher
neg_scores = np.random.beta(2, 5, n_neg)   # non-defaulters score lower

y_true  = np.array([1]*n_pos + [0]*n_neg)
y_score = np.concatenate([pos_scores, neg_scores])

# ── Method 1: sklearn ──────────────────────────────────────────────────
auc_sklearn = roc_auc_score(y_true, y_score)

# ── Method 2: probabilistic — count concordant pairs ──────────────────
concordant = 0
discordant = 0
tied       = 0

# Sample 10,000 random pairs for speed (exact needs n_pos × n_neg pairs)
np.random.seed(0)
n_pairs = 10_000
pos_sample = np.random.choice(pos_scores, n_pairs)
neg_sample = np.random.choice(neg_scores, n_pairs)

concordant = (pos_sample > neg_sample).sum()
discordant = (pos_sample < neg_sample).sum()
tied       = (pos_sample == neg_sample).sum()

auc_manual = (concordant + 0.5 * tied) / n_pairs

print(f"AUC (sklearn):               {auc_sklearn:.4f}")
print(f"AUC (concordant pairs):      {auc_manual:.4f}  ← same interpretation")
print(f"
Concordant pairs: {concordant:,}/{n_pairs:,} ({concordant/n_pairs*100:.1f}%)")
print(f"Discordant pairs: {discordant:,}/{n_pairs:,} ({discordant/n_pairs*100:.1f}%)")
print(f"Tied pairs:       {tied:,}/{n_pairs:,}")
print(f"
Interpretation: if you pick one random defaulter and one random")
print(f"non-defaulter, the model ranks the defaulter higher {auc_sklearn*100:.1f}% of the time.")

# ── AUC on different datasets — what the numbers mean in practice ──────
print("
AUC benchmarks by domain:")
benchmarks = [
    ('Fraud detection (Stripe)',    0.94, 'Production quality'),
    ('Credit scoring (Brex)',         0.88, 'Good, acceptable'),
    ('Churn prediction (DoorDash)',     0.81, 'Fair, investigate features'),
    ('Random model (baseline)',       0.50, 'No signal at all'),
    ('Reverse model (worse than rnd)',0.30, 'AUC < 0.5 — flip predictions'),
]
for name, auc_val, verdict in benchmarks:
    bar = '█' * int(auc_val * 30)
    print(f"  {name:<35}: {bar:<30} {auc_val:.2f}  {verdict}")

import numpy as np
from sklearn.metrics import (roc_auc_score, average_precision_score,
                              roc_curve, precision_recall_curve)
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import warnings
warnings.filterwarnings('ignore')

np.random.seed(42)

# ── Demonstrate ROC vs PR on severely imbalanced data ──────────────────
# 0.5% fraud rate — extreme imbalance
n = 20_000
X = np.random.randn(n, 8)
# Fraud: only 100 cases out of 20,000 (0.5%)
y = np.zeros(n, dtype=int)
fraud_idx = np.random.choice(n, 100, replace=False)
y[fraud_idx] = 1
# Add some signal
X[fraud_idx, 0] += 2.5
X[fraud_idx, 1] += 1.8

X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)
sc = StandardScaler()
model = GradientBoostingClassifier(
    n_estimators=200, learning_rate=0.1, max_depth=3,
    subsample=0.8, random_state=42,
    scale_pos_weight=None,
)
model.fit(sc.fit_transform(X_tr), y_tr)
y_prob = model.predict_proba(sc.transform(X_te))[:, 1]

roc_auc = roc_auc_score(y_te, y_prob)
pr_auc  = average_precision_score(y_te, y_prob)
base_pr = y_te.mean()   # PR-AUC baseline = fraud rate

print(f"Dataset: {y_te.sum()} fraud / {(y_te==0).sum()} legit ({y_te.mean()*100:.1f}% fraud)")
print(f"
ROC-AUC:  {roc_auc:.4f}  ← looks great")
print(f"PR-AUC:   {pr_auc:.4f}  ← reveals poor precision at high recall")
print(f"PR base:  {base_pr:.4f}  ← random model PR-AUC (= fraud rate)")
print(f"PR skill: {(pr_auc - base_pr)/(1 - base_pr):.4f}  ← normalised PR improvement")

# ── Precision at specific recall levels ───────────────────────────────
prec, rec, thresh = precision_recall_curve(y_te, y_prob)

print(f"
Precision at specific recall levels (what matters for ops):")
for target_recall in [0.90, 0.80, 0.70, 0.60, 0.50]:
    idx     = np.argmin(np.abs(rec - target_recall))
    print(f"  At recall={target_recall:.0%}: precision={prec[idx]:.3f}  "
          f"threshold={thresh[idx] if idx < len(thresh) else 'n/a':.3f}  "
          f"({prec[idx]*100:.1f}% of flagged are genuine fraud)")

# ── Decision guide ─────────────────────────────────────────────────────
print(f"
When to use which:")
print(f"  ROC-AUC: balanced classes, ranking quality, model comparison")
print(f"  PR-AUC:  imbalanced classes (<10% positive), precision matters")
print(f"  Both:    always report both — they capture different aspects")

import numpy as np
from sklearn.metrics import roc_curve, roc_auc_score
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import warnings
warnings.filterwarnings('ignore')

np.random.seed(42)
n = 10_000

# Stripe fraud dataset
amount        = np.abs(np.random.normal(1200, 2000, n)).clip(10, 50_000)
merchant_risk = np.random.uniform(0, 1, n)
n_tx_hour     = np.random.randint(0, 20, n).astype(float)
device_age    = np.abs(np.random.normal(200, 150, n)).clip(0, 1000)
is_new_device = np.random.randint(0, 2, n).astype(float)
fraud_score   = (
    (amount/50_000)*0.30 + merchant_risk*0.25
    + (n_tx_hour/20)*0.25 + is_new_device*0.15
    + np.random.randn(n)*0.05
)
y = (fraud_score > 0.55).astype(int)
X = np.column_stack([amount, merchant_risk, n_tx_hour, device_age, is_new_device])

X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)
sc      = StandardScaler()
X_tr_sc = sc.fit_transform(X_tr)
X_te_sc = sc.transform(X_te)

# Use validation split for threshold selection
X_tv, X_val, y_tv, y_val = train_test_split(
    X_tr_sc, y_tr, test_size=0.2, stratify=y_tr, random_state=42
)
model = GradientBoostingClassifier(
    n_estimators=200, learning_rate=0.1, max_depth=3,
    subsample=0.8, random_state=42,
)
model.fit(X_tv, y_tv)
val_prob = model.predict_proba(X_val)[:, 1]

fpr, tpr, thresholds = roc_curve(y_val, val_prob)

# ── Method 1: Youden Index ─────────────────────────────────────────────
youden     = tpr - fpr
best_idx_j = np.argmax(youden)
t_youden   = thresholds[best_idx_j]
print(f"Method 1 — Youden Index:")
print(f"  Best threshold: {t_youden:.3f}")
print(f"  TPR={tpr[best_idx_j]:.3f}  FPR={fpr[best_idx_j]:.3f}  J={youden[best_idx_j]:.3f}")

# ── Method 2: Cost minimisation ────────────────────────────────────────
cost_fn = 2500   # $ cost of missing one fraud
cost_fp = 50     # $ cost of one false alarm

n_pos = y_val.sum()
n_neg = (1 - y_val).sum()

costs = []
for t, tp_rate, fp_rate in zip(thresholds, tpr, fpr):
    fn  = n_pos * (1 - tp_rate)   # false negatives
    fp  = n_neg * fp_rate          # false positives
    costs.append(fn * cost_fn + fp * cost_fp)

best_idx_cost = np.argmin(costs)
t_cost        = thresholds[best_idx_cost]
print(f"
Method 2 — Cost minimisation (FN=${cost_fn}, FP=${cost_fp}):")
print(f"  Best threshold: {t_cost:.3f}")
print(f"  TPR={tpr[best_idx_cost]:.3f}  FPR={fpr[best_idx_cost]:.3f}  Cost=${min(costs):,.0f}")

# ── Method 3: Fixed recall constraint ─────────────────────────────────
target_recall  = 0.90
recall_thresh_idx = np.argmin(np.abs(tpr - target_recall))
t_recall       = thresholds[recall_thresh_idx]
print(f"
Method 3 — Fixed recall constraint (TPR ≥ {target_recall:.0%}):")
print(f"  Best threshold: {t_recall:.3f}")
print(f"  TPR={tpr[recall_thresh_idx]:.3f}  FPR={fpr[recall_thresh_idx]:.3f}")

# ── Final evaluation on test set ──────────────────────────────────────
test_prob = model.predict_proba(X_te_sc)[:, 1]
print(f"
Test set AUC: {roc_auc_score(y_te, test_prob):.4f}")
print(f"
Comparison at each threshold on test set:")
print(f"{'Method':<28} {'Threshold':>10} {'TPR':>7} {'FPR':>7} {'Cost $':>10}")
print("─" * 67)

for name, t in [('Youden Index',    t_youden),
                 ('Cost-minimising', t_cost),
                 ('90% recall fix',  t_recall)]:
    pred  = (test_prob >= t).astype(int)
    tp    = ((pred == 1) & (y_te == 1)).sum()
    fp    = ((pred == 1) & (y_te == 0)).sum()
    fn    = ((pred == 0) & (y_te == 1)).sum()
    tpr_t = tp / y_te.sum()
    fpr_t = fp / (y_te == 0).sum()
    cost  = fn * cost_fn + fp * cost_fp
    print(f"  {name:<26}  {t:>10.3f}  {tpr_t:>7.3f}  {fpr_t:>7.3f}  {cost:>10,.0f}")

import numpy as np
from sklearn.metrics import roc_auc_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, label_binarize
import warnings
warnings.filterwarnings('ignore')

np.random.seed(42)
n = 4000

# DoorDash support ticket categories: 4 classes
X = np.random.randn(n, 10)
y = np.random.choice([0, 1, 2, 3], n, p=[0.40, 0.25, 0.20, 0.15])
classes = ['delivery', 'food_quality', 'payment', 'general']

# Add signal
for cls in range(4):
    X[y == cls, cls] += 2.5

X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)
sc = StandardScaler()
model = RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1)
model.fit(sc.fit_transform(X_tr), y_tr)
y_prob = model.predict_proba(sc.transform(X_te))

# ── Multi-class AUC — two strategies ─────────────────────────────────
# OvR: one AUC per class vs all others (macro average)
auc_ovr_macro    = roc_auc_score(y_te, y_prob, multi_class='ovr',  average='macro')
auc_ovr_weighted = roc_auc_score(y_te, y_prob, multi_class='ovr',  average='weighted')

# OvO: one AUC per class pair (all n×(n-1)/2 pairs)
auc_ovo_macro    = roc_auc_score(y_te, y_prob, multi_class='ovo',  average='macro')
auc_ovo_weighted = roc_auc_score(y_te, y_prob, multi_class='ovo',  average='weighted')

print("Multi-class AUC:")
print(f"  OvR macro:    {auc_ovr_macro:.4f}  (unweighted mean per class)")
print(f"  OvR weighted: {auc_ovr_weighted:.4f}  (weighted by class frequency)")
print(f"  OvO macro:    {auc_ovo_macro:.4f}  (unweighted mean per class pair)")
print(f"  OvO weighted: {auc_ovo_weighted:.4f}  (weighted by pair frequency)")

# ── Per-class AUC ─────────────────────────────────────────────────────
y_bin = label_binarize(y_te, classes=list(range(4)))
print(f"
Per-class AUC (OvR):")
for i, cls in enumerate(classes):
    auc_i = roc_auc_score(y_bin[:, i], y_prob[:, i])
    bar   = '█' * int(auc_i * 30)
    print(f"  {cls:<14}: {bar:<30} {auc_i:.4f}")

# ── Which to use? ─────────────────────────────────────────────────────
print(f"
OvR vs OvO guidance:")
print(f"  OvR:  each class vs all others — faster, standard default")
print(f"  OvO:  each pair of classes — less influenced by class imbalance")
print(f"  macro:    treat all classes equally — use when minority class matters")
print(f"  weighted: weight by support — use for overall performance summary")

Instead of starting from a cost ratio, some teams start from an operational limit: a fraud review team can manually check only a small, fixed share of daily transaction volume, no matter how good the model looks on paper. That caps the acceptable false positive rate directly — the threshold is not chosen to minimise cost or hit a target recall, it is chosen to be the least restrictive threshold that still keeps the false positive rate within the review team's actual staffing capacity. Read straight off the ROC curve: among every threshold whose false positive rate fits inside that capacity budget, pick the one with the highest true positive rate.

import numpy as np
from sklearn.metrics import roc_curve, roc_auc_score
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import warnings
warnings.filterwarnings('ignore')

np.random.seed(42)
n = 10_000
amount        = np.abs(np.random.normal(1200, 2000, n)).clip(10, 50_000)
merchant_risk = np.random.uniform(0, 1, n)
n_tx_hour     = np.random.randint(0, 20, n).astype(float)
fraud_score   = (amount/50_000)*0.30 + merchant_risk*0.35 + (n_tx_hour/20)*0.25 + np.random.randn(n)*0.05
y             = (fraud_score > 0.55).astype(int)
X             = np.column_stack([amount, merchant_risk, n_tx_hour])

X_tr, X_val, y_tr, y_val = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
sc = StandardScaler()
model = GradientBoostingClassifier(n_estimators=200, learning_rate=0.1, max_depth=3, random_state=42)
model.fit(sc.fit_transform(X_tr), y_tr)
val_prob = model.predict_proba(sc.transform(X_val))[:, 1]

# ── Real evaluation reports rarely stop at a single AUC number ────────
# Capacity constraint: the review team can manually check at most 2%
# of daily transaction volume, no matter how good the model looks.
fpr, tpr, thresholds = roc_curve(y_val, val_prob)

max_fpr_capacity = 0.02
eligible   = np.where(fpr <= max_fpr_capacity)[0]
best_idx   = eligible[np.argmax(tpr[eligible])]   # highest recall within capacity
t_capacity = thresholds[best_idx]

print(f"Capacity-constrained threshold: {t_capacity:.3f}")
print(f"  Achieved recall at this threshold: {tpr[best_idx]:.3f}")
print(f"  Achieved FPR (within capacity):    {fpr[best_idx]:.3f}")

# ── Bootstrap confidence interval on AUC — a single point estimate
#    is not enough on its own to greenlight a launch ──────────────────
def bootstrap_auc(y_true, y_score, n_boot=500):
    n = len(y_true)
    aucs = []
    for _ in range(n_boot):
        idx = np.random.randint(0, n, n)
        if len(np.unique(y_true[idx])) < 2:
            continue
        aucs.append(roc_auc_score(y_true[idx], y_score[idx]))
    return np.percentile(aucs, [2.5, 50, 97.5])

lo, mid, hi = bootstrap_auc(y_val, val_prob)
print(f"\nAUC 95% CI: {lo:.3f} to {hi:.3f}  (median {mid:.3f})")
print("A report showing only the point estimate hides how much that")
print("number could move on a different sample of the same population.")

AUC deliberately ignores thresholds so it can summarize a model's underlying ranking ability in one number, but that is a property of the evaluation, not of deployment. In production the model still has to output a single yes-or-no decision for every prediction, which means someone still has to pick one specific threshold before the model can be used. AUC tells you whether the model is capable of separating the classes well across the board; it never tells you which threshold to actually deploy — that is always a separate decision, driven by the real costs of false positives and false negatives.

AUC is an average taken across every possible threshold, including many that nobody would ever use in production. A model can post an excellent overall AUC while performing quite badly in the specific region of the curve where you actually need to operate — for example, at the high-recall end where precision matters most for a fraud team's review queue. A single aggregate number can hide a weak stretch in exactly the operating range that matters, which is why you should always inspect the curve, or precision at your target recall, rather than trusting the summary statistic alone.

An AUC of 0.5 has a precise meaning that has nothing to do with being half correct — it means the model's scores contain no usable signal for ranking positives above negatives, equivalent to guessing at random. This is a common mix-up with accuracy, where 50 percent genuinely does mean getting half of the predictions right. An AUC of 0.5 is not "somewhat helpful"; it says the model could be replaced with a coin flip and perform identically, which is a much stronger and more useful statement than "half correct."

The dangerous case is not a suspiciously high AUC from leakage — that at least tends to get double-checked. The quieter trap is a perfectly plausible, honestly-earned high AUC on a severely imbalanced dataset, where a huge pool of true negatives makes the false positive rate look tiny even while the model is generating large numbers of false positives in absolute terms. A model can post a legitimate, high ROC-AUC while its precision-recall curve tells a much worse story, with precision collapsing at any recall level worth operating at. This is exactly why PR-AUC exists as a second check specifically for imbalanced problems.

AUC depends on how separable the two classes are in a given population, which is itself a property of the dataset, not just the model — a task with a rare, very distinctive positive class can be easy to achieve a high AUC on, while a task with subtle or overlapping classes may cap out well below that even for a strong model. Comparing AUC across different datasets, different populations, or even the same problem measured in different time periods treats two different exams as if they were the same exam. AUC is only a fair comparison between models evaluated on the exact same test set and population.

An AUC of 0.5 means the model's scores carry no information that separates the two classes — it is mathematically equivalent to ranking transactions at random, the same performance you would get from a coin flip. An AUC of 0 is actually more informative than it sounds: it means the model consistently ranks every negative above every positive, a perfectly inverted ranking. In practice, an AUC near 0 is a strong signal of a bug, most often flipped labels or an inverted score, not a hopeless model — flipping the prediction (or the label encoding) turns an AUC of 0 into an AUC of 1.

I would skip the formula entirely and describe the experiment it corresponds to: pick one random example the model should have flagged and one random example it should not have, and ask the model to score both. AUC is simply the probability the model gives the one that should have been flagged a higher score. An AUC of 0.9 means that if you ran that experiment many times, the model would get the ranking right 90 percent of the time. That framing makes it clear why AUC needs no threshold to be meaningful — it is purely about whether the model's relative ordering of the two groups is correct.

Not necessarily, and this is a common trap. AUC reflects how separable the classes are in that specific population as much as how good the model is, so the same AUC number can represent a much easier or much harder task depending on the class balance and how distinctive the positive class is. Before calling them equally good I would look at the precision-recall curve for both, especially Model B's, since PR-AUC is far more sensitive to what is actually happening on the rare positive class — it is entirely possible for Model B to have excellent ROC-AUC and mediocre precision at any usable recall level.

Whenever the positive class is a small minority of the data, because ROC-AUC's false positive rate is calculated against a huge pool of true negatives, which makes it look forgiving even when the model produces a large absolute number of false positives. A concrete case: a fraud dataset with a very low fraud rate can show an excellent ROC-AUC around 0.97 while its PR-AUC is only around 0.41, revealing that precision collapses badly at any recall level a review team could actually operate at. PR-AUC ignores true negatives entirely, so it reflects that reality directly instead of averaging it away.

Because AUC evaluates the model as a ranking system across every possible cutoff, but a production system has to make one concrete decision — flag this transaction or do not — for every single prediction, and that requires exactly one threshold. I would pick it using whichever method fits the situation: the Youden index when there is no cost information and both error types are equally bad, a cost-minimizing threshold when I know the dollar cost of a false positive versus a false negative, or a fixed-recall constraint when a regulator or the business has set a minimum catch rate. AUC tells you the model is worth deploying at all; choosing the threshold is the separate step that turns it into an actual decision-making system.