Evaluation Metrics — Beyond Accuracy

A doctor is screening patients for a rare disease affecting 1 in 100 people. A doctor who says "healthy" to everyone achieves 99% accuracy. But they miss every sick patient. The medical community does not measure doctors by "how often are you right overall?" They measure: "of the people you said were sick, how many actually were?" (precision) and "of all the people who were actually sick, how many did you catch?" (recall).

These two questions — precision and recall — are the core of all classification evaluation. Every other metric (F1, ROC-AUC, PR-AUC) is built on top of them.

import numpy as np
from sklearn.metrics import (confusion_matrix, classification_report,
                              accuracy_score, precision_score,
                              recall_score, f1_score)
import warnings
warnings.filterwarnings('ignore')

np.random.seed(42)
n = 10_000

# Simulate Stripe fraud predictions
# 1.5% fraud rate
y_true  = (np.random.random(n) < 0.015).astype(int)
# Realistic model: catches 70% of fraud, 3% false alarm rate
y_pred  = np.zeros(n, dtype=int)
fraud_idx = np.where(y_true == 1)[0]
legit_idx = np.where(y_true == 0)[0]
# True positives: catch 70% of actual fraud
tp_idx = np.random.choice(fraud_idx, int(len(fraud_idx)*0.70), replace=False)
y_pred[tp_idx] = 1
# False positives: flag 3% of legitimate transactions
fp_idx = np.random.choice(legit_idx, int(len(legit_idx)*0.03), replace=False)
y_pred[fp_idx] = 1

# ── Confusion matrix ──────────────────────────────────────────────────
cm = confusion_matrix(y_true, y_pred)
tn, fp, fn, tp = cm.ravel()

print("Confusion matrix:")
print(f"  {'':15} {'Pred: Fraud':>14} {'Pred: Legit':>14}")
print(f"  {'Actual: Fraud':<15} {'TP = ' + str(tp):>14} {'FN = ' + str(fn):>14}")
print(f"  {'Actual: Legit':<15} {'FP = ' + str(fp):>14} {'TN = ' + str(tn):>14}")

# ── Derive every metric manually ─────────────────────────────────────
accuracy  = (tp + tn) / (tp + tn + fp + fn)
precision = tp / (tp + fp)          # of all flagged, how many were fraud?
recall    = tp / (tp + fn)          # of all fraud, how many did we catch?
f1        = 2 * precision * recall / (precision + recall)
specificity = tn / (tn + fp)        # of all legit, how many were correctly allowed?
fpr       = fp / (fp + tn)          # false positive rate = 1 - specificity

print(f"
Metrics (computed manually):")
print(f"  Accuracy:    {accuracy:.4f}   ← misleading — 98.5% if we predict all legit")
print(f"  Precision:   {precision:.4f}   ← of flagged transactions, {precision*100:.1f}% are fraud")
print(f"  Recall:      {recall:.4f}   ← we catch {recall*100:.1f}% of all fraud")
print(f"  F1 score:    {f1:.4f}   ← harmonic mean of precision and recall")
print(f"  Specificity: {specificity:.4f}   ← {specificity*100:.1f}% of legit transactions pass through")
print(f"  FPR:         {fpr:.4f}   ← {fpr*100:.1f}% of legit flagged as fraud")

# Business impact translation
fraud_value_per_tx = 2500  # avg $2500 per fraudulent transaction
false_alarm_cost   = 50    # $50 cost per false alarm (support call, friction)

fraud_caught    = tp * fraud_value_per_tx
fraud_missed    = fn * fraud_value_per_tx
false_alarm_cost_total = fp * false_alarm_cost

print(f"
Business impact:")
print(f"  Fraud caught:         ${fraud_caught:,.0f} protected")
print(f"  Fraud missed (FN):    ${fraud_missed:,.0f} lost")
print(f"  False alarm cost:     ${false_alarm_cost_total:,.0f} in friction")
print(f"  Net value of model:   ${fraud_caught - fraud_missed - false_alarm_cost_total:,.0f}")

# sklearn classification_report gives everything at once
print("
sklearn classification_report:")
print(classification_report(y_true, y_pred, target_names=['Legit', 'Fraud']))

import numpy as np
from sklearn.svm import SVC
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import (precision_score, recall_score, f1_score,
                              precision_recall_curve, average_precision_score)
import warnings
warnings.filterwarnings('ignore')

np.random.seed(42)
n = 8_000

# Stripe fraud features
amount         = np.abs(np.random.normal(1200, 2000, n)).clip(10, 50_000)
merchant_risk  = np.random.uniform(0, 1, n)
n_tx_hour      = np.random.randint(0, 20, n).astype(float)
device_age     = np.abs(np.random.normal(200, 150, n)).clip(0, 1000)
is_new_device  = np.random.randint(0, 2, n).astype(float)

fraud_score = (
    (amount/50_000)*0.30 + merchant_risk*0.25
    + (n_tx_hour/20)*0.25 + is_new_device*0.15
    + np.random.randn(n)*0.05
)
y = (fraud_score > 0.55).astype(int)

X = np.column_stack([amount, merchant_risk, n_tx_hour, device_age, is_new_device])
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2,
                                            stratify=y, random_state=42)
sc = StandardScaler()
X_tr_sc = sc.fit_transform(X_tr)
X_te_sc = sc.transform(X_te)

model = GradientBoostingClassifier(
    n_estimators=200, learning_rate=0.1, max_depth=3,
    subsample=0.8, random_state=42,
)
model.fit(X_tr_sc, y_tr)
y_proba = model.predict_proba(X_te_sc)[:, 1]

# ── Precision-recall at different thresholds ──────────────────────────
print("Precision-recall trade-off at different thresholds:")
print(f"{'Threshold':<12} {'Precision':>11} {'Recall':>9} {'F1':>9} {'Flagged':>9}")
print("─" * 54)

for threshold in np.arange(0.1, 0.91, 0.1):
    y_pred_t  = (y_proba >= threshold).astype(int)
    prec      = precision_score(y_te, y_pred_t, zero_division=0)
    rec       = recall_score(y_te, y_pred_t, zero_division=0)
    f1        = f1_score(y_te, y_pred_t, zero_division=0)
    flagged   = y_pred_t.mean() * 100
    print(f"  t={threshold:.1f}      {prec:>9.4f}  {rec:>9.4f}  {f1:>9.4f}  {flagged:>7.1f}%")

# ── F-beta score — weight recall over precision ────────────────────────
from sklearn.metrics import fbeta_score

# At Stripe: missing fraud costs 50× more than false alarm
# β=2 weights recall twice as heavily as precision
# β=0.5 weights precision twice as heavily as recall
print("
F-beta score — adjusting the precision/recall balance:")
for beta in [0.5, 1.0, 2.0, 3.0]:
    y_pred_default = (y_proba >= 0.5).astype(int)
    fb = fbeta_score(y_te, y_pred_default, beta=beta, zero_division=0)
    desc = '(precision weighted)' if beta < 1 else '(equal weight)' if beta == 1 else '(recall weighted)'
    print(f"  F{beta}: {fb:.4f}  {desc}")

# ── PR-AUC — area under the precision-recall curve ────────────────────
# Better than ROC-AUC for highly imbalanced problems
pr_auc = average_precision_score(y_te, y_proba)
print(f"
PR-AUC (Average Precision): {pr_auc:.4f}")
print("PR-AUC = 1.0: perfect model")
print(f"PR-AUC = {y_te.mean():.4f}: random baseline (= fraud rate)")

You have 100 fraud cases and 9,900 legit transactions — all shuffled randomly. You ask the model to score all 10,000 and sort them by fraud probability, highest first. How many of the actual 100 fraud cases appear in the top 100? Top 200? Top 500? If the model is perfect, all 100 fraud cases appear before any legitimate transaction. The ROC curve plots this across every possible cutpoint. AUC is the probability that a randomly chosen fraud transaction scores higher than a randomly chosen legit one.

AUC = 0.95 means: take one random fraud transaction and one random legit transaction. There is a 95% chance the model assigns a higher fraud score to the fraud transaction. This is the most intuitive interpretation of AUC.

import numpy as np
from sklearn.metrics import roc_auc_score, roc_curve, average_precision_score
from sklearn.ensemble import GradientBoostingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
import warnings
warnings.filterwarnings('ignore')

np.random.seed(42)
n = 8_000

amount        = np.abs(np.random.normal(1200, 2000, n)).clip(10, 50_000)
merchant_risk = np.random.uniform(0, 1, n)
n_tx_hour     = np.random.randint(0, 20, n).astype(float)
device_age    = np.abs(np.random.normal(200, 150, n)).clip(0, 1000)
is_new_device = np.random.randint(0, 2, n).astype(float)

fraud_score = (
    (amount/50_000)*0.30 + merchant_risk*0.25
    + (n_tx_hour/20)*0.25 + is_new_device*0.15
    + np.random.randn(n)*0.05
)
y = (fraud_score > 0.55).astype(int)
X = np.column_stack([amount, merchant_risk, n_tx_hour, device_age, is_new_device])

X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2,
                                            stratify=y, random_state=42)
sc      = StandardScaler()
X_tr_sc = sc.fit_transform(X_tr)
X_te_sc = sc.transform(X_te)

models = {
    'LogisticRegression': LogisticRegression(max_iter=1000, random_state=42),
    'RandomForest':       RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1),
    'GradientBoosting':   GradientBoostingClassifier(n_estimators=200, learning_rate=0.1,
                                                       max_depth=3, random_state=42),
}

print(f"{'Model':<22} {'ROC-AUC':>9} {'PR-AUC':>9} {'CV AUC (5-fold)':>16}")
print("─" * 60)

for name, model in models.items():
    model.fit(X_tr_sc, y_tr)
    y_prob  = model.predict_proba(X_te_sc)[:, 1]
    roc_auc = roc_auc_score(y_te, y_prob)
    pr_auc  = average_precision_score(y_te, y_prob)
    cv_auc  = cross_val_score(model, X_tr_sc, y_tr, cv=5,
                               scoring='roc_auc').mean()
    print(f"  {name:<20}  {roc_auc:>9.4f}  {pr_auc:>9.4f}  {cv_auc:>14.4f}")

# ── ROC curve — operating points ──────────────────────────────────────
best_model = models['GradientBoosting']
y_prob     = best_model.predict_proba(X_te_sc)[:, 1]
fpr, tpr, thresholds = roc_curve(y_te, y_prob)

print("
ROC curve operating points (selected thresholds):")
print(f"{'Threshold':<12} {'TPR (Recall)':>13} {'FPR':>8} {'Specificity':>13}")
print("─" * 50)

# Find specific operating points
for target_recall in [0.95, 0.90, 0.80, 0.70, 0.60]:
    idx  = np.argmin(np.abs(tpr - target_recall))
    spec = 1 - fpr[idx]
    print(f"  t={thresholds[idx]:.3f}     TPR={tpr[idx]:.3f}       FPR={fpr[idx]:.3f}    Spec={spec:.3f}")

# ── When to use ROC-AUC vs PR-AUC ─────────────────────────────────────
print("
ROC-AUC vs PR-AUC:")
print("  ROC-AUC: affected equally by both classes — good for balanced datasets")
print("  PR-AUC:  focuses on the positive (minority) class — use for fraud, churn")
print(f"  Fraud rate in this dataset: {y_te.mean()*100:.1f}%")
print(f"  → At {y_te.mean()*100:.1f}% positive rate, PR-AUC is the more informative metric")

import numpy as np
from sklearn.metrics import (mean_absolute_error, mean_squared_error,
                              mean_absolute_percentage_error, r2_score)
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import train_test_split
import warnings
warnings.filterwarnings('ignore')

np.random.seed(42)
n = 3000
distance = np.abs(np.random.normal(4.0, 2.0, n)).clip(0.5, 15)
traffic  = np.random.randint(1, 11, n).astype(float)
prep     = np.abs(np.random.normal(15, 5, n)).clip(5, 35)
delivery = (8.6 + 7.3*distance + 0.8*prep + 1.5*traffic
            + np.random.normal(0, 4, n)).clip(10, 120)

X = np.column_stack([distance, traffic, prep])
y = delivery
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)

model = GradientBoostingRegressor(n_estimators=200, learning_rate=0.1,
                                   max_depth=3, random_state=42)
model.fit(X_tr, y_tr)
y_pred = model.predict(X_te)

# ── All four metrics ──────────────────────────────────────────────────
mae  = mean_absolute_error(y_te, y_pred)
rmse = np.sqrt(mean_squared_error(y_te, y_pred))
mape = mean_absolute_percentage_error(y_te, y_pred) * 100
r2   = r2_score(y_te, y_pred)

print(f"DoorDash delivery time model evaluation:")
print(f"  MAE:   {mae:.4f} min   ← average error in minutes")
print(f"  RMSE:  {rmse:.4f} min  ← penalises large errors more")
print(f"  MAPE:  {mape:.4f}%    ← percentage error relative to actual")
print(f"  R²:    {r2:.4f}       ← model explains {r2*100:.1f}% of variance")

# ── When RMSE >> MAE: outliers are present ────────────────────────────
print(f"
RMSE / MAE ratio: {rmse/mae:.2f}")
print("  Ratio near 1.0: errors are uniform, no major outliers")
print("  Ratio > 2.0: large outlier errors are inflating RMSE")

# ── Baseline comparison — always compare against naive models ─────────
mean_pred    = np.full_like(y_te, y_tr.mean())
median_pred  = np.full_like(y_te, np.median(y_tr))

print(f"
Baseline comparisons:")
print(f"  Always-predict-mean  MAE={mean_absolute_error(y_te, mean_pred):.4f}  R²={r2_score(y_te, mean_pred):.4f}")
print(f"  Always-predict-median MAE={mean_absolute_error(y_te, median_pred):.4f}  R²={r2_score(y_te, median_pred):.4f}")
print(f"  Our GBM model        MAE={mae:.4f}  R²={r2:.4f}")
print(f"  Improvement over mean: {(mean_absolute_error(y_te, mean_pred) - mae) / mean_absolute_error(y_te, mean_pred) * 100:.1f}%")

import numpy as np
from sklearn.metrics import (precision_recall_curve, roc_curve,
                              f1_score, fbeta_score,
                              precision_score, recall_score)
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import warnings
warnings.filterwarnings('ignore')

np.random.seed(42)
n = 8_000
amount = np.abs(np.random.normal(1200,2000,n)).clip(10,50_000)
merchant_risk = np.random.uniform(0,1,n)
n_tx_hour = np.random.randint(0,20,n).astype(float)
device_age = np.abs(np.random.normal(200,150,n)).clip(0,1000)
is_new_device = np.random.randint(0,2,n).astype(float)
fraud_score = (amount/50_000)*0.30 + merchant_risk*0.25 + (n_tx_hour/20)*0.25 + is_new_device*0.15 + np.random.randn(n)*0.05
y = (fraud_score > 0.55).astype(int)
X = np.column_stack([amount,merchant_risk,n_tx_hour,device_age,is_new_device])
X_tr,X_te,y_tr,y_te = train_test_split(X,y,test_size=0.2,stratify=y,random_state=42)
X_tv,X_val,y_tv,y_val = train_test_split(X_tr,y_tr,test_size=0.2,stratify=y_tr,random_state=42)
sc = StandardScaler()
X_tv_sc  = sc.fit_transform(X_tv)
X_val_sc = sc.transform(X_val)
X_te_sc  = sc.transform(X_te)

model = GradientBoostingClassifier(n_estimators=200, learning_rate=0.1,
                                    max_depth=3, random_state=42)
model.fit(X_tv_sc, y_tv)
val_proba = model.predict_proba(X_val_sc)[:, 1]

# ── Method 1: Maximise F1 ─────────────────────────────────────────────
thresholds   = np.arange(0.05, 0.96, 0.01)
f1_scores    = [f1_score(y_val, (val_proba >= t).astype(int), zero_division=0)
                for t in thresholds]
best_t_f1    = thresholds[np.argmax(f1_scores)]
print(f"Best threshold by F1:     {best_t_f1:.2f}  (F1={max(f1_scores):.4f})")

# ── Method 2: Maximise Fβ (recall-weighted for fraud) ─────────────────
fb_scores    = [fbeta_score(y_val, (val_proba >= t).astype(int),
                             beta=2, zero_division=0) for t in thresholds]
best_t_fb    = thresholds[np.argmax(fb_scores)]
print(f"Best threshold by F2:     {best_t_fb:.2f}  (F2={max(fb_scores):.4f})")

# ── Method 3: Business cost optimisation ─────────────────────────────
# Cost of false negative (missed fraud):  $2,500 avg transaction
# Cost of false positive (blocked legit): $50 friction
fn_cost = 2500
fp_cost = 50

costs = []
for t in thresholds:
    pred = (val_proba >= t).astype(int)
    fn   = ((pred == 0) & (y_val == 1)).sum()
    fp   = ((pred == 1) & (y_val == 0)).sum()
    costs.append(fn * fn_cost + fp * fp_cost)

best_t_biz = thresholds[np.argmin(costs)]
print(f"Best threshold by cost:   {best_t_biz:.2f}  (cost=${min(costs):,.0f})")

# ── Compare all thresholds on test set ────────────────────────────────
test_proba = model.predict_proba(X_te_sc)[:, 1]
print(f"
Test set performance at different thresholds:")
print(f"{'Method':<22} {'Threshold':>10} {'Precision':>11} {'Recall':>9} {'F1':>9}")
print("─" * 65)

for label, t in [('Default (0.5)', 0.5),
                  ('Max F1', best_t_f1),
                  ('Max F2 (recall++)', best_t_fb),
                  ('Min business cost', best_t_biz)]:
    pred = (test_proba >= t).astype(int)
    p  = precision_score(y_te, pred, zero_division=0)
    r  = recall_score(y_te, pred, zero_division=0)
    f1 = f1_score(y_te, pred, zero_division=0)
    print(f"  {label:<20}  {t:>10.2f}  {p:>11.4f}  {r:>9.4f}  {f1:>9.4f}")

import numpy as np
from sklearn.metrics import (classification_report, confusion_matrix,
                              f1_score, precision_score, recall_score)
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import warnings
warnings.filterwarnings('ignore')

np.random.seed(42)
n = 5000

# DoorDash support ticket categories: 4 classes
# delivery_issue, food_quality, payment_issue, general
X = np.random.randn(n, 10)
y = np.random.choice([0,1,2,3], n, p=[0.40, 0.25, 0.20, 0.15])
# Class 0=delivery (40%), 1=food (25%), 2=payment (20%), 3=general (15%)

X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2,
                                            stratify=y, random_state=42)
sc = StandardScaler()
model = RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1)
model.fit(sc.fit_transform(X_tr), y_tr)
y_pred = model.predict(sc.transform(X_te))

class_names = ['delivery', 'food_quality', 'payment', 'general']

# ── Classification report ──────────────────────────────────────────────
print("Full classification report:")
print(classification_report(y_te, y_pred, target_names=class_names))

# ── Three averaging strategies explained ──────────────────────────────
print("Averaging strategies for multi-class F1:")
for avg in ['macro', 'weighted', 'micro']:
    f1 = f1_score(y_te, y_pred, average=avg)
    if avg == 'macro':
        desc = 'unweighted mean of per-class F1 — treats all classes equally'
    elif avg == 'weighted':
        desc = 'weighted by support (class size) — accounts for imbalance'
    else:
        desc = 'global TP/FP/FN — equivalent to accuracy for multi-class'
    print(f"  {avg:<10}: {f1:.4f}  ← {desc}")

print("
When to use each:")
print("  macro:    when all classes matter equally (even rare ones)")
print("  weighted: when class frequency should influence the metric")
print("  micro:    rarely used for multi-class; equivalent to accuracy")

# ── Per-class metrics ──────────────────────────────────────────────────
print("
Per-class F1 scores:")
per_class_f1 = f1_score(y_te, y_pred, average=None)
class_counts = np.bincount(y_te)
for name, f1, count in zip(class_names, per_class_f1, class_counts):
    bar = '█' * int(f1 * 25)
    print(f"  {name:<14}: {bar:<25} {f1:.4f}  (n={count})")

The offline metric used to pick the winning model during development — PR-AUC, F2, whatever balances precision and recall for that cost ratio — is usually not what gets watched in production. Production dashboards translate the same confusion matrix into numbers a non-technical stakeholder can act on: dollars protected per week for a fraud model, confirmed cases per thousand screened for a medical model, tickets deflected per day for a support-routing model. A model whose PR-AUC quietly drops over a month is invisible to most executives; the same drop showing up as "twelve thousand fewer dollars protected this week" gets a message within the hour. Translating the abstract metric into a business number is part of the job, not an afterthought once the model ships.

import numpy as np

# ── Production monitoring: same formula, opposite cost ratios ─────────
# Every week, recompute the confusion matrix on newly labelled outcomes
# and translate it into whatever number the business actually watches.

def weekly_fraud_report(tp, fp, fn, tn, fraud_value=2500, alarm_cost=50):
    dollars_protected = tp * fraud_value
    dollars_lost       = fn * fraud_value
    friction_cost       = fp * alarm_cost
    net_value           = dollars_protected - dollars_lost - friction_cost
    precision           = tp / (tp + fp)
    recall              = tp / (tp + fn)
    return {
        'net_value': net_value,
        'precision_at_threshold': precision,
        'recall_at_threshold': recall,
    }

def weekly_screening_report(tp, fp, fn, tn, recall_floor=0.95):
    recall    = tp / (tp + fn)
    precision = tp / (tp + fp)
    breach    = recall < recall_floor
    return {
        'recall': recall,
        'precision': precision,
        'below_mandated_floor': breach,   # this triggers an incident, not a shrug
    }

# Same shapes, same formulas — the only difference is which number
# is allowed to move and which number is a hard floor.
fraud_week     = weekly_fraud_report(tp=612, fp=1840, fn=203, tn=97_345)
screening_week = weekly_screening_report(tp=142, fp=880, fn=6, tn=8_972)

print("Fraud team weekly report:")
for k, v in fraud_week.items():
    print(f"  {k:<24}: {v:,.2f}" if isinstance(v, float) else f"  {k:<24}: {v}")

print("\nScreening team weekly report:")
for k, v in screening_week.items():
    print(f"  {k:<24}: {v:.4f}" if isinstance(v, float) else f"  {k:<24}: {v}")

Accuracy itself is not flawed — it is a perfectly good summary when classes are roughly balanced and errors cost about the same in both directions. What breaks it is imbalance: predicting the majority class every time can score 98 percent or higher while catching none of the minority class. The real rule is not "never use accuracy" but "check the class balance and the cost of each error type before trusting any single number" — for a roughly balanced classification problem with symmetric costs, accuracy is often the simplest metric that tells you exactly what you need.

For a fixed, already-trained model, precision and recall move in opposite directions as you slide the classification threshold — there is no threshold that maximizes both simultaneously, because raising it to cut false alarms always costs you some true positives, and lowering it to catch more positives always lets in more false alarms. What can genuinely improve is the model itself: a better model shifts the entire precision-recall curve upward, so at the same recall you get higher precision than before. So "improve the model" and "pick the threshold" are two different jobs — the first can raise both, the second always trades one for the other.

F1 looks neutral because it is a symmetric formula, but that symmetry itself is a hidden assumption — it is the harmonic mean of precision and recall weighted equally, which silently assumes a false positive and a false negative cost the business the same amount. That is rarely true: at Stripe a missed fraud transaction costs thousands of dollars while a false alarm costs a support ticket, so equal weighting understates how much recall actually matters there. The honest tool for asymmetric costs is the F-beta score, where beta lets you state explicitly how many times more costly one error type is than the other.

Every metric encodes a different set of priorities baked into its formula, so "best" only makes sense relative to a specific cost structure and a specific deployment context — the metric that is right for a fraud model (recall-heavy, because missed fraud is expensive) is often wrong for a spam filter (precision-heavy, because wrongly blocking a real email is worse than letting one spam message through). Reaching for the same default metric — usually accuracy or F1 — on every project skips the step that actually matters: writing down what each type of error costs before choosing how to measure success.

The test set only tells you how the model performs on data drawn from the same distribution it was trained and evaluated on — it says nothing about tomorrow's data if that distribution shifts. Fraud patterns evolve as fraudsters adapt, customer behavior changes seasonally, and the mix of transaction types drifts over time; a 0.94 ROC-AUC measured in March can quietly become 0.80 by August without any change to the model itself. Trusting a single offline number without ongoing production monitoring is one of the most common ways a "validated" model quietly stops working.

Not without more information. The first question is the class balance: on a dataset where the positive class is rare, 99 percent accuracy can be achieved by a model that never predicts the positive class at all — as happened with the 98.5 percent "always predict legitimate" fraud model. I would ask for the confusion matrix first, then precision and recall on the positive class specifically, then ROC-AUC or PR-AUC depending on how imbalanced the data is. Only after seeing those would I have an informed opinion about whether the model does anything useful.

Precision asks, of everything the model flagged as positive, how many actually were; recall asks, of everything that was actually positive, how many the model caught. You optimize for precision when a false positive is the expensive error — a spam filter that wrongly blocks a real business email causes real harm, so you would rather let a few spam messages through than risk that. You optimize for recall when a false negative is the expensive error — missing an actual fraud transaction costs far more than the friction of one extra manual review, so a fraud model should lean toward flagging more and accepting a lower precision.

F1 is useful as a single ranking number when comparing many models or many hyperparameter settings quickly, since sorting by one number is easier than eyeballing two. The risk is that F1 hides which of the two errors is actually being made — two models can have the same F1 while one has high precision and low recall and the other has the opposite profile, and F1 alone will not tell you that. In practice I would use F1 to narrow down candidates, then go back to the full precision/recall breakdown, or an explicit F-beta score matching the real cost ratio, before making the final call.

I would start by writing down, in dollars if possible, what a false positive costs and what a false negative costs — that ratio drives everything else. If the costs are roughly symmetric and the classes are balanced, accuracy or F1 is fine. If the classes are imbalanced, I would move to ROC-AUC for overall ranking quality and PR-AUC if the positive class is rare, then pick a threshold using the actual cost ratio rather than the default of 0.5. I would also confirm what happens downstream of each prediction — a false positive that triggers an automatic account block is a much bigger deal than one that triggers a single review, and that changes the metric choice too.

First I would rule out a pipeline bug — confirm the production feature computation matches training exactly, since train-serve skew is one of the most common causes and is easy to overlook. Next I would check for distribution shift by comparing the statistical profile of production inputs against the training data — a change in the mix of transaction types or a new fraud pattern the model never saw would show up here. Finally I would set up ongoing shadow evaluation against labelled outcomes so a drop like this gets caught within days instead of being discovered after damage is done, and treat a meaningful metric drop as an automatic trigger for retraining.