Support Vector Machines

Imagine drawing a road between two rows of houses. You could draw the road anywhere between them — but the safest road is the one exactly in the middle, with equal distance to both rows. Any car staying on the road has the maximum buffer before hitting a house.

SVM finds that middle road — the decision boundary equidistant from both classes, giving the maximum safety margin to new data points. The houses closest to the road are the support vectors — they are the only training points that actually determine where the road goes.

import numpy as np
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import classification_report
import warnings
warnings.filterwarnings('ignore')

np.random.seed(42)
n = 2000

# Stripe transaction features
transaction_amount = np.abs(np.random.normal(500, 400, n)).clip(10, 5000)
time_since_last    = np.abs(np.random.normal(24, 20, n)).clip(0.1, 200)
merchant_risk      = np.random.uniform(0, 1, n)
device_age_days    = np.abs(np.random.normal(180, 100, n)).clip(0, 730)
n_transactions_24h = np.random.randint(0, 20, n).astype(float)

# Fraud signal: high amount + high merchant risk + many transactions in 24h
fraud_score = (
    (transaction_amount / 5000) * 0.35
    + merchant_risk * 0.30
    + (n_transactions_24h / 20) * 0.20
    + np.random.randn(n) * 0.15
)
y = (fraud_score > 0.55).astype(int)

X = np.column_stack([transaction_amount, time_since_last, merchant_risk,
                      device_age_days, n_transactions_24h])

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

# ── ALWAYS scale before SVM ───────────────────────────────────────────
# SVM is one of the most scaling-sensitive algorithms
# Unscaled: dominated by transaction_amount (up to 5000)
# Scaled: all features contribute equally
scaler     = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)
X_test_sc  = scaler.transform(X_test)

# ── Effect of C parameter ─────────────────────────────────────────────
print(f"{'C value':<12} {'Train acc':<12} {'Test acc':<12} {'CV acc (5-fold)'}")
print("─" * 52)
for C in [0.001, 0.01, 0.1, 1.0, 10.0, 100.0]:
    model = SVC(C=C, kernel='rbf', random_state=42)
    model.fit(X_train_sc, y_train)
    tr_acc = model.score(X_train_sc, y_train)
    te_acc = model.score(X_test_sc, y_test)
    cv     = cross_val_score(model, X_train_sc, y_train, cv=5).mean()
    gap    = ' ← overfit' if tr_acc - te_acc > 0.05 else ''
    print(f"  {C:<10}  {tr_acc:<12.4f} {te_acc:<12.4f} {cv:.4f}{gap}")

# ── Number of support vectors ─────────────────────────────────────────
for C in [0.01, 1.0, 100.0]:
    m = SVC(C=C, kernel='rbf', random_state=42).fit(X_train_sc, y_train)
    print(f"  C={C}: {m.n_support_} support vectors per class  "
          f"(total {sum(m.n_support_)}/{len(X_train)})")

Imagine two groups of ants on a table — one group in the centre, one group around the edges. You cannot draw a straight line between them. But if you lift the table into the air and fold it into a bowl shape, suddenly the centre ants are at the bottom and the edge ants are up high — and you can cut them apart with a flat knife.

The kernel function is like the bowl shape — it transforms the space so a linear separator works. The kernel trick means you never actually have to fold the table — you just compute as if you did.

import numpy as np
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.datasets import make_circles, make_moons

np.random.seed(42)

# ── Demonstrate kernel effect on non-linearly separable data ──────────

# Dataset 1: circles — inner cluster vs outer ring
X_circ, y_circ = make_circles(n_samples=500, noise=0.1, factor=0.4, random_state=42)
X_tr_c, X_te_c, y_tr_c, y_te_c = train_test_split(X_circ, y_circ, test_size=0.2, random_state=42)

sc = StandardScaler()
X_tr_cs = sc.fit_transform(X_tr_c)
X_te_cs = sc.transform(X_te_c)

print("Circles dataset (inner vs outer ring):")
for kernel in ['linear', 'poly', 'rbf', 'sigmoid']:
    m   = SVC(kernel=kernel, C=1.0, random_state=42).fit(X_tr_cs, y_tr_c)
    acc = accuracy_score(y_te_c, m.predict(X_te_cs))
    bar = '█' * int(acc * 30)
    print(f"  {kernel:<10}: {bar} {acc:.4f}")

# RBF and poly should dominate — linear cannot separate circles

# Dataset 2: moons — two interleaved crescent shapes
X_moon, y_moon = make_moons(n_samples=500, noise=0.15, random_state=42)
X_tr_m, X_te_m, y_tr_m, y_te_m = train_test_split(X_moon, y_moon, test_size=0.2, random_state=42)
X_tr_ms = sc.fit_transform(X_tr_m)
X_te_ms = sc.transform(X_te_m)

print("
Moons dataset (two crescents):")
for kernel in ['linear', 'poly', 'rbf']:
    m   = SVC(kernel=kernel, C=1.0, random_state=42).fit(X_tr_ms, y_tr_m)
    acc = accuracy_score(y_te_m, m.predict(X_te_ms))
    bar = '█' * int(acc * 30)
    print(f"  {kernel:<10}: {bar} {acc:.4f}")

# ── The gamma parameter — RBF kernel ─────────────────────────────────
# gamma = how much influence each training point has
# Small gamma: smooth, wide decision boundary (underfits if too small)
# Large gamma: tight, complex boundary that wraps around training points (overfits)
print("
RBF kernel — effect of gamma:")
for gamma in [0.001, 0.01, 0.1, 1.0, 10.0, 100.0]:
    m    = SVC(kernel='rbf', C=1.0, gamma=gamma).fit(X_tr_cs, y_tr_c)
    tr_a = m.score(X_tr_cs, y_tr_c)
    te_a = m.score(X_te_cs, y_te_c)
    flag = ' ← overfit' if tr_a - te_a > 0.05 else ''
    print(f"  gamma={gamma:<8}: train={tr_a:.4f}  test={te_a:.4f}{flag}")

# 'scale' (default) = 1/(n_features * X.var()) — usually a good starting point
m_scale = SVC(kernel='rbf', gamma='scale').fit(X_tr_cs, y_tr_c)
print(f"
  gamma='scale': test={m_scale.score(X_te_cs, y_te_c):.4f}")

import numpy as np
from sklearn.svm import SVR
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error

np.random.seed(42)
n = 1000

distance = np.abs(np.random.normal(4.0, 2.0, n)).clip(0.5, 15)
traffic  = np.random.randint(1, 11, n).astype(float)
prep     = np.abs(np.random.normal(15, 5, n)).clip(5, 35)
delivery = (8.6 + 7.3*distance + 0.8*prep + 1.5*traffic
            + np.random.normal(0, 4, n)).clip(10, 120)

X = np.column_stack([distance, traffic, prep])
y = delivery

X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)
sc = StandardScaler()
X_tr_sc = sc.fit_transform(X_tr)
X_te_sc = sc.transform(X_te)

print(f"{'Model':<35} {'Test MAE':>10} {'Support Vectors':>16}")
print("─" * 64)

for kernel in ['linear', 'rbf', 'poly']:
    svr = SVR(kernel=kernel, C=1.0, epsilon=0.5)
    svr.fit(X_tr_sc, y_tr)
    mae = mean_absolute_error(y_te, svr.predict(X_te_sc))
    n_sv = svr.n_support_vectors_
    print(f"  SVR(kernel='{kernel}'){'':<15} {mae:>10.4f} {n_sv:>16}")

# epsilon parameter: the tube width
# Points within epsilon of the prediction contribute zero loss
# Larger epsilon = wider tube = fewer support vectors = smoother model
print("
Epsilon effect (SVR with rbf kernel):")
for eps in [0.01, 0.1, 0.5, 1.0, 2.0, 5.0]:
    svr = SVR(kernel='rbf', C=1.0, epsilon=eps)
    svr.fit(X_tr_sc, y_tr)
    mae = mean_absolute_error(y_te, svr.predict(X_te_sc))
    print(f"  epsilon={eps:<6}: MAE={mae:.4f}  support_vectors={svr.n_support_vectors_}")

import numpy as np
import pandas as pd
from sklearn.svm import SVC
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import GridSearchCV, StratifiedKFold
from sklearn.metrics import classification_report, roc_auc_score
import warnings
warnings.filterwarnings('ignore')

np.random.seed(42)
n = 3000

# Stripe fraud detection — production pipeline
transaction_amount = np.abs(np.random.normal(500, 400, n)).clip(10, 5000)
time_since_last    = np.abs(np.random.normal(24, 20, n)).clip(0.1, 200)
merchant_risk      = np.random.uniform(0, 1, n)
device_age_days    = np.abs(np.random.normal(180, 100, n)).clip(0, 730)
n_tx_24h           = np.random.randint(0, 20, n).astype(float)
velocity_score     = (n_tx_24h / 20) * 0.5 + (transaction_amount / 5000) * 0.5

fraud_score = (
    (transaction_amount / 5000) * 0.35 + merchant_risk * 0.30
    + (n_tx_24h / 20) * 0.20 + np.random.randn(n) * 0.15
)
y = (fraud_score > 0.55).astype(int)

X = pd.DataFrame({
    'transaction_amount': transaction_amount,
    'time_since_last':    time_since_last,
    'merchant_risk':      merchant_risk,
    'device_age_days':    device_age_days,
    'n_tx_24h':           n_tx_24h,
    'velocity_score':     velocity_score,
})

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

# ── Production SVM pipeline ───────────────────────────────────────────
# ALWAYS scale — SVM is one of the most scaling-sensitive algorithms
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('model',  SVC(kernel='rbf', probability=True, random_state=42)),
])

# ── Hyperparameter search ─────────────────────────────────────────────
# C and gamma are the two most important parameters for RBF SVM
param_grid = {
    'model__C':     [0.1, 1.0, 10.0, 100.0],
    'model__gamma': ['scale', 'auto', 0.001, 0.01, 0.1],
}

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
grid = GridSearchCV(
    pipeline, param_grid,
    cv=cv, scoring='roc_auc',
    n_jobs=-1, verbose=0,
)
grid.fit(X_train, y_train)

print(f"Best params: {grid.best_params_}")
print(f"Best CV AUC: {grid.best_score_:.4f}")

# ── Final evaluation ──────────────────────────────────────────────────
best_model = grid.best_estimator_
y_pred     = best_model.predict(X_test)
y_proba    = best_model.predict_proba(X_test)[:, 1]

print(f"
Test ROC-AUC: {roc_auc_score(y_test, y_proba):.4f}")
print("
Classification report:")
print(classification_report(y_test, y_pred, target_names=['Legitimate', 'Fraud']))

# ── Calibration — SVM probabilities need calibration ──────────────────
# SVC(probability=True) uses Platt scaling internally
# For better calibration use CalibratedClassifierCV
from sklearn.calibration import CalibratedClassifierCV
base_svm   = Pipeline([
    ('scaler', StandardScaler()),
    ('model',  SVC(kernel='rbf', C=10.0, gamma='scale', random_state=42)),
])
calibrated = CalibratedClassifierCV(base_svm, cv=5, method='isotonic')
calibrated.fit(X_train, y_train)
y_calib    = calibrated.predict_proba(X_test)[:, 1]
print(f"
Calibrated SVM ROC-AUC: {roc_auc_score(y_test, y_calib):.4f}")

The classic SVM training problem — a quadratic program with linear constraints — really is convex, and for a fixed kernel and C, solvers genuinely guarantee the global optimum. But that "global optimum" is only global with respect to the choices you already handed the solver. Switch the kernel from linear to RBF, or change gamma, and you get an entirely different convex problem with a different, possibly worse, global optimum. Model quality is not automatically solved by convexity; you still have to search over kernel and hyperparameter choices, and a perfectly optimal fit for a badly chosen kernel can lose to a mediocre fit for a well-chosen one.

Nothing is ever projected into that space in memory. The kernel trick is an algebraic shortcut: it computes what the dot product between two points would be if they had been projected into a higher, even infinite, dimensional space, without ever performing or storing that projection. The RBF kernel formula is just a number computed directly from the original low-dimensional inputs — no infinite-dimensional vector is ever created. This is exactly why SVM with an RBF kernel is not more memory-hungry than one with a linear kernel: the computation happens entirely inside the kernel function, never in some conjured high-dimensional space.

C controls a real tradeoff, not a dial to crank up for better performance. A high C tells the optimiser to penalise margin violations heavily, producing a narrow margin that hugs the training data closely — this can raise training accuracy while hurting generalisation, since the boundary becomes sensitive to individual points, including outliers. A low C allows more violations in exchange for a wider, more robust margin. The best C is whichever value generalises best on held-out data, found through cross-validation — not the largest value that fits the training set most tightly.

Standard kernel SVM training scales roughly quadratically to cubically in the number of training samples, because it works with a kernel matrix comparing every pair of points. At fifty thousand rows that is already a matrix with billions of entries. Past roughly a hundred thousand rows, SVM training routinely becomes impractically slow or memory-heavy, in a way logistic regression or gradient boosting simply do not. Where SVM genuinely wins is on small to medium, high-dimensional datasets — it is not a safe general default the way it is sometimes treated, and reaching for it on a large dataset without checking runtime first is a common practical misstep.

Strictly, only the support vectors — the points closest to, or violating, the margin — determine the final position of the boundary. Every other training point could be removed entirely, the model retrained, and the boundary would land in exactly the same place. This is exactly why SVM predictions only require storing the support vectors, often a small fraction of the training set, rather than the whole dataset — a genuine memory advantage at inference time. It also means the model is somewhat blind to the bulk distribution of "obvious" points that sit far from the boundary, unlike a model such as logistic regression whose loss is shaped by every single point.

Some datasets cannot be separated by any straight line in their original feature space — circles nested inside a ring is the classic example. Projecting the data into a higher-dimensional space can make a linear separator exist there, but explicitly computing that projection for every point would be expensive, or in some cases impossible if the target space is infinite-dimensional. The kernel trick sidesteps this: it uses a kernel function that returns exactly the dot product two points would have had if you had projected them first, computed directly from the original inputs. You get all the separating power of the higher dimension without ever visiting it.

A hard margin SVM requires the two classes to be perfectly separable, with no training point allowed inside the margin or on the wrong side of the boundary — that only works on data that is cleanly separable, which real-world data rarely is. A soft margin SVM introduces slack variables that allow some violations, each penalised in the objective, with the C parameter controlling how harshly those violations are punished. In practice the soft margin formulation is the default: it degrades gracefully on noisy or overlapping classes instead of failing to find any solution at all, which is what a hard margin would do if the data is not perfectly separable.

SVM's entire notion of a margin is a distance concept — it is measured in Euclidean terms, and kernels like RBF are explicit functions of the distance between two points. A feature with a much larger numeric range than the others dominates that distance calculation and effectively decides the boundary on its own, ignoring the rest. A tree-based model like random forest or XGBoost instead splits on one feature at a time using a threshold — whether that threshold is 5 or 5,000 does not change which side of the split a point falls on, so the split logic is invariant to scale in a way SVM's geometry simply is not.

I would push back before committing to it. Kernel SVM training scales roughly quadratically to cubically in the number of rows, so two million rows means a kernel matrix computation that is likely infeasible in reasonable time or memory. I would suggest either LinearSVC, which uses a solver that scales closer to linearly for a linear-kernel approximation, SGDClassifier with a hinge loss for a stochastic approximation of the same boundary, or moving to gradient boosting or a neural network if the data genuinely needs non-linear separation at that scale. Full kernel SVM is realistically a small-to-medium-dataset tool.

I would start with a linear kernel if the number of features is very large relative to the number of samples, as with text data represented by TF-IDF, since a linear boundary in that many dimensions is often already expressive enough, and RBF's extra flexibility mainly risks overfitting while slowing training down. If a linear kernel clearly underperforms and the dataset is small to medium in size, I would try RBF next and tune gamma and C together through cross-validation, watching for a growing gap between training and validation accuracy as a signal that gamma is too high and the boundary is overfitting to individual points.