ML System Design — End to End

An architect designing a building does not start by choosing the colour of the walls. They start with: who lives here, how many people, what activities happen inside, what is the budget, what are the structural constraints of the land. The colour comes last. ML system design is the same — the model choice (colour of the walls) comes after you understand the data availability, latency requirements, and scale constraints. Most candidates start with model selection and never get to the questions that actually determine system feasibility.

In ML system design interviews, an interviewer would rather see you ask the right clarifying questions than immediately jump to "I would use a Transformer." The right questions demonstrate systems thinking. The immediate model answer demonstrates pattern matching.

# ── Fraud detection system design — capacity and tradeoff analysis ───

# Question 1: Problem framing
print("PROBLEM FRAMING")
print("=" * 55)
print("""
ML task:     Binary classification — fraud vs legitimate
             Output: fraud probability score (0-1), not binary
             Thresholds set by risk team based on business cost

Business metric:
  Primary:   Transaction fraud rate (TFR) — fraud_amount / total_amount
  Secondary: False positive rate (FPR) — declined_legitimate / all_legitimate

Why score not binary:
  Different thresholds for different transaction types:
  - Low value card (< $500): high threshold (0.95) — FP very costly
  - High value bank transfer (> $10K): low threshold (0.30) — FN very costly
  - International: medium threshold (0.60)
""")

# Question 6: Scale (do this early for fraud — it's the key constraint)
print("SCALE ANALYSIS")
print("=" * 55)

transactions_per_day = 10_000_000        # 10M transactions/day
peak_tps             = transactions_per_day / 86400 * 10   # 10× peak = ~1,157 TPS
fraud_rate           = 0.001             # 0.1% = 10,000 fraud transactions/day

print(f"Transaction volume:  {transactions_per_day:,}/day = {transactions_per_day/86400:.0f} avg TPS")
print(f"Peak TPS:            {peak_tps:.0f} TPS  (10× peak factor)")
print(f"Expected fraud:      {int(transactions_per_day * fraud_rate):,} transactions/day")
print(f"Latency budget:      < 50ms total  (before payment clears)")
print(f"Model budget:        < 10ms  (50ms - feature fetch - network)")

# Capacity for 1,157 TPS at 10ms each:
import math
model_time_ms    = 10
replicas_needed  = math.ceil(peak_tps * model_time_ms / 1000)
print(f"
Replicas needed:     {replicas_needed} pods × (1000ms/10ms) = handles {1000//model_time_ms} RPS each")

# Question 3: Features — three tiers by computation cost
print("
FEATURE TIERS")
print("=" * 55)
feature_tiers = {
    'Tier 1 — Request (0ms)': [
        'transaction_amount', 'merchant_category', 'payment_method',
        'device_fingerprint', 'ip_address', 'billing_zip',
        'hour_of_day', 'is_international',
    ],
    'Tier 2 — Feature Store (1-3ms Redis)': [
        'user_7d_transaction_count', 'user_7d_total_spend',
        'user_30d_distinct_merchants', 'user_30d_avg_amount',
        'merchant_fraud_rate_30d', 'ip_country_mismatch_flag',
        'velocity_1h_amount', 'device_new_flag',
    ],
    'Tier 3 — Computed (5ms, parallel)': [
        'transaction_vs_user_avg_ratio',   # txn / user 30d avg
        'amount_round_number_flag',         # $10000.00 exact
        'time_since_last_transaction_s',
        'distance_from_last_merchant_km',
    ],
}
for tier, features in feature_tiers.items():
    print(f"
  {tier}:")
    for f in features:
        print(f"    → {f}")

# Question 4: Model — why an ensemble
print("
MODEL SELECTION")
print("=" * 55)
print("""
Primary:  LightGBM score (tabular features, fast, interpretable)
          Trained on 6-month rolling window
          Feature importance for regulatory compliance (CFPB audit)

Secondary: Rule engine running in parallel (100% recall for known patterns)
           Hard rules: velocity limits, blacklisted IPs, blocked BINs
           Soft rules: unusual merchant for user, odd hours

Output: max(lgbm_score, rule_score) → final risk score

Why not deep learning:
  - Regulatory: CFPB requires explainability for declined transactions
  - Latency: transformer inference > 50ms for tabular data
  - Data: 0.1% fraud rate with 10M daily → only 10k fraud labels/day
    LightGBM is far more data-efficient than deep learning for this

Class imbalance handling:
  - scale_pos_weight=999 in LightGBM (ratio of negatives to positives)
  - Focal loss as alternative: penalises easy negatives more
  - Evaluation: precision-recall AUC not ROC-AUC (imbalanced classes)
""")

# Question 7: Monitoring — adversarial drift is the hardest problem
print("MONITORING — ADVERSARIAL DRIFT")
print("=" * 55)
print("""
Normal drift: feature distributions shift gradually (monthly retraining)
Adversarial drift: fraudsters adapt to the model within DAYS

Detection:
  - Monitor fraud rate per merchant category hourly
  - Monitor false positive rate daily (declined legit transactions)
  - Monitor feature distribution of FRAUD transactions (not all)
    → New fraud pattern appears as a cluster in a previously clean region

Response:
  - New fraud pattern detected → add hard rule IMMEDIATELY (same day)
  - Retrain model weekly with new fraud samples
  - Deploy rule before model update — rules are instant, models take days

Feedback loop:
  - Chargebacks confirm fraud (7-30 day delay)
  - Merchant disputes confirm false positives (same day)
  - Manual review team labels 2% of flagged transactions same-day
    → Creates fast label feedback for concept drift detection
""")

# ── Recommendation system capacity analysis ───────────────────────────
print("MEESHO RECOMMENDATION SCALE ANALYSIS")
print("=" * 55)

# User and product scale
dau           = 15_000_000   # 15M daily active users
products      = 50_000_000   # 50M products in catalogue
sessions_per_user = 3        # avg 3 sessions per day per DAU
recs_per_session = 4         # recommendation requests per session
total_rps = dau * sessions_per_user * recs_per_session / 86400

print(f"DAU:                    {dau/1e6:.0f}M")
print(f"Product catalogue:      {products/1e6:.0f}M")
print(f"Recommendation RPS:     {total_rps:.0f} avg  (~{total_rps*5:.0f} peak)")

# Why two-stage is necessary
print("
WHY TWO-STAGE IS NECESSARY:")
ranking_model_ms  = 5    # ms per (user, product) pair
full_catalogue_ms = ranking_model_ms * products / 1000
print(f"  Ranking all {products/1e6:.0f}M products: {full_catalogue_ms/1000:.0f} seconds — impossible")
print(f"  Ranking 500 candidates:              {ranking_model_ms * 500:.0f}ms — feasible")

# Offline training cadence
print("
TRAINING CADENCE:")
print("""
  Retrieval models (ALS, CLIP embeddings):  Weekly
    → Catalogue changes slowly, embeddings expensive to compute

  Ranking model:  Daily
    → Click patterns change quickly, model relatively cheap to retrain

  Popularity/trending features:  Hourly
    → Trending products change fast, must be fresh

  Cold start (new product):  Real-time
    → New product uploaded → compute CLIP embedding → add to index immediately
    → Use content-based retrieval until enough interactions for collaborative
""")

# Evaluation metrics — what actually matters
print("EVALUATION METRICS:")
metrics = [
    ('CTR (Click-Through Rate)',   'Online',  '% users who click a recommendation'),
    ('Conversion rate',            'Online',  '% recommendations that lead to purchase'),
    ('Diversity (ILD)',            'Offline', 'Intra-list diversity — are recommendations varied?'),
    ('Coverage',                   'Offline', '% of catalogue that appears in recommendations'),
    ('Novelty',                    'Offline', '% of recommendations user has never seen'),
    ('NDCG@10',                    'Offline', 'Normalised Discounted Cumulative Gain at position 10'),
]
for metric, where, desc in metrics:
    print(f"  {metric:<28}: {where:<8}  {desc}")

# ── What interviewers score — the actual rubric ───────────────────────

SCORING_RUBRIC = {
    'Problem framing (10%)': [
        'Asks clarifying questions before designing',
        'Correctly identifies ML task type (regression/classification/ranking)',
        'Defines business metric separate from ML metric',
        'States success criteria quantitatively',
    ],
    'Data and features (25%)': [
        'Identifies relevant data sources including non-obvious ones',
        'Addresses data quality and bias',
        'Addresses cold start problem for new entities',
        'Explains feature computation (batch vs real-time)',
        'Mentions point-in-time correctness for training',
    ],
    'Model selection (20%)': [
        'Proposes a simple baseline before complex model',
        'Justifies model choice with explicit tradeoffs',
        'Addresses class imbalance if present',
        'Mentions explainability requirements if regulatory context',
    ],
    'System design (25%)': [
        'Correctly determines online vs batch serving',
        'Mentions feature store for pre-computed features',
        'Provides latency breakdown (feature fetch + model + network)',
        'Capacity estimates with numbers',
        'Fallback when model unavailable',
    ],
    'Monitoring (15%)': [
        'Identifies what drifts in this specific system',
        'Proposes concrete monitoring metrics',
        'States retraining trigger condition',
        'Addresses feedback loop and label delay',
    ],
    'Communication (5%)': [
        'Draws architecture diagram',
        'States assumptions explicitly',
        'Handles interviewer's probing questions without getting flustered',
    ],
}

for category, criteria in SCORING_RUBRIC.items():
    print(f"
{category}:")
    for c in criteria:
        print(f"  ✓ {c}")

# ── Common mistakes that kill interview scores ────────────────────────
print("

COMMON MISTAKES:")
mistakes = [
    ('Jumping to model selection',    'First 5 minutes spent on LightGBM vs XGBoost. Interviewer wants system design.'),
    ('Forgetting cold start',         '"New restaurant has no history" — always asked as a follow-up. Address proactively.'),
    ('No fallback plan',              '"What if the model is down?" — must have a rule-based or static fallback.'),
    ('Missing label strategy',        '"How do you get training labels?" — must be answered for every system.'),
    ('Batch when online needed',      '"Pre-compute all (user, merchant) pairs" — combinatorial explosion.'),
    ('Ignoring data leakage',         '"Use future features in training" — always mention point-in-time correctness.'),
    ('No numbers',                    '"Large scale, many requests" — always give orders of magnitude.'),
]
print(f"  {'Mistake':<30} {'Why it hurts'}")
print("  " + "─" * 70)
for mistake, why in mistakes:
    print(f"  {mistake:<30} {why}")

It is rarely knowledge of a fancier model. It is asking about label strategy and failure modes unprompted, before the interviewer has to drag it out of you — the same behaviour a staff engineer expects to see in a design document review, where an author who anticipates the hard questions is trusted with more scope than one who has to be walked through them.

Model selection is one of eight equally necessary questions, and the scoring rubric in the previous section weights data and system design far higher than model selection alone. Most real systems fail from a data quality gap, a missing cold-start plan, or no fallback when the model is unavailable — not from choosing the second-best model family. A LightGBM baseline with solid data and monitoring beats a state-of-the-art model with no failure plan every time.

It is about the entire lifecycle end to end: how labels are obtained, how features reach the model consistently at both training time and serving time, what happens when the model is unavailable, and how drift gets detected. The model is one box in a diagram that also contains a feature store, a fallback path, and a monitoring dashboard — and in a well-drawn diagram, the data flow usually takes up more space than the model box itself.

Multiple architectures can satisfy the same constraints — online versus batch serving, one global model versus per-segment models — and reasonable engineers land in different places depending on team size, existing infrastructure, and risk tolerance. What is actually being evaluated is whether the chosen tradeoff is justified given the stated constraints, not whether the answer matches a fixed key.

Reaching for complexity the problem does not need is graded down, not up, in both interviews and real design reviews. It signals that the actual constraint — usually latency, data volume, or an explainability requirement — was never identified. Both interviewers and real staff engineers are more impressed by "I considered a transformer, but the latency budget rules it out, so I would start with gradient boosting" than by simply naming the newest model.

Real ML systems keep hitting the monitoring and failure-modes questions long after the first architecture ships — drift, a new fraud pattern, a data pipeline outage — so the design keeps getting revisited. This is exactly why interviewers spend the last five to ten minutes on "what would you do differently at ten times scale" — a design that only ever gets defended once, at kickoff, was never actually a complete design.

Resist the urge to start drawing an architecture immediately. Spend the first three to five minutes purely on clarifying questions covering scale, latency, and the business metric, then explicitly state the ML task type and label definition before touching data or model. Say the structure out loud — problem framing, then data, features, model, serving, scale, monitoring, failure modes — so the interviewer can redirect you toward whichever area they care about most. Announcing the structure signals more seniority than any single model choice.

Batch serving pre-computes predictions for every entity on a schedule and looks them up at request time — cheap and simple, but stale, and infeasible when there are too many context combinations to precompute, like a fraud score that depends on the exact transaction amount and merchant at that instant. Online serving computes the prediction at request time using live features, required whenever the prediction depends on information only available at request time, at the cost of a tighter latency budget and supporting infrastructure. Decide by asking whether the prediction depends on real-time context: if yes, online; if the entity list is fixed and small enough to enumerate, batch is dramatically cheaper.

Fall back to a coarser aggregate that does have data — a city or category average instead of entity-specific features, a popularity-based recommendation instead of a personalised one, or a simple rule instead of a learned estimate — and explicitly flag the entity as cold so the fallback is a deliberate choice, not missing data silently treated as zero. As the entity accumulates interactions, blend from the coarse fallback toward the entity-specific model. Mention this proactively — cold start is a near-universal follow-up question if you do not raise it first.

Start with a single global model — it pools data, is simpler to maintain, and avoids duplicating monitoring and retraining work across many models. Split into per-segment models only when there is evidence the global model materially underperforms on a specific segment, that segment has enough data to support its own model, and the business impact justifies the ongoing maintenance of several models instead of one. Segmenting too early is a common overengineering mistake that adds operational burden without adding accuracy.

Treat it as a chance to show you understand where your design's current assumptions break, not a chance to redesign from scratch. Name the first concrete thing that breaks — a feature store lookup pattern fine at hundreds of requests per second becomes a bottleneck at thousands, a single model trained daily may need to become segment-specific once there is enough data per segment, a rule-based fallback that handled rare outages may need real redundancy once outages become frequent enough to matter. This question tests whether your original design had implicit assumptions you can now name.