Model Deployment — FastAPI, Docker, Kubernetes

A chef (your model) → a restaurant (FastAPI API) → a restaurant chain (Docker) → a restaurant franchise (Kubernetes). One chef cooking in their kitchen is a model in a notebook. Opening a restaurant adds a standardised environment, a menu (API contract), and a way for customers to order. Franchising the restaurant (Docker) means any city can run the same restaurant with the same recipe, regardless of local conditions. The franchise management company (Kubernetes) opens more locations when demand spikes and closes underperforming ones.

Docker solves "works on my machine." Kubernetes solves "stays running at scale." FastAPI solves "speaks HTTP." Together they are how every production ML model at top tech companies is served.

# main.py — production FastAPI model serving
# pip install fastapi uvicorn pydantic scikit-learn numpy

from fastapi import FastAPI, HTTPException, Request, status
from fastapi.responses import JSONResponse
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel, Field, validator
from contextlib import asynccontextmanager
from typing import Optional
import numpy as np
import pickle, time, logging, os
from datetime import datetime

logging.basicConfig(level=logging.INFO,
                    format='%(asctime)s %(levelname)s %(message)s')
logger = logging.getLogger(__name__)

# ── Request and response schemas ──────────────────────────────────────
class DeliveryPredictionRequest(BaseModel):
    restaurant_id:   str    = Field(..., description="Restaurant identifier")
    driver_id:       str    = Field(..., description="Driver identifier")
    distance_km:     float  = Field(..., ge=0.1, le=50.0,
                                     description="Delivery distance in km")
    is_peak_hour:    int    = Field(..., ge=0, le=1,
                                     description="1 if 7-9 PM or 12-2 PM else 0")
    order_value:     float  = Field(..., ge=0.0,
                                     description="Order total in USD")
    # Features fetched from feature store at runtime
    restaurant_avg_delivery_time: Optional[float] = None
    driver_avg_delivery_time:     Optional[float] = None

    @validator('distance_km')
    def distance_must_be_reasonable(cls, v):
        if v > 30:
            logger.warning(f"Unusually large distance: {v} km")
        return v

class DeliveryPredictionResponse(BaseModel):
    predicted_delivery_time_min: float
    confidence_interval_lower:   float
    confidence_interval_upper:   float
    model_version:               str
    prediction_id:               str
    latency_ms:                  float

class HealthResponse(BaseModel):
    status:        str
    model_loaded:  bool
    model_version: str
    uptime_s:      float
    timestamp:     str

# ── Global model state ────────────────────────────────────────────────
class ModelState:
    model           = None
    model_version   = 'unknown'
    scaler          = None
    feature_columns = []
    load_time       = None

state = ModelState()

# ── Feature store client (simulated) ─────────────────────────────────
ONLINE_FEATURE_STORE = {
    'RST001': {'restaurant_avg_delivery_time': 33.2},
    'RST002': {'restaurant_avg_delivery_time': 41.5},
    'DRV01':  {'driver_avg_delivery_time': 29.8},
    'DRV02':  {'driver_avg_delivery_time': 35.1},
}

def fetch_online_features(restaurant_id: str, driver_id: str) -> dict:
    """Fetch latest features from online store (~1ms in production)."""
    features = {}
    features.update(ONLINE_FEATURE_STORE.get(restaurant_id, {}))
    features.update(ONLINE_FEATURE_STORE.get(driver_id, {}))
    # Default values for unknown entities
    features.setdefault('restaurant_avg_delivery_time', 35.0)
    features.setdefault('driver_avg_delivery_time', 32.0)
    return features

# ── Model loading ─────────────────────────────────────────────────────
def load_model():
    """Load model from disk or model registry at startup."""
    model_path = os.environ.get('MODEL_PATH', '/tmp/model.pkl')

    if os.path.exists(model_path):
        with open(model_path, 'rb') as f:
            artifact = pickle.load(f)
        state.model           = artifact.get('model')
        state.scaler          = artifact.get('scaler')
        state.feature_columns = artifact.get('feature_columns', [])
        state.model_version   = artifact.get('version', 'unknown')
        logger.info(f"Model loaded: version={state.model_version}")
    else:
        # For demo: create and use a dummy model
        from sklearn.ensemble import GradientBoostingRegressor
        from sklearn.preprocessing import StandardScaler

        state.feature_columns = [
            'restaurant_avg_delivery_time', 'driver_avg_delivery_time',
            'distance_km', 'is_peak_hour', 'order_value',
        ]
        state.model = GradientBoostingRegressor(n_estimators=100, random_state=42)
        # Fit on tiny synthetic data for demo
        np.random.seed(42)
        X_demo = np.random.randn(200, len(state.feature_columns))
        y_demo = X_demo[:, 2] * 6 + 30 + np.random.randn(200) * 5
        state.model.fit(X_demo, y_demo)
        state.model_version = 'demo-v1.0'
        logger.info("Demo model initialised")

    state.load_time = time.time()

# ── Lifespan: load model at startup ──────────────────────────────────
@asynccontextmanager
async def lifespan(app: FastAPI):
    logger.info("Starting up — loading model...")
    load_model()
    yield
    logger.info("Shutting down")

# ── FastAPI app ───────────────────────────────────────────────────────
app = FastAPI(
    title='DoorDash Delivery Time Prediction API',
    description='Predicts delivery time for a given order and driver.',
    version='1.0.0',
    lifespan=lifespan,
)

app.add_middleware(
    CORSMiddleware,
    allow_origins=['*'],   # restrict in production
    allow_methods=['*'],
    allow_headers=['*'],
)

# ── Request logging middleware ────────────────────────────────────────
@app.middleware('http')
async def log_requests(request: Request, call_next):
    start = time.time()
    response = await call_next(request)
    latency = (time.time() - start) * 1000
    logger.info(
        f"{request.method} {request.url.path} "
        f"status={response.status_code} latency={latency:.1f}ms"
    )
    return response

# ── Health and readiness endpoints ────────────────────────────────────
@app.get('/health', response_model=HealthResponse)
async def health():
    """Kubernetes liveness probe — is the container alive?"""
    return HealthResponse(
        status='healthy',
        model_loaded=state.model is not None,
        model_version=state.model_version,
        uptime_s=time.time() - (state.load_time or time.time()),
        timestamp=datetime.now().isoformat(),
    )

@app.get('/ready')
async def ready():
    """Kubernetes readiness probe — is the model loaded and ready?"""
    if state.model is None:
        raise HTTPException(
            status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
            detail='Model not yet loaded',
        )
    return {'status': 'ready', 'model_version': state.model_version}

# ── Prediction endpoint ────────────────────────────────────────────────
@app.post('/v1/predict', response_model=DeliveryPredictionResponse)
async def predict(request: DeliveryPredictionRequest):
    """Predict delivery time for a given order."""
    if state.model is None:
        raise HTTPException(status_code=503, detail='Model not loaded')

    start = time.time()

    # Fetch features from online store
    online_features = fetch_online_features(
        request.restaurant_id, request.driver_id,
    )

    # Override with request-provided values if present
    if request.restaurant_avg_delivery_time is not None:
        online_features['restaurant_avg_delivery_time'] = request.restaurant_avg_delivery_time
    if request.driver_avg_delivery_time is not None:
        online_features['driver_avg_delivery_time'] = request.driver_avg_delivery_time

    # Build feature vector in correct order
    feature_values = [
        online_features.get('restaurant_avg_delivery_time', 35.0),
        online_features.get('driver_avg_delivery_time', 32.0),
        request.distance_km,
        float(request.is_peak_hour),
        request.order_value,
    ]
    X = np.array(feature_values).reshape(1, -1)

    # Predict
    pred = float(state.model.predict(X)[0])
    pred = max(5.0, min(120.0, pred))   # clip to reasonable range

    # Confidence interval (±1 MAE — simplified)
    ci_margin = 6.0
    latency   = (time.time() - start) * 1000

    import uuid
    return DeliveryPredictionResponse(
        predicted_delivery_time_min=round(pred, 1),
        confidence_interval_lower=round(max(5.0, pred - ci_margin), 1),
        confidence_interval_upper=round(pred + ci_margin, 1),
        model_version=state.model_version,
        prediction_id=str(uuid.uuid4()),
        latency_ms=round(latency, 2),
    )

@app.post('/v1/predict/batch')
async def predict_batch(requests_list: list[DeliveryPredictionRequest]):
    """Batch predictions — up to 100 per call."""
    if len(requests_list) > 100:
        raise HTTPException(status_code=400, detail='Max batch size is 100')
    results = []
    for req in requests_list:
        result = await predict(req)
        results.append(result)
    return results

# ── Run with: uvicorn main:app --host 0.0.0.0 --port 8000 --workers 4
if __name__ == '__main__':
    import uvicorn
    uvicorn.run('main:app', host='0.0.0.0', port=8000,
                workers=4, log_level='info')

# Dockerfile — production ML model serving
# Multi-stage build: builder installs dependencies, runtime is lean

# ── Stage 1: Builder — install dependencies ──────────────────────────
FROM python:3.11-slim AS builder

WORKDIR /build

# Install build dependencies (not needed at runtime)
RUN apt-get update && apt-get install -y --no-install-recommends \
    gcc g++ \
    && rm -rf /var/lib/apt/lists/*

# Copy and install Python dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir --user -r requirements.txt

# ── Stage 2: Runtime — lean final image ───────────────────────────────
FROM python:3.11-slim AS runtime

WORKDIR /app

# Copy installed packages from builder (not gcc/g++ build tools)
COPY --from=builder /root/.local /root/.local
ENV PATH=/root/.local/bin:$PATH

# Non-root user for security
RUN groupadd -r mluser && useradd -r -g mluser mluser

# Copy application code
COPY main.py .
COPY model/ ./model/

# Model path — overridden via env var or Kubernetes ConfigMap
ENV MODEL_PATH=/app/model/model.pkl
ENV PYTHONUNBUFFERED=1

USER mluser

EXPOSE 8000

# Health check — Docker restarts container if this fails
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
    CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"

# Use gunicorn in production for multi-worker handling
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "4"]

# requirements.txt — pin every version for reproducibility
fastapi==0.110.0
uvicorn[standard]==0.29.0
pydantic==2.6.4
scikit-learn==1.4.1
numpy==1.26.4
pandas==2.2.1

# ── Docker build and run commands ─────────────────────────────────────

# Build the image
# docker build -t doordash-delivery-model:v1.0.0 .

# Run locally to test
# docker run -p 8000:8000 \
#   -e MODEL_PATH=/app/model/model.pkl \
#   doordash-delivery-model:v1.0.0

# Test the running container
# curl -X POST http://localhost:8000/v1/predict \
#   -H "Content-Type: application/json" \
#   -d '{"restaurant_id":"RST001","driver_id":"DRV01",
#         "distance_km":3.5,"is_peak_hour":1,"order_value":450}'

# Push to container registry (AWS ECR / GCP Artifact Registry / Docker Hub)
# docker tag doordash-delivery-model:v1.0.0 \
#   123456789.dkr.ecr.us-east-1.amazonaws.com/doordash-delivery-model:v1.0.0
# docker push 123456789.dkr.ecr.us-east-1.amazonaws.com/doordash-delivery-model:v1.0.0

# ── Image size optimisation ────────────────────────────────────────────
print("""
Image size reduction techniques:
  Technique                    Before    After     Savings
  ─────────────────────────────────────────────────────────
  python:3.11 (full)           ~1.0 GB   —
  python:3.11-slim             ~180 MB   180 MB    820 MB
  Multi-stage build            180 MB    120 MB     60 MB
  --no-cache-dir in pip        120 MB     95 MB     25 MB
  .dockerignore (tests/docs)    95 MB     90 MB      5 MB

  Total: ~1.0 GB → ~90 MB for a typical scikit-learn model

.dockerignore file:
  __pycache__/
  *.pyc
  .pytest_cache/
  tests/
  notebooks/
  .git/
  mlruns/
  *.egg-info/
""")

# ── Kubernetes YAML manifests ─────────────────────────────────────────
# Apply with: kubectl apply -f deployment.yaml

DEPLOYMENT_YAML = """
apiVersion: apps/v1
kind: Deployment
metadata:
  name: doordash-delivery-model
  namespace: ml-serving
  labels:
    app: delivery-model
    version: v1.0.0
    team: delivery-ml
spec:
  replicas: 3
  selector:
    matchLabels:
      app: delivery-model
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1        # temporarily create 1 extra pod during update
      maxUnavailable: 0  # never have fewer than desired replicas
  template:
    metadata:
      labels:
        app: delivery-model
        version: v1.0.0
    spec:
      containers:
        - name: model-server
          image: 123456789.dkr.ecr.us-east-1.amazonaws.com/doordash-delivery-model:v1.0.0
          ports:
            - containerPort: 8000
          env:
            - name: MODEL_PATH
              value: /app/model/model.pkl
            - name: WORKERS
              value: "4"
          resources:
            requests:
              cpu: "500m"     # 0.5 CPU core guaranteed
              memory: "512Mi" # 512 MB RAM guaranteed
            limits:
              cpu: "2000m"    # never use more than 2 CPU cores
              memory: "2Gi"   # never use more than 2 GB RAM
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 60   # wait for model to load
            periodSeconds: 30
            failureThreshold: 3       # restart after 3 consecutive failures
          readinessProbe:
            httpGet:
              path: /ready
              port: 8000
            initialDelaySeconds: 30
            periodSeconds: 10
            failureThreshold: 3       # remove from load balancer if not ready
"""

SERVICE_YAML = """
apiVersion: v1
kind: Service
metadata:
  name: delivery-model-service
  namespace: ml-serving
spec:
  type: LoadBalancer   # ClusterIP for internal, LoadBalancer for external
  selector:
    app: delivery-model
  ports:
    - protocol: TCP
      port: 80          # external port
      targetPort: 8000  # container port
"""

HPA_YAML = """
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: delivery-model-hpa
  namespace: ml-serving
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: doordash-delivery-model
  minReplicas: 3    # always run at least 3 for availability
  maxReplicas: 20   # cap at 20 to control cost
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70   # scale up when avg CPU > 70%
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
"""

print("Kubernetes deployment commands:")
commands = [
    ('kubectl apply -f deployment.yaml',
     'Deploy or update the model serving deployment'),
    ('kubectl get pods -n ml-serving',
     'List all pods and their status'),
    ('kubectl logs -f deployment/doordash-delivery-model -n ml-serving',
     'Stream logs from the deployment'),
    ('kubectl rollout status deployment/doordash-delivery-model',
     'Watch rollout progress during update'),
    ('kubectl rollout undo deployment/doordash-delivery-model',
     'Rollback to previous version instantly'),
    ('kubectl scale deployment/doordash-delivery-model --replicas=10',
     'Manual scale up during peak'),
    ('kubectl top pods -n ml-serving',
     'CPU and memory usage per pod'),
]
for cmd, desc in commands:
    print(f"  $ {cmd}")
    print(f"    {desc}")
    print()

# ── Update strategies ──────────────────────────────────────────────────

# ── Strategy 1: Rolling Update (default, simplest) ────────────────────
# Kubernetes replaces pods one at a time.
# Old version serves traffic until new version is ready.
# Zero downtime but both versions serve traffic briefly.

ROLLING_UPDATE = """
# Update the image — Kubernetes handles the rest
kubectl set image deployment/doordash-delivery-model \
  model-server=.../doordash-delivery-model:v2.0.0

# Watch the rollout
kubectl rollout status deployment/doordash-delivery-model
# Waiting for deployment to finish: 1 out of 3 new replicas updated...
# Waiting for deployment to finish: 2 out of 3 new replicas updated...
# deployment "doordash-delivery-model" successfully rolled out

# Rollback if something goes wrong
kubectl rollout undo deployment/doordash-delivery-model
"""

# ── Strategy 2: Canary Release (safest for ML models) ─────────────────
# Send 5% of traffic to new model, monitor quality, then promote.
# If new model degrades, only 5% of users are affected before rollback.

CANARY_CONFIG = """
# Canary deployment: v2 gets 5% of traffic (1 of 20 replicas)
# v1 deployment: 19 replicas
# v2 deployment: 1 replica

# Both deployments share the same Service label selector
# Traffic is distributed proportionally to replica count

apiVersion: apps/v1
kind: Deployment
metadata:
  name: doordash-delivery-model-canary
spec:
  replicas: 1    # 1 of 20 total = 5% of traffic
  selector:
    matchLabels:
      app: delivery-model   # same label — Service routes to both
  template:
    metadata:
      labels:
        app: delivery-model
        track: canary
    spec:
      containers:
        - name: model-server
          image: .../doordash-delivery-model:v2.0.0   # new version
          # same ports, probes, resources as main deployment
"""

# ── Strategy 3: Blue-Green (instant cutover, instant rollback) ────────
# Two identical deployments (blue=current, green=new).
# Switch Service selector to point from blue to green.
# Rollback = switch back. No gradual rollout.

BLUE_GREEN = """
# Blue deployment (current production): label track=blue
# Green deployment (new version):       label track=green

# Service currently points to blue:
# selector:
#   app: delivery-model
#   track: blue

# Cutover: update Service to point to green
kubectl patch service delivery-model-service \
  -p '{"spec":{"selector":{"track":"green"}}}'

# Rollback: update Service to point back to blue
kubectl patch service delivery-model-service \
  -p '{"spec":{"selector":{"track":"blue"}}}'

# After validating green: delete blue deployment
kubectl delete deployment doordash-delivery-model-blue
"""

print("Deployment strategy comparison:")
strategies = [
    ('Rolling Update', 'Simple, zero downtime', 'Brief mixed traffic', 'Always'),
    ('Canary',         '5% traffic to new, gradual', 'More complex', 'New models'),
    ('Blue-Green',     'Instant cutover/rollback', 'Double resources briefly', 'High-risk updates'),
]
print(f"  {'Strategy':<15} {'Benefit':<35} {'Cost':<25} {'Use when'}")
print("  " + "─" * 85)
for s, b, c, u in strategies:
    print(f"  {s:<15} {b:<35} {c:<25} {u}")

# pip install locust httpx

# ── Load test with Locust ─────────────────────────────────────────────
# Save as locustfile.py and run: locust --host http://localhost:8000

LOCUST_FILE = """
import random
from locust import HttpUser, task, between

class DeliveryModelUser(HttpUser):
    wait_time = between(0.1, 0.5)   # 100-500ms between requests per user

    RESTAURANT_IDS = ['RST001', 'RST002', 'RST003', 'RST004', 'RST005']
    DRIVER_IDS     = ['DRV01', 'DRV02', 'DRV03', 'DRV04', 'DRV05']

    @task(10)   # weight 10 = most common
    def predict_single(self):
        payload = {
            "restaurant_id": random.choice(self.RESTAURANT_IDS),
            "driver_id":     random.choice(self.DRIVER_IDS),
            "distance_km":   round(random.uniform(0.5, 15.0), 1),
            "is_peak_hour":  random.randint(0, 1),
            "order_value":   round(random.uniform(100, 2000), 0),
        }
        with self.client.post(
            '/v1/predict',
            json=payload,
            catch_response=True,
        ) as response:
            if response.status_code != 200:
                response.failure(f"Status {response.status_code}")
            elif response.elapsed.total_seconds() > 0.5:
                response.failure("Response time > 500ms")

    @task(1)   # weight 1 = rare
    def health_check(self):
        self.client.get('/health')

# Run: locust --host http://localhost:8000 --users 100 --spawn-rate 10
# Users: 100 concurrent users
# Spawn-rate: 10 new users per second until target reached
"""

# ── Benchmark with httpx directly ────────────────────────────────────
import asyncio
import time
import statistics
import json

async def benchmark_endpoint():
    """Quick async benchmark without locust."""
    try:
        import httpx
        url     = 'http://localhost:8000/v1/predict'
        payload = {
            'restaurant_id': 'RST001',
            'driver_id':     'DRV01',
            'distance_km':   3.5,
            'is_peak_hour':  1,
            'order_value':   450.0,
        }
        latencies = []

        async with httpx.AsyncClient() as client:
            # Warmup
            for _ in range(3):
                await client.post(url, json=payload)

            # Benchmark: 50 concurrent requests
            tasks = [client.post(url, json=payload) for _ in range(50)]
            start     = time.time()
            responses = await asyncio.gather(*tasks, return_exceptions=True)
            total_s   = time.time() - start

            for r in responses:
                if isinstance(r, Exception):
                    continue
                latencies.append(r.elapsed.total_seconds() * 1000)

        if latencies:
            print(f"Benchmark results (50 concurrent requests):")
            print(f"  Throughput:  {50/total_s:.0f} req/s")
            print(f"  p50 latency: {statistics.median(latencies):.1f}ms")
            print(f"  p95 latency: {sorted(latencies)[int(0.95*len(latencies))]:.1f}ms")
            print(f"  p99 latency: {sorted(latencies)[int(0.99*len(latencies))]:.1f}ms")
            print(f"  Success:     {len(latencies)}/50")

    except Exception as e:
        print(f"Benchmark requires running server: {e}")

# asyncio.run(benchmark_endpoint())

# ── SLO targets for DoorDash delivery prediction ────────────────────────
print("Production SLO targets:")
slos = [
    ('Availability',   '99.9%',   '< 45 min downtime per month'),
    ('p50 latency',    '< 50ms',  'Median response under 50ms'),
    ('p95 latency',    '< 200ms', '95th percentile under 200ms'),
    ('p99 latency',    '< 500ms', '99th percentile under 500ms'),
    ('Error rate',     '< 0.1%',  'Less than 1 error per 1000 requests'),
    ('Throughput',     '> 200 RPS', 'Per replica, single core'),
]
for metric, target, desc in slos:
    print(f"  {metric:<18}: {target:<10}  {desc}")

Running a Python script that loads a model and prints a prediction is a demo, not a deployment. Production deployment means the model is behind an API contract that other services can depend on without breaking, survives a crashed process by restarting automatically, handles many concurrent requests without falling over, can be updated without taking the service offline, and exposes health signals so an orchestrator knows when it is broken. Every one of those properties is infrastructure work that has nothing to do with the model itself — it is the same work whether the model is a linear regression or a neural network.

A notebook has no latency budget, no concurrent users, and features computed leisurely from a clean, already-joined dataframe. Production has none of those luxuries: features must be fetched from a live feature store in single-digit milliseconds, malformed input has to be rejected instead of crashing the process, and the exact same preprocessing that ran during training has to run identically on a single incoming request. Training-serving skew — where the notebook's feature computation subtly differs from the API's — is one of the most common causes of a model that scored well offline and performs badly in production, and it is invisible until you actually deploy and check.

The two are different systems with different failure modes, not the same predict call on a different schedule. Batch prediction runs on a schedule over a warehouse table, has no per-request latency budget, and can retry an entire failed run overnight. Real-time serving answers a single request synchronously inside a strict latency budget, needs an online feature store for fresh values, and a single slow dependency can take down the whole request path. A model built assuming batch-style feature freshness (daily aggregates) often cannot simply be dropped into a real-time endpoint — the features it needs may not exist yet at request time with the freshness the model was trained on.

Docker guarantees the software environment is reproducible — same Python version, same pinned package versions, same OS libraries. It says nothing about the data flowing into that environment. A model can run inside the exact same container in staging and production and still produce different predictions if the feature store returns different values, if a categorical encoder sees a category it never saw during training, or if an upstream data pipeline change silently altered a feature's scale. Reproducible software is necessary but is a completely separate problem from reproducible data.

A rolling update guarantees the service stays up while pods are replaced — it says absolutely nothing about whether the new model's predictions are any good. Kubernetes has no concept of prediction quality; it only checks HTTP health endpoints and resource usage. A model that returns a plausible-looking number for every request but is systematically wrong will sail through a rolling update with every probe green. That is exactly why canary releases and shadow deployment exist as separate concerns from the rolling update mechanism — they check whether the new version's outputs are trustworthy, which is a question infrastructure tooling cannot answer on its own.

A canary release sends a small slice of real user traffic to the new version and those users receive its actual predictions — if the new version is bad, a small number of real users are affected before you catch it and roll back. A shadow deployment sends a copy of every request to the new version as well, but its prediction is only logged, never returned to the user — zero user impact, at the cost of running the new model's compute twice for every request. I would reach for shadow deployment first, before any user is exposed at all, to validate the new model on real live traffic distribution. Once shadow results look good, I would move to a canary to validate real-world business outcomes, since some effects (actual conversion, actual delivery accuracy) only show up when the model's prediction is the one acted on.

Latency is how long one request takes; throughput is how many requests the system handles per second — and improving one can hurt the other. A larger, more accurate model usually has higher per-request latency, so hitting a strict latency budget might mean choosing a smaller model, adding a result cache for repeat inputs, or batching several requests together on the server side to use hardware more efficiently — but batching trades a little latency (waiting to fill the batch) for a lot more throughput. I would start from the actual product requirement — a fraud check blocking a checkout needs sub-100-millisecond p99 latency, while a nightly recommendation batch job cares only about total throughput and can trade individual-request latency freely for it.

First, stop the bleeding: if it was a gradual rollout (canary or staged traffic shift), immediately shift traffic back to zero percent on the new version — kubectl rollout undo, or flipping the Service selector back to the previous deployment if it was blue-green, either of which should take seconds, not minutes. Second, confirm the rollback actually restored good predictions by watching the same dashboards that flagged the problem. Only after production is stable would I investigate root cause — was it a genuinely worse model, a feature pipeline bug, or a data issue upstream — because debugging under live production pressure risks making the incident worse. I would also check whether anything besides the model artifact shipped alongside it (a feature schema change, a new dependency) that also needs reverting.

A liveness probe answers "is this process alive or should it be restarted" — if it fails repeatedly, Kubernetes kills and restarts the pod. A readiness probe answers "should this pod currently receive traffic" — if it fails, the pod is pulled out of the load balancer rotation but is not restarted. ML services need both because loading a model into memory can take anywhere from a few seconds to a couple of minutes, and during that window the process is alive and healthy but not yet able to serve a correct prediction. Without a separate readiness probe, Kubernetes would either send traffic to a pod whose model has not finished loading, or the liveness probe's timeout would need to be set so generously that a genuinely hung process would take far too long to get restarted.

I would choose batch when the prediction does not need to reflect something that just happened — a weekly churn-risk score, a nightly recommendation refresh, a monthly credit-limit recalculation. Batch is simpler to operate (no always-on API, no per-request latency budget, no online feature store), cheaper per prediction at high volume, and failures can simply be retried on the next scheduled run. I would choose real-time serving when the prediction depends on something that just happened in this exact request — a fraud check at checkout, a delivery time estimate for the order being placed right now — where the value of the prediction depends entirely on it reflecting the current moment. Some systems genuinely need both: a nightly batch job scores every user for a baseline, and a real-time endpoint adjusts that score with in-session signals.