Experiment Tracking with MLflow and Weights & Biases

A chef's recipe book vs cooking from memory. A chef who cooks from memory might produce excellent dishes — but cannot replicate them exactly next week, cannot scale the recipe for 200 people, and cannot hand the recipe to a junior chef. A chef who writes down every recipe with precise measurements can reproduce any dish, compare two versions of the same dish scientifically, and build on past experiments. Experiment tracking is the recipe book for ML.

The discipline of logging experiments also forces clarity of thought. When you must decide what to log before running an experiment, you think more carefully about what you are trying to learn. Untracked experiments are usually under-thought experiments.

# pip install mlflow scikit-learn pandas numpy

import mlflow
import mlflow.sklearn
import numpy as np
import pandas as pd
from sklearn.ensemble import GradientBoostingRegressor, RandomForestRegressor
from sklearn.linear_model import Ridge
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
from sklearn.preprocessing import StandardScaler
import time, json, os

# ── Start MLflow server locally ───────────────────────────────────────
# mlflow server --host 0.0.0.0 --port 5000
# Then set: mlflow.set_tracking_uri("http://localhost:5000")
# For this demo: use local file store
mlflow.set_tracking_uri('file:///tmp/mlruns')

# ── Create or set experiment ──────────────────────────────────────────
mlflow.set_experiment('doordash-delivery-time-prediction')

# ── Synthetic dataset ─────────────────────────────────────────────────
np.random.seed(42)
n = 3000
X = pd.DataFrame({
    'restaurant_avg_delivery_time':  np.random.normal(35, 8, n),
    'driver_avg_delivery_time':      np.random.normal(32, 7, n),
    'distance_km':                   np.random.exponential(3, n),
    'is_peak_hour':                  np.random.randint(0, 2, n),
    'order_value':                   np.random.exponential(400, n),
    'restaurant_7d_order_count':     np.random.randint(50, 500, n),
    'driver_completed_orders':       np.random.randint(100, 1000, n),
})
y = (X['distance_km'] * 6 +
     X['restaurant_avg_delivery_time'] * 0.4 +
     X['is_peak_hour'] * 8 +
     np.random.normal(0, 5, n)).clip(10, 90)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# ── Run 1: GradientBoosting baseline ─────────────────────────────────
def train_and_log(model, params: dict, model_name: str,
                   tags: dict = None) -> str:
    """Train a model and log everything to MLflow."""
    with mlflow.start_run(run_name=model_name) as run:
        # Log parameters
        mlflow.log_params(params)

        # Log tags for filtering
        mlflow.set_tag('model_type', model_name)
        mlflow.set_tag('dataset_version', 'v1.0')
        mlflow.set_tag('team', 'delivery-ml')
        if tags:
            for k, v in tags.items():
                mlflow.set_tag(k, str(v))

        # Train
        start = time.time()
        model.fit(X_train, y_train)
        train_time = time.time() - start

        # Evaluate
        train_preds = model.predict(X_train)
        test_preds  = model.predict(X_test)

        train_mae = mean_absolute_error(y_train, train_preds)
        test_mae  = mean_absolute_error(y_test,  test_preds)
        test_rmse = mean_squared_error(y_test,  test_preds, squared=False)
        test_r2   = r2_score(y_test, test_preds)

        # Cross-validation
        cv_scores = cross_val_score(model, X_train, y_train,
                                     cv=5, scoring='neg_mean_absolute_error')
        cv_mae = -cv_scores.mean()

        # Log metrics
        mlflow.log_metric('train_mae',   train_mae)
        mlflow.log_metric('test_mae',    test_mae)
        mlflow.log_metric('test_rmse',   test_rmse)
        mlflow.log_metric('test_r2',     test_r2)
        mlflow.log_metric('cv_mae',      cv_mae)
        mlflow.log_metric('cv_mae_std',  cv_scores.std())
        mlflow.log_metric('train_time_s', train_time)

        # Log step-level metrics (simulated epochs)
        for step in range(1, params.get('n_estimators', 100) + 1, 20):
            mlflow.log_metric('staged_test_mae', test_mae * (1 + 0.5/step), step=step)

        # Log model
        mlflow.sklearn.log_model(
            model, artifact_path='model',
            registered_model_name=f'delivery-time-{model_name}',
        )

        # Log feature importance as artifact
        if hasattr(model, 'feature_importances_'):
            fi = pd.DataFrame({
                'feature':   X.columns,
                'importance': model.feature_importances_,
            }).sort_values('importance', ascending=False)
            fi_path = '/tmp/feature_importance.csv'
            fi.to_csv(fi_path, index=False)
            mlflow.log_artifact(fi_path, artifact_path='analysis')

        print(f"  {model_name:<30}: test_mae={test_mae:.3f}  "
              f"cv_mae={cv_mae:.3f}  time={train_time:.1f}s  "
              f"run_id={run.info.run_id[:8]}...")
        return run.info.run_id

print("Running experiments:")
run_ids = {}

# Experiment 1: GradientBoosting
gb_params = {'n_estimators': 200, 'max_depth': 4, 'learning_rate': 0.05,
              'min_samples_leaf': 5, 'subsample': 0.8}
run_ids['gb'] = train_and_log(
    GradientBoostingRegressor(**gb_params, random_state=42),
    gb_params, 'GradientBoosting',
)

# Experiment 2: RandomForest
rf_params = {'n_estimators': 200, 'max_depth': 8, 'min_samples_leaf': 5}
run_ids['rf'] = train_and_log(
    RandomForestRegressor(**rf_params, random_state=42),
    rf_params, 'RandomForest',
)

# Experiment 3: Ridge regression baseline
ridge_params = {'alpha': 10.0}
scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)
X_test_sc  = scaler.transform(X_test)
with mlflow.start_run(run_name='Ridge-baseline') as run:
    mlflow.log_params(ridge_params)
    mlflow.set_tag('model_type', 'linear')
    model = Ridge(**ridge_params).fit(X_train_sc, y_train)
    preds = model.predict(X_test_sc)
    mlflow.log_metric('test_mae', mean_absolute_error(y_test, preds))
    mlflow.log_metric('test_r2',  r2_score(y_test, preds))
    run_ids['ridge'] = run.info.run_id
    print(f"  {'Ridge-baseline':<30}: test_mae={mean_absolute_error(y_test, preds):.3f}")

# ── Compare runs programmatically ─────────────────────────────────────
client = mlflow.tracking.MlflowClient()
experiment = mlflow.get_experiment_by_name('doordash-delivery-time-prediction')
runs = client.search_runs(
    experiment_ids=[experiment.experiment_id],
    order_by=['metrics.test_mae ASC'],
)

print(f"
Experiment comparison (sorted by test MAE):")
print(f"  {'Run name':<30} {'test_mae':>10} {'test_r2':>8} {'train_time':>12}")
print("  " + "─" * 62)
for run in runs[:5]:
    name      = run.data.tags.get('mlflow.runName', run.info.run_id[:8])
    test_mae  = run.data.metrics.get('test_mae', float('nan'))
    test_r2   = run.data.metrics.get('test_r2',  float('nan'))
    train_t   = run.data.metrics.get('train_time_s', float('nan'))
    print(f"  {name:<30} {test_mae:>10.4f} {test_r2:>8.4f} {train_t:>12.1f}s")

import mlflow
from mlflow.tracking import MlflowClient

mlflow.set_tracking_uri('file:///tmp/mlruns')
client = MlflowClient()

MODEL_NAME = 'delivery-time-GradientBoosting'

# ── Register a model from a run ───────────────────────────────────────
# (Already done via registered_model_name in log_model() above)
# Manual registration from a run:
# result = mlflow.register_model(
#     model_uri=f"runs:/{run_id}/model",
#     name=MODEL_NAME,
# )

# ── List all versions of a model ──────────────────────────────────────
try:
    versions = client.search_model_versions(f"name='{MODEL_NAME}'")
    print(f"Model versions for '{MODEL_NAME}':")
    for v in versions:
        print(f"  Version {v.version}: stage={v.current_stage}  "
              f"run_id={v.run_id[:8]}  status={v.status}")
except Exception:
    print(f"Model '{MODEL_NAME}' not yet registered (run experiments first)")

# ── Promote a version through stages ─────────────────────────────────
def promote_model(model_name: str, version: int,
                   stage: str, comment: str = ''):
    """
    Promote a model version to a new stage.
    stage: 'Staging', 'Production', or 'Archived'
    """
    client.transition_model_version_stage(
        name=model_name,
        version=version,
        stage=stage,
        archive_existing_versions=(stage == 'Production'),
    )
    if comment:
        client.update_model_version(
            name=model_name, version=version, description=comment,
        )
    print(f"  Model {model_name} v{version} → {stage}")

# promote_model(MODEL_NAME, version=1, stage='Staging',
#               comment='Passed automated quality gate: test_mae=5.8')
# promote_model(MODEL_NAME, version=1, stage='Production',
#               comment='Approved by ML lead. Replacing v0.')

# ── Load production model in inference service ─────────────────────────
def load_production_model(model_name: str):
    """
    Load the current Production-stage model.
    This is how the inference service should load models —
    always by name and stage, never by run_id.
    """
    model_uri = f"models:/{model_name}/Production"
    # model = mlflow.sklearn.load_model(model_uri)
    # return model
    print(f"  Loading: {model_uri}")
    print(f"  (In production: mlflow.sklearn.load_model('{model_uri}'))")

load_production_model(MODEL_NAME)

# ── Automated promotion pipeline ──────────────────────────────────────
def auto_promote_if_better(model_name: str, new_run_id: str,
                             metric: str = 'test_mae',
                             better_fn = min) -> bool:
    """
    Compare new model to current production model.
    Promote to staging if better, reject otherwise.
    """
    # Get new model metrics
    new_run   = client.get_run(new_run_id)
    new_value = new_run.data.metrics.get(metric)
    if new_value is None:
        print(f"  New run missing metric '{metric}'")
        return False

    # Get current production metrics
    try:
        prod_versions = [v for v in client.search_model_versions(f"name='{model_name}'")
                          if v.current_stage == 'Production']
        if prod_versions:
            prod_run_id = prod_versions[0].run_id
            prod_run    = client.get_run(prod_run_id)
            prod_value  = prod_run.data.metrics.get(metric, float('inf'))
            is_better   = better_fn(new_value, prod_value) == new_value
            print(f"  New {metric}={new_value:.4f} vs Production {metric}={prod_value:.4f}: "
                  f"{'✓ better' if is_better else '✗ worse'}")
            return is_better
        else:
            print(f"  No current production model — promoting as first production version")
            return True
    except Exception:
        return True

print("
Automated promotion check:")
if run_ids.get('gb'):
    result = auto_promote_if_better(MODEL_NAME, run_ids['gb'])
    print(f"  Promote to staging: {result}")

# pip install wandb torch torchvision

import wandb
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np

# ── W&B quickstart — three lines to add full tracking ─────────────────
# wandb.login()   # first time: paste API key from wandb.ai/authorize

# ── Full W&B training loop for a neural network ───────────────────────
def train_with_wandb(config: dict = None):
    """
    Train a model with full W&B experiment tracking.
    Pass config dict or use wandb.config for hyperparameter sweeps.
    """
    run = wandb.init(
        project='doordash-delivery-prediction',
        name=f"mlp-lr{config.get('lr', 0.001):.0e}",
        config=config or {},
        tags=['neural-network', 'delivery-time'],
        notes='MLP baseline for delivery time regression',
        # group='experiment-batch-1',   # group related runs
    )

    cfg = wandb.config   # use config from W&B for sweep support

    # Synthetic data
    np.random.seed(42)
    n = 2000
    X = torch.randn(n, 7)
    y = (X[:, 2].abs() * 6 + X[:, 0] * 3 + torch.randn(n) * 5).unsqueeze(1)

    X_train, X_val = X[:1600], X[1600:]
    y_train, y_val = y[:1600], y[1600:]

    # Model
    model = nn.Sequential(
        nn.Linear(7, 64), nn.ReLU(), nn.Dropout(0.2),
        nn.Linear(64, 32), nn.ReLU(),
        nn.Linear(32, 1),
    )

    # W&B: watch model — logs gradients and parameters every N steps
    wandb.watch(model, log='all', log_freq=50)

    optimizer = optim.Adam(model.parameters(), lr=cfg.lr)
    criterion = nn.MSELoss()

    best_val_loss = float('inf')
    for epoch in range(cfg.epochs):
        model.train()
        optimizer.zero_grad()
        pred      = model(X_train)
        train_loss = criterion(pred, y_train)
        train_loss.backward()
        optimizer.step()

        model.eval()
        with torch.no_grad():
            val_pred = model(X_val)
            val_loss = criterion(val_pred, y_val)
            val_mae  = (val_pred - y_val).abs().mean().item()

        # W&B: log metrics every epoch
        wandb.log({
            'epoch':      epoch,
            'train_loss': train_loss.item(),
            'val_loss':   val_loss.item(),
            'val_mae':    val_mae,
            'lr':         optimizer.param_groups[0]['lr'],
        })

        if val_loss.item() < best_val_loss:
            best_val_loss = val_loss.item()
            # W&B: save best model as artifact
            torch.save(model.state_dict(), '/tmp/best_model.pt')
            artifact = wandb.Artifact(
                name='delivery-time-mlp',
                type='model',
                description=f'Best model at epoch {epoch}, val_mae={val_mae:.3f}',
                metadata={'val_mae': val_mae, 'epoch': epoch},
            )
            artifact.add_file('/tmp/best_model.pt')
            run.log_artifact(artifact)

    wandb.summary['best_val_mae'] = val_mae
    wandb.finish()
    return best_val_loss

print("""
# Run a single experiment:
train_with_wandb(config={'lr': 0.001, 'epochs': 50})

# W&B automatically creates:
# - Learning curve plots (train/val loss over epochs)
# - Gradient flow visualisations
# - System metrics (GPU utilisation, memory)
# - Model artifact with version history
# - Shareable run URL for team collaboration
""")

# ── W&B Sweeps — automated hyperparameter search ──────────────────────
SWEEP_CONFIG = {
    'method': 'bayes',   # bayesian optimisation, or 'random', 'grid'
    'metric': {
        'name':  'val_mae',
        'goal':  'minimize',
    },
    'parameters': {
        'lr':     {'distribution': 'log_uniform_values', 'min': 1e-4, 'max': 1e-2},
        'epochs': {'values': [30, 50, 100]},
    },
    'early_terminate': {
        'type':   'hyperband',
        'min_iter': 10,
    },
}

print("W&B Sweep configuration:")
print(f"  Method:     {SWEEP_CONFIG['method']} (bayesian optimisation)")
print(f"  Metric:     minimize val_mae")
print(f"  LR range:   1e-4 to 1e-2 (log scale)")
print(f"  Epochs:     30, 50, or 100")
print("""
# Start sweep:
# sweep_id = wandb.sweep(SWEEP_CONFIG, project='doordash-delivery-prediction')
# wandb.agent(sweep_id, function=train_with_wandb, count=20)
# → Runs 20 experiments, bayesian-optimises hyperparameters
# → Best config highlighted automatically in W&B dashboard
""")

# ── MLflow vs W&B comparison ──────────────────────────────────────────
print("MLflow vs W&B — when to use each:")
comparison = [
    ('Setup',           'pip install + local server',  'pip install + wandb.login()'),
    ('Hosting',         'Self-hosted or managed',       'Cloud (free tier available)'),
    ('Model registry',  'Built-in, enterprise-ready',  'W&B Artifacts (less mature)'),
    ('Training curves', 'Basic charts',                'Rich interactive charts'),
    ('Image logging',   'As files only',               'First-class: wandb.Image()'),
    ('Sweeps/HPO',      'External (Optuna)',            'Built-in sweep agent'),
    ('Team features',   'Limited',                     'Reports, alerts, comments'),
    ('Best for',        'Classical ML, prod registry', 'Deep learning, research'),
    ('Cost at scale',   'Free (self-hosted)',           'Paid above free tier'),
]
print(f"  {'Dimension':<18} {'MLflow':>30} {'W&B':>30}")
print("  " + "─" * 80)
for dim, mlf, wb in comparison:
    print(f"  {dim:<18} {mlf:>30} {wb:>30}")

import mlflow
import os
from functools import wraps
from datetime import datetime

mlflow.set_tracking_uri('file:///tmp/mlruns')

# ── Standardised experiment wrapper ───────────────────────────────────
# Define once, use everywhere — prevents inconsistent logging

class MLExperiment:
    """
    Standardised experiment context manager.
    Ensures every run logs the same baseline metadata.
    Enforced across the team via shared utility library.
    """

    REQUIRED_PARAMS = ['model_type', 'dataset_version', 'feature_set']
    REQUIRED_TAGS   = ['team', 'purpose']   # 'purpose': baseline, ablation, production-candidate

    def __init__(self, experiment_name: str, run_name: str,
                  params: dict, tags: dict):
        # Validate required fields
        missing_params = [p for p in self.REQUIRED_PARAMS if p not in params]
        missing_tags   = [t for t in self.REQUIRED_TAGS   if t not in tags]
        if missing_params:
            raise ValueError(f"Missing required params: {missing_params}")
        if missing_tags:
            raise ValueError(f"Missing required tags: {missing_tags}")

        self.experiment_name = experiment_name
        self.run_name        = run_name
        self.params          = params
        self.tags            = tags
        self.run             = None

    def __enter__(self):
        mlflow.set_experiment(self.experiment_name)
        self.run = mlflow.start_run(run_name=self.run_name)

        # Log all params
        mlflow.log_params(self.params)

        # Log all tags + standard metadata
        for k, v in self.tags.items():
            mlflow.set_tag(k, str(v))
        mlflow.set_tag('git_commit',   os.environ.get('GIT_COMMIT', 'unknown'))
        mlflow.set_tag('run_by',       os.environ.get('USER', 'unknown'))
        mlflow.set_tag('timestamp',    datetime.now().isoformat())
        mlflow.set_tag('python_env',   os.environ.get('CONDA_ENV', 'unknown'))

        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        if exc_type is not None:
            mlflow.set_tag('status', 'failed')
            mlflow.set_tag('error',  str(exc_val))
        else:
            mlflow.set_tag('status', 'success')
        mlflow.end_run()

    def log_metric(self, key: str, value: float, step: int = None):
        mlflow.log_metric(key, value, step=step)

    def log_metrics(self, metrics: dict, step: int = None):
        mlflow.log_metrics(metrics, step=step)

    def log_model(self, model, artifact_path: str = 'model', **kwargs):
        mlflow.sklearn.log_model(model, artifact_path=artifact_path, **kwargs)

# ── Example: standardised experiment usage ────────────────────────────
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import mean_absolute_error
import numpy as np

np.random.seed(42)
X = np.random.randn(500, 5)
y = X[:, 0] * 3 + np.random.randn(500)

try:
    with MLExperiment(
        experiment_name='fraud-detection',
        run_name='gb-feature-set-v3',
        params={
            'model_type':      'GradientBoosting',
            'dataset_version': 'v3.2',
            'feature_set':     'v3',
            'n_estimators':    200,
            'max_depth':       4,
            'learning_rate':   0.05,
        },
        tags={
            'team':    'fraud-ml',
            'purpose': 'production-candidate',
            'ticket':  'ML-247',
        },
    ) as exp:
        model = GradientBoostingRegressor(
            n_estimators=200, max_depth=4,
            learning_rate=0.05, random_state=42,
        )
        model.fit(X[:400], y[:400])
        val_mae = mean_absolute_error(y[400:], model.predict(X[400:]))

        exp.log_metrics({'val_mae': val_mae, 'train_samples': 400})
        exp.log_model(model, registered_model_name='fraud-score')

    print(f"Experiment logged successfully: val_mae={val_mae:.4f}")

except ValueError as e:
    print(f"Experiment blocked: {e}")

# ── CI-triggered training run — the pattern behind "tracking as policy" ──
# .github/workflows/train-and-gate.yml (abridged)

name: train-and-gate
on:
  pull_request:
    paths: ['model_training/**', 'features/**']

jobs:
  train:
    runs-on: ubuntu-latest
    container: ml-platform/training-image:pinned-v14   # pinned env — reproducibility
    steps:
      - uses: actions/checkout@v4

      - name: Run training with tracking
        env:
          MLFLOW_TRACKING_URI: https://mlflow.internal.company.com
        run: |
          python train.py \
            --experiment "fraud-detection" \
            --run-name "pr-${{ github.event.number }}-${{ github.sha }}" \
            --git-commit "${{ github.sha }}"
          # train.py logs params/metrics/model via mlflow as a normal side
          # effect of running — nobody has to remember to track anything

      - name: Gate on champion metric
        run: |
          python compare_to_production.py \
            --new-run-tag "${{ github.sha }}" \
            --metric val_mae --require-improvement 0.0
          # Fails the CI check (blocks merge) if the new run does not
          # match or beat the current Production-stage model's metric

      - name: Post run link to PR
        run: python post_pr_comment.py --run-tag "${{ github.sha }}"
          # Comments the MLflow run URL + metric diff directly on the PR
          # so reviewers see experiment results without leaving GitHub

A metric number by itself is close to useless — 'val_mae was 5.8' tells you nothing actionable unless it is paired with exactly what produced it: the hyperparameters, the model type, the data version, and ideally the code version. The four categories this module opened with — parameters, metrics, artifacts, tags — exist together specifically because metrics alone cannot answer the question that actually matters: what should change to get a better number next time. A tracking setup that logs only metrics has built a scoreboard, not an experiment record.

Tracking and version control solve different problems and neither substitutes for the other. Git records how the training code itself evolved over time — every change, who made it, and why. An experiment tracking run records what happened in one specific execution of that code — which parameters, which metrics, which resulting artifact. A run is only actually reproducible if it also logged which git commit produced it, which is why the standardised wrapper pattern in this module logs git_commit as a tag on every run. Skip that link and a perfectly detailed MLflow run becomes unreproducible the moment the training script changes again.

Matching parameters and metrics is necessary but not sufficient. True reproducibility also depends on the exact data snapshot used (not just 'dataset_version=v3' as a label, but the actual rows, which can drift if the underlying table is mutable), the library versions in the environment (a scikit-learn point release can change a default and shift results slightly), every random seed across NumPy, the framework, and the language's own random module, and sometimes hardware-level non-determinism in GPU operations. Logging only hyperparameters and a final metric captures the easy twenty percent of what reproducibility actually requires.

For a team past a couple of people, the tracking server is often the only shared source of truth for what has already been tried — without it, two engineers routinely burn compute re-running an experiment a teammate already tried and rejected three weeks earlier, because that knowledge lived only in their head or a Slack message nobody can find again. In regulated industries it is stronger than a convenience: being able to show exactly which data, code, and parameters produced a production model on demand is frequently an actual compliance requirement, not an engineering nicety a team can choose to skip under deadline pressure.

A metric is only comparable across runs if the evaluation conditions were actually the same — the same test split, the same preprocessing, the same evaluation window of data. A run that scores better because it was accidentally evaluated on an easier slice of data, an older test set before a distribution shift, or with subtle data leakage in its features will show a higher number on the dashboard while being a genuinely worse model in production. Comparing experiments fairly means checking that the comparison itself is apples to apples before trusting the ranking, not just sorting the experiment table by the metric column and promoting whichever run sits on top.

More than hyperparameters and a final metric. You need the exact data version or snapshot used (ideally a hash or a pointer to an immutable dataset version, not a mutable table name), the code version — a git commit hash logged as a tag — the full library and environment versions the run executed under, and every random seed involved across NumPy, the deep learning framework, and Python's own random module. Miss the data version and the 'same' training code can silently train on different rows six months later; miss the environment version and a library's changed default can shift results even with identical code and data.

First, confirm they were evaluated under identical conditions — same test split, same preprocessing pipeline, same evaluation window of data — because a metric computed on a different slice is not actually comparable no matter how similar the number looks. Second, standardise what gets logged across the team (the required-params-and-tags pattern this module covers) so every run records dataset_version and feature_set consistently, making it possible to filter to only genuinely comparable runs in the first place. Third, look beyond the single headline metric — cross-validation standard deviation, overfitting gap between train and validation, and inference latency all matter for a real promotion decision, not just whichever run has the lowest MAE on the dashboard.

Because a model is a function of far more than its hyperparameters — it is also a function of exactly which data rows it saw, in what order, under which library versions, and with which random seeds. Two runs with identical logged hyperparameters can produce meaningfully different models if the underlying data table was mutated between them, if a dependency was silently upgraded, or if a seed was never set for one of several independent random number generators a typical training script touches. Genuine reproducibility means treating data version and environment version as first-class logged fields, the same as any hyperparameter, not as background assumptions that are 'probably fine.'

The training job itself runs inside CI, triggered by a pull request or a schedule, inside a pinned container image so the environment is captured by construction rather than logged after the fact. The training script logs to the shared tracking server as a normal part of running, tagged with the triggering commit hash. A comparison step then checks the new run's key metric against the current Production-stage model and can block the merge or the deploy if it does not improve or at least hold steady. The result is that every model that ever reaches production has a CI-generated tracking run behind it automatically, with no reliance on an engineer remembering to log anything by hand.

I would not try to retroactively log the 200 old runs — that effort rarely pays for itself. Instead I would start requiring tracking only for new work going forward, beginning with a lightweight shared wrapper that enforces a small set of required tags and params so adoption has almost no friction on day one. I would pick one active, valuable experiment stream to migrate first as a proof of the workflow, then wire the CI gate in once the team trusts the tooling rather than mandating it everywhere at once. The goal is making tracking the path of least resistance — logging a run should be easier than not logging one — rather than treating it as an audit requirement bolted on top of how the team already works.