DVC — Data Version Control

Git is like a library catalogue — it tracks which books exist and where they are filed, but the books themselves are stored on shelves. DVC is the cataloguing system for your ML datasets — it tracks which version of your data exists and stores a reference in Git, while the actual data lives in a remote storage warehouse (S3). When you need the book (data), you check the catalogue (Git + DVC), find the shelf (S3 path), and retrieve it. The catalogue is tiny. The warehouse can hold terabytes.

The .dvc pointer file committed to Git is typically 200 bytes. The actual dataset it references can be 200GB. Git stores the pointer. S3 stores the data. DVC coordinates between them so every git checkout brings the right data version automatically.

# ── DVC project setup from scratch ───────────────────────────────────

# 1. Initialise Git and DVC in the same directory
git init
dvc init
# Creates: .dvc/ directory, .dvcignore, .dvc/config

# 2. Configure remote storage (S3 example)
dvc remote add -d myremote s3://company-ml-data/dvc-store
# -d = set as default remote
# For GCS: dvc remote add -d myremote gs://bucket/dvc-store
# For local (testing): dvc remote add -d myremote /tmp/dvc-remote

# 3. Track a large data file
# This MOVES the file to the DVC cache and creates a pointer
dvc add data/train.csv
# Creates:  data/train.csv.dvc   (tiny pointer file — commit this to Git)
# Adds:     data/train.csv to .gitignore  (never commit the actual data)
# Caches:   .dvc/cache/a3/f2b1c4...  (the actual file content)

cat data/train.csv.dvc
# outs:
# - md5: a3f2b1c4d5e6f7a8b9c0d1e2f3a4b5c6
#   size: 245678901
#   path: train.csv

# 4. Commit the pointer file to Git
git add data/train.csv.dvc data/.gitignore dvc.yaml
git commit -m "add training dataset v1.0"

# 5. Push data to remote storage
dvc push
# Uploads .dvc/cache/a3/f2b1c4... to s3://company-ml-data/dvc-store/a3/f2b1c4...

# ── Typical workflow for a team member ────────────────────────────────
# Clone the repo and get data:
git clone https://github.com/company/ml-project.git
cd ml-project
dvc pull              # downloads data matching current Git commit
# Downloads: data/train.csv from S3 to local workspace

# ── Updating the dataset ──────────────────────────────────────────────
# After adding new data to train.csv:
dvc add data/train.csv          # rehash, update .dvc file
git add data/train.csv.dvc
git commit -m "update training data: added March 2026 orders"
dvc push                        # upload new version to S3

# ── Reverting to a previous data version ──────────────────────────────
git checkout abc1234            # checkout old code commit
dvc checkout                    # restore data version for that commit
# DVC reads the .dvc pointer from that commit, fetches from cache/S3

# ── dvc.yaml — complete ML pipeline ──────────────────────────────────
# This file lives in Git alongside your code

DVCYAML = """
stages:

  prepare_data:
    cmd: python src/prepare.py --input data/raw/orders.csv --output data/processed/
    deps:
      - data/raw/orders.csv
      - src/prepare.py
    outs:
      - data/processed/train.parquet
      - data/processed/val.parquet
      - data/processed/test.parquet
    params:
      - params.yaml:
          - prepare.train_fraction
          - prepare.val_fraction
          - prepare.cutoff_date

  featurise:
    cmd: python src/featurise.py
    deps:
      - data/processed/train.parquet
      - data/processed/val.parquet
      - src/featurise.py
    outs:
      - data/features/train_features.parquet
      - data/features/val_features.parquet

  train:
    cmd: python src/train.py
    deps:
      - data/features/train_features.parquet
      - data/features/val_features.parquet
      - src/train.py
    params:
      - params.yaml:
          - train.n_estimators
          - train.max_depth
          - train.learning_rate
    outs:
      - models/model.pkl
    metrics:
      - metrics/train_metrics.json:
          cache: false           # metrics stored in Git, not DVC cache

  evaluate:
    cmd: python src/evaluate.py
    deps:
      - models/model.pkl
      - data/processed/test.parquet
      - src/evaluate.py
    metrics:
      - metrics/eval_metrics.json:
          cache: false
    plots:
      - plots/confusion_matrix.csv:
          cache: false
"""

# ── params.yaml — hyperparameters tracked alongside data ──────────────
PARAMSYAML = """
prepare:
  train_fraction: 0.70
  val_fraction:   0.15
  cutoff_date:    "2026-03-01"

train:
  n_estimators: 200
  max_depth:    4
  learning_rate: 0.05
  random_seed:  42
"""

print("DVC pipeline commands:")
commands = [
    ('dvc repro',
     'Run all stages where deps have changed. Skip unchanged stages.'),
    ('dvc repro train',
     'Run only the train stage (and its unmet dependencies).'),
    ('dvc repro --force',
     'Force rerun all stages regardless of what changed.'),
    ('dvc dag',
     'Print ASCII diagram of the pipeline DAG.'),
    ('dvc status',
     'Show which stages are changed/outdated vs dvc.lock.'),
    ('dvc params diff',
     'Show parameter changes between current and last run.'),
    ('dvc metrics show',
     'Show metrics from metrics/*.json files.'),
    ('dvc metrics diff',
     'Compare metrics between two Git commits.'),
    ('dvc plots show plots/confusion_matrix.csv',
     'Open interactive plot in browser.'),
]
for cmd, desc in commands:
    print(f"  $ {cmd}")
    print(f"    {desc}")

# src/prepare.py — data preparation stage
import pandas as pd
import numpy as np
import yaml, os, argparse
from pathlib import Path

def prepare(input_path: str, output_dir: str, params: dict):
    """Load raw data, split by date, save processed splits."""
    print(f"Loading raw data from {input_path}")
    # In production: pd.read_csv(input_path) or query from warehouse
    np.random.seed(42)
    n = 10000
    df = pd.DataFrame({
        'order_id':       [f'ORD{i:05d}' for i in range(n)],
        'order_date':     pd.date_range('2026-01-01', periods=n, freq='1h'),
        'distance_km':    np.random.exponential(3.5, n),
        'is_peak_hour':   np.random.randint(0, 2, n),
        'order_value':    np.random.exponential(400, n),
        'restaurant_avg': np.random.normal(35, 8, n),
        'driver_avg':     np.random.normal(32, 7, n),
        'actual_time':    np.clip(
            np.random.exponential(3.5, n)*5.5 + np.random.randint(0,2,n)*7 +
            np.random.normal(0,5,n), 10, 90,
        ),
    })

    cutoff = pd.Timestamp(params['cutoff_date'])
    df_before = df[df['order_date'] < cutoff]

    train_frac = params['train_fraction']
    val_frac   = params['val_fraction']
    n_train    = int(len(df_before) * train_frac)
    n_val      = int(len(df_before) * val_frac)

    train = df_before.iloc[:n_train]
    val   = df_before.iloc[n_train:n_train+n_val]
    test  = df_before.iloc[n_train+n_val:]

    Path(output_dir).mkdir(parents=True, exist_ok=True)
    train.to_parquet(f'{output_dir}/train.parquet', index=False)
    val.to_parquet(f'{output_dir}/val.parquet', index=False)
    test.to_parquet(f'{output_dir}/test.parquet', index=False)

    print(f"  Train: {len(train):,}  Val: {len(val):,}  Test: {len(test):,}")

# ────────────────────────────────────────────────────────────────────────
# src/train.py — training stage
import pandas as pd, numpy as np, pickle, json, yaml
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import mean_absolute_error
from pathlib import Path

def train(features_dir: str, model_dir: str, metrics_dir: str, params: dict):
    """Train model, save artifact, log training metrics."""
    X_train = pd.read_parquet(f'{features_dir}/train_features.parquet')
    X_val   = pd.read_parquet(f'{features_dir}/val_features.parquet')

    feature_cols = [c for c in X_train.columns if c != 'actual_time']
    y_train = X_train['actual_time']
    y_val   = X_val['actual_time']

    model = GradientBoostingRegressor(
        n_estimators=params['n_estimators'],
        max_depth=params['max_depth'],
        learning_rate=params['learning_rate'],
        random_state=params.get('random_seed', 42),
    )
    model.fit(X_train[feature_cols], y_train)

    train_mae = mean_absolute_error(y_train, model.predict(X_train[feature_cols]))
    val_mae   = mean_absolute_error(y_val,   model.predict(X_val[feature_cols]))

    Path(model_dir).mkdir(parents=True, exist_ok=True)
    Path(metrics_dir).mkdir(parents=True, exist_ok=True)

    with open(f'{model_dir}/model.pkl', 'wb') as f:
        pickle.dump({'model': model, 'feature_cols': feature_cols}, f)

    # Metrics written to JSON — tracked by DVC metrics, not cached
    metrics = {'train_mae': round(train_mae, 4), 'val_mae': round(val_mae, 4)}
    with open(f'{metrics_dir}/train_metrics.json', 'w') as f:
        json.dump(metrics, f, indent=2)

    print(f"  Train MAE: {train_mae:.4f}  Val MAE: {val_mae:.4f}")
    return metrics

# ── Simulate running the pipeline stages ─────────────────────────────
with open('/tmp/params.yaml', 'w') as f:
    import yaml
    yaml.dump({
        'prepare': {
            'train_fraction': 0.70, 'val_fraction': 0.15,
            'cutoff_date': '2026-03-01',
        },
        'train': {
            'n_estimators': 200, 'max_depth': 4,
            'learning_rate': 0.05, 'random_seed': 42,
        },
    }, f)

with open('/tmp/params.yaml') as f:
    params = yaml.safe_load(f)

print("Running DVC pipeline stages:")
print("Stage 1: prepare_data")
prepare('/tmp/raw.csv', '/tmp/processed', params['prepare'])

print("
Stage 2: train (simulated with direct call)")
# In real DVC: this runs via dvc repro

# ── DVC experiment workflow ───────────────────────────────────────────

# Run baseline (current params.yaml)
# dvc repro
# git add dvc.lock metrics/ && git commit -m "baseline: lr=0.05 n_est=200"

# Run experiment with modified hyperparameters
# dvc exp run --set-param train.learning_rate=0.01
# dvc exp run --set-param train.n_estimators=300
# dvc exp run --set-param train.learning_rate=0.1 --set-param train.max_depth=6

# Compare all experiments
# dvc exp show
# Output:
# ┌────────────────┬──────────┬──────────┬──────────────┬───────────┐
# │ Experiment     │ train_mae│ val_mae  │ learning_rate│ n_estimators│
# ├────────────────┼──────────┼──────────┼──────────────┼───────────┤
# │ workspace      │ 4.2103   │ 5.8241   │ 0.05         │ 200       │
# │ exp-abc123     │ 4.8912   │ 6.1034   │ 0.01         │ 200       │
# │ exp-def456     │ 3.9201   │ 5.7123   │ 0.05         │ 300  ← best│
# │ exp-ghi789     │ 3.5102   │ 6.3401   │ 0.10         │ 200 ← overfit│
# └────────────────┴──────────┴──────────┴──────────────┴───────────┘

# Promote the best experiment to a Git commit
# dvc exp apply exp-def456   # update workspace to best experiment
# git add .
# git commit -m "best: n_estimators=300 val_mae=5.71"

# ── Python: programmatic experiment comparison ─────────────────────────
import json, yaml
import numpy as np
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import mean_absolute_error

np.random.seed(42)
n = 2000
X_tr = np.random.randn(1600, 5)
y_tr = X_tr[:, 0] * 6 + X_tr[:, 2] * 3 + np.random.randn(1600) * 5
X_val = np.random.randn(400, 5)
y_val = X_val[:, 0] * 6 + X_val[:, 2] * 3 + np.random.randn(400) * 5

experiment_grid = [
    {'n_estimators': 100, 'max_depth': 3, 'learning_rate': 0.10},
    {'n_estimators': 200, 'max_depth': 4, 'learning_rate': 0.05},
    {'n_estimators': 300, 'max_depth': 4, 'learning_rate': 0.05},
    {'n_estimators': 200, 'max_depth': 6, 'learning_rate': 0.05},
    {'n_estimators': 200, 'max_depth': 4, 'learning_rate': 0.01},
]

print(f"{'n_est':>6} {'depth':>6} {'lr':>6} {'train_mae':>10} {'val_mae':>10} {'overfit_gap':>12}")
print("─" * 55)

results = []
for params in experiment_grid:
    model = GradientBoostingRegressor(**params, random_state=42)
    model.fit(X_tr, y_tr)
    train_mae = mean_absolute_error(y_tr, model.predict(X_tr))
    val_mae   = mean_absolute_error(y_val, model.predict(X_val))
    gap       = val_mae - train_mae
    results.append({**params, 'train_mae': train_mae, 'val_mae': val_mae})
    best_mark = ' ←' if val_mae == min(r['val_mae'] for r in results) else ''
    print(f"  {params['n_estimators']:>4}  {params['max_depth']:>5}  "
          f"{params['learning_rate']:>5.2f}  {train_mae:>10.4f}  "
          f"{val_mae:>10.4f}  {gap:>12.4f}{best_mark}")

best = min(results, key=lambda r: r['val_mae'])
print(f"
Best experiment: n_estimators={best['n_estimators']} "
      f"max_depth={best['max_depth']} lr={best['learning_rate']} "
      f"val_mae={best['val_mae']:.4f}")

# ── Complete Git + DVC workflow for a DoorDash ML team ────────────────

# ── Project structure ─────────────────────────────────────────────────
PROJECT_STRUCTURE = """
ml-project/
├── .git/                      ← Git tracks code and pointer files
├── .dvc/
│   ├── config                 ← remote storage URL, cache settings
│   └── cache/                 ← local content-addressable data cache
├── data/
│   ├── raw/
│   │   ├── orders.csv.dvc     ← GIT: pointer to raw data in S3
│   │   └── .gitignore         ← ignores orders.csv itself
│   ├── processed/             ← DVC output: ignored by Git
│   └── features/              ← DVC output: ignored by Git
├── models/
│   └── model.pkl              ← DVC output: ignored by Git
├── metrics/
│   └── eval_metrics.json      ← GIT: small metrics file committed
├── plots/
│   └── feature_importance.csv ← GIT: small plot data committed
├── src/
│   ├── prepare.py             ← GIT: code always in Git
│   ├── featurise.py
│   ├── train.py
│   └── evaluate.py
├── dvc.yaml                   ← GIT: pipeline definition
├── dvc.lock                   ← GIT: locked dep/output hashes after last run
├── params.yaml                ← GIT: hyperparameters
└── requirements.txt           ← GIT: Python dependencies
"""

# ── Day-in-the-life workflow ──────────────────────────────────────────
DAILY_WORKFLOW = """
Morning: Start work on a new feature experiment
──────────────────────────────────────────────────────────────────────

# 1. Get latest code and data
git pull
dvc pull                       # sync any new data versions from S3

# 2. Create a branch for your experiment
git checkout -b experiment/add-weather-features

# 3. Add new raw data (e.g. weather data enrichment)
cp /data/weather_march.csv data/raw/
dvc add data/raw/weather_march.csv
git add data/raw/weather_march.csv.dvc data/raw/.gitignore

# 4. Update the featurise script to use weather data
# ... edit src/featurise.py ...

# 5. Update params for this experiment
# ... edit params.yaml: add weather_features: true ...

# 6. Run the pipeline (only changed stages rerun)
dvc repro
# DVC detects: featurise.py changed + new data → reruns featurise + train + evaluate
# DVC skips:   prepare_data → raw data unchanged

# 7. Check metrics
dvc metrics show
# eval_metrics.json:
#   mae: 5.21   (improved from 5.87!)
#   rmse: 7.43

dvc metrics diff main          # compare to main branch
# Path                   Metric   HEAD        main      Change
# metrics/eval_metrics   mae      5.21        5.87      -0.66 ✓

# 8. Commit and push
git add dvc.lock params.yaml metrics/ src/featurise.py
git commit -m "add weather features: val_mae 5.87 → 5.21"
dvc push                       # push new data version to S3
git push origin experiment/add-weather-features

# 9. Open PR — reviewer can reproduce your experiment exactly:
# git checkout experiment/add-weather-features
# dvc pull
# dvc repro
# → Gets identical metrics

────────────────────────────────────────────────────────────────────
CI/CD pipeline (GitHub Actions) runs on every PR:
────────────────────────────────────────────────────────────────────

# .github/workflows/ml-ci.yml
CI_YAML = """\
name: ML CI
on: [pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: pip install "dvc[s3]" -r requirements.txt
      - run: dvc pull                    # download data from S3
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_KEY }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET }}
      - run: dvc repro                   # run full pipeline
      - run: |
          python -c "
          import json
          with open('metrics/eval_metrics.json') as f:
              m = json.load(f)
          assert m['mae'] < 6.5, f'MAE too high: {m[\"mae\"]}  (threshold: 6.5)'
          print(f'MAE gate passed: {m[\"mae\"]}')
          "
"""

print("Git + DVC command reference:")
cmd_ref = [
    ('dvc init',                    'Initialise DVC in a Git repo'),
    ('dvc remote add -d r s3://…',  'Set default remote storage'),
    ('dvc add file.csv',            'Track a file with DVC'),
    ('dvc repro',                   'Run pipeline (skip unchanged stages)'),
    ('dvc push',                    'Upload data to remote'),
    ('dvc pull',                    'Download data from remote'),
    ('dvc checkout',                'Restore data for current Git commit'),
    ('dvc status',                  'Show what has changed'),
    ('dvc metrics show',            'Print all tracked metrics'),
    ('dvc metrics diff HEAD~1',     'Compare metrics to previous commit'),
    ('dvc dag',                     'Print pipeline DAG'),
    ('dvc exp run --set-param k=v', 'Run experiment with modified param'),
    ('dvc exp show',                'Compare all experiments in a table'),
    ('dvc gc -w',                   'Garbage collect unused cache entries'),
]
for cmd, desc in cmd_ref:
    print(f"  {cmd:<40}  {desc}")

# ── Lineage lookup used during an incident postmortem ─────────────────
# "A bad prediction went out last Tuesday. What data trained that model?"

# 1. Find which model version was serving in production at that time
#    (from the model registry / deployment log)
MODEL_VERSION="v2.3.1"

# 2. Find the Git commit that built that model version
git log --all --grep="model: build v2.3.1" --format="%H %s"
# → a1b2c3d  "model: build v2.3.1 — train_mae=4.21"

# 3. Check out that commit and inspect the .dvc pointer for the training data
git checkout a1b2c3d -- data/processed/train.parquet.dvc
cat data/processed/train.parquet.dvc
# outs:
# - md5: 9f8e7d6c5b4a...
#   size: 184320112
#   path: train.parquet

# 4. Pull that exact data version to inspect it locally
dvc pull data/processed/train.parquet
# Now you have the EXACT bytes that trained the model in production —
# not "roughly the data from that week," the literal file.

# 5. Confirm which raw source rows fed the snapshot (params.yaml, dvc.lock)
dvc dag --outs      # shows: raw.csv → prepare_data → train.parquet → model.pkl
cat dvc.lock         # shows exact hash of raw.csv used to build train.parquet

# Answer: "The model was trained on train.parquet (hash 9f8e7d6c...),
# built from raw.csv (hash 3a2b1c0d...) with a cutoff_date of 2026-03-01."

A zip file with a date in its name records that a version existed, but not what changed between versions, which code commit used it, or how to get back to it programmatically. DVC's pointer file ties a specific data hash to a specific Git commit automatically — checking out an old commit and running dvc checkout restores the exact matching data without anyone needing to remember which zip file went with which experiment. The zip folder approach also duplicates the full dataset for every version; DVC's content-addressable cache stores each unique chunk of data once, no matter how many versions reference it.

Both move large files out of the Git object database, but that is where the similarity ends. Git LFS still requires a Git remote that understands LFS pointers and has no concept of a data pipeline — it cannot express "rerun this stage only if this specific dependency changed." DVC adds pipeline stages (dvc.yaml), parameter tracking, metrics tracked across commits, and an experiment-comparison layer (dvc exp show) on top of the same content-addressable storage idea. Choosing Git LFS over DVC for an ML project usually means rebuilding all of that tooling yourself later, badly.

A backup answers whether you can recover this file if it is deleted. It does not answer which version of this file trained the model that is in production right now, because backups are not linked to the code commit or model artifact that consumed them. Versioning is a linkage problem, not a durability problem — you can have perfect nightly backups of every dataset that ever existed and still have no way to say which one a given model saw. DVC solves the linkage: the .dvc pointer committed alongside the training code is the permanent record of exactly which snapshot produced exactly which model.

DVC versions files, not queryable tables — it has no notion of point-in-time correctness, no SQL interface, and no low-latency lookups for online serving. A feature store (Module 69) solves a different problem: serving consistent, point-in-time-correct feature values to both training and real-time inference. In practice the two work together — the feature store produces a training snapshot, and DVC versions that snapshot as a file so it can be reproduced later. Neither tool replaces the other.

DVC guarantees that the same input files are available for a given commit — it does not guarantee the code that processes them is deterministic. A training script with an unseeded random split, a model that depends on library versions not pinned in requirements.txt, or a feature computation that calls an external API for live values will still produce different results on rerun even with byte-identical data. DVC removes one major source of irreproducibility; seeding randomness and pinning dependencies remove the others, and all three are required together.

A model is a function of three things: the code that trained it, the hyperparameters, and the data. Git already versions the first two. Without versioning the data too, reproducing an experiment from six months ago is impossible even with the exact code and params checked out, because the underlying dataset has likely been overwritten, appended to, or regenerated since. Data versioning closes that gap: given a Git commit, you can recover the exact data that commit was trained and evaluated against, which is what makes a past result auditable, debuggable, and legally defensible in regulated domains like lending or healthcare.

Git LFS solves storage — it keeps large binary files out of the Git object database while still tracking them with Git-like commands. DVC solves storage the same way, using its own remote rather than a Git-LFS-aware host, but it also adds a pipeline layer on top: stages with explicit dependencies and outputs (dvc.yaml), a lockfile that lets dvc repro skip unchanged stages, tracked metrics and parameters comparable across commits, and a lightweight experiment-tracking mode (dvc exp) that runs variations without creating a Git commit for each one. Git LFS is a storage backend; DVC is a storage backend plus an ML workflow tool built on top of it.

Every model artifact should be tagged with the Git commit hash that produced it, ideally embedded directly in the model file or its accompanying metadata. From that commit, git show or git checkout on the .dvc pointer file reveals the MD5 hash and size of every dataset used to build it, and dvc pull retrieves the exact matching bytes from remote storage. In practice I would automate this into a model registry: every model version record stores its Git commit, and a single lookup — commit to .dvc hash to dvc pull — recovers the training data in a couple of commands, which is invaluable during an incident when someone needs to know exactly what a model saw.

DVC uses content-addressable storage: every file is stored in the cache under a path derived from its MD5 hash, not its filename. If a new dataset version changes 5 percent of rows and DVC tracks it as a single file, the whole file gets a new hash and is stored in full — DVC does not do byte-level diffing within a file. The real savings come from identical files being stored exactly once: if two branches or two experiments happen to produce byte-identical outputs, or if a file reverts to a previous exact state, DVC recognizes the matching hash and reuses the existing cached copy instead of storing it again.

First, stop it from getting worse — do not let that commit get merged or pushed further if it can still be avoided, since a large binary blob committed to Git history is expensive to remove later and bloats every future clone. If it has already been pushed, the fix is to run dvc add on the file to move it into DVC-managed storage, commit the resulting small .dvc pointer in place of the raw file, and then rewrite history to strip the large blob from earlier commits using something like git filter-repo, coordinating with the team since that rewrites shared history. Long term, I would add a pre-commit hook or CI check that rejects commits containing files above a size threshold that are not tracked by DVC.