Chapter 5 — Data and Pipeline Versioning

An ML system is the marriage of code, data, environment, and stochastic processes. Each can drift independently and silently invalidate yesterday's results. A reproducible experiment is one where a tuple of (git commit, data version, container image digest, seed/config) uniquely defines the run.

The Four Dimensions of ML Reproducibility

Reproducibility Levels

• Bitwise: identical floats, requires identical hardware + deterministic kernels.
• Numerical: within small tolerance (loss within 1e-4) — feasible same-GPU-family.
• Statistical: distributions match but individual runs differ — fine for research.
• Conceptual: same methodology, reimplemented — academic bar, not production.

PyTorch Determinism Playbook

def set_seed(seed: int):
    os.environ["PYTHONHASHSEED"] = str(seed)
    os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":16:8"
    random.seed(seed); np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.use_deterministic_algorithms(True)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

For DistributedDataParallel, derive each rank's seed as base_seed + rank. None of these flags guarantee bitwise reproducibility across different GPU architectures.

Large datasets do not fit in git, and git's text-oriented diffing makes binary diffs useless. Three dominant patterns: a git-like project tool (DVC), a branchable layer over object storage (lakeFS), and a transactional table format (Delta Lake).

DVC: Git-Like at the Project Level

dvc add computes a content hash, moves bytes into a cache, and writes a tiny .dvc metadata file you commit to git. Actual bytes live in a DVC remote (S3/GCS/Azure). The git history of .dvc files is the git history of your data.

git checkout v1.3-paper-submission
dvc pull          # downloads exact data hashes for this commit
dvc repro         # re-runs the pipeline defined in dvc.yaml

lakeFS: Branchable Over Object Storage

lakeFS versions an entire bucket, exposing S3 (or GCS/Azure) through a versioning layer with branches, commits, and merges. Branches are cheap because of copy-on-write — objects are copied only when modified on the branch.

Delta Lake and Iceberg: ACID Time Travel for Tables

Delta Lake is a transactional table format: Parquet files plus a _delta_log/ directory that records every transaction as JSON or checkpoint entries. Engines read the log to determine which files constitute a given table version.

-- Read the exact features table the model was trained on
SELECT * FROM features.user_features VERSION AS OF 37;
SELECT * FROM features.user_features TIMESTAMP AS OF '2026-05-01 09:00:00';

Tool Comparison

Large organizations often stack all three: lakeFS at the bucket layer, Delta Lake for ACID tables inside it, DVC for project-local slices and models.

Docker as the Environment Unit

A container image encodes the OS, CUDA/cuDNN, Python interpreter, system libs, and all Python dependencies into a single immutable artifact identified by a SHA-256 digest. Two engineers running docker pull myorg/ml@sha256:abc123... execute against byte-identical environments.

FROM nvidia/cuda:12.1.0-cudnn9-runtime-ubuntu22.04
RUN apt-get update && apt-get install -y \
    git python3 python3-pip && \
    rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY requirements.lock /app/
RUN pip install --no-cache-dir -r requirements.lock
COPY . /app
CMD ["python3", "train.py", "--config", "config/exp1.yaml"]

Two rules: (1) log the image digest, never the tag; (2) treat the image as immutable — rebuild and re-tag instead of docker exec-ing fixes.

Lockfile Tools

• pip-tools (pip-compile) produces a fully pinned requirements.txt from requirements.in, with transitive deps + hashes.
• Poetry resolves pyproject.toml into poetry.lock with exact versions and content hashes.
• conda-lock generates platform-specific lockfiles from environment.yml, capturing non-Python packages too.

Pipeline-as-Code

Pipeline definitions belong in version-controlled source files, not in a UI someone clicked six months ago. Forms include Airflow DAGs (Python), Kubeflow/Argo (YAML), DVC pipelines (dvc.yaml), and Python-native frameworks (Dagster, Prefect, Metaflow).

stages:
  prepare_features:
    cmd: python src/features.py --input data/raw --output data/features
    deps: [data/raw, src/features.py]
    outs: [data/features]
  train:
    cmd: python src/train.py --features data/features --model models/churn.pt
    deps: [data/features, src/train.py]
    outs: [models/churn.pt]
    metrics:
      - metrics.json:
          cache: false

Compute Environment Metadata

Beyond the image digest, mature teams log runtime configuration alongside each experiment so a run is auditable months later:

{
  "image_digest": "sha256:abc123...",
  "git_sha": "f4a2c8e",
  "data_dvc_hash": "md5:9f1ec4...",
  "gpu_model": "NVIDIA A100-SXM4-80GB",
  "cuda_version": "12.1",
  "driver_version": "535.104.05",
  "seed": 1234
}

Data lineage is the graph that ties raw sources to features to models to predictions. It is the difference between "this model was trained on user data" and "this exact run, with this commit SHA, on these specific Delta table versions, produced this artifact."

OpenLineage Data Model

• Job: a logical unit of work (Airflow task, Spark job, dbt model). Identified by name + namespace.
• Run: a single execution of a Job, identified by a runId (UUID).
• Dataset: a logical dataset read or written — tables, files, streams, model artifacts.
• Event: START / COMPLETE / FAIL JSON messages with inputs, outputs, and extensible facets.

Facets carry rich metadata: schema, columnLineage, dataQualityMetrics, errorMessage, sourceCode, sql, plus custom ML facets for hyperparameters.

Auto-Integrations

• Airflow: OpenLineage provider hooks task-lifecycle callbacks; inspects operators to infer inputs/outputs.
• Spark: A listener jar attaches to logical/physical plans; emits jobs + datasets with column-level lineage where derivable.
• dbt: A plugin reads manifest.json + run_results.json for rich column-level lineage from compiled SQL.

End-to-End Lineage Graph

In Marquez (the reference open-source store/UI), start at predictions.churn_scores and walk upstream through the model artifact, feature table, dbt staging model, all the way to raw events. Every node carries run history, schema, and facets.

Compliance Queries Become Graph Walks

• "Show all models trained on PII-tagged datasets" → filter by sensitivity: PII, walk downstream.
• "User X revoked consent — which models need retraining?" → identify their datasets, walk forward to models.
• "Prove this credit model didn't use applicant_race" → column-level lineage from training inputs.

Debugging via Lineage

Churn model AUC drops 0.80 → 0.72 overnight. Walk: open latest training run, inspect facets (unchanged), walk upstream to features.user_features (row count down 30%), walk upstream to stg.events (schema-mismatch error in dbt). Fix once, re-run, verify the downstream graph turns green.