Chapter 7: MLOps and ML Pipelines on Kubernetes

Section 1: ML Pipeline Orchestration

An ML pipeline is a directed acyclic graph (DAG) of computational steps. Each node is a containerized task; edges carry data artifacts or parameter values from one task to the next. Getting this orchestration right on Kubernetes is the foundation of production MLOps.

Kubeflow Pipelines v2 Architecture and SDK

Kubeflow Pipelines (KFP) provides a Python SDK for defining pipelines plus backend services for running them on any Kubernetes cluster. KFP v2 introduced major improvements over v1:

The @dsl.component decorator converts a plain Python function into a self-contained pipeline step that KFP can containerize and schedule. The @dsl.container_component decorator gives precise control over the container command — useful when wrapping non-Python tools or shell scripts.

A Three-Step Training Pipeline

from kfp import dsl
from kfp.dsl import Dataset, Model, Output, Input

@dsl.component(base_image="python:3.11-slim",
               packages_to_install=["pandas", "scikit-learn"])
def preprocess(raw_data_path: str, dataset: Output[Dataset]):
    import pandas as pd
    df = pd.read_csv(raw_data_path).dropna()
    df.to_csv(dataset.path, index=False)

@dsl.component(base_image="python:3.11-slim",
               packages_to_install=["pandas", "scikit-learn"])
def train(dataset: Input[Dataset], model: Output[Model],
          n_estimators: int = 100):
    import pandas as pd
    from sklearn.ensemble import RandomForestClassifier
    import joblib
    df = pd.read_csv(dataset.path)
    X, y = df.drop("label", axis=1), df["label"]
    clf = RandomForestClassifier(n_estimators=n_estimators)
    clf.fit(X, y)
    joblib.dump(clf, model.path)

@dsl.pipeline(name="random-forest-pipeline")
def rf_pipeline(data_path: str, n_estimators: int = 100):
    prep = preprocess(raw_data_path=data_path)
    fit  = train(dataset=prep.outputs["dataset"],
                 n_estimators=n_estimators)
    evaluate(dataset=prep.outputs["dataset"],
             model=fit.outputs["model"])

Notice how Dataset and Model types are declared as Output and Input parameters. KFP v2 surfaces these as first-class artifact nodes in the pipeline visualization, so engineers can trace exactly which model artifact came from which training run.

Argo Workflows for ML DAGs

KFP compiles Python pipelines into an intermediate representation executed by Argo Workflows as the default backend. Argo can also be used directly — teams working in multiple languages or wrapping legacy tools often prefer writing Argo YAML. Argo provides DAG-based execution, parameter passing, S3/GCS artifact management, and configurable retry policies.

Tekton Pipelines for ML CI/CD

Tekton occupies a different niche: it is a general-purpose CI/CD framework on Kubernetes. Its strength in MLOps is in the CI/CD layer — building container images, running linting and unit tests before training, and triggering downstream deployments after model validation. Tekton integrates naturally with ArgoCD to create a full GitOps deployment chain.

Pipeline Parameterization and Caching

Effective pipelines treat all tunable values as explicit parameters. This enables:

• Hyperparameter sweeps: run the same pipeline with different learning rates or batch sizes
• Environment promotion: swap a data_path parameter to move from staging to production data
• Reproducibility auditing: every run records its exact parameter set for recreation on demand

Pipeline caching is a standout feature: when a component's inputs (parameter values and artifact content hashes) match a previous execution, KFP reuses the cached output. Change n_estimators and only train and evaluate re-execute; preprocess returns instantly from cache.

Section 2: Experiment Tracking and Model Registry

MLflow on Kubernetes

Every training run produces metrics, parameters, and artifacts. Without a system to capture them, teams lose track of which configuration produced which result. MLflow is the most widely adopted open-source solution, deployed on Kubernetes as a three-tier architecture:

import mlflow

mlflow.set_tracking_uri(
    "http://mlflow-service.mlops.svc.cluster.local:5000"
)
mlflow.set_experiment("fraud-detection-v2")

with mlflow.start_run():
    mlflow.log_param("n_estimators", 200)
    mlflow.log_param("max_depth", 8)
    mlflow.log_metric("accuracy", 0.942)
    mlflow.log_metric("f1_score", 0.917)
    mlflow.sklearn.log_model(clf, "model")

Model Versioning with the MLflow Registry

The MLflow Model Registry provides a versioned catalog of model artifacts with semantic aliases that communicate production status:

Promotion between stages is a deliberate, audited action — not an automatic side-effect of training. Downstream systems reference aliases rather than hard-coded version numbers, enabling safe canary promotions and rollbacks.

W&B and Neptune Integration

For richer visualization and real-time collaboration, managed services like Weights & Biases and Neptune extend the MLflow pattern. Both integrate via environment variables and lightweight SDK calls. The key trade-off: self-hosted MLflow keeps data governance simple; managed SaaS reduces operational burden.

Section 3: Feature Stores on Kubernetes

The Training-Serving Skew Problem

Imagine building a fraud detection model. During training, you compute "transactions in the last 24 hours" using a batch SQL query. In production, the same feature must be computed in real time from a live event stream. If these two computations diverge even slightly, the model degrades silently. This is training-serving skew — one of the most common causes of silent model degradation in production.

Feast Architecture

Feast solves this by providing a single versioned feature definition shared between training and serving. Its architecture separates two concerns with distinct storage backends:

Online and Offline Feature Serving

The workflow is:

1. Feature definitions are written as Python FeatureView objects and committed to Git
2. Materialization runs on a schedule to push computed features from the offline store into Redis
3. Training pipelines retrieve historical point-in-time features via get_historical_features
4. Serving infrastructure retrieves features at inference time via get_online_features — using identical definitions

# Training: historical features
training_df = store.get_historical_features(
    entity_df=entity_df,
    features=["user_stats:transactions_24h",
              "user_stats:avg_amount_7d"]
).to_df()

# Serving: online features (identical feature names)
online_features = store.get_online_features(
    features=["user_stats:transactions_24h",
              "user_stats:avg_amount_7d"],
    entity_rows=[{"user_id": "u-12345"}]
).to_dict()

Section 4: CI/CD for Machine Learning

ML CI/CD must do everything traditional CI/CD does and: retrain models on new data, validate statistical model quality before promotion, manage large binary artifacts (model weights), and handle the fact that "tests passing" does not guarantee a model will perform well on tomorrow's data distribution.

GitOps with ArgoCD

GitOps makes Git the single source of truth for cluster state. A GitOps controller (ArgoCD or Flux) watches a Git repository and continuously reconciles the cluster to match what is declared there. For ML, this means model configuration changes are committed to Git, ArgoCD detects the drift, and applies the update automatically.

Automated Model Validation Gates

A validation gate blocks promotion unless a model meets defined quality thresholds:

Container Image Management

ML images have unique challenges: a PyTorch training image with CUDA can exceed 10 GB, dependencies must be mutually compatible (CUDA + cuDNN + PyTorch + Python versions), and exact reproducibility months later is required.

Reproducibility

Reproducibility requires discipline at four layers:

1. Environment: Docker image digest (not tag) pinned in pipeline
2. Code versioning: Git SHA pinned in pipeline metadata
3. Data versioning: DVC or Delta Lake snapshot for training datasets
4. Random seeds: torch.manual_seed(), numpy.random.seed()

Pinning pipeline steps to image digests (sha256:...) rather than tags is the single most impactful reproducibility practice — digests are content-addressed and immutable, unlike tags which can be overwritten.