Chapter 2: GPU and Accelerator Management

The GPU Landscape for AI

GPUs were originally designed for rendering pixels in parallel. That same parallel architecture — thousands of smaller compute cores working simultaneously — is exactly what neural network training and inference require. Three major vendors compete in the Kubernetes AI workload space:

Vendor	Key Products	Kubernetes Support	Notes
NVIDIA	A100, H100, L40, RTX 40-series	Mature (device plugin + GPU Operator)	Dominant; CUDA ecosystem widely supported
AMD	Instinct MI300, RX 7900 XTX	Stable (ROCm device plugin)	Growing adoption; ROCm is the CUDA equivalent
Intel	Gaudi 2/3, Arc GPUs	Emerging (Intel Device Plugin)	Strong for data center inference; OpenVINO

NVIDIA dominates Kubernetes AI workloads because the CUDA (Compute Unified Device Architecture) ecosystem has been the default target for nearly every major AI framework (PyTorch, TensorFlow, JAX). However, the architectural patterns — device plugins, resource requests, sharing strategies — apply across all vendors.

Device Plugin Architecture and Discovery

Kubernetes was designed around a generic resource model. GPUs are not built-in concepts — they are extended resources: arbitrary named quantities a node advertises and a pod can request. The device plugin framework bridges physical GPU hardware and the Kubernetes resource model.

A device plugin is a small program running as a DaemonSet on each GPU node. It communicates with the kubelet over a Unix socket using gRPC and performs three core functions:

1. Discovery — Detects available devices using NVML (NVIDIA Management Library)
2. Advertisement — Registers devices as extended resources (e.g., nvidia.com/gpu: 4) so the scheduler is aware of them
3. Allocation — When a pod needs a GPU, the plugin injects the correct device files and environment variables into the container

Container Runtime GPU Passthrough

Even with a device plugin advertising GPU resources, the container itself still needs to use those GPUs. The NVIDIA Container Toolkit (formerly nvidia-docker2) hooks into the container runtime (containerd or CRI-O) and, at container launch time, injects the correct GPU devices and driver libraries without requiring those libraries in every container image.

What happens when a GPU pod starts:

1. Pod spec requests nvidia.com/gpu: 1
2. Scheduler finds a node with available nvidia.com/gpu capacity
3. kubelet calls the NVIDIA device plugin to allocate one GPU
4. Device plugin returns device paths (/dev/nvidia0) and environment variables
5. NVIDIA Container Toolkit intercepts the container start, mounts /dev/nvidia0, and injects CUDA libraries
6. Container starts with full GPU access

The Operator Pattern for GPU Management

Managing GPU nodes manually — installing drivers, configuring the container toolkit, deploying the device plugin, setting up monitoring — is a fragile, error-prone process. The NVIDIA GPU Operator encodes this operational knowledge into a Kubernetes Operator: a controller that continuously watches cluster state and reconciles it toward the desired GPU software configuration.

Components Managed by the GPU Operator

Component	Purpose
NVIDIA Drivers	Kernel module enabling CUDA; deployed as a privileged container
Container Toolkit	Hooks the container runtime to inject GPU access into pods
Device Plugin	Advertises nvidia.com/gpu extended resources to the scheduler
GPU Feature Discovery	Labels nodes with GPU metadata (model, memory, CUDA version)
DCGM Exporter	Exposes GPU metrics (utilization, temperature, memory) for Prometheus
MIG Manager	Configures MIG partitioning on supported hardware
CUDA Validator	Runs a test workload to confirm CUDA is functional before scheduling

Installation via Helm

# Add the NVIDIA Helm repository
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

# Create the namespace and label it for privileged pod security
kubectl create namespace gpu-operator
kubectl label --overwrite namespace gpu-operator \
  pod-security.kubernetes.io/enforce=privileged

# Install the GPU Operator
helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator --wait

Automated Provisioning Workflow

For each new GPU node, the Operator follows a three-step workflow:

1. Discovery — Identifies nodes with NVIDIA GPUs using labels and hardware detection
2. Installation and Configuration — Deploys the driver container, configures the Container Toolkit, starts the device plugin and monitoring stack
3. Validation — The CUDA Validator runs a test workload; only after it passes does the node accept GPU workloads

The validation gate is critical: it prevents misconfigured nodes from silently accepting GPU pods that would then fail at runtime.

Node Labeling and GPU Feature Discovery

GPU Feature Discovery (GFD) automatically labels nodes with rich GPU metadata, enabling fine-grained scheduling decisions. Example labels on a node with an A100:

nvidia.com/gpu.present=true
nvidia.com/gpu.product=A100-SXM4-80GB
nvidia.com/gpu.memory=81920
nvidia.com/gpu.count=8
nvidia.com/cuda.driver.major=525
nvidia.com/mig.capable=true

With these labels, pods can express hardware requirements as node affinity rules rather than relying on manually maintained node pools.

Requesting GPUs in Pod Specs

Requesting a GPU follows the same resources.limits pattern as CPU and memory, with one constraint: GPU requests and limits must match, and they are whole numbers (no fractional GPUs by default).

apiVersion: v1
kind: Pod
metadata:
  name: gpu-training-job
spec:
  containers:
  - name: trainer
    image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
    resources:
      limits:
        nvidia.com/gpu: 1
    command: ["python", "train.py"]
  tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule

The tolerations section is critical: GPU nodes are typically tainted with nvidia.com/gpu:NoSchedule to prevent non-GPU workloads from consuming expensive GPU node resources. A pod must explicitly tolerate this taint.

Node Affinity and GPU Topology Awareness

When a cluster has multiple GPU types, raw GPU count is insufficient. Node affinity rules allow targeting specific GPU models using GFD-generated labels:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: nvidia.com/gpu.product
            operator: In
            values:
            - A100-SXM4-80GB

GPU Interconnect Bandwidth Comparison

Interconnect	Bandwidth	Typical Use Case
NVLink (4th gen, H100)	900 GB/s	Multi-GPU LLM training on a single node
NVLink (3rd gen, A100)	600 GB/s	Distributed training, large model sharding
PCIe 4.0 x16	32 GB/s	Inference, single-GPU training
InfiniBand HDR (inter-node)	200 Gb/s	Multi-node distributed training

Extended Resources and Custom Schedulers

The standard Kubernetes scheduler treats extended resources as opaque integer counts. It can count and subtract, but it cannot reason about GPU topology, memory bandwidth, or NUMA locality. For topology-aware placement, the GPU Operator integrates with the Topology Manager and NUMA-aware scheduler to co-locate GPU and CPU resources on the same NUMA node.

The GPU Utilization Problem

A single NVIDIA A100 costs $10,000–$30,000. Running one small inference workload at 5% utilization is financially untenable. GPU sharing runs multiple workloads on a single GPU to increase utilization and reduce per-workload cost. NVIDIA provides three distinct sharing mechanisms:

Strategy	Isolation	Hardware Req.	Latency Impact	Best For
MIG	Hardware (hard)	Ampere+ (A100, H100)	None	Production inference with SLAs
Time-Slicing	None (soft)	Any NVIDIA GPU	Context switch jitter	Dev/test, bursty workloads
MPS	Process-level (soft)	Any NVIDIA GPU	Low overhead	Throughput-focused inference replicas

MIG (Multi-Instance GPU) Partitioning

MIG is a hardware feature on Ampere+ architecture that divides one physical GPU into up to 7 independent instances, each with dedicated compute engines, memory bandwidth, L2 cache, and DRAM. One workload in a MIG instance cannot read memory from another instance or interfere with its performance.

MIG Partition Profiles (A100 80GB)

Profile	Compute Fraction	Memory	Max Instances
1g.10gb	1/7	~10 GB	7
2g.20gb	2/7	~20 GB	3
3g.40gb	3/7	~40 GB	2
4g.40gb	4/7	~40 GB	1
7g.80gb	7/7	~80 GB	1 (whole GPU)

Enabling MIG with the GPU Operator

# Label the node with the desired MIG strategy
kubectl label node <gpu-node> nvidia.com/mig.config=all-1g.10gb

# The GPU Operator's MIG Manager detects the label change and
# reconfigures the GPU automatically. Verify the result:
kubectl describe node <gpu-node> | grep nvidia.com/mig

After configuration, the node advertises nvidia.com/mig-1g.10gb: 7 as an extended resource. Pods request MIG slices explicitly:

resources:
  limits:
    nvidia.com/mig-1g.10gb: 1

Time-Slicing GPUs Across Pods

Time-slicing works on any NVIDIA GPU. It configures the device plugin to advertise each physical GPU as multiple virtual GPUs via a ConfigMap. The GPU rapidly context-switches between active workloads.

apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: gpu-operator
data:
  any: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        replicas: 4

Critical limitations: No memory isolation (all pods share VRAM), no compute isolation, latency jitter from context switching, no fair scheduling guarantees. Time-slicing is appropriate for dev/test clusters and bursty workloads, not production inference with SLAs.

MPS (Multi-Process Service)

MPS funnels multiple CUDA processes through a single shared CUDA context via a server daemon, reducing context-switching overhead. Ideal for replicated inference: running multiple copies of the same model with higher aggregate throughput than time-slicing.

Constraints: MPS and time-slicing cannot coexist. MPS is not supported on MIG-enabled devices. A crash in one CUDA process can destabilize the shared context — best for trusted workloads only.

Combining Strategies: MIG + Time-Slicing

For maximum density on A100/H100 hardware, MIG and time-slicing can be combined: 7 MIG instances × 4 time-sliced replicas = 28 pod allocations from a single physical GPU. Each MIG instance provides hardware memory isolation, while time-slicing adds density within each partition.

Cost Optimization Impact

Production clusters using advanced GPU sharing can move utilization from the typical 13% baseline to over 80%. The decision framework: isolation requirements first, hardware capabilities second, workload patterns third.