Chapter 6: Resource Scheduling and Cluster Optimization

Section 1: Kubernetes Scheduling for AI

Priority Classes and Preemption

Every pod in Kubernetes can be assigned a PriorityClass, a cluster-scoped object that assigns a numerical priority value. When the scheduler cannot fit a new pod onto any node, it preempts (evicts) lower-priority pods to make room. A typical AI platform uses a tiered scheme:

With this setup, a newly submitted production inference pod can preempt a running development notebook without disrupting higher-priority work. Research jobs at training-standard cannot block production inference — the scheduler always prefers the higher value.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: training-high
value: 1000
globalDefault: false
description: "High-priority training jobs with team SLAs"
---
apiVersion: batch/v1
kind: Job
metadata:
  name: gpt-finetune-v3
spec:
  template:
    spec:
      priorityClassName: training-high
      containers:
        - name: trainer
          image: my-registry/trainer:latest
          resources:
            limits:
              nvidia.com/gpu: "4"

Node Affinity, Taints, and Tolerations

GPU nodes are specialized and expensive. Without controls, ordinary CPU workloads can land on GPU nodes and waste hardware. Two mechanisms prevent this:

• Taints and Tolerations act as a lock-and-key system. A taint on a GPU node repels all pods that do not carry a matching toleration.
• Node Affinity allows pods to express a preference or requirement for specific node characteristics such as GPU model, memory capacity, or network interconnect.

# Taint the GPU node
kubectl taint nodes gpu-node-01 nvidia.com/gpu=present:NoSchedule

# Pod toleration
spec:
  tolerations:
    - key: "nvidia.com/gpu"
      operator: "Equal"
      value: "present"
      effect: "NoSchedule"
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: nvidia.com/gpu.product
                operator: In
                values: ["NVIDIA-A100-SXM4-80GB"]

Topology-Aware Scheduling

For distributed training across multiple GPUs, physical placement directly impacts performance. Pods sharing NVLink fabric communicate orders of magnitude faster than pods on separate nodes connected by Ethernet. The topologySpreadConstraints feature lets jobs request co-location on the same high-bandwidth domain.

Section 2: Kueue — Kubernetes-Native Job Queuing

The Problem Kueue Solves

The default Kubernetes scheduler assigns pods as fast as they arrive. In a multi-tenant GPU cluster, a single team can submit thousands of jobs and consume all GPUs immediately. There is no mechanism for fairness, burst borrowing, or holding a job until all resources are simultaneously available. Kueue intercepts Jobs before pod creation, places them in a managed queue, and admits entire workloads only when resources are guaranteed.

ClusterQueue, LocalQueue, and Workloads

Cluster administrators control resource pools through ClusterQueues. Teams interact only with their namespaced LocalQueue. This cleanly separates platform governance from user access.

Fair-Sharing and Borrowing

When multiple ClusterQueues belong to the same cohort, their unused nominal quota becomes a shared pool. Fair sharing uses a scoring system: the queue consuming the least relative to its entitlement receives the highest admission priority. The queue consuming the most is the first target for preemption. This creates a self-balancing system converging toward equitable utilization.

Priority-Based Preemption

Kueue supports two queueing strategies: StrictFIFO and BestEffortFIFO. Three preemption policies govern behavior when a high-priority workload cannot fit:

The LessThanInitialShare policy adds a fairness guard: preemption is only allowed if the preempting queue's share would remain strictly less than the target queue's current share.

Integration with Jobs, RayJobs, and Training Operator

Kueue integrates natively with batch/v1 Job, RayJob, PyTorchJob, TFJob, and JobSet. Each integration wraps the custom resource in a Kueue Workload and manages its spec.suspend field.

Section 3: Volcano Batch Scheduler

Gang Scheduling and the Deadlock Problem

Consider a distributed training job needing 16 GPUs across 4 nodes. The default scheduler may place 12 pods, leaving 4 waiting. The placed pods hold GPUs while waiting for peers — creating a deadlock where everyone waits and nobody makes progress.

Gang scheduling solves this with an all-or-nothing guarantee. A PodGroup is admitted only when all requested resources are simultaneously available. If 16 GPUs are not free at once, the entire job waits without holding any resources.

Volcano Plugins

Preemption vs. Reclaim

• Preemption: A higher-priority job evicts lower-priority pods in the same queue
• Reclaim: A queue exceeding its fair share has pods evicted to return capacity to underserved queues

Kueue + Volcano: Better Together

Section 4: Cluster Bin-Packing and Cost Optimization

GPU Utilization Monitoring

Many clusters appear busy while actual GPU compute utilization sits below 30%. NVIDIA DCGM exposes per-GPU metrics (SM utilization, memory utilization, NVLink bandwidth) that flow into Prometheus and Grafana.

NVIDIA MIG

MIG partitions a single A100 or H100 into isolated slices with dedicated memory and compute. An A100 80GB can split into up to seven 1g.10gb MIG instances, each appearing as an independent GPU to Kubernetes.

Spot and Preemptible Instances

GPU spot instances offer up to 70% discounts with the tradeoff that the cloud provider can reclaim them. Training workloads with checkpointing are well-suited; inference serving live traffic is less suitable.

Karpenter vs. Cluster Autoscaler

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-nodepool
spec:
  template:
    spec:
      requirements:
        - key: "karpenter.k8s.aws/instance-family"
          operator: In
          values: ["g4dn", "g5", "p4d"]
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["spot", "on-demand"]
        - key: "topology.kubernetes.io/zone"
          operator: In
          values: ["us-east-1a", "us-east-1b", "us-east-1c"]
  disruption:
    consolidationPolicy: WhenUnderutilized
    consolidateAfter: 30s