Chapter 5: Model Serving and Inference

Online vs. Batch Inference

Not all inference looks the same. Two fundamental patterns govern how predictions are generated:

Online inference prioritizes low latency: a user waits for each response, so the serving system must return results quickly. Batch inference prioritizes throughput: thousands of records are processed together. On Kubernetes, online serving uses Deployment or InferenceService resources fronted by a Service, while batch inference is better expressed as a Job or CronJob.

Model Server Options

Choosing the right model server is one of the most consequential decisions in your serving stack:

Sidecar and Standalone Topologies

Standalone: The model server runs as the primary container in a pod. This is simpler and appropriate for most cases.

Sidecar: A lightweight proxy or transformer runs alongside the model server container. KServe uses this pattern to inject its storage initializer as an init container that downloads model weights, and optionally attaches a transformer container for pre/post-processing.

KServe is the leading Kubernetes-native model serving platform, providing a single InferenceService CRD that abstracts away autoscaling, networking, health checking, and runtime configuration. It graduated from the Kubeflow project and is now a standalone CNCF project.

The InferenceService Custom Resource

A minimal InferenceService definition specifies a predictor with a model format and storage URI. When applied, KServe launches a storage initializer init container, starts the serving runtime, configures ingress routing, and optionally enables autoscaling.

apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "sklearn-iris"
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      storageUri: "gs://kfserving-examples/models/sklearn/1.0/model"
      resources:
        requests:
          cpu: "100m"
          memory: "128Mi"

Predictor, Transformer, and Explainer Components

A full InferenceService can include up to three logical components:

Model Storage and Model Mesh

The storage initializer is an init container that KServe injects into every predictor pod. It supports multiple storage backends: GCS (gs://), S3 (s3://), Azure Blob (azureblob://), HTTP/HTTPS, and PersistentVolumeClaims (pvc://).

Model Mesh is KServe's multi-model serving capability. Instead of one pod per model, Model Mesh co-locates many models within a shared pool of serving pods, dramatically reducing per-model overhead when maintaining hundreds of smaller models. It maintains an in-memory cache and loads/unloads models on demand.

Large language models present unique serving challenges. A 70B parameter model in FP16 requires approximately 140GB of GPU VRAM. The autoregressive generation pattern creates inherently sequential workloads that are hard to batch naively.

vLLM and PagedAttention

vLLM is the dominant open-source LLM serving engine. Its defining innovation is PagedAttention, which manages the KV cache using virtual memory techniques borrowed from OS page tables. Instead of reserving large contiguous VRAM blocks, PagedAttention stores the KV cache in non-contiguous "pages," eliminating internal fragmentation and enabling up to 24x more concurrent requests.

A critical Kubernetes-specific detail: pods default to only 64MB of shared memory (/dev/shm). Without an emptyDir with medium: Memory, large models will crash during initialization.

volumes:
  - name: shm
    emptyDir:
      medium: Memory
      sizeLimit: 8Gi    # critical for LLM IPC

Text Generation Inference (TGI)

HuggingFace's TGI is an alternative LLM server supporting Flash Attention 2, continuous batching, and tensor parallelism. It integrates deeply with the HuggingFace Hub ecosystem.

Quantization: GPTQ, AWQ, GGUF

Quantization converts model weights from high-precision formats (FP16) to lower-precision formats (INT4), trading a small amount of accuracy for significant memory and speed gains:

A 13B parameter model in FP16 requires ~26GB VRAM. The GPTQ INT4 version requires ~7GB -- fitting on a single A10G or consumer RTX 3090.

Continuous Batching and Speculative Decoding

Continuous batching keeps the GPU pipeline full at all times. As soon as a sequence completes, a new request from the queue immediately fills its slot in the next forward pass, keeping GPU utilization high and minimizing time-to-first-token.

Speculative decoding uses a small draft model to propose several tokens, which the large target model verifies in parallel. When the draft model is right, multiple tokens are produced per forward pass.

HPA with Custom Metrics

The Horizontal Pod Autoscaler (HPA) scales workloads based on observed metrics. For inference services, useful signals include CPU utilization, GPU utilization (DCGM metrics), request latency (p99), request concurrency, and pending queue depth.

Scale-to-Zero with Knative or KEDA

Scale-to-zero completely removes all pods when no traffic is present, and scales back up on the first request. Knative Pod Autoscaler (KPA) is KServe's default autoscaler in serverless mode. KEDA extends HPA with support for custom event sources and LLM-specific metrics.

The tradeoff with scale-to-zero for LLMs is cold start time. A 7B parameter model takes 30-90 seconds to load, making scale-to-zero unsuitable for latency-sensitive endpoints. For LLMs, minReplicas: 1 is usually the right choice.

Canary Rollouts and Traffic Splitting

Deploying a new model version to 100% of traffic immediately is high risk. Canary deployments mitigate this by routing a small percentage of traffic to the new version while the majority continues on the stable version.

spec:
  predictor:
    canaryTrafficPercent: 20    # 20% to new version
    model:
      modelFormat:
        name: sklearn
      storageUri: "gs://my-bucket/models/classifier/v2"

For automated promotion, Flagger integrates with KServe to progressively shift traffic based on Prometheus metrics, starting with a small canary percentage and automatically promoting or rolling back.

A/B Testing and Load Balancing

A/B testing routes traffic based on user identity or request metadata, enabling controlled experiments. KServe, backed by Istio, supports header-based routing (e.g., X-Experiment-Group: B).

For load balancing, LLM inference benefits from prefix cache affinity (routing users to replicas that have cached their system prompt), request length awareness, and least-connection routing rather than naive round-robin.