Chapter 8: Networking and Security for AI Workloads

Ordinary TCP/IP networking was designed for flexibility and correctness, not for the microsecond-latency, near-zero-CPU-overhead requirements of distributed deep learning. When a training job executes an AllReduce collective across 256 GPUs, each process must exchange gradient tensors with its peers hundreds of times per training step. The bottleneck is not GPU computation — it is the time gradients spend waiting in CPU buffers and kernel queues.

RDMA and InfiniBand with Kubernetes

Remote Direct Memory Access (RDMA) allows a network adapter to read from and write to the memory of a remote host without involving the remote CPU. This eliminates the copy-to-kernel-buffer and context-switch overhead that dominates conventional networking at high message rates.

InfiniBand is the dominant physical transport for RDMA in HPC and AI data centers, providing link-level reliability, congestion management, and switch fabrics rated at 400 Gbps (HDR) or 800 Gbps (NDR) per port, with end-to-end latency measured in single-digit microseconds.

The NVIDIA Network Operator manages the full driver and plugin lifecycle to expose InfiniBand or RoCE devices to Kubernetes pods: MOFED/DOCA-OFED drivers, the RDMA shared device plugin (advertising rdma/ib resources), and the nvidia-peermem module for GPU-NIC peer-to-peer DMA.

GPUDirect RDMA for GPU-to-GPU Communication

GPUDirect RDMA extends RDMA into the GPU memory address space. Without it, gradient data travels: GPU memory → CPU pinned buffer → NIC → Network → NIC → CPU pinned buffer → GPU memory. With GPUDirect, the NIC can DMA directly into and out of the GPU's BAR memory region, eliminating both CPU copies.

NCCL can achieve 90–95% of theoretical InfiniBand bandwidth with GPUDirect RDMA, versus 40–60% through a CPU-mediated path.

SR-IOV for High-Bandwidth Network Interfaces

SR-IOV (Single Root I/O Virtualization) partitions a single physical NIC into multiple lightweight Virtual Functions (VFs). Each VF has its own DMA engine, queues, and interrupt lines. When assigned exclusively to a pod, the pod gets near-native NIC performance with hardware-enforced isolation.

Deploying SR-IOV in Kubernetes requires three components: the SR-IOV CNI plugin (moves a VF into the pod network namespace), the SR-IOV device plugin (advertises VF resources to the scheduler), and the SR-IOV Network Operator (automates VF creation).

Multus CNI for Secondary Network Attachments

Multus CNI acts as a meta-CNI plugin multiplexer. It creates the primary interface using the cluster's default CNI, then creates additional interfaces using other CNI plugins. This separates management traffic (Kubernetes API, health probes) from high-speed gradient communication on a dedicated InfiniBand or SR-IOV network.

In a shared AI cluster, different teams run workloads with different data sensitivity levels. Without explicit network isolation, those workloads can communicate freely at the pod level. Network isolation is a defense-in-depth strategy that starts with denying everything and explicitly allowing only what is needed.

Kubernetes NetworkPolicy for AI Namespace Isolation

The most important first step is a default-deny policy applied to every AI namespace. This blocks all ingress and egress by default, then you add explicit allow rules only for what each workload legitimately needs.

Training pods should have no internet egress — there is no legitimate reason for a gradient synchronization process to reach the public internet, and allowing it creates an exfiltration path.

Service Mesh (Istio, Linkerd)

For inference workloads, a NetworkPolicy alone is insufficient. A service mesh injects a sidecar proxy into every pod, transparently intercepting all connections to provide: traffic management (canary deployments, A/B testing), observability (per-route latency p50/p95/p99), and security (mTLS between all pods).

mTLS for Secure Inter-Pod Communication

Mutual TLS (mTLS) means both ends of a connection present and verify certificates. Each pod's identity is cryptographically bound to its Kubernetes service account. In Istio's STRICT mode, unencrypted or unauthenticated connections are rejected at the sidecar proxy before reaching the application.

RBAC for Multi-Team AI Clusters

RBAC misconfigurations account for over 35% of Kubernetes breaches. The most common mistake is granting cluster-admin to training job service accounts. A well-structured cluster uses namespace-scoped roles matching what each job type actually needs.

A training job needs to: read ConfigMaps (hyperparameters), read Secrets (data credentials), and list Pods (workers). It does not need to list nodes, modify RBAC policies, or access other namespaces.

Pod Security Standards and Admission Controllers

Pod Security Admission (PSA) enforces one of three built-in profiles at the namespace level:

• Privileged — No restrictions. Only for system namespaces like kube-system.
• Baseline — Prevents known privilege escalations. Default for AI training namespaces.
• Restricted — Enforces hardened configuration. For inference and multi-tenant environments.

The warn and audit modes surface violations without blocking pods, useful during migration before switching to enforce.

Image Scanning for ML Container Vulnerabilities

ML containers can reach 20–30 GB, meaning more surface area and more CVEs. Integrate scanning (Trivy, Grype, Snyk Container) into CI/CD, and use admission controllers (Kyverno, OPA/Gatekeeper) to reject pods from untrusted registries or with critical CVEs.

Secrets Management

The most important rule: mount secrets as files, not environment variables. Environment variables are visible in /proc/<pid>/environ, readable by any process as the same user. File-mounted secrets with 0400 permissions are far more restrictive.

For frequently changing secrets or audit trail requirements, use External Secrets Operator or Vault Agent Injector with HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault.

Encryption at Rest and in Transit

Encryption at rest applies at two levels in Kubernetes: etcd encryption (Secrets and sensitive API objects encrypted with aescbc or aesgcm via KMS provider) and persistent volume encryption (AWS EBS encryption, GCP CMEK, or LUKS-encrypted block devices on-premises).

Encryption in transit means TLS on all paths carrying training data. Note that RDMA bypasses the CPU and therefore bypasses TLS — for workloads requiring encrypted gradient communication, NCCL supports encrypted collectives or the cluster can enforce IPsec at the NIC level.

Data Access Auditing and Lineage Tracking

Configure Kubernetes audit logging to capture RequestResponse-level events for sensitive resources (secrets, PVCs). For AI-specific data lineage, tools like MLflow, DVC, and Kubeflow Pipelines track which dataset version produced which model checkpoint.

Compliance Frameworks

Treat compliance as a set of Kubernetes namespaces with different security boundaries. A HIPAA-scoped namespace gets: enforce=restricted PSA, dedicated node pool with encrypted root volumes, default-deny NetworkPolicy, mandatory mTLS, and audit logs exported to an immutable log store.

Runtime threat detection using Falco watches the Linux syscall stream and alerts on compromise indicators: spawning shells inside containers, reading /etc/shadow, or outbound connections on unusual ports.