Chapter 5: Network Architecture for AI Workloads

Bandwidth: Training vs. Inference

AI workloads split into two categories with very different network demands. Training is the bandwidth-hungry phase: hundreds or thousands of GPUs exchange gradient updates after every forward and backward pass. Modern LLM clusters require 800 Gbps to 1.6 Tbps per node. In Cisco's reference architecture, a single Scalable Unit (SU) houses 256 GPUs across 32 HGX H200 systems, all communicating at full line rate.

Inference requires less per-node bandwidth (100-400 Gbps) but demands consistent low-latency responses at scale for user-facing SLAs.

Figure 5.1: Bulk-Synchronous Training -- GPU Synchronization Cycle

Latency Sensitivity

Collective communication operations (AllReduce, parameter sync) are exquisitely latency-sensitive because distributed training follows a bulk-synchronous model: every GPU must finish communicating before any GPU can begin the next computation step. The slowest flow dictates cluster pace.

Figure 5.2: RDMA Latency Comparison -- TCP/IP vs. RoCE v2

Redundancy and High Availability

Unplanned downtime in an AI cluster wastes expensive GPU compute hours. Redundancy is achieved through:

• Full-mesh leaf-spine connectivity -- every leaf connects to every spine, eliminating single points of failure.
• ECMP routing -- distributes flows across multiple equal-cost paths for load balancing and fault tolerance.
• Graceful degradation -- losing one spine in an 8-spine fabric reduces bandwidth by 12.5%, not a full outage.

Figure: ECMP Load Balancing and Graceful Degradation

Security Considerations

• Model and data confidentiality: VRFs and micro-segmentation isolate the AI backend from general traffic.
• Lossless fabric isolation: Dedicated RDMA/RoCE fabric prevents non-AI traffic from triggering PFC storms.
• Access control: GPU nodes reachable only from authorized management and storage networks; east-west firewall policies restrict lateral movement.

Spine-Leaf Architecture

The spine-leaf (Clos) topology is the dominant architecture in AI-ready data centers. Leaf switches connect to endpoints (GPUs, storage). Spine switches interconnect all leaf switches in a full mesh. Any endpoint can reach any other endpoint in exactly two hops, providing predictable, uniform latency.

For larger deployments, a super-spine (third tier) is added. Cisco's three-tier design scales to 32 planes with 32 switches each, maintaining 1:1 oversubscription within an SU and typically 3:1 at the spine-to-super-spine boundary.

Figure: Spine-Leaf Full-Mesh Topology

East-West vs. North-South Traffic

When 1,024 GPUs perform AllReduce, every GPU sends data to many other GPUs across the fabric. This east-west traffic can saturate spine links if the network lacks sufficient bisection bandwidth.

Figure 5.3: East-West vs. North-South Traffic Patterns

Rail-Optimized Topology

In a rail-optimized design, each "rail" is a dedicated parallel data path connecting one specific GPU position across all servers to a single leaf switch. For an 8-GPU-per-server cluster: GPU 0 in every server connects to Leaf A (Rail 0), GPU 1 to Leaf B (Rail 1), and so on.

This exploits sparse communication patterns in LLM training. Rail-only networks (without spine) can achieve the same training performance while reducing network cost by 38-77% and power by 37-75%.

Figure: Rail-Optimized Topology

Oversubscription and Capacity Planning

Oversubscription is the ratio of total downlink to total uplink bandwidth on a leaf switch -- the single most critical capacity planning metric.

Scalability in Large GPU Clusters

Scaling from hundreds to thousands of GPUs introduces compounding challenges:

• Port density: A 1,024-GPU cluster (8 GPUs/server, 128 servers, 2 ports each) needs 256 leaf-facing ports across multiple leaf switches.
• Cabling complexity: 8 leaf + 8 spine = 64 inter-switch links, each cabled, labeled, and managed.
• Hash polarization: Larger clusters risk ECMP hash functions unevenly distributing flows across paths.

Adding leaf switches without proportionally adding spine ports degrades the oversubscription ratio.

Bottlenecks in Distributed Training

1. Synchronized congestion: Collective operations cause all GPUs to burst traffic simultaneously -- far more damaging than random congestion.
2. Tail latency dominance: If 1,023 GPUs finish in 5 us but one takes 50 us, all 1,023 idle for 45 us.
3. Packet loss amplification: A single dropped packet triggers RDMA retransmission, stalling all GPUs in the cluster.

Figure 5.4: Lossless Fabric Mechanisms -- PFC, ECN, and DCQCN Interaction

Connectivity Models: RoCE v2 vs. InfiniBand

Copper Interconnects

Direct Attach Cables (DACs) are passive copper assemblies with pre-attached transceivers, saving 50-70% vs. optical at distances up to 3-5 meters. Zero power consumption and near-zero latency contribution make them ideal for within-rack connections.

Active Electrical Cables (AECs) embed signal-conditioning electronics, extending copper reach to 5-10 meters while consuming 25-50% less power than Active Optical Cables. AECs are emerging as preferred for top-of-rack to end-of-row in AI clusters.

Copper limitation: At 800G+, the skin effect causes signal loss to increase with frequency. Industry consensus: copper cannot scale beyond 800G -- at 1.6 Tbps, cables become too short, thick, and impractical.

Optical Interconnects

Fiber optics deliver data faster and over longer distances with minimal signal loss. Optical cables are thinner and lighter, enabling higher density. However, they cost up to 7x equivalent copper connections and consume more power.

For AI workloads, optical is essential for any link exceeding 5-10 meters: all spine uplinks, inter-rack, and inter-building links.

Transceiver Form Factors

Both use PAM4 modulation (4-level Pulse Amplitude Modulation), encoding 2 bits per symbol to double the bit rate per lane compared to NRZ.

Transceiver Reach Designations

Figure 5.5: Interconnect Technology Selection by Distance

Emerging Technologies

Co-Packaged Optics (CPO) integrates optical transceivers directly onto the switch ASIC package, eliminating pluggable transceivers. Expected to be critical for 1.6 Tbps and beyond, reducing electrical-to-optical conversion losses.

e-Tube uses RF data transmission through a plastic dielectric waveguide -- a third option between copper and optical offering copper-like simplicity with optical-like bandwidth density for short reach.

The optical interconnect market for AI data centers reached $3.75 billion in 2025, projected to reach $18.36 billion by 2033 (21.87% CAGR). Analysts predict all AI DC interconnects will be optical within five years.