Chapter 9: High-Performance Networking — RDMA, RoCE, and QoS

What Is RDMA?

Remote Direct Memory Access (RDMA) is a technology that enables direct memory-to-memory data transfers between two computers without involving the operating system or CPU of either machine. By bypassing the kernel network stack entirely, RDMA dramatically reduces latency, eliminates CPU overhead, and increases throughput -- three properties essential for distributed AI training workloads.

Think of traditional networking like sending a package through a corporate mailroom: your data leaves the application, passes through several departments (the kernel, the TCP/IP stack, device drivers), gets wrapped in packaging at each stop, and eventually reaches the wire. RDMA is more like a private conveyor belt installed directly between two offices -- the data moves straight from the sender's desk to the receiver's desk with no intermediary handling.

Why RDMA Matters for AI/ML

In AI/ML clusters, GPUs across multiple servers must exchange gradient data at extremely high speed during distributed training. NVIDIA's GPUDirect RDMA enables GPU-to-GPU memory transfers over the network without staging data through system memory or the host CPU. This is critical because:

• Latency sensitivity: Collective operations like AllReduce require synchronization across hundreds or thousands of GPUs. Every microsecond of network latency compounds across iterations.
• CPU offload: Without RDMA, the host CPU must copy data between GPU memory, system memory, and the NIC. RDMA eliminates these copies.
• Throughput: Modern AI fabrics operate at 100G, 200G, and 400G per port. RDMA's zero-copy architecture can saturate these links efficiently.

The RDMA Protocol Stack

In a conventional stack, application data traverses the socket API, TCP/UDP transport, IP layer, and device driver before reaching the NIC hardware. Each layer involves kernel context switches and memory copies. With RDMA, applications interact directly with the NIC through a user-space verbs API. The NIC itself handles transport reliability, segmentation, and reassembly in hardware.

Three RDMA Implementations

InfiniBand vs. Ethernet RDMA: The Trade-Off

InfiniBand remains the gold standard for raw RDMA performance. Its dedicated fabric provides the lowest, most consistent latency because every component is purpose-built for RDMA. Flow control is handled natively through a credit-based mechanism, so there is no need for complex PFC/ECN configuration.

However, InfiniBand requires a completely separate network infrastructure with specialized switches, cables, and management tools (such as the OpenSM subnet manager). Ethernet-based RoCEv2 deployments can achieve up to 55% total cost of ownership (TCO) savings -- including 56% OpEx savings and 55% CapEx savings over three years. For organizations already invested in Ethernet infrastructure, RoCEv2 allows them to add RDMA capabilities without building a parallel network.

RoCE v1 vs. RoCE v2

RoCE v1, introduced by the InfiniBand Trade Association (IBTA) in 2010, encapsulates InfiniBand transport packets directly inside Ethernet frames using EtherType 0x8915. Because it operates at Layer 2 only, RoCE v1 traffic cannot cross router boundaries -- both communicating hosts must reside in the same Ethernet broadcast domain.

RoCE v2, introduced in 2014, solves this by wrapping the InfiniBand transport header inside a UDP/IP packet (IPv4 or IPv6) with UDP destination port 4791. This "Routable RoCE" can traverse any IP network, making it fully compatible with leaf-spine Clos topologies. RoCEv2 is the dominant Ethernet-based RDMA transport for AI workloads today.

RoCEv2 Packet Format

The RoCEv2 packet adds two key headers on top of the original InfiniBand transport:

• IP Header: Provides Layer 3 addressing and enables routing. The DSCP field is used for QoS classification and ECN signaling.
• UDP Header: Uses destination port 4791. The source port is typically derived from a hash of flow parameters, which is important for ECMP load balancing.
• IB Base Transport Header (BTH): Contains the InfiniBand operation code, partition key, destination queue pair (QP) number, and packet sequence number. NX-OS 10.6(1)F supports filtering on BTH fields in access lists.
• ICRC (Invariant CRC): Provides end-to-end data integrity verification that is not affected by IP header modifications during routing.

Deploying RoCEv2 on Cisco Nexus 9000

Deploying RoCEv2 requires configuring several interdependent features: PFC for lossless transport, ECN for congestion signaling, ETS for bandwidth allocation, and DCBX for parameter negotiation. Here is a basic RoCEv2 QoS configuration using the Modular QoS CLI (MQC):

! Step 1: Classification -- match RoCEv2 data and CNP traffic
class-map type qos match-all class-rocev2-data
  match dscp 26
class-map type qos match-all class-cnp
  match dscp 48

! Step 2: QoS policy -- set CoS and queue group for matched traffic
policy-map type qos rocev2-ingress-policy
  class class-rocev2-data
    set qos-group 3
  class class-cnp
    set qos-group 6

! Step 3: Queuing policy -- configure ECN thresholds and scheduling
policy-map type queuing rocev2-queuing-policy
  class type queuing c-out-8q-q3
    bandwidth percent 50
    random-detect minimum-threshold 150 kbytes maximum-threshold 3000 kbytes
    congestion-control ecn
  class type queuing c-out-8q-q6
    priority level 1

! Step 4: Network QoS -- enable PFC on the RoCEv2 traffic class
policy-map type network-qos rocev2-network-policy
  class type network-qos c-8q-nq3
    pause no-drop
    mtu 9216

! Step 5: Apply policies
system qos
  service-policy type network-qos rocev2-network-policy
  service-policy type queuing output rocev2-queuing-policy

interface Ethernet1/1
  service-policy type qos input rocev2-ingress-policy

Priority Flow Control (PFC) -- IEEE 802.1Qbb

Standard Ethernet is inherently lossy -- when switch buffers overflow, packets are silently dropped. For RoCEv2, packet loss is catastrophic: it triggers expensive transport-layer retransmissions that can increase AI training job completion times by orders of magnitude.

PFC extends the legacy IEEE 802.3x PAUSE mechanism to operate on a per-priority basis. Rather than pausing all traffic on a link, PFC can selectively pause only the congested Class of Service (CoS) value while allowing other priorities to continue flowing.

How PFC operates:

1. Each switch port monitors ingress buffer utilization per CoS priority (0 through 7).
2. When buffer usage exceeds the xoff threshold, the switch sends a PFC PAUSE frame upstream, specifying which priority to pause.
3. The upstream device stops transmitting that priority for the specified pause quanta.
4. When buffer usage drops below the xon threshold, the switch signals resume.
5. Traffic on all other priorities continues uninterrupted.

PFC Risks and Mitigations

Explicit Congestion Notification (ECN) -- RFC 3168

While PFC is a reactive, hop-by-hop emergency brake, ECN is a proactive, end-to-end congestion signaling mechanism. ECN marks packets rather than dropping them, allowing endpoints to reduce their sending rate before buffers overflow.

1. The sender sets the ECN field in the IP header to ECN-Capable Transport (ECT) -- binary 01 or 10.
2. When a switch queue depth crosses a configured threshold, it changes the ECN field to Congestion Experienced (CE) -- binary 11.
3. The receiver detects the CE marking and generates a Congestion Notification Packet (CNP) back to the sender (DSCP 48, strict-priority queue).
4. The sender receives the CNP and reduces its transmission rate.

ECN Threshold Parameters

DCQCN: Putting ECN and PFC Together

Enhanced Transmission Selection (ETS) -- IEEE 802.1Qaz

ETS provides bandwidth allocation and scheduling across traffic classes. It ensures RoCEv2 traffic receives a guaranteed share of link bandwidth while preventing best-effort traffic from starving.

• Traffic Class Mapping: Up to 8 traffic classes (TC 0-7), each mapped to one or more 802.1p CoS priorities.
• Bandwidth Allocation: Each traffic class receives a guaranteed minimum percentage (must sum to 100%). Unused bandwidth is redistributed proportionally.
• Scheduling: ETS supports both strict priority (for CNPs) and weighted scheduling (for data classes).

Example ETS Allocation for an AI Fabric

Data Center Bridging Exchange (DCBX) -- IEEE 802.1Qaz

DCBX is the discovery and negotiation protocol that automates DCB parameter exchange between connected devices. It rides on top of LLDP and negotiates three key TLV parameters:

1. PFC Configuration TLV: Which CoS priorities are lossless (PFC-enabled).
2. ETS Configuration / Recommendation TLV: Bandwidth allocation percentages and traffic class mappings.
3. Application Priority TLV: Which applications (RoCEv2, FCoE) map to which CoS priorities.

DCBX supports a willing/non-willing negotiation model. Typically, switches are non-willing (authoritative) and host NICs are willing, so the switch pushes consistent policy to all endpoints.

QoS Policy Design with MQC

Cisco NX-OS implements QoS through the Modular QoS CLI (MQC) framework with three distinct policy-map types:

Traffic Classification and Marking

A critical design rule: never mix drop-eligible and lossless (no-drop) traffic in the same queue. When PFC pauses a queue, all traffic in that queue is paused. Cisco recommends dedicating a queue exclusively to no-drop RoCEv2 traffic.

Load Distribution Strategies

The ECMP Problem with Elephant Flows

Traditional ECMP assigns entire flows to a single path based on a static 5-tuple hash. AI training generates large, persistent "elephant flows" between GPUs that can last minutes or hours. When multiple elephant flows hash to the same path, that link becomes congested while parallel links sit idle.

Dynamic Load Balancing (DLB)

Cisco Nexus 9000 switches (NX-OS 10.5(1)F and later) support Layer 3 ECMP Dynamic Load Balancing:

• Flowlet Load Balancing (FLB) -- Default Mode: Identifies natural idle gaps within long-lived flows. When a gap exceeds a configurable threshold, the next burst (flowlet) can be routed to a different ECMP path. Packets within a single flowlet always follow the same path, avoiding reordering. Cisco benchmarks showed 18.6% performance gain (43.48 GB/s vs. 36.66 GB/s) over static ECMP.
• Per-Packet Load Balancing (PLB): Distributes individual packets across all available ECMP paths. Provides the most even distribution but requires endpoints that can handle packet reordering.

Approximate Fair Drop (AFD)

AFD distinguishes high-bandwidth "elephant flows" from short-lived "mice flows." It applies more aggressive drop probability to elephant flows during congestion, preventing them from starving mice flows. This is useful on front-end (north-south) networks where AI inference traffic coexists with smaller management and storage flows.

Fabric Design: Spine-Leaf (Clos) Topology

The spine-leaf (Clos) topology is the recommended architecture for AI/ML clusters:

• Non-blocking: With 1:1 oversubscription ratio, every GPU can communicate at full line rate simultaneously.
• Consistent latency: Every source-destination pair traverses the same number of hops (leaf-spine-leaf).
• Horizontal scalability: Add capacity by adding more spine or leaf switches without redesigning the fabric.
• ECMP-friendly: All leaf-to-leaf paths are equal cost, maximizing DLB effectiveness.

Cisco Nexus Dashboard Management Tools

• Nexus Dashboard Fabric Controller (NDFC): Automates fabric provisioning including QoS configuration for PFC/ECN, template-based AI/ML network profiles, and consistent policy deployment.
• Nexus Dashboard Insights (NDI): Provides real-time visibility into congestion hot spots, flow telemetry, PFC/ECN statistics, and anomaly detection.