Chapter 10: Building Lossless Ethernet Fabrics

Why Lossless Matters for AI/ML

When hundreds or thousands of GPUs exchange gradients during a training run, the network is the single largest determinant of job completion time. RoCEv2 relies on lossless Ethernet because a dropped packet does not trigger a simple retransmission -- it triggers a go-back-N recovery at the transport layer that can stall an entire collective operation across every participating GPU.

Design Principles for AI Fabrics

A high-throughput AI fabric must satisfy four simultaneous constraints:

1. Non-blocking bandwidth -- every port-to-port path must sustain line-rate forwarding with no oversubscription in GPU-to-GPU switching tiers.
2. Ultra-low latency -- collective communication operations (all-reduce, all-gather, reduce-scatter) are latency-sensitive; even microseconds of additional delay compound across thousands of iterations.
3. Zero packet loss under congestion -- the fabric must absorb bursts and signal congestion without discarding frames.
4. Deterministic performance at scale -- these guarantees must hold whether the cluster contains 32 GPUs or 32,000.

Topologies for AI/ML Workloads

Multi-Tier Leaf-Spine Topology

Buffer Management and Queue Design

Buffers are the shock absorbers of a data center switch. When multiple flows converge on the same egress port simultaneously -- a pattern called incast -- the switch must temporarily store excess packets in buffers rather than dropping them.

ECMP Enhancements for RoCEv2: Standard 5-tuple ECMP hashing suffers from low entropy with RoCEv2 because all traffic shares UDP destination port 4791. Nexus 9000 switches with CloudScale H2R and H1 ASICs parse Base Transport Header (BTH) fields -- including opcode, destination queue pair, and packet sequence number -- to distribute flows more evenly across spine uplinks.

The Three Pillars of Lossless Ethernet

Achieving lossless Ethernet is a fabric-wide property that must be configured consistently on every hop between source NIC and destination NIC.

1. Priority Flow Control (PFC) -- IEEE 802.1Qbb: Pauses a specific Class of Service on an ingress port when receive buffers approach capacity. Unlike legacy 802.3x PAUSE (which halts all traffic), PFC is per-priority -- only the lossless class is paused while best-effort continues.
2. Explicit Congestion Notification (ECN) -- RFC 3168: A switch marks the IP ECN field (CE -- Congestion Experienced) in transit packets rather than dropping them. The receiver echoes this back, and the sender reduces its rate. ECN is preferred because it avoids the latency penalty of PFC pauses.
3. Weighted Random Early Detection (WRED): Probabilistically marks (ECN) or drops packets before queue saturation. Combined with ECN, WRED provides early congestion signaling that keeps queues short.

Hierarchical Congestion Control

NX-OS Lossless QoS Configuration Walkthrough

The following steps enable lossless transport for RoCEv2 on Cisco Nexus 9000 using CoS 3 as the no-drop class and CoS 0 as best-effort.

Configuration Workflow

Step 1: Enable PFC globally and on interfaces.

feature priority-flow-control

interface Ethernet1/1
  priority-flow-control mode on

Step 2: Define QoS classification.

class-map type qos match-all class-rocev2
  match cos 3

policy-map type qos rocev2-ingress
  class class-rocev2
    set qos-group 3
  class class-default
    set qos-group 0

Step 3: Configure queuing with no-drop and ECN/WRED.

policy-map type queuing rocev2-egress
  class type queuing c-out-8q-q3
    priority level 1
    no-drop
    random-detect minimum-threshold 150 kbytes maximum-threshold 3000 kbytes drop-probability 100 weight 0 ecn
  class type queuing c-out-8q-q-default
    bandwidth remaining percent 100

Step 4: Apply policies to interfaces.

interface Ethernet1/1
  service-policy type qos input rocev2-ingress
  service-policy type queuing output rocev2-egress

Step 5: Enable PFC Watchdog.

priority-flow-control watch-dog-interval on

Fabric-Wide QoS Consistency

Testing and Validation

1. PFC frame verification -- generate controlled congestion and confirm PFC pause frames using show interface priority-flow-control.
2. ECN marking verification -- capture packets at the receiver and confirm CE bit; use show queuing interface for WRED ECN counters.
3. Zero-drop confirmation -- under sustained load, verify drop counters remain at zero for the no-drop class.
4. PFC Watchdog test -- simulate a PFC storm and confirm PFCWD activates and isolates the offending queue.
5. ECMP balance validation -- verify RoCEv2 flow distribution across spine uplinks using show routing hash with BTH-aware hashing.

NDI for Congestion Monitoring

Configuring a lossless fabric is only half the battle -- you must also see what the fabric is doing in real time. NDI collects telemetry data from Nexus switches and correlates it with advanced algorithms for baselining, anomaly detection, and forecasting.

ECN and PFC Analytics

ECN Metrics:

• ECN mark counters per device, per interface, and per flow
• Historical trending of ECN marking rates to identify worsening congestion
• Correlation of ECN marks with specific application flows to pinpoint affected workloads

PFC Metrics:

• PFC frames issued and received per interface, broken down by Class of Service
• PFC duration tracking -- how long each pause event lasts
• PFC Watchdog activation events, showing when and where storms were detected

Congestion Tree Visualization

When PFC pause frames propagate upstream from a congestion point, they create a tree-shaped pattern of affected switches and links. The congestion root is the bottleneck source (often a slow receiver or oversubscribed link), and the congestion branches include all upstream switches and hosts whose traffic is paused as a result.

PFC Pause Propagation -- Congestion Tree Formation

NDI visualizes these congestion trees on a topology map, allowing administrators to:

• Identify the congestion root (often a slow receiver or oversubscribed link)
• Trace all affected branches to understand the blast radius
• Correlate congestion trees with specific AI training jobs or collective operations
• Track how congestion trees form and dissolve over time

Traffic Patterns in AI Workloads

Training vs. Inference: Different Stress on the Network

Training: Transient Incast. GPUs execute a compute phase (forward/backward pass) followed by a communication phase (gradient synchronization). The communication phase produces highly synchronized bursts where many GPUs transmit simultaneously. This transient incast can exceed link capacity by 10x or more for microseconds. Training congestion is self-limiting: buffers absorb the burst, traffic drains during the compute phase.

Inference: Sustained Elephant Flows. Inference (especially for LLMs with extended context windows) creates sustained, long-lived flows as models distribute memory and KV-caches. These elephant flows do not drain between phases. A fabric designed solely for training assumptions will experience sustained congestion during inference.

Microburst Detection

A microburst is a short-lived surge of traffic -- typically microseconds to low milliseconds -- that exceeds link capacity even when average utilization appears low. Traditional polling-based monitoring (sampling every 30-60 seconds) completely misses them.

NDI Microburst Detection uses hardware telemetry from the Nexus 9000 ASIC at sub-millisecond granularity:

• Per-port burst tracking: Monitors instantaneous queue depth and flags when utilization exceeds thresholds.
• Temporal correlation: Correlates microbursts across multiple ports and switches to identify collective operation boundaries.
• Historical trending: Records microburst frequency, duration, and magnitude over time.
• Flow-level attribution: Attributes specific microbursts to individual RoCEv2 flows, identifying which GPU pairs cause congestion.

NDI Congestion Monitoring Workflow for AI/ML Traffic

NDI's traffic analytics also auto-discovers services by well-known Layer 4 ports or user-defined custom categories, baselines them, and tracks latency, congestion, and drops over time. This is particularly valuable for converged fabrics where RoCEv2 AI traffic coexists with storage (NVMe-oF) and management traffic.