Chapter 12: Orchestration — Nexus Dashboard, APIC, Hyperfabric, and Intersight

Platform Architecture and Unified Services

Cisco Nexus Dashboard is the unified operations platform for data center network management. With version 4.1, three previously separate services -- Fabric Controller (NDFC), Orchestrator (NDO), and Insights -- were consolidated into a single-image installation. Exam questions may reference them individually, but they operate within one platform.

NDFC for AI Fabric Provisioning

NDFC is the comprehensive management and automation solution for Cisco Nexus and MDS platforms running NX-OS. For AI workloads it provides:

• Built-in AI fabric templates that pre-configure PFC and ECN for lossless RoCEv2 transport -- hours of manual work completed in minutes.
• Support for Nexus 9364E 800G switches with Cisco Intelligent Packet Flow algorithms optimized for AI training and inference.
• Centralized configuration push with drift detection that alarms when devices deviate from intended state.
• VXLAN EVPN fabric management with BGP EVPN control plane for multi-tenancy and mobility across data centers.

Multi-Site Orchestration with NDO

NDO provides centralized network and policy management across ACI, Cloud ACI, and VXLAN EVPN sites:

• Maintains VLAN, VRF, and network configuration consistency across sites
• Automates multi-fabric connectivity between NX-OS VXLAN EVPN and ACI
• Enables seamless L2/L3 network extension with end-to-end policy
• Streamlines security and segmentation for GPO-aware VXLAN fabrics

Telemetry and Insights for AI Fabrics

Nexus Dashboard Insights collects streaming telemetry from every fabric node. For AI workloads -- where a single congested link can stall training across thousands of GPUs -- this visibility is critical.

Infrastructure-as-Code with NDFC

NDFC exposes full-featured REST APIs supporting multiple IaC frameworks:

ACI Fabric Architecture for AI

Cisco ACI is the SDN solution for data centers, providing application-driven policy management through a declarative, object-oriented framework. It uses a spine-leaf topology where all policy is defined declaratively and enforced by APIC.

The ACI Policy Model

Understanding this hierarchy is essential for the exam:

Worked Example: AI Training Tenant Design

Tenant: AI-Training-Prod
  VRF: Backend-GPU-VRF
    Bridge Domain: GPU-Cluster-A-BD (subnet 10.100.1.0/24)
    Bridge Domain: GPU-Cluster-B-BD (subnet 10.100.2.0/24)
  VRF: Storage-VRF
    Bridge Domain: NFS-Storage-BD (subnet 10.200.1.0/24)
  Application Profile: LLM-Training-App
    EPG: GPU-Workers (bound to GPU-Cluster-A-BD)
    EPG: Parameter-Servers (bound to GPU-Cluster-B-BD)
    EPG: Training-Storage (bound to NFS-Storage-BD)
    Contract: GPU-to-GPU (permits RoCEv2, ICMP)
    Contract: GPU-to-Storage (permits NFS/iSCSI)

AI/ML Traffic Patterns in ACI

East-West (Inter-GPU): The dominant pattern during training. GPUs exchange gradient updates via all-reduce operations. Requires ultra-low latency and lossless delivery -- a single dropped packet can stall the entire collective across thousands of GPUs.

North-South (Storage/External): Data ingestion, model checkpointing, and inference API serving. High-bandwidth but more tolerant of minor latency variations.

APIC as the SDN Controller

The APIC cluster (typically three controllers) serves as the centralized policy engine:

• Fabric discovery -- automatically discovers spine and leaf switches via LLDP
• Policy programming -- translates high-level intent into concrete forwarding rules
• Health monitoring -- tracks health scores at tenant, application, EPG, and endpoint levels
• External integration -- VMware vCenter, Kubernetes, OpenStack for dynamic endpoint mapping
• Multiple access methods -- GUI, CLI, and REST API

Network Requirements for AI/ML in ACI

Architecture and Cloud-Managed Model

Cisco Nexus Hyperfabric uses a cloud controller managed by Cisco to design, deploy, and manage fabrics located anywhere -- primary data centers, colocation facilities, and edge sites. This eliminates the operational burden of maintaining on-premises controllers.

Hyperfabric Full Stack AI Infrastructure

The full stack option is a turnkey, vertically integrated AI platform that is NVIDIA ERA-compliant:

Day-0 Through Day-2 Operations

Day-0 (Design and Order): Preconfigured templates for AI clusters, cloud-assisted capacity planning, integrated ordering workflow.

Day-1 (Deployment): Hardware arrives, switches power on and auto-connect to the cloud controller via zero-touch plug-and-play. Fully operational fabric in minutes -- no manual CLI.

Day-2 (Operations): Continuous loss/latency monitoring, assertion-based validation against defined intent, cloud-managed firmware upgrades, and auto-provisioned scaling.

Three-Tier AI Cluster Network Architecture

Integration with Cisco Ecosystem

Hyperfabric does not operate in isolation in the full stack AI model:

• Cisco Intersight (purchased separately) manages GPU servers and storage servers in detail.
• The Hyperfabric cloud controller handles high-level assertion-based control across network, GPU, and storage layers.
• Together they provide centralized control from configuration through daily operations.

Platform Overview

Cisco Intersight provides unified, intelligent management of Cisco UCS compute infrastructure from core to edge. While Nexus Dashboard focuses on network fabric and Hyperfabric provides cloud-managed full-stack solutions, Intersight concentrates on compute and cross-domain infrastructure lifecycle management.

Core Capabilities

Deployment Options

Infrastructure-as-Code with Intersight

Intersight provides full API access with pre-packaged clients:

• Terraform Provider: Manages UCS server profiles, policies, and firmware declaratively
• Ansible Modules: Playbooks for server provisioning, policy config, and compliance checks
• PowerShell SDK: Windows-native automation for Microsoft-centric toolchains
• Python SDK: Custom automation and ML pipeline orchestrator integration

Terraform Server Profile Example

resource "intersight_server_profile" "ai_gpu_node" {
  name            = "AI-GPU-Worker-01"
  target_platform = "FIAttached"

  organization {
    object_type = "organization.Organization"
    moid        = data.intersight_organization_organization.default.moid
  }

  policy_bucket {
    object_type = "bios.Policy"
    moid        = intersight_bios_policy.ai_optimized.moid
  }

  policy_bucket {
    object_type = "boot.PrecisionPolicy"
    moid        = intersight_boot_precision_policy.pxe_boot.moid
  }
}

Firmware and Compliance Management

• HCL checking: Automated validation of server components against Cisco's validated firmware/driver combinations
• Firmware policies: Define target firmware versions enforced across all managed servers
• Compliance dashboards: Real-time visibility into compliant vs. non-compliant servers and hardware advisories
• Rolling firmware upgrades: Orchestrated updates minimizing disruption to running AI workloads

Intersight and Nexus Dashboard Integration

Intersight provides global management of UCS, HyperFlex, APIC, and Nexus Dashboard from a single pane of glass. This cross-domain visibility is essential for AI troubleshooting: when a training job slows, the root cause could be network congestion, compute thermal throttling, or storage I/O bottleneck. Unified visibility across all three domains accelerates root cause analysis.