Chapter 16: Troubleshooting Controller-Based Network Automation

The Diagnostic Mindset: Narrowing the Blast Radius

Troubleshooting a broken automation script is an exercise in elimination. When a script fails, the failure could live in at least four places: your client code, the network path, the controller itself, or the API server process. Running the broken code again with extra print statements is the least efficient path forward.

The "does the battery work" test is reproducing the call manually in Postman or curl. If it works there and not in your code, the problem is in your code. If it fails there too, the problem is the server, network, or credentials — and you can stop looking at the code entirely.

Using curl as a First-Responder Tool

A minimal authentication test against Catalyst Center:

curl -X POST \
  https://sandboxdnac.cisco.com/dna/system/api/v1/auth/token \
  -H "Content-Type: application/json" \
  -u admin:Cisco1234! \
  -k \
  --verbose

The --verbose flag prints the TLS handshake, request headers, response headers, and response body. The -k flag disables SSL verification — acceptable only in sandbox environments. In production, use --cacert /path/to/ca-bundle.pem.

HTTP Status Codes as a Diagnostic Chart

Handling Asynchronous Responses: The 202 Pattern

Catalyst Center uses an asynchronous execution model for operations that take more than a few seconds. An automation script that assumes a 202 means success will produce silent failures. The correct pattern polls until endTime is set:

def wait_for_task(base_url, token, task_id, max_polls=30, poll_interval=5):
    headers = {"X-Auth-Token": token}
    url = f"{base_url}/dna/intent/api/v1/task/{task_id}"
    for attempt in range(max_polls):
        data = requests.get(url, headers=headers, verify=False).json()["response"]
        if data.get("isError"):
            raise RuntimeError(f"Task failed: {data.get('failureReason')}")
        if data.get("endTime"):
            return data  # Task complete
        time.sleep(poll_interval)
    raise TimeoutError(f"Task {task_id} timed out")

The Authentication Zoo: Three Platforms, Three Models

Each Cisco controller platform uses a fundamentally different authentication architecture. There is no universal pattern.

vManage: The XSRF Token and Session Limit

vManage enforces a hard limit of 100 concurrent sessions. When the 101st session is created, vManage invalidates the oldest session. Always logout in a finally block:

def logout(session, vmanage_host):
    session.get(f"https://{vmanage_host}/logout", verify=False)
    session.close()

# Usage
try:
    session = create_vmanage_session(host, user, password)
    # ... automation work ...
finally:
    logout(session, host)

SSL/TLS Certificate Failures

verify=False disables all certificate validation — any attacker between your host and controller can intercept credentials. In production, always pass the CA cert path:

# Production
response = requests.get(url, verify="/etc/ssl/certs/corporate-ca-bundle.pem")

# Or via environment variable (applies globally)
# export REQUESTS_CA_BUNDLE=/etc/ssl/certs/corporate-ca-bundle.pem

Certificate failures caused by NTP misconfiguration (clock skew) are documented in Cisco Field Notice FN-72406 — the root cause is X.509 validity windows being violated when system time drifts.

Catalyst Center: Tracking Asynchronous Tasks

Meraki: Taming the 429 Rate Limiter

Meraki enforces 10 requests/second per organization. The response headers tell you exactly how long to wait via Retry-After. A production-grade retry handler:

def meraki_get(url, api_key, max_retries=5):
    headers = {"X-Cisco-Meraki-API-Key": api_key}
    for attempt in range(max_retries):
        response = requests.get(url, headers=headers)
        if response.status_code == 200:
            return response.json()
        if response.status_code == 429:
            retry_after = int(response.headers.get("Retry-After", 60))
            jitter = random.uniform(0, 5)
            time.sleep(retry_after + jitter)
            continue
        response.raise_for_status()
    raise RuntimeError(f"Max retries exceeded for {url}")

For large-scale Meraki deployments, Action Batches are the architecturally correct solution — a single API call can contain up to 100 configuration operations, reducing total request count by two orders of magnitude. The official meraki Python SDK handles 429 responses automatically with no custom retry code needed.

SD-WAN vManage: The CSRF Token Trap

The most common vManage failure after session-cookie mishandling is the missing XSRF token. It presents as HTTP 403 but the fix is different from RBAC issues:

• 403 on a GET: suspect RBAC role assignment
• 403 on a POST/PUT/DELETE: check the XSRF token first

# WRONG — raises JSONDecodeError; token endpoint returns plain text
xsrf_token = session.get(token_url).json()

# CORRECT
xsrf_token = session.get(token_url).text

ISE ERS API: RBAC and Content Type Pitfalls

ISE ERS uses HTTP Basic Auth on every request (stateless by design). Common failures:

• HTTP 401: Account needs the "ERS Admin" role in ISE Administration > System > Admin Access > Administrators
• HTTP 415 Unsupported Media Type: ISE requires both Content-Type: application/json and Accept: application/json — omitting either header triggers this error
• HTTP 403 on specific resources: ERS uses fine-grained resource-level permissions

Controller-Specific Quick Reference

The Six-Step API Debugging Protocol

Ad-hoc debugging produces slow, unpredictable results. A systematic methodology produces consistent, reproducible resolution paths.

Building Automation Test Suites

Production-grade network automation requires automated tests. Use pytest with session-scoped fixtures for the authentication token and test both positive paths (smoke tests) and negative paths (error handling):

@pytest.fixture(scope="session")
def dnac_token():
    response = requests.post(
        f"{DNAC_BASE}/dna/system/api/v1/auth/token",
        auth=(DNAC_USER, DNAC_PASS), verify=False
    )
    assert response.status_code == 200
    return response.json()["Token"]

def test_invalid_token_returns_401():
    headers = {"X-Auth-Token": "invalid-token-value"}
    response = requests.get(f"{DNAC_BASE}/dna/intent/api/v1/network-device",
                            headers=headers, verify=False)
    assert response.status_code == 401

Integrate with CI/CD using Postman Newman — the command-line Postman runner — to execute API collections on every code push via GitHub Actions, GitLab CI, or Jenkins.

API Version Management

Pin API version paths in a single configuration file rather than scattering them across the codebase. After controller software upgrades, deprecated endpoint paths return 404 errors as a wave — a compatibility matrix and full test-suite run against the new version in a lab prevents production surprises.

Operational Runbook Essentials

A runbook is not optional for enterprise automation. The highest-value section is a living catalog of known error conditions with verified resolutions — built before the first production incident, not during it: