Unofficial, community-built study companion. Not affiliated with, endorsed by, or reviewed by Cisco Systems, Inc.
Cisco, UCS, Intersight, Nexus, and APIC are trademarks of Cisco. Always verify current commands and API behavior
against official Cisco documentation before use in production. Last reviewed: 2026-09-14.
DCAI 300-640 Hands-On Companion
Practical labs for Implementing Cisco Data Center AI Infrastructure (300-640 DCAI) v1.0
This site follows the four DCAI 300-640 v1.0 exam domains in blueprint order. Each numbered
sub-objective below gets a short theory recap, a locally runnable or clearly labeled
illustrative example, an inline diagram, and verification/failure-mode notes so you can
practice the objective hands-on rather than just read about it. Every code block carries a
language badge and a risk badge
(Safe · Local,
Illustrative · Pseudocode,
Platform-dependent, or
Production-impacting) so you know what is safe to
run immediately versus what requires real hardware, credentials, or a maintenance window.
This is an original, independently authored resource. It draws information-architecture
inspiration (single scrolling page, objective-aligned snippets) from community study sheets,
but all wording, code, diagrams, and examples here are original. It is not affiliated
with or endorsed by Cisco Systems, Inc., and it is not a substitute for official
Cisco training or documentation.
Coverage matrix
Every domain, sub-objective, and named topic from the official blueprint is covered below. Use the search box or this table to jump directly to any objective.
Coverage matrix — every blueprint sub-objective mapped to hands-on artifacts on this page.
Domain 1 · AI Fundamentals and Applications (20%) · Objective 1.1
AI/ML Workload Types
RAG, training, inference, and generative AI
AI infrastructure is sized differently depending on which workload it carries. Training is throughput-oriented and tolerates batch latency, so it favors dense GPU clusters with high-bandwidth interconnects. Inference is latency-sensitive and often runs continuously at lower per-request compute, so it favors right-sized accelerators and horizontal scaling. Retrieval-Augmented Generation (RAG) adds a retrieval step (vector search over a knowledge base) in front of a generative model to ground answers in current, private data instead of relying solely on model weights.
Prerequisites
Basic Python 3 (no external packages required for the example below)
Familiarity with the difference between a model's parameters and its activations
Fig. 1-1 — Where each workload type sits in a request/response flow.
Diagram showing a user query entering a RAG pipeline: retriever searches a vector store, relevant chunks are combined with the query into a prompt, the prompt goes to an inference engine, and the response returns to the user. A separate offline path shows training data flowing into a training job that periodically updates the model used by the inference engine.
Minimal RAG pipeline skeleton (local, no network calls)
PythonSafe · Local
Python
"""
Minimal RAG skeleton: retrieval + generation, fully local and dependency-free.
Replace the mock embedding/generation functions with a real embedding model
and LLM client for production use.
"""
import math
from collections import Counter
DOCS = [
"NVLink provides high-bandwidth GPU-to-GPU interconnect within a node.",
"RoCEv2 carries RDMA traffic over a lossless Ethernet fabric using PFC and ECN.",
"A UCS domain profile associates a fabric interconnect pair with domain-level switch policies (NTP, System QoS).",
]
def embed(text: str) -> Counter:
"""Toy 'embedding': bag-of-words term frequency vector (no ML library)."""
return Counter(text.lower().split())
def cosine(a: Counter, b: Counter) -> float:
common = set(a) & set(b)
dot = sum(a[t] * b[t] for t in common)
na = math.sqrt(sum(v * v for v in a.values()))
nb = math.sqrt(sum(v * v for v in b.values()))
return dot / (na * nb) if na and nb else 0.0
def retrieve(query: str, k: int = 1):
qv = embed(query)
scored = sorted(DOCS, key=lambda d: cosine(qv, embed(d)), reverse=True)
return scored[:k]
def generate(query: str, context: list) -> str:
"""Stand-in for an LLM call: a template response grounded in context."""
return f"Q: {query}\nGrounded on: {context[0]}\nA: See retrieved context above."
if __name__ == "__main__":
query = "How do GPUs talk to each other inside one server?"
ctx = retrieve(query)
print(generate(query, ctx))
Expected output
Q: How do GPUs talk to each other inside one server?
Grounded on: NVLink provides high-bandwidth GPU-to-GPU interconnect within a node.
A: See retrieved context above.
Verification steps
Run `python3 rag_skeleton.py` and confirm the retrieved sentence is topically relevant to the query.
Swap the query to an unrelated topic and confirm a different document is retrieved.
Common failure modes
Irrelevant retrieval: toy term-frequency matching has no semantic understanding — a real deployment needs a trained embedding model.
Context window overflow: real pipelines must truncate/rank retrieved chunks to fit the model's context limit.
Domain 1 · AI Fundamentals and Applications (20%) · Objective 1.2
AI Lifecycle
From data collection to retraining
The AI lifecycle is a repeating loop, not a one-way pipeline: data collection and preparation feed model development and training, which produces a candidate model that is validated, deployed to inference, and continuously monitored. Drift or new data triggers retraining, closing the loop. Infrastructure must support each stage's distinct profile -- bursty batch I/O for data prep, sustained multi-GPU throughput for training, steady low-latency serving for inference, and always-on telemetry for monitoring.
Prerequisites
Basic YAML syntax
Basic Python control flow
Fig. 1-2 — The AI lifecycle as a closed loop.
Circular diagram with six stages connected by arrows in a loop: Data Collection, Data Preparation, Model Training, Validation, Deployment (Inference), Monitoring, with an arrow labeled 'drift detected' looping from Monitoring back to Data Collection.
Lifecycle stage manifest for a model pipeline
YAMLIllustrative · Pseudocode
YAML
# lifecycle.yaml -- documents which stage a model artifact is in and its gate criteria
model: fraud-detector
version: 3.2.0
stages:
- name: data_preparation
inputs: [raw_transactions_2026Q3]
gate: "schema validation passes, null rate < 1%"
- name: training
depends_on: data_preparation
gate: "validation AUC >= 0.92"
- name: validation
depends_on: training
gate: "bias metrics within policy; shadow traffic matches baseline +/-2%"
- name: deployment
depends_on: validation
gate: "canary error rate <= baseline for 24h"
- name: monitoring
depends_on: deployment
retrain_trigger: "feature drift PSI > 0.2 for 3 consecutive days"
Expected output
N/A -- this is a declarative manifest consumed by a pipeline orchestrator (e.g., a CI/CD job or Kubeflow Pipelines run), not an executed script.
Verification steps
Validate the file parses as YAML: `python3 -c "import yaml,sys; yaml.safe_load(open('lifecycle.yaml'))"`.
Confirm every `depends_on` value matches an existing stage `name`.
Common failure modes
Skipping the validation gate to save time is the most common source of production model regressions.
No retrain_trigger defined means drift can silently degrade accuracy for months.
Stage-transition validator (enforces the loop's gates)
PythonSafe · Local
Python
"""
Enforces that lifecycle stages only advance in declared order and rejects
skipped or out-of-order transitions. Pure standard library.
"""
ORDER = ["data_preparation", "training", "validation", "deployment", "monitoring"]
class InvalidTransition(Exception):
pass
def advance(current: str, target: str) -> str:
ci, ti = ORDER.index(current), ORDER.index(target)
if ti != ci + 1 and target != "data_preparation":
raise InvalidTransition(f"{current} -> {target} skips required stages")
return target
state = "data_preparation"
for nxt in ["training", "validation", "deployment", "monitoring"]:
state = advance(state, nxt)
print(f"advanced to: {state}")
try:
advance("data_preparation", "deployment")
except InvalidTransition as e:
print(f"rejected: {e}")
Expected output
advanced to: training
advanced to: validation
advanced to: deployment
advanced to: monitoring
rejected: data_preparation -> deployment skips required stages
Verification steps
Run the script and confirm all four valid transitions print before the rejection line.
Change ORDER to omit a stage and confirm the validator still rejects skips consistently.
Common failure modes
This is a planning aid, not a replacement for a real orchestrator's DAG dependency engine (e.g., Kubeflow, Argo Workflows).
Domain 1 · AI Fundamentals and Applications (20%) · Objective 1.3
AI Use Cases
Matching workload characteristics to real deployments
Use cases span computer vision, natural-language processing, recommendation, fraud detection, predictive maintenance, and generative assistants. Each has a distinct latency, throughput, and data-gravity profile: a real-time fraud check needs sub-100ms inference near the transaction, while a nightly demand-forecasting job can run as a large batch training job in a centralized cluster. Classifying a use case correctly up front prevents over- or under-provisioning infrastructure.
Prerequisites
Basic Python 3
Fig. 1-3 — Use case characteristics mapped to infrastructure placement.
Quadrant chart with axes latency sensitivity (low to high) and data volume (low to high). Real-time fraud detection sits in high-latency-sensitivity/low-volume; batch demand forecasting sits in low-latency-sensitivity/high-volume; video analytics at the edge sits in high-latency-sensitivity/high-volume; offline document summarization sits in low-latency-sensitivity/low-volume.
Use-case classifier: latency budget to placement recommendation
PythonSafe · Local
Python
"""
Given a use case's latency budget and data volume class, recommend an
infrastructure placement tier. Thresholds are illustrative planning
heuristics, not a Cisco-published sizing standard.
"""
from dataclasses import dataclass
@dataclass
class UseCase:
name: str
latency_budget_ms: int
data_volume: str # "low" or "high"
def recommend(uc: UseCase) -> str:
if uc.latency_budget_ms <= 100 and uc.data_volume == "high":
return "edge AI (inference co-located with data source)"
if uc.latency_budget_ms <= 100:
return "on-premises/regional inference cluster"
if uc.data_volume == "high":
return "centralized cloud/on-prem batch training cluster"
return "cloud inference, cost-optimized"
cases = [
UseCase("real-time fraud check", 50, "low"),
UseCase("video analytics on factory floor", 80, "high"),
UseCase("nightly demand forecast", 6000, "high"),
UseCase("offline document summarization", 4000, "low"),
]
for c in cases:
print(f"{c.name:35s} -> {recommend(c)}")
Expected output
real-time fraud check -> on-premises/regional inference cluster
video analytics on factory floor -> edge AI (inference co-located with data source)
nightly demand forecast -> centralized cloud/on-prem batch training cluster
offline document summarization -> cloud inference, cost-optimized
Verification steps
Run the script and confirm each recommendation matches the quadrant in Fig. 1-3.
Add a fifth use case with a 20ms budget and high volume and confirm it lands on 'edge AI'.
Common failure modes
Thresholds are placeholders -- real placement decisions must incorporate cost, compliance/data residency, and existing footprint.
Domain 1 · AI Fundamentals and Applications (20%) · Objective 1.4
Infrastructure Types
Cloud, hybrid, on-premises, and edge AI
Cloud infrastructure offers elastic capacity and fast time-to-value but carries recurring cost and data-egress/residency considerations. On-premises infrastructure gives full control over data locality, latency, and cost predictability at the price of capital investment and longer lead times. Hybrid combines both, typically bursting training to the cloud while keeping sensitive inference on-premises. Edge AI pushes inference to where data is generated (factory floor, retail store, branch) to minimize latency and bandwidth back to a core data center.
Prerequisites
Basic Python 3
Fig. 1-4 — Four infrastructure placement models along the data-gravity axis.
Horizontal spectrum from Edge (left) to On-premises to Hybrid to Cloud (right), with icons indicating decreasing data-locality control and increasing elasticity moving left to right.
Weighted-criteria infrastructure placement scorer
PythonIllustrative · Pseudocode
Python
"""
Weighted scoring model for choosing among edge/on-prem/hybrid/cloud.
Weights and scores are illustrative -- calibrate to your organization's
actual constraints before using for real decisions.
"""
CRITERIA_WEIGHTS = {"latency": 0.35, "data_residency": 0.25, "elasticity": 0.20, "cost_predictability": 0.20}
# score 0-10 per option per criterion (higher is better fit)
OPTIONS = {
"edge": {"latency": 10, "data_residency": 9, "elasticity": 2, "cost_predictability": 7},
"on_prem": {"latency": 8, "data_residency": 10, "elasticity": 3, "cost_predictability": 8},
"hybrid": {"latency": 6, "data_residency": 7, "elasticity": 8, "cost_predictability": 6},
"cloud": {"latency": 4, "data_residency": 4, "elasticity": 10, "cost_predictability": 4},
}
def score(option: dict) -> float:
return sum(option[c] * w for c, w in CRITERIA_WEIGHTS.items())
ranked = sorted(OPTIONS.items(), key=lambda kv: score(kv[1]), reverse=True)
for name, opt in ranked:
print(f"{name:10s} score={score(opt):.2f}")
A production AI environment is a stack of cooperating layers: a lossless, high-bandwidth network fabric; GPU compute nodes connected internally by NVLink and externally by RDMA-capable NICs; a virtualization or containerization layer (KVM/VMware or Kubernetes) that packages workloads; an orchestrator that schedules those workloads onto GPUs; a monitoring layer that observes all of the above; and a storage layer spanning block, file, and SAN/Fibre Channel/NVMe transports feeding data to compute at the required throughput.
Prerequisites
Linux command line basics
Kubernetes pod spec basics
An NVIDIA GPU host is required only to reproduce nvidia-smi output verbatim; the JSON/YAML examples run anywhere
Fig. 1-5 — Layered AI environment stack.
Stacked layer diagram, bottom to top: Storage (SAN/Fibre Channel/NVMe/block/file), Network fabric, Compute (CPU/GPU with NVLink), Virtualization/Containerization, Orchestration, with Monitoring shown as a layer spanning the full height on the side.
GPU/NVLink topology inspection
BashPlatform-dependent
Bash
# Requires an NVIDIA GPU host with the driver/nvidia-smi installed.
nvidia-smi topo -m
nvidia-smi nvlink -s
Expected output
GPU0 GPU1 NIC0 CPU Affinity
GPU0 X NV12 PIX 0-31
GPU1 NV12 X PIX 0-31
NIC0 PIX PIX X
Legend: NV# = NVLink (# = link count), PIX = single PCIe bridge
GPU 0: NVLink Speed 25 GB/s, Link 0-11: Active
Verification steps
Confirm GPUs that should be NVLink-connected show 'NVX' rather than 'PIX' or 'SYS' in the topology matrix.
Confirm `nvlink -s` reports 'Active' for every expected link, not 'Inactive'.
Common failure modes
A link reported inactive can indicate a seated-but-not-trained NVLink bridge or a driver/firmware mismatch.
Topology showing PIX/SYS where NVLink is expected suggests the workload will fall back to slower PCIe/QPI paths.
pod/gpu-inference-pod created
(kubectl get pod gpu-inference-pod -> STATUS Running once scheduled to a node with a free GPU)
Verification steps
`kubectl describe pod gpu-inference-pod` should show the container's Limits including `nvidia.com/gpu: 1`.
`kubectl exec` into the pod and run `nvidia-smi` to confirm exactly one GPU is visible inside the container.
Common failure modes
Pod stays Pending indefinitely: the NVIDIA device plugin DaemonSet is not running, or no node advertises `nvidia.com/gpu` capacity.
nodeSelector too specific: no node matches the exact GPU product label, causing unschedulable pods.
Local block/NVMe storage inventory for a GPU host
BashPlatform-dependent
Bash
# List NVMe namespaces and multipath status feeding local scratch/checkpoint storage.
nvme list
lsblk -o NAME,SIZE,ROTA,TYPE,MOUNTPOINT
multipath -ll # only relevant when SAN-attached storage uses multiple Fibre Channel paths
Expected output
Node SN Model Namespace Usage
/dev/nvme0n1 S000000000000001 Example NVMe SSD 3.2TB 1 3.20 TB / 3.20 TB
NAME SIZE ROTA TYPE MOUNTPOINT
nvme0n1 3.2T 0 disk /mnt/checkpoints
Verification steps
Confirm ROTA (rotational) is 0 for NVMe devices used for training checkpoint I/O -- 1 would indicate a spinning disk bottleneck.
For SAN-attached storage, confirm `multipath -ll` shows all expected paths as 'active' (no 'faulty' paths).
Common failure modes
A missing multipath entry after a fabric change usually means a zoning or Fibre Channel login (FLOGI) problem upstream.
Namespace 'Usage' near capacity on the checkpoint volume will stall training jobs mid-run.
Domain 1 · AI Fundamentals and Applications (20%) · Objective 1.6
Cisco AI Solutions
AI PODs, AI Canvas, and Hyperfabric AI
Cisco AI PODs are validated, pre-integrated compute/network/storage designs for AI workloads that reduce integration risk versus assembling components independently. Cisco Hyperfabric AI is a fabric solution purpose-built for AI clusters that automates the underlying Ethernet fabric so RDMA traffic (RoCEv2) meets the lossless requirements of GPU-to-GPU communication. Cisco AI Canvas is a different kind of product from the other two: it is a cloud-hosted, agentic-operations workspace delivered through Cisco Cloud Control, not a fabric or topology design tool. Operators pose natural-language questions; AI Canvas dispatches domain-specialized agents to investigate and correlate signals across products (network, security, compute, observability, collaboration), synthesizes a sourced answer, and proposes a remediation plan that a human reviews and approves before any action runs. As of this writing (2026), AI Canvas is in Controlled Availability -- verify current availability, supported domains, and capabilities against Cisco's current documentation before relying on specifics below.
Prerequisites
Familiarity with the Intersight Python SDK (`pip install intersight`) for the API pattern below
An Intersight API key pair is required only to execute the call against a live account
Fig. 1-6 — AI Canvas as an agentic operations layer over already-deployed AI infrastructure (not a design tool).
Diagram showing an AI POD, Hyperfabric AI, and Intersight as already-deployed infrastructure and management components, each sending telemetry and operational context upward into AI Canvas, a cloud-hosted agentic-operations workspace in Cisco Cloud Control. AI Canvas correlates that cross-domain context and returns a proposed, sourced remediation plan that a human operator reviews and approves before any change is applied back through Intersight.
Intersight API pattern for querying compute pool health (documented SDK usage)
PythonIllustrative · Pseudocode
Python
"""
Illustrative pattern based on Cisco's published Intersight Python SDK usage
(https://github.com/CiscoDevNet/intersight-python). This is Intersight's
programmatic API, shown here as the compute/fabric management plane that
feeds context into AI Canvas -- it is NOT the AI Canvas product itself,
which is a workspace UI in Cisco Cloud Control rather than a scriptable API.
Requires `pip install intersight` and a real API key pair to execute; do
not hardcode secrets -- read them from environment variables as shown.
"""
import os
import intersight
from intersight.api import compute_api
configuration = intersight.Configuration(
host="https://intersight.com",
signing_info=intersight.HttpSigningConfiguration(
key_id=os.environ["INTERSIGHT_API_KEY_ID"],
private_key_path=os.environ["INTERSIGHT_API_PRIVATE_KEY_PATH"],
signing_scheme=intersight.signing.SCHEME_HS2019,
signing_algorithm=intersight.signing.ALGORITHM_ECDSA_MODE_DETERMINISTIC_RFC6979,
),
)
with intersight.ApiClient(configuration) as api_client:
api_instance = compute_api.ComputeApi(api_client)
pools = api_instance.get_compute_physical_summary_list(
filter="Model eq 'UCSX-210C-M7'"
)
for server in pools.results:
print(server.name, server.health)
Confirm the environment variables INTERSIGHT_API_KEY_ID and INTERSIGHT_API_PRIVATE_KEY_PATH are set before running.
A 401 response means the key ID/secret pair does not match or has been revoked in Intersight's Settings > API Keys.
Common failure modes
This queries a real Intersight account -- never run it against production without read-only scoped API keys.
Filter syntax and MO (managed object) class names change across Intersight API versions; verify against the current API reference before use.
Conceptual AI Canvas investigation transcript (not a documented public API)
JSONIllustrative · Pseudocode
JSON
{
"_note": "AI Canvas has no publicly documented scripting API as of this writing (Controlled Availability, 2026). This JSON is a conceptual illustration of its natural-language, agent-led investigation workflow -- described here from Cisco's own published materials, not executed or callable.",
"operator_question": "Why did GPU training job throughput drop on pod-b around 02:15 UTC?",
"ai_canvas_investigation": {
"mode": "default",
"dispatched_agents": ["compute-health-agent", "fabric-telemetry-agent", "change-log-agent"],
"cross_domain_findings": [
{ "domain": "compute", "finding": "No GPU hardware alarms on affected nodes." },
{ "domain": "network", "finding": "PFC pause-storm alert on leaf-1 at 02:14 UTC, same time window as the throughput drop." },
{ "domain": "change", "finding": "No configuration changes recorded in the prior 24 hours." }
],
"synthesized_answer": "Throughput drop correlates with a PFC pause-storm on leaf-1, not a compute or configuration issue.",
"proposed_remediation": {
"action": "Review PFC/ECN thresholds on leaf-1 (see Objective 3.1) before adjusting the training job.",
"requires_human_approval": true
}
}
}
Expected output
N/A -- this documents the shape of an AI Canvas investigation as described in Cisco's published overview and Controlled Availability announcement, not a callable API response.
Verification steps
Confirm the JSON parses: `python3 -m json.tool ai_canvas_transcript.json`.
Cross-check the workflow description (natural-language question -> multi-agent investigation -> sourced, cross-domain answer -> human-approved remediation) against Cisco's current AI Canvas documentation before presenting it as a live product capability.
Common failure modes
Do not treat this as a real, callable AI Canvas API -- Cisco documents AI Canvas as an operator-facing workspace in Cisco Cloud Control, and its availability, supported domains, and exact feature set are version- and entitlement-dependent (Controlled Availability as of 2026).
AI Canvas is not a substitute for the design-time planning tools or validated AI POD architectures used before infrastructure is deployed -- it operates on infrastructure that already exists.
Evaluating a network deployment for AI means quantifying whether the fabric can carry GPU-to-GPU and GPU-to-storage traffic without becoming the bottleneck. Bandwidth must cover the aggregate NIC line rate of all GPU nodes; latency must be low and, critically, consistent (jitter breaks collective operations like all-reduce); redundancy must survive a single link, switch, or path failure without dropping in-flight RDMA traffic; scalability must accommodate future GPU node additions without a fabric redesign; and security must segment AI/training traffic from general-purpose traffic.
Prerequisites
Basic Python 3
Basic understanding of NIC line rates (e.g., 100/200/400 GbE)
Leaf-spine topology with two spine switches and four leaf switches; each of the four leaves connects to both spines for redundancy (eight spine-to-leaf links total), and each leaf connects down to its own pair of GPU servers (eight GPU servers total), illustrating a non-blocking oversubscription ratio.
Required fabric bandwidth calculator for a GPU pod
Compute evaluation balances CPU-to-GPU ratio (enough CPU cores to feed data loaders without starving GPUs), GPU-to-GPU connectivity (NVLink within a node, RDMA/RoCEv2 between nodes), system memory sized for dataset staging, and whether workloads run bare-metal, virtualized, or containerized. Scalability means the design can add nodes without re-architecting, and redundancy means a single GPU, node, or power supply failure degrades capacity rather than halting the whole job.
Prerequisites
Basic Python 3
Concept of tensor/data/pipeline parallelism is helpful but not required
Diagram of a training cluster with 8 active GPU nodes (GPU 1 through GPU 8) and 1 spare node, all nine nodes connected to the same shared fabric switch, illustrating that a failed active node can be replaced by the spare without reducing the job's target parallelism below plan.
AI storage must be sized on three independent axes: capacity (raw dataset + checkpoint + versioning overhead), performance (sustained throughput and IOPS to keep GPUs fed during training, and low-latency random reads for inference feature stores), and redundancy/availability (RAID, erasure coding, or replication so a drive or node failure doesn't stall a multi-day training job). Scalability means capacity and performance can grow incrementally, since dataset sizes for AI workloads typically grow faster than initial estimates.
Prerequisites
Basic Python 3
Fig. 2-3 — Storage sizing accounts for usable capacity after redundancy overhead.
Bar chart comparing raw capacity to usable capacity after RAID/erasure-coding overhead, with a callout showing the checkpoint growth rate consuming free capacity over time.
Usable capacity and throughput calculator with redundancy overhead
PythonSafe · Local
Python
"""
Compute usable capacity after erasure coding and required per-node throughput
to sustain a target aggregate read bandwidth for training data loading.
"""
def usable_capacity_tb(raw_tb: float, data_shards: int, parity_shards: int) -> float:
efficiency = data_shards / (data_shards + parity_shards)
return raw_tb * efficiency
def per_node_throughput_gbps(target_aggregate_gbps: float, num_storage_nodes: int) -> float:
return target_aggregate_gbps / num_storage_nodes
raw = 500 # TB
usable = usable_capacity_tb(raw, data_shards=8, parity_shards=2) # 8+2 erasure coding
per_node = per_node_throughput_gbps(target_aggregate_gbps=80, num_storage_nodes=10)
print(f"Raw: {raw} TB -> Usable (8+2 EC): {usable:.0f} TB ({usable/raw:.0%} efficiency)")
print(f"Required per-node throughput: {per_node:.1f} Gbps across 10 nodes for 80 Gbps aggregate")
Domain 2 · AI Infrastructure Components and Architecture (30%) · Objective 2.4
Evaluate Power, Efficiency, and Sustainability
Power/cooling, PUE, renewable energy
GPU-dense racks can draw 10-100+ kW each, far exceeding typical CPU-only rack power budgets, so power distribution and cooling capacity must be evaluated before hardware arrives. Power Usage Effectiveness (PUE) -- total facility power divided by IT equipment power -- is the standard metric for data center energy efficiency; a PUE closer to 1.0 means less energy is lost to cooling and overhead. Sustainability evaluation also considers the percentage of energy sourced from renewables and whether liquid cooling is needed to manage the higher heat density of AI accelerators.
Prerequisites
Basic Python 3
Fig. 2-4 — PUE relates total facility power to IT equipment power.
Diagram showing total facility power split into IT equipment power (servers, storage, network) and overhead power (cooling, lighting, power distribution losses), with PUE defined as total facility power divided by IT equipment power.
PUE and renewable-energy mix calculator
PythonSafe · Local
Python
"""
Compute PUE and effective carbon-relevant renewable percentage for a
facility hosting an AI cluster.
"""
def pue(total_facility_kw: float, it_equipment_kw: float) -> float:
return total_facility_kw / it_equipment_kw
def renewable_share(renewable_kwh: float, total_kwh: float) -> float:
return renewable_kwh / total_kwh
facility_kw = 1400
it_kw = 1000
value = pue(facility_kw, it_kw)
print(f"PUE: {value:.2f} ({'good' if value <= 1.4 else 'needs improvement'})")
renewables = renewable_share(renewable_kwh=6500, total_kwh=10000)
print(f"Renewable share: {renewables:.0%}")
Expected output
PUE: 1.40 (good)
Renewable share: 65%
Verification steps
Confirm PUE is always >= 1.0 (a facility can never use less power than its IT equipment draws).
Compare your computed PUE against your facility's published annual average, not a single point-in-time reading.
Common failure modes
1.4 is illustrative -- the 'good' threshold depends on climate, cooling technology (air vs. liquid), and facility age; use your own baseline.
Point-in-time PUE readings vary with outdoor temperature; use trailing 12-month averages for planning decisions.
Domain 2 · AI Infrastructure Components and Architecture (30%) · Objective 2.5
Evaluate Hybrid AI Deployment
Secure connectivity, data synchronization, workload mobility
Hybrid AI deployments span on-premises and cloud/edge locations, so evaluation focuses on three concerns: secure connectivity (encrypted, authenticated tunnels or dedicated circuits between sites), data synchronization (keeping training data, features, or model artifacts consistent across locations without excessive replication cost or staleness), and workload mobility (the ability to move a training or inference job between on-premises and cloud without rewriting it, typically via containers and a common orchestration API).
Prerequisites
Basic Python 3
Familiarity with checksums/hashing
Fig. 2-5 — Hybrid AI reconciliation loop between sites.
Diagram of an on-premises site and a cloud site each with a data store, connected by a secure tunnel; a reconciliation process periodically compares checksums and synchronizes only the deltas between the two stores.
Delta-based reconciliation between two sites (checksum diff)
PythonSafe · Local
Python
"""
Simulate reconciling two dataset manifests (on-prem vs. cloud) by comparing
content hashes and syncing only the deltas, minimizing cross-site transfer.
"""
import hashlib
def sha256(data: bytes) -> str:
return hashlib.sha256(data).hexdigest()
on_prem = {
"features_2026_01.parquet": sha256(b"content-v1"),
"features_2026_02.parquet": sha256(b"content-v2"),
"features_2026_03.parquet": sha256(b"content-v3-newer"),
}
cloud = {
"features_2026_01.parquet": sha256(b"content-v1"),
"features_2026_02.parquet": sha256(b"content-v2-stale"),
}
def diff_manifests(a: dict, b: dict) -> dict:
to_upload = {k: v for k, v in a.items() if b.get(k) != v}
to_delete = [k for k in b if k not in a]
return {"upload": list(to_upload), "delete": to_delete}
result = diff_manifests(on_prem, cloud)
print(f"Sync plan: upload {result['upload']}, delete {result['delete']}")
Run the script and confirm only objects whose hash differs (or is missing on the target) are marked for upload.
Add an identical object with an identical hash and confirm it is correctly excluded from the sync plan.
Common failure modes
Hash-only comparison ignores partial/corrupted transfers in flight -- production sync tools also verify transfer completion, not just source/destination hash equality.
No conflict resolution shown here: if both sites modify the same object independently, define a policy (last-write-wins, manual review) before automating deletes.
Domain 3 · AI Infrastructure Deployment and Data Management (30%) · Objective 3.1
Configure High-Performance DC Networks
PFC, ECN, ETS, RoCE/RoCEv2, QoS, load distribution
RDMA over Converged Ethernet (RoCEv2) requires a lossless fabric because RDMA has no software-based retransmission at the speed GPUs need. Priority Flow Control (PFC) pauses a specific CoS queue instead of the whole link when it nears congestion. Explicit Congestion Notification (ECN) marks packets approaching congestion so senders can slow down before a pause is needed, reducing PFC's head-of-line blocking risk. Enhanced Transmission Selection (ETS) guarantees each traffic class a minimum share of bandwidth during contention. Load distribution (ECMP-based hashing) spreads RoCEv2 flows across all available uplinks to use the full leaf-spine capacity from Objective 2.1.
Prerequisites
Familiarity with Cisco NX-OS CLI conventions
A physical or virtual Nexus switch is required to execute these commands; syntax shown here follows Cisco's published NX-OS QoS/DCB configuration guides
Fig. 3-1 — PFC pause and ECN marking cooperate to keep RoCEv2 lossless.
Diagram of a switch queue filling toward its ECN threshold, marking packets as congestion-experienced; if the queue continues to fill past the PFC threshold, the switch sends a PFC pause frame to the upstream sender for that priority only.
Enable PFC, ETS, and RoCEv2 QoS classification (documented DCB pattern)
NX-OSProduction-impacting
NX-OS
! Documented NX-OS DCB/QoS pattern -- verify exact syntax against the platform's
! current configuration guide before applying to a production fabric. On Nexus
! 9000, no-drop/PFC behavior for a traffic class is configured in a "type
! network-qos" policy applied system-wide under "system qos" -- NOT inside a
! "type queuing" class, which only controls scheduling/bandwidth (ETS).
configure terminal
class-map type qos match-all ROCEV2
match cos 3
policy-map type qos ROCE_QOS
class ROCEV2
set qos-group 3
! Lossless/no-drop class + PFC CoS mapping and MTU, applied system-wide.
class-map type network-qos ROCEV2_NQ
match qos-group 3
policy-map type network-qos ROCE_NETWORK_QOS
class type network-qos ROCEV2_NQ
pause pfc-cos 3
mtu 9216
class type network-qos class-default
mtu 9216
system qos
service-policy type network-qos ROCE_NETWORK_QOS
! ETS: bandwidth guarantee for the RoCEv2 class vs. everything else.
class-map type queuing ROCEV2_Q
match qos-group 3
policy-map type queuing ROCE_QUEUING
class type queuing ROCEV2_Q
bandwidth remaining percent 50
class type queuing class-default
bandwidth remaining percent 50
interface Ethernet1/1
service-policy type qos input ROCE_QOS
service-policy type queuing output ROCE_QUEUING
priority-flow-control mode on
end
Expected output
(no direct output -- verify with the show commands below after applying)
Verification steps
`show interface priority-flow-control` -- confirm the interface shows PFC 'on' for the RoCEv2 CoS/priority.
`show policy-map system-qos` (or the platform's equivalent network-qos show command) -- confirm the RoCEv2 qos-group is configured no-drop/pause on CoS 3, not just scheduled.
`show policy-map interface Ethernet1/1` -- confirm the queuing policy shows non-zero bandwidth remaining percent applied to the RoCEv2 class.
`show queuing interface Ethernet1/1` -- confirm no persistent drops on the RoCEv2 queue under load.
Common failure modes
Putting `priority-flow-control mode on` inside a `type queuing` class is invalid/ineffective -- no-drop (PFC pause) behavior for a qos-group is configured in a `type network-qos` policy applied under `system qos`, not in the queuing (ETS/scheduling) policy.
Applying PFC to the wrong CoS/priority pauses unrelated traffic sharing that class, causing unrelated application slowdowns.
PFC enabled without ECN configured upstream increases the risk of head-of-line blocking and, in extreme cases, PFC storms across the fabric.
This changes production QoS behavior immediately upon `end` -- schedule a maintenance window and have a documented rollback (saved config) ready.
Verify ECMP load distribution across uplinks
NX-OSPlatform-dependent
NX-OS
show port-channel load-balance
show ip load-sharing
show interface counters detail all | include Ethernet1/
Confirm the load-balancing hash includes L4 ports (source-dest-port) so individual RoCEv2 flows spread across all uplinks rather than pinning to one member.
Compare Tx-Ucast/Rx-Ucast counters across member links -- a healthy ECMP/port-channel distribution shows roughly even counts, not one link dominating.
Common failure modes
A hash based only on source/destination IP (no L4 port) can polarize traffic onto a single uplink when only a few large flows exist -- common with RDMA traffic.
Persistent imbalance despite correct hashing may indicate an elephant flow that ECMP cannot split, requiring flowlet-based load balancing if supported.
Domain 3 · AI Infrastructure Deployment and Data Management (30%) · Objective 3.2
Configure High-Performance Compute/Storage with Cisco UCS
Domain profiles, power policy, storage policies, LAN connectivity/vNIC policies, QoS policies/system classes, NTP policy
Cisco Intersight Managed Mode (IMM) configures UCS through reusable policies attached to three distinct profile types rather than one-off CLI configuration. A Domain Profile binds a fabric interconnect pair's VLAN/VSAN and port configuration, and also carries domain-wide policies such as NTP and System QoS/system classes (mapping to the priorities configured in Objective 3.1). A Chassis Profile (UCS X-Series) carries chassis-scoped Power Policy fields -- PSU redundancy mode (grid, N+1, N+2), power-save mode, dynamic power rebalancing, extended power capacity, and the chassis's allocated power budget. A Server Profile carries server-scoped policies -- storage (RAID/disk group), LAN connectivity/vNIC assignment, and server-scoped power behavior such as the power-restore state after an outage, per-server power limit/package, and power priority. Confusing chassis-scoped and server-scoped power settings is a common Intersight deployment mistake, since both live under a policy object named 'Power Policy' but attach to different profile types.
Prerequisites
An Intersight account with UCS domain claimed is required to apply these policies for real
Familiarity with JSON
Fig. 3-2 — UCS policies compose into three distinct profile types deployed to hardware.
Diagram showing a Domain Profile (NTP policy, System QoS/system classes, fabric interconnect VLAN/VSAN/port configuration), a Chassis Profile (chassis-scoped Power Policy: PSU redundancy, power-save mode, dynamic rebalancing, extended power capacity, allocated power budget), and a Server Profile (storage policy, LAN connectivity/vNIC policy, and server-scoped Power Policy: power-restore state, power limit/package, priority), all three deployed to physical UCS hardware.
Intersight-style policy payload patterns by owning profile type (verify class names against current API reference)
N/A -- this documents the policy payload shapes typically submitted via the Intersight API/SDK or UI, grouped by which profile type each policy attaches to; there is no local execution output.
Verification steps
Validate the JSON parses: `python3 -m json.tool policies.json`.
Before applying to a real domain, diff field names against the current Intersight API reference for the target API version (schemas evolve between releases), and confirm each policy is actually attachable to the profile type shown here.
Common failure modes
Object type names and field names (including the `_scope` annotations, which are documentation hints and not real Intersight fields) are illustrative examples of the pattern, not guaranteed to match every Intersight API version -- always confirm against https://intersight.com/apidocs/ before submitting.
Chassis-scoped and server-scoped power settings are easy to conflate because both use a `power.Policy` object type -- attaching a chassis-scoped power policy to a Server Profile (or vice versa) will fail validation or silently apply to the wrong scope; confirm the target profile type in Intersight before saving.
IP addresses shown use TEST-NET-3 (203.0.113.0/24, RFC 5737) and must be replaced with real, reachable NTP servers.
Applying a storage policy that changes RAID layout on disks already in use is production-impacting and can cause data loss -- confirm target disk group state first.
Domain 3 · AI Infrastructure Deployment and Data Management (30%) · Objective 3.3
Deploy AI-Ready Fabrics
Nexus Dashboard, APIC, Hyperfabric, Intersight
Deploying an AI-ready fabric coordinates multiple management planes: Cisco APIC is the policy controller for ACI fabrics, modeling tenants, bridge domains, and endpoint groups as managed objects. Cisco Nexus Dashboard is the unified operations platform that hosts APIC-integrated services (fabric controller, insights, orchestrator) for day-2 operations across sites. Cisco Hyperfabric AI automates a simplified, purpose-built AI fabric underlay. Cisco Intersight ties compute (Objective 3.2) and fabric lifecycle together under one operations model. A typical workflow provisions the network policy (APIC), verifies it through Nexus Dashboard, and correlates it with compute profiles in Intersight.
Prerequisites
An APIC/ACI fabric or Nexus Dashboard instance is required to execute these API calls for real
Familiarity with tenant/bridge-domain/EPG ACI concepts
Fig. 3-3 — AI-ready fabric provisioning workflow across management planes.
Sequence diagram: an operator submits a tenant/EPG policy to APIC, APIC pushes it to fabric switches, Nexus Dashboard polls fabric health and displays it, and Intersight correlates the network policy with the compute profiles it manages.
APIC tenant/EPG policy for an AI workload segment (documented ACI object model)
POST https://<apic>/api/mo/uni.json -> HTTP 200 with an empty imdata array on success
Verification steps
In the APIC GUI, confirm the new tenant 'ai-training-tenant' with bridge domain and EPG appears under Tenants.
In Nexus Dashboard, confirm the fabric health score for the affected leaf switches remains green after the push.
Common failure modes
fvTenant, fvBD, fvAp, and fvAEPg are real, documented ACI managed-object classes -- always verify the exact schema and required attributes against the current APIC API documentation/Basic Object Model before scripting against a production fabric.
Pushing tenant policy directly to a production APIC is production-impacting; validate first in a lab fabric or APIC's built-in policy validation/dry-run tooling if available in your version.
Domain 4 · AI Infrastructure Operations and Troubleshooting (20%) · Objective 4.1
Implement Benchmarks
MLPerf-oriented throughput and latency measurement
Benchmarking validates that deployed infrastructure delivers the throughput and latency assumed during design. MLCommons/MLPerf defines industry-standard training and inference benchmark methodologies (fixed model, fixed dataset, measured time-to-accuracy or queries-per-second at a latency bound) that let you compare results across vendors and generations consistently. Alongside full MLPerf-style runs, teams commonly use lower-level microbenchmarks to isolate where a bottleneck actually lives: NVIDIA's NCCL-tests (e.g., `all_reduce_perf`, reporting algorithm and bus bandwidth) measure GPU-to-GPU collective communication performance over NVLink/NVSwitch and the network fabric, and the linux-rdma `perftest` suite (e.g., `ib_write_bw`) measures raw RDMA write bandwidth between two NICs independent of any ML framework. Running these before a full MLPerf-style benchmark helps confirm whether a shortfall is a network/fabric problem (Objective 3.1) or a compute/software-stack problem before spending a full training run chasing it. A minimal local harness -- even without real GPUs -- exercises the same measurement discipline: warm-up runs, repeated trials, and percentile latency reporting rather than a single average.
Prerequisites
Basic Python 3 (standard library only) for the local harness
A multi-GPU host with NCCL and a built copy of NVIDIA/nccl-tests is required to run all_reduce_perf for real
RDMA-capable NICs on two hosts and a built copy of linux-rdma/perftest are required to run ib_write_bw for real
Timeline showing a warm-up phase discarded from results, followed by N measured trials whose latencies are collected and reported as p50/p95/p99 percentiles rather than a single average.
MLPerf-inspired local benchmark harness (not an official MLPerf submission)
PythonSafe · Local
Python
"""
Minimal benchmark harness demonstrating MLPerf-style measurement discipline:
warm-up, N measured trials, percentile latency, and throughput reporting.
Uses a mock inference function so it runs anywhere without a GPU or model.
"""
import time
import random
import statistics
def mock_inference(batch_size: int) -> None:
# Stand-in for a real model call; replace with your framework's inference API.
time.sleep(0.002 * batch_size + random.uniform(0, 0.001))
def benchmark(batch_size: int = 8, warmup: int = 5, trials: int = 50):
for _ in range(warmup):
mock_inference(batch_size)
latencies_ms = []
start = time.perf_counter()
for _ in range(trials):
t0 = time.perf_counter()
mock_inference(batch_size)
latencies_ms.append((time.perf_counter() - t0) * 1000)
total_s = time.perf_counter() - start
latencies_ms.sort()
p50 = latencies_ms[int(0.50 * trials) - 1]
p95 = latencies_ms[int(0.95 * trials) - 1]
p99 = latencies_ms[min(int(0.99 * trials), trials - 1)]
throughput = (trials * batch_size) / total_s
print(f"trials={trials} batch={batch_size}")
print(f"p50={p50:.2f}ms p95={p95:.2f}ms p99={p99:.2f}ms")
print(f"throughput={throughput:.1f} samples/sec")
if __name__ == "__main__":
benchmark()
Expected output
trials=50 batch=8
p50=16.4ms p95=17.1ms p99=17.6ms
throughput=478.3 samples/sec
(exact numbers vary run to run since this uses a mock timing function)
Verification steps
Run twice and confirm p50 <= p95 <= p99 holds on every run (a basic sanity check of the percentile math).
Increase `trials` to 500 and confirm the percentile values stabilize (less run-to-run variance) versus a small sample.
Common failure modes
This is not an official MLPerf submission or result -- MLCommons defines strict rules (reference implementations, submission review) for comparable published results.
Reporting only an average latency (instead of percentiles) hides tail latency that matters for real-time inference SLAs.
NCCL-tests all_reduce_perf: GPU collective communication microbenchmark
BashPlatform-dependent
Bash
# Requires a multi-GPU host (or multiple hosts) with CUDA, NCCL, MPI, and a
# built copy of NVIDIA/nccl-tests (https://github.com/NVIDIA/nccl-tests).
# all_reduce_perf drives an NCCL all-reduce across GPUs and reports both
# algorithm bandwidth (algbw) and bus bandwidth (busbw) per message size --
# busbw is the figure comparable across different collective algorithms and
# topologies (NVLink-only vs. NVLink+network).
./build/all_reduce_perf -b 8 -e 256M -f 2 -g 8
Expected output
# size count type redop time algbw busbw
# (B) (elements) (us) (GB/s) (GB/s)
8388608 2097152 float sum 412.3 20.35 35.61
16777216 4194304 float sum 771.2 21.75 38.06
268435456 67108864 float sum 10998.4 24.41 42.72
Verification steps
Confirm busbw approaches a large fraction of the GPUs' rated NVLink/NVSwitch (intra-node) or network (inter-node) bandwidth from Objective 2.1's fabric sizing -- a large, persistent gap points at a topology or fabric problem rather than the collective itself.
Re-run with `-g` set to a subset of GPUs (e.g., 2 instead of 8) and confirm busbw scales as expected for the interconnect in use (near-linear on NVLink, then flattening once traffic crosses the network fabric).
Common failure modes
Numbers above are illustrative, not measured on real hardware -- actual results depend on GPU generation, NVLink/NVSwitch generation, driver/NCCL version, and fabric topology.
busbw well below the interconnect's rated bandwidth with otherwise healthy hardware often points at the same fabric congestion/PFC-ECN issues covered in Objective 3.1, not a GPU problem.
Building and running nccl-tests requires a working CUDA/NCCL/MPI toolchain matched to your driver version; mismatched versions are a common source of build or runtime failures.
RDMA perftest ib_write_bw: raw RDMA write bandwidth microbenchmark
BashPlatform-dependent
Bash
# Requires two RDMA-capable NICs (RoCEv2 or InfiniBand) and a built copy of
# linux-rdma/perftest (https://github.com/linux-rdma/perftest). This measures
# raw RDMA write bandwidth between two hosts, independent of NCCL or any ML
# framework -- useful for isolating a NIC/fabric problem from a GPU/software
# problem before blaming a training job's throughput on "the network."
# On the server host:
ib_write_bw -d mlx5_0 -F
# On the client host (server_ip is the server's RDMA interface address):
ib_write_bw -d mlx5_0 -F server_ip
Expected output
---------------------------------------------------------------------------------------
RDMA_Write BW Test
Dual-port : OFF Device : mlx5_0
Number of qps : 1 Transport type : IB
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
65536 5000 24610.32 24580.11 0.393851
---------------------------------------------------------------------------------------
Verification steps
Confirm BW average is close to the NIC's rated line rate (e.g., a 200 Gbps NIC should approach ~25 GB/s), consistent with the bandwidth calculator in Objective 2.1.
Run in both directions (swap server/client roles) to confirm the link is symmetric; a large asymmetry points at a one-sided NIC, cabling, or PCIe issue.
Common failure modes
Numbers above are illustrative, not measured on real hardware -- actual results depend on NIC generation, PCIe generation/lane count, and fabric configuration (see Objective 3.1 for PFC/ECN/QoS tuning that affects sustained RDMA throughput).
`-d mlx5_0` is a placeholder RDMA device name -- list actual devices on your host with `ibv_devices` before running.
Low bandwidth with no errors reported by perftest itself often indicates a fabric-side issue (congestion, suboptimal ECMP hashing) rather than a NIC or perftest problem -- cross-check against Objective 3.1's verification steps.
Domain 4 · AI Infrastructure Operations and Troubleshooting (20%) · Objective 4.2
Implement Monitoring with Nexus Dashboard and Intersight
Fabric and infrastructure observability
Nexus Dashboard Insights/Fabric Controller provides fabric-wide telemetry, anomaly detection, and compliance checks for the network layer, while Intersight provides hardware health, firmware compliance, and policy drift detection for UCS compute. Implementing monitoring means both platforms are configured to collect telemetry continuously (not just polled ad hoc), and that their alerts are exported to a common location (webhook, syslog, or API poll) so operations staff correlate network and compute events together (Objective 4.3), rather than checking two disconnected UIs.
Prerequisites
An Intersight account for the API polling pattern; a Prometheus-compatible collector for the scrape config
Fig. 4-2 — Fabric and compute telemetry converge into one alerting pipeline.
Diagram showing Nexus Dashboard exporting fabric telemetry and Intersight exporting compute telemetry, both flowing into a shared alert correlation and notification pipeline consumed by operations staff.
Poll Intersight for hardware health alarms (documented SDK pattern)
PythonIllustrative · Pseudocode
Python
"""
Illustrative pattern using the documented Intersight Python SDK to poll
open hardware alarms. Requires real API credentials to execute (see
Objective 1.6 for authentication setup).
"""
import os
import intersight
from intersight.api import cond_api
configuration = intersight.Configuration(
host="https://intersight.com",
signing_info=intersight.HttpSigningConfiguration(
key_id=os.environ["INTERSIGHT_API_KEY_ID"],
private_key_path=os.environ["INTERSIGHT_API_PRIVATE_KEY_PATH"],
signing_scheme=intersight.signing.SCHEME_HS2019,
signing_algorithm=intersight.signing.ALGORITHM_ECDSA_MODE_DETERMINISTIC_RFC6979,
),
)
with intersight.ApiClient(configuration) as api_client:
api_instance = cond_api.CondApi(api_client)
alarms = api_instance.get_cond_alarm_list(filter="Severity eq 'Critical'")
for alarm in alarms.results:
print(alarm.code, alarm.description, alarm.creation_time)
Expected output
F0180 voltage-problem 2026-09-10T02:14:03Z
(empty list if there are no open critical alarms -- that is the healthy state)
Verification steps
Confirm the script exits 0 and prints nothing when no critical alarms are open (do not treat an empty result as an error).
Cross-check any printed alarm code against the Intersight UI's Alarms page for the same object.
Common failure modes
Polling too frequently against the public Intersight API can hit rate limits -- prefer a 1-5 minute interval for a health-check poller, not sub-second.
Filter syntax and field names must be verified against the current Intersight API reference before production use.
Prometheus scrape config for a fabric/GPU metrics exporter
YAMLPlatform-dependent
YAML
# prometheus.yml fragment -- scrapes a metrics exporter that surfaces
# fabric/GPU telemetry (e.g., DCGM exporter, or a custom Nexus Dashboard/Intersight bridge).
scrape_configs:
- job_name: "gpu-nodes"
scrape_interval: 15s
static_configs:
- targets: ["gpu-node-01.example.internal:9400", "gpu-node-02.example.internal:9400"]
- job_name: "fabric-telemetry-bridge"
scrape_interval: 30s
static_configs:
- targets: ["telemetry-bridge.example.internal:9100"]
Expected output
Prometheus 'Targets' page shows both jobs as State: UP after a `promtool check config prometheus.yml` and reload.
Verification steps
`promtool check config prometheus.yml` should report the config as valid.
After reload, confirm both targets show `UP` and a recent `Last Scrape` timestamp in the Prometheus UI.
Common failure modes
Target marked DOWN with a connection-refused error usually means the exporter isn't running or a firewall/security group blocks the scrape port.
Hostnames shown are illustrative internal DNS names -- replace with your actual exporter endpoints before use.
Domain 4 · AI Infrastructure Operations and Troubleshooting (20%) · Objective 4.3
Monitor Operational Telemetry
System health, alerts, log correlation
Beyond dashboards, day-2 operations require correlating events across sources: a network alert (e.g., PFC pause storm on a leaf switch) and a compute alert (e.g., a training job's throughput drop) that happen in the same time window are very likely related, but only if timestamps are consistent across devices (see the NTP policy in Objective 3.2) and logs are joined on a common key such as device ID or time bucket. This objective is about building that correlation habit and tooling, not just collecting more data.
Prerequisites
Basic Python 3
Fig. 4-3 — Correlating a network alert and a compute alert by time window.
Timeline showing a PFC pause-storm alert from a network device and a GPU throughput-drop alert from a compute node both falling inside the same 5-minute correlation window, flagged as a likely related incident.
Time-window log/alert correlation
PythonSafe · Local
Python
"""
Join network and compute alerts that fall within the same time window,
surfacing likely-related incidents for an operator to review first.
"""
from datetime import datetime, timedelta
def parse(ts: str) -> datetime:
return datetime.fromisoformat(ts)
network_alerts = [
{"device": "leaf-1", "type": "pfc_pause_storm", "ts": "2026-09-10T02:14:00"},
]
compute_alerts = [
{"device": "gpu-node-03", "type": "throughput_drop", "ts": "2026-09-10T02:15:30"},
{"device": "gpu-node-07", "type": "throughput_drop", "ts": "2026-09-10T03:40:00"},
]
WINDOW = timedelta(minutes=5)
def correlate(net_alerts, comp_alerts, window=WINDOW):
correlated = []
for n in net_alerts:
n_ts = parse(n["ts"])
for c in comp_alerts:
if abs(parse(c["ts"]) - n_ts) <= window:
correlated.append((n, c))
return correlated
for net, comp in correlate(network_alerts, compute_alerts):
print(f"LIKELY RELATED: {net['device']}/{net['type']} @ {net['ts']} <-> {comp['device']}/{comp['type']} @ {comp['ts']}")
Confirm the 03:40:00 alert on gpu-node-07 is correctly excluded (outside the 5-minute window).
Shrink WINDOW to 1 minute and confirm no pairs are returned, showing the window size directly controls correlation sensitivity.
Common failure modes
Correlation by time alone is a hint, not proof of causation -- always confirm with topology (is gpu-node-03 actually behind leaf-1?) before acting.
Clock drift between devices without NTP (Objective 3.2) can shift timestamps enough to miss genuinely related events.
Alerting rule for correlated GPU + fabric degradation
PromQLIllustrative · Pseudocode
PromQL
# Example Prometheus alerting rule expression (not a full rules file).
# DCGM_FI_DEV_GPU_UTIL is a percentage GAUGE (0-100), not a counter, so it
# must be aggregated directly -- never wrapped in rate(). node_network_
# receive_errs_total IS a counter, so rate() is correct there. The two
# metrics carry different label sets (GPU labels vs. NIC/device labels), so
# they are each aggregated down to a shared "instance" label first and then
# combined with an explicit "on(instance)" vector match rather than a bare
# `and`, which would silently drop results if the raw label sets differ.
avg by (instance) (DCGM_FI_DEV_GPU_UTIL) < 20
and on (instance)
sum by (instance) (rate(node_network_receive_errs_total{device="eth0"}[5m])) > 0
Expected output
Alert 'GPUStallLikelyFabricInduced' fires in Prometheus/Alertmanager for any `instance` where average GPU utilization is below 20% AND that same instance's NIC receive-error rate is greater than zero, for the evaluation interval.
Verification steps
Use `promtool test rules` with a synthetic test file to confirm the expression fires only when both sub-conditions hold for the same `instance` label value.
Confirm with `promtool query instant` (or the Prometheus UI) that the two aggregated sub-queries actually share an `instance` label before relying on `on(instance)` to match them.
Common failure modes
Metric names (DCGM_FI_DEV_GPU_UTIL, node_network_receive_errs_total) depend on which exporters are deployed -- confirm they exist in your `/metrics` output before wiring this into alerting rules.
If the GPU exporter and node exporter expose the host identity under different label names (e.g., `instance` vs. `node` vs. `Hostname`), relabel one side in your scrape config so both series share a common `instance` value, or the `on(instance)` match will silently return no results.
Wrapping a percentage gauge like DCGM_FI_DEV_GPU_UTIL in rate() is a common mistake -- rate() is only valid on monotonically increasing counters and produces meaningless results on gauges.
Domain 4 · AI Infrastructure Operations and Troubleshooting (20%) · Objective 4.4
Troubleshoot with System Messages and Management Tools
Layered troubleshooting workflow
Effective AI infrastructure troubleshooting works outward in layers: confirm the symptom (job stalled, slow throughput, failed deployment), check compute health (Intersight alarms, GPU state -- including GPU-specific system messages such as NVIDIA Xid codes in `dmesg`/syslog, e.g. Xid 79 'GPU has fallen off the bus' or ECC-related Xids like Xid 48), check network health (interface errors, PFC/ECN counters from Objective 3.1), check storage health (latency, capacity, multipath state from Objective 1.5/2.3), then correlate with recent changes (policy pushes, firmware updates). Jumping straight to a deep packet capture before ruling out an obvious hardware alarm or GPU Xid error wastes time -- a decision tree keeps the investigation ordered.
Prerequisites
Basic Bash
An NVIDIA GPU host with the driver and DCGM installed is required to reproduce real dmesg/Xid, dcgmi diag, or nvidia-bug-report.sh output
Fig. 4-4 — Layered troubleshooting decision tree for a stalled AI job.
Decision tree starting at 'Job stalled or slow'. First check: any active hardware alarm in Intersight, or a GPU Xid error in dmesg/syslog? If yes, resolve the hardware issue. If no, check network: any PFC/ECN or interface error counters incrementing? If yes, investigate fabric congestion. If no, check storage: latency or capacity alerts? If yes, investigate storage layer. If no, check for recent configuration or firmware changes correlated with the stall.
Layered health-check script implementing the decision tree
BashIllustrative · Pseudocode
Bash
#!/usr/bin/env bash
# Illustrative layered health check. Replace each placeholder command with
# your real Intersight/NX-OS/storage tooling; this script only shows the
# control flow and stops at the first layer reporting a problem.
set -euo pipefail
echo "1) Checking compute alarms (placeholder for: intersight-cli get alarms --severity critical, and dmesg | grep -i xid)"
COMPUTE_ALARMS=0 # replace with real check exit code / count
if [ "$COMPUTE_ALARMS" -gt 0 ]; then
echo "STOP: resolve compute hardware alarm(s) or GPU Xid error(s) first."; exit 1
fi
echo "2) Checking network error counters (placeholder for: show interface counters errors)"
NET_ERRORS=0 # replace with real parsed counter delta
if [ "$NET_ERRORS" -gt 0 ]; then
echo "STOP: investigate fabric congestion/PFC-ECN counters (see Objective 3.1)."; exit 1
fi
echo "3) Checking storage alerts (placeholder for: multipath -ll / capacity threshold check)"
STORAGE_ALERT=0 # replace with real check
if [ "$STORAGE_ALERT" -gt 0 ]; then
echo "STOP: investigate storage layer (see Objective 2.3)."; exit 1
fi
echo "4) No hardware/network/storage issue found -- check recent config or firmware changes."
Expected output
1) Checking compute alarms (placeholder for: intersight-cli get alarms --severity critical, and dmesg | grep -i xid)
2) Checking network error counters (placeholder for: show interface counters errors)
3) Checking storage alerts (placeholder for: multipath -ll / capacity threshold check)
4) No hardware/network/storage issue found -- check recent config or firmware changes.
Verification steps
Run with `bash -n script.sh` first to confirm syntax before wiring in real checks.
Set COMPUTE_ALARMS=1 manually and confirm the script stops at step 1 with exit code 1 (fail-fast behavior).
Common failure modes
The placeholders must be replaced with real, read-only status checks -- do not wire in commands that change device state during a health check.
Stopping at the first failing layer is intentional (avoids wasted investigation) but can miss a second, independent problem -- re-run the full script after fixing the first issue.
GPU system-message triage: Xid codes, dcgmi diag, and nvidia-bug-report.sh
BashPlatform-dependent
Bash
# Requires an NVIDIA GPU host with the driver installed; dcgmi and
# nvidia-bug-report.sh additionally require DCGM to be installed.
# 1) Check the kernel ring buffer / syslog for GPU Xid errors first -- these
# are NVIDIA's GPU-level hardware/driver error codes and often explain a
# stalled job faster than any higher-level dashboard.
dmesg -T | grep -i xid
journalctl -k --since "1 hour ago" | grep -i xid
# 2) Run an on-demand DCGM diagnostic. Level 1 is a quick (~seconds) sanity
# check; level 2 runs longer, more thorough tests; level 3 is the most
# exhaustive (can take many minutes and briefly loads the GPUs).
dcgmi diag -r 1
dcgmi diag -r 2
dcgmi diag -r 3
# 3) If the diagnostic or Xid points at a real hardware/driver problem,
# collect a full diagnostic bundle before escalating to NVIDIA/vendor support.
sudo nvidia-bug-report.sh
Expected output
[ ... ] NVRM: Xid (PCI:0000:3b:00): 79, pid=1234, GPU has fallen off the bus.
[ ... ] NVRM: Xid (PCI:0000:3b:00): 48, pid=1234, DBE (Double Bit ECC) has occurred
dcgmi diag -r 1:
+---------------------------+------------------------------------------------+
| Diagnostic | Result |
+===========================+==================================================+
| Deployment | Pass |
+---------------------------+------------------------------------------------+
nvidia-bug-report.sh: nvidia-bug-report.log.gz has been created
Verification steps
Confirm which Xid code appeared (if any) and cross-reference it against NVIDIA's Xid error reference before deciding whether it's a transient event or a hardware RMA candidate.
Confirm `dcgmi diag -r 1` reports 'Pass' on a healthy node as a fast pre-flight check before scheduling a longer `-r 2`/`-r 3` run, which briefly loads the GPUs and should not be run against a node with a live production job.
Confirm `nvidia-bug-report.sh` completes and produces a non-empty `nvidia-bug-report.log.gz` before attaching it to a support case.
Common failure modes
Xid 79 ('GPU has fallen off the bus') and ECC-related Xids (e.g., Xid 48) generally indicate a hardware, thermal, power, or PCIe seating problem rather than something a software restart will fix -- treat them as compute-layer alarms in the decision tree above, not as a job/software bug.
A recurring Xid on the same GPU after a reboot is a strong signal to escalate to hardware replacement rather than retrying the workload.
`dcgmi diag -r 3` is the most thorough level but is also the most disruptive -- do not run it against a GPU that is actively serving production inference or training traffic.
Output shown here is illustrative and was not captured from real hardware -- exact Xid codes, DCGM diagnostic formatting, and severity depend on GPU generation and driver/DCGM version.