NVIDIA DSX OS: Cluster baseline management, readiness validation

1. Define configuration baselines in files and connect existing deployment tools

NVIDIA AI Cluster Runtime (AICR) creates baseline recipes that pin versions of components such as networking, storage, and drivers. A recipe is a configuration blueprint specifying which software to install on a cluster and in what order.

AICR converts this blueprint into artifacts: deployment files that existing tools can apply. Those tools handle deployment and reconciliation, letting teams retain their current pipelines, the flows that run successive work steps.

Recipes also specify dependencies, or prerequisite relationships between components, and post-installation checks. Teams can version and review baselines and detect drift from manual changes. Kubernetes can mark pods with unhealthy containers as running. This makes component health checks more important than checking pod status alone.

The commands below compare a live cluster against a recipe baseline and return a non-zero exit code, a number indicating the command's result, if it has drifted.

# Check a live cluster against the recipe's constraints
aicr validate --recipe recipe.yaml

# Compare a golden baseline against the live cluster, non-zero exit on drift
aicr diff --baseline ./golden.yaml --target cm://default/aicr-snapshot --fail-on-drift

2. Run real workloads to identify slow nodes and failure causes

Basic health checks establish whether services work, while NVIDIA Cluster Readiness Engine (NVCRE) tests whether multiple nodes can handle real workloads together. It runs training and GPU communication tests and records results and failure causes.

A certification specification documents test targets and acceptance criteria. It defines nodes, test types, and pass thresholds: numerical criteria for passing. The example selects communication and training tests and requires bus bandwidth, the capacity of a data transfer path, of at least 300GB per second. This is an example of a team-defined pass threshold, not a measurement.

Tests include level 4 diagnostics from DCGM, a GPU management tool, and pretraining with NVIDIA NeMo Framework: training a model ahead of full-scale work. Teams can also add their own container-based tests to certification checks.

If bandwidth is low in a 64-node communication test, NVCRE repeatedly splits the failing group and tests down to the minimum group size, reporting suspect nodes and reasons. One degraded node can slow all distributed training, where multiple GPUs share the work. This helps find bottlenecks, points that slow overall processing, that simple health metrics miss.

3. Link fault monitoring to isolation, recovery, and revalidation

NVSentinel monitors GPU diagnostics and error signals, the health of network interfaces that handle network connections, and cloud maintenance events. NVIDIA says monitoring uses no GPU time, leaving those resources available for running workloads.

The default monitoring mode reports faults as node conditions. After evaluating reports, teams can enable automatic isolation, draining to remove running workloads from nodes, and recovery. Safeguards also limit how many nodes can be isolated within a given period.

Revalidation can be requested when new nodes join or recovered nodes return. New workload assignments are blocked during validation. The block is lifted once tests pass and no other checks require it. Nodes that still fail after retries remain blocked.

NVCRE identifies which nodes failed and why, while NVSentinel handles isolation and recovery. This separation reduces duplicate blocking and inconsistent cleanup states between the systems. Platform teams can also pass certification results to their own node lifecycle tools, which manage nodes from creation to retirement, and let those tools handle operations.

4. Start with monitoring, but let teams define pass thresholds and recovery rules

All three tools are open source under Apache 2.0 and accept external contributions. Adoption requires installing and configuring each tool and its test environment. On live clusters, teams can start by reviewing monitoring reports, then link isolation and recovery rules to trusted signals.

Configuring automatic recovery and revalidation can reduce manual handoffs when repairing, testing, and returning nodes to available capacity. Teams must still design and maintain cluster-specific tests, thresholds, isolation rules, recovery limits, and staff response procedures.

Performance thresholds are not set by default, so nodes slower than the target may pass if none are specified. Validation does not start automatically after driver or management component upgrades or network changes; teams must request it.

DSX OS derives its value from connecting GPU cluster baselines, validation, and recovery, while the scope of automation depends on the tests and rules defined by operations teams.