NVIDIA has unveiled NVIDIA Mission Control 2.5, extending automated AI training recovery to Kubernetes. Teams running large-scale training can restart jobs while investigating failures and isolate suspect hardware from future jobs.

1. Separating workload recovery from hardware action

The self-healing engine previously covered HPC workloads using Slurm for large-scale computation. It now covers Kubernetes workloads managed by the included NVIDIA Run:ai. Teams running both can use the same recovery model.

A scheduler, workload recovery service, and hardware recovery service share the process. The scheduler selects resources. Workload recovery uses failure history and placement to plan retries, while hardware recovery checks faulty resources.

The hardware self-healing service monitors nodes and GPUs. It removes faulty resources from scheduling and begins diagnostics or remediation. Workload recovery need not wait for repairs, and suspect nodes cannot receive the next job.

2. Different recovery paths for different failures

When a GPU failure stops training, error signals, job context, and execution state help identify suspect nodes. Workload recovery restarts or relocates jobs through the scheduler. Hardware recovery isolates and diagnoses nodes in parallel.

Transient software faults, such as worker exits, recoverable timeouts, or unresponsiveness, allow retries under scheduler policy. Healthy hardware remains available and monitored. GPU alerts or connection issues can exclude nodes before jobs fail.

3. Run:ai integration and installation preparation

Workload recovery integrates with Run:ai's KAI scheduler. Logs and status are matched to failure timing to identify hardware or software causes. Job specifications are updated to exclude implicated nodes before jobs return to the queue.

The Autonomous Resiliency Engine comprises autonomous job recovery, autonomous hardware recovery, and NVIDIA Resiliency Extensions (NVRx). NVRx is not installed with NVIDIA Mission Control and must be installed separately, following the instructions at https://nvidia.github.io/nvidia-resiliency-ext/.

Separately, training apps can integrate the Autonomous Job Recovery Training Telemetry Client Library to record state. It automatically produces log messages monitored by the job recovery service — covering throughput, checkpoints, and hang detection without manual log formatting — so the app can supply recovery services with the data they need.

Install the training telemetry client from NVIDIA PyPI with the following command.

pip install --extra-index-url https://pypi.nvidia.com --upgrade aidot_training_telemetry

4. Check eligible workloads and hardware first

The release notes list workload management and both self-healing components as supported on DGX B200, B300, GB200 NVL72, and GB300 NVL72 systems. Teams using supported hardware gain an option to extend recovery to Kubernetes training.

The release notes limit workload recovery to jobs using the Kubeflow PyTorch training operator. Vera Rubin NVL72 has only essential bring-up support in this release, without Run:ai or the self-healing engine.

5. Less manual response requires integration work

Automated recovery lets jobs resume without waiting for hardware repairs and reduces needless isolation of healthy hardware after transient software faults. NVIDIA says this separation reduces downtime and wasted resources.

Teams still need to integrate training app logs and install recovery components. Hardware self-healing has dropped Datadog, FluentBit, and Elastic integrations, so reviewing monitoring paths that rely on them adds to adoption costs.

Mission Control 2.5's value lies in separating job resumption from hardware diagnostics, moving Kubernetes training recovery from manual response to an integrated operational workflow.