As distributed AI training scales, failures across hardware and software dependencies increasingly disrupt training progress, making fast, reliable recovery essential. A stalled training rank can trigger an NVIDIA Collective Communications Library (NCCL) timeout, a GPU node can stop responding to the scheduler, or a straggler can slow the entire job. Each disruption is manageable on its own, but the cumulative impact and recovery overhead determine how efficiently the system converts wall-clock time into productive training progress or goodput.

Manual recovery becomes unsustainable when it must cover every training job across a cluster, around the clock, at scale. Each interruption demands log analysis, fault isolation, and a restart from a saved checkpoint, while affected GPUs remain idle during recovery. As the number and size of training jobs grow, overlapping failures increase the engineering burden and wasted GPU-hours. Without continuous coverage, those losses can accumulate, increasing overall compute costs and extending time to train.

NVIDIA Recovery, part of NVIDIA DSX OS, the modular software portfolio with the NVIDIA DSX AI Factory Platform, is an open-source, workload-level service that automates recovery across a cluster’s training jobs, around the clock. It detects interruptions using application logs, GPU metrics, and scheduler signals, uses failure attribution to identify implicated resources, and coordinates isolation and checkpoint restarts from checkpoints on healthy capacity. By automating this recovery loop across workloads, it handles concurrent interruptions, reduces manual intervention, and helps sustain training goodput. This post explains how the recovery loop works, how NVIDIA Recovery integrates with Slurm, and how to deploy it on your own cluster.