Training large-scale AI models on thousands of GPUs is a high-stakes operation where monitoring and automation must be built-in to optimize cluster resiliency and uptime at scale. This concept isn’t new to data center operators who have measured mean-time-to-repair (MTTR) and mean-time-between-failure (MTBF) long before AI factories existed.

Now the stakes are higher. Modern AI factories run distributed training jobs across hundreds of nodes for days or weeks, and at this scale the operating model changes: hardware anomalies, software errors, network timeouts, and resource contention must be managed proactively to keep workloads on track. Traditional approaches that rely on manual intervention, static checkpointing, and reactive monitoring simply don’t scale when organizations are running dozens of these concurrent multi-node workloads in shared infrastructure.

This is why software services, like the autonomous recovery engine offered in NVIDIA Mission Control, are critical to scaling production AI. Previously, the autonomous recovery engine, which can detect, diagnose, and recover from anomalies across the stack, was limited to high-performance computing (HPC) workloads leveraging Slurm. As a part of Mission Control 2.5, this service is now extended to Kubernetes-based workloads managed using NVIDIA Run:ai.