At cluster scale, GPU faults are a reality. An Xid error, a double-bit ECC error, a GPU falling off the PCIe bus or a flapping InfiniBand link can stall a distributed training job or take an inference replica down. Spotting that something is wrong is usually the easy part. The hard part is everything between the fault and a healthy node again

Earlier this year, we announced the launch of our fully Automatic node remediation features. Our GPU clusters customers and our internal Inference platform has been using this capability to help drive workload resiliency and minimize down time. Our customers love this capability, and we wanted to take some time break the system down and explain how it works and why we designed it this way.

We first implemented “Passive health” checks system, that watches every node continuously, both inside your VM and on the host underneath it. When a check finds a fault, auto repair recommends the lightest action to fix and shows you supporting evidence behind the recommendation. You decide up front, fault by fault, whether a repair waits for approval or runs on its own. You also decide how long auto repair waits for running workload to finish. Your own detectors can open repairs through the same pipeline by writing a single annotation on a node.