Together AI Details Generally Available Auto Repair for GPU Clusters
Together AI Blog says automatic node remediation for Together GPU Clusters is generally available, following an earlier launch announcement. The system pairs passive health checks with evidence-backed repair recommendations, configurable approvals, workload-aware delays, API and console repair, and automatic host re-validation; node-label remediation contracts remain in preview. Together AI says customers and its internal Inference platform have used the capability to improve resiliency and reduce downtime, but the report provides no independent measurements of those results.
The release gives GPU-cluster operators a policy-driven workflow for detecting, approving, executing, and validating node repairs, though its reported resilience benefits remain vendor claims.
At cluster scale, GPU faults are a reality. An Xid error, a double-bit ECC error, a GPU falling off the PCIe bus or a flapping InfiniBand link can stall a distributed training job or take an inference replica down. Spotting that something is wrong is usually the easy part. The hard part is everything between the fault and a healthy node again
Earlier this year, we announced the launch of our fully Automatic node remediation features. Our GPU clusters customers and our internal Inference platform has been using this capability to help drive workload resiliency and minimize down time. Our customers love this capability, and we wanted to take some time break the system down and explain how it works and why we designed it this way.
We first implemented “Passive health” checks system, that watches every node continuously, both inside your VM and on the host underneath it. When a check finds a fault, auto repair recommends the lightest action to fix and shows you supporting evidence behind the recommendation. You decide up front, fault by fault, whether a repair waits for approval or runs on its own. You also decide how long auto repair waits for running workload to finish. Your own detectors can open repairs through the same pipeline by writing a single annotation on a node.