SkyPilot: topology-aware scheduling for Grace Blackwell NVL72 racks
SkyPilot says running the same distributed training job over InfiniBand instead of NVLink can degrade performance by 43%, and that topology-aware scheduling can yield 27% more usable GPUs from the same Grace Blackwell racks. The post describes NVIDIA's NVL72 systems, which connect all 72 GPUs in a rack over a unified NVLink fabric (18 NVLink ports per GPU at 100 GB/s each, 130 TB/s aggregate internally), and how the SkyPilot Platform keeps multi-node jobs rack-local through Topology-Aware Scheduling, in-place recovery from hardware failures in Frontier Trainer, and Automatic Defragmentation of shared clusters. It also cites Meta's measured mean time between interruptions of roughly 490 days per 4-GPU node, implying about 27 days for an 18-node rack, and recommends leaving spare nodes in each rack.
For Grace Blackwell clusters, job placement and failure handling determine how much of a rack is actually usable, and SkyPilot claims topology-aware scheduling recovers 27% more usable GPUs from the same hardware, though the performance and cost figures are vendor-supplied.
Not all high-speed interconnects are created equal, and using the wrong one can be quite costly: running the same distributed training job over InfiniBand rather than NVLink can degrade performance by 43%.
Historically, NVLink was limited to communication within a single node, meaning that multi-node AI workloads were forced to communicate over InfiniBand or a cloud equivalent (EFA on AWS, GPUDirect-TCPX on GCP). NVIDIA’s NVL72 systems remove this limitation by connecting all 72 GPUs in a rack with a unified NVLink fabric, so GPUs on different nodes can exchange data with an order of magnitude more bandwidth.
However, maximizing the benefits of NVL72 systems is highly dependent on workload placement: a multi-node job spread across racks is still bottlenecked by InfiniBand. Preserving rack-locality gets harder as hardware failures take nodes offline and jobs finishing at different times scatter free capacity across racks.
#SkyPilot#NVIDIA#GB200 NVL72#NVLink#InfiniBand#DeepSeek-R1#Meta#Megatron-LM#Frontier Trainer#SkyPilot Platform