SkyPilot 谈 Grace Blackwell 机架级调度
SkyPilot 称，同一分布式训练任务若走 InfiniBand 而非 NVLink，性能可能下降 43%；其拓扑感知调度可让同一批 Grace Blackwell 机架多出 27% 可用 GPU。该文介绍 NVIDIA NVL72 将整机架 72 张 GPU 以统一 NVLink fabric 互联（每卡 18 个 NVLink 端口、每个 100 GB/s，机架内部聚合带宽 130 TB/s），并说明 SkyPilot Platform 如何通过拓扑感知调度、Frontier Trainer 的原地故障恢复与自动碎片整理，让多节点任务保持在机架内。文章引用 Meta 测得的 4-GPU 节点平均中断间隔约 490 天，推算 18 节点机架约 27 天一次中断，因此建议每个机架保留备用节点。
在 Grace Blackwell 集群中，作业放置与故障处理决定机架实际可用率；SkyPilot 声称拓扑感知调度可多获得 27% 可用 GPU，但相关性能与成本数字均为厂商口径。
Not all high-speed interconnects are created equal, and using the wrong one can be quite costly: running the same distributed training job over InfiniBand rather than NVLink can degrade performance by 43%.
Historically, NVLink was limited to communication within a single node, meaning that multi-node AI workloads were forced to communicate over InfiniBand or a cloud equivalent (EFA on AWS, GPUDirect-TCPX on GCP). NVIDIA’s NVL72 systems remove this limitation by connecting all 72 GPUs in a rack with a unified NVLink fabric, so GPUs on different nodes can exchange data with an order of magnitude more bandwidth.
However, maximizing the benefits of NVL72 systems is highly dependent on workload placement: a multi-node job spread across racks is still bottlenecked by InfiniBand. Preserving rack-locality gets harder as hardware failures take nodes offline and jobs finishing at different times scatter free capacity across racks.
#SkyPilot#NVIDIA#GB200 NVL72#NVLink#InfiniBand#DeepSeek-R1#Meta#Megatron-LM#Frontier Trainer#SkyPilot Platform