AWS为HyperPod训练加入静默数据损坏检测
AWS Machine Learning Blog介绍了一套用于PyTorch DistributedDataParallel训练循环的静默数据损坏检测方案,包含非有限梯度检查、环形重放审计和权重指纹比较三项检查。检查触发后,方案会把可疑rank映射到对应节点,隔离污染,并请求SageMaker HyperPod替换该节点。AWS称已在配备32块NVIDIA H100 GPU的四实例集群上使用软件注入故障进行测试。
该方案针对一种不会被GPU健康监控发现的硬件故障:计算结果虽然错误但看起来合理且不报错,而训练任务内检查可直接定位并隔离受影响的节点。
Silent data corruption (SDC) happens when hardware, such as a GPU, returns a wrong but plausible number and raises no error. In distributed training, the wrong number can flow into the gradients and weights of every replica. Loss can still look normal for hours, and by the time a team notices an unexplained divergence or a weaker model, the compute is spent. It happens at scale: during a 54-day period of Llama 3 405B pre-training on up to 16K H100 GPUs, Meta attributed 6 of 419 unexpected job interruptions to silent data corruption, and those were only the cases that interrupted the job and were traced to SDC.
Amazon SageMaker HyperPod already replaces nodes that report a fault: the health-monitoring agent watches for GPU errors, deep health checks stress-test nodes, and automatic node recovery swaps out bad hardware. SDC slips past all of these features, because the hardware never reports anything. Detection has to come from the training job itself.
In this post, we add three checks to a PyTorch DistributedDataParallel (DDP) training loop to detect SDC. When a check fires, we map the suspect rank to its node, contain the corruption, and have SageMaker HyperPod replace the node. Each check covers a different place where a fault can land:
#Amazon SageMaker HyperPod#AWS#PyTorch DistributedDataParallel#NVIDIA H100#Meta#Llama 3 405B