创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

AWS为HyperPod训练加入静默数据损坏检测

AI 总结

AWS Machine Learning Blog介绍了一套用于PyTorch DistributedDataParallel训练循环的静默数据损坏检测方案,包含非有限梯度检查、环形重放审计和权重指纹比较三项检查。检查触发后,方案会把可疑rank映射到对应节点,隔离污染,并请求SageMaker HyperPod替换该节点。AWS称已在配备32块NVIDIA H100 GPU的四实例集群上使用软件注入故障进行测试。

深度见解

该方案针对一种不会被GPU健康监控发现的硬件故障:计算结果虽然错误但看起来合理且不报错,而训练任务内检查可直接定位并隔离受影响的节点。

Silent data corruption (SDC) happens when hardware, such as a GPU, returns a wrong but plausible number and raises no error. In distributed training, the wrong number can flow into the gradients and weights of every replica. Loss can still look normal for hours, and by the time a team notices an unexplained divergence or a weaker model, the compute is spent. It happens at scale: during a 54-day period of Llama 3 405B pre-training on up to 16K H100 GPUs, Meta attributed 6 of 419 unexpected job interruptions to silent data corruption, and those were only the cases that interrupted the job and were traced to SDC.

Amazon SageMaker HyperPod already replaces nodes that report a fault: the health-monitoring agent watches for GPU errors, deep health checks stress-test nodes, and automatic node recovery swaps out bad hardware. SDC slips past all of these features, because the hardware never reports anything. Detection has to come from the training job itself.

In this post, we add three checks to a PyTorch DistributedDataParallel (DDP) training loop to detect SDC. When a check fires, we map the suspect rank to its node, contain the corruption, and have SageMaker HyperPod replace the node. Each check covers a different place where a fault can land:

#Amazon SageMaker HyperPod#AWS#PyTorch DistributedDataParallel#NVIDIA H100#Meta#Llama 3 405B

原文约 32 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。