创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

SageMaker HyperPod新增工作负载恢复检查

AI 总结

AWS为Amazon SageMaker HyperPod推出三项韧性功能:MultiNodeTrainingValidation、Kubernetes资源监控和挂起作业检测。它们分别用于识别数值错误、Kubernetes基础设施故障,以及停止取得进展的Slurm作业;后者还能取消并重新排队受影响作业。

深度见解

AWS表示,这些功能可更早发现分布式训练故障、缩短排查时间,并减少无法产生有效进展的工作负载所占用的GPU时间。

Large training jobs can fail even when every node looks healthy. A GPU with a subtle fault can pass its stress tests and still return wrong values. When a job stalls on an NCCL timeout, it keeps its GPUs, and Slurm still shows it as running. If a device plugin crashes on one node, the job fails with an error that never mentions the plugin. Teams usually find these problems only after the loss diverges or the job fails, and by then the compute is spent. Each of these failures lowers goodput, the share of your cluster’s time that goes to useful training. Today, we’re introducing workload recovery and infrastructure checks on Amazon SageMaker HyperPod that catch these failures sooner and raise the goodput of your training workloads.

SageMaker HyperPod already handles nodes that report hardware faults. The health monitoring agent (HMA) watches each node for GPU errors, deep health checks test nodes during provisioning, and automatic node recovery reboots or replaces faulty hardware. These features look at one node’s hardware at a time. They can’t tell whether the GPUs train correctly, whether a job is still making progress, or whether the Kubernetes components a job depends on are working.

#Amazon SageMaker HyperPod#MultiNodeTrainingValidation#HyperPod Kubernetes Resource Monitoring Agent#Amazon Web Services#Amazon Elastic Kubernetes Service#Amazon CloudWatch#Slurm

原文约 26 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。