Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Amazon SageMaker HyperPod adds workload recovery checks

AI summary

AWS introduced three resilience capabilities for Amazon SageMaker HyperPod: MultiNodeTrainingValidation, Kubernetes resource monitoring, and hung-job detection. They target numerical faults, Kubernetes infrastructure failures, and Slurm jobs that stop progressing, with the latter able to cancel and requeue affected jobs.

In-depth insight

AWS says the features can identify distributed-training failures earlier, shorten investigations, and reduce GPU time spent on workloads that cannot make useful progress.

Large training jobs can fail even when every node looks healthy. A GPU with a subtle fault can pass its stress tests and still return wrong values. When a job stalls on an NCCL timeout, it keeps its GPUs, and Slurm still shows it as running. If a device plugin crashes on one node, the job fails with an error that never mentions the plugin. Teams usually find these problems only after the loss diverges or the job fails, and by then the compute is spent. Each of these failures lowers goodput, the share of your cluster’s time that goes to useful training. Today, we’re introducing workload recovery and infrastructure checks on Amazon SageMaker HyperPod that catch these failures sooner and raise the goodput of your training workloads.

SageMaker HyperPod already handles nodes that report hardware faults. The health monitoring agent (HMA) watches each node for GPU errors, deep health checks test nodes during provisioning, and automatic node recovery reboots or replaces faulty hardware. These features look at one node’s hardware at a time. They can’t tell whether the GPUs train correctly, whether a job is still making progress, or whether the Kubernetes components a job depends on are working.

#Amazon SageMaker HyperPod#MultiNodeTrainingValidation#HyperPod Kubernetes Resource Monitoring Agent#Amazon Web Services#Amazon Elastic Kubernetes Service#Amazon CloudWatch#Slurm

Original · 26 min readRead the original
Group chat

No comments yet. Start the conversation.