Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

AWS Adds Silent Data Corruption Checks to HyperPod Training

AI summary

AWS Machine Learning Blog describes three checks for a PyTorch DistributedDataParallel training loop that detect silent data corruption before corrupted values can spread through gradients and replica weights. The checks cover non-finite gradients, ring replay of micro-batches, and post-update weight fingerprint comparison; when triggered, the workflow maps a suspect rank to its node and asks SageMaker HyperPod to replace it. AWS says it tested the approach on a four-instance cluster with 32 NVIDIA H100 GPUs using software-injected faults.

In-depth insight

The workflow addresses a hardware failure mode that can escape GPU health monitoring because a corrupted compute result looks valid and raises no error, while providing a concrete path to detect and contain it during distributed training.

Silent data corruption (SDC) happens when hardware, such as a GPU, returns a wrong but plausible number and raises no error. In distributed training, the wrong number can flow into the gradients and weights of every replica. Loss can still look normal for hours, and by the time a team notices an unexplained divergence or a weaker model, the compute is spent. It happens at scale: during a 54-day period of Llama 3 405B pre-training on up to 16K H100 GPUs, Meta attributed 6 of 419 unexpected job interruptions to silent data corruption, and those were only the cases that interrupted the job and were traced to SDC.

Amazon SageMaker HyperPod already replaces nodes that report a fault: the health-monitoring agent watches for GPU errors, deep health checks stress-test nodes, and automatic node recovery swaps out bad hardware. SDC slips past all of these features, because the hardware never reports anything. Detection has to come from the training job itself.

In this post, we add three checks to a PyTorch DistributedDataParallel (DDP) training loop to detect SDC. When a check fires, we map the suspect rank to its node, contain the corruption, and have SageMaker HyperPod replace the node. Each check covers a different place where a fault can land:

#Amazon SageMaker HyperPod#AWS#PyTorch DistributedDataParallel#NVIDIA H100#Meta#Llama 3 405B

Original · 32 min readRead the original
Group chat

No comments yet. Start the conversation.