Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

NVIDIA's NeMo-DCR: bit-exact delta refit for trillion-parameter agentic RL

AI summary

NVIDIA researchers posted NeMo-DCR (Delta-Compressed Refit), a method that synchronizes policy updates between training and rollout clusters by sending only weight changes while remaining bit-exact against a dense refit. The paper reports that about 1% of BF16 training weights change stored values per step, and that at 3% and 5% change rates refits of 30B–1T models run 12–40x faster than a transport-only full-checkpoint reference; a 1T relay-tree refit at 3% takes 150 seconds versus 87.5 minutes to move a full checkpoint between two AWS regions. The code is open-sourced in NVIDIA NeMo RL PR #2444.

In-depth insight

Cutting cross-cluster weight sync from roughly 90 minutes to minutes could make continuous training-and-rollout loops practical at trillion-parameter scale.

Computer Science > Distributed, Parallel, and Cluster Computing

arXiv:2610.08430 (cs)

[Submitted on 6 Oct 2026]

Title:NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale

Authors:Songlin Jiang, Zhiyu Li, Terry Kong, Yu Yao, Youngeun Kwon, Bernard Nguyen, Ashwath Aithal, Mario Di Francesco

View a PDF of the paper titled NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale, by Songlin Jiang and 7 other authors

#NVIDIA#NeMo RL#NeMo-DCR#AWS

Original · 3 min readRead the original
Group chat

No comments yet. Start the conversation.