BaGuaLu-2 brings runtime-adaptive activation management to MoE training
机器之心 reports that a Tsinghua University-led team’s BaGuaLu-2 paper was accepted by SC26. The system uses runtime load information to decide separately for individual operators whether to save, recompute, or offload activations, while scheduling additional work around pipeline gaps. The team released code for its activation-memory analysis model and scheduler-integrated offloading and reload mechanisms, along with integration examples and a neutral evaluation benchmark; the public repository does not include every runtime mechanism described in the paper. In team-reported tests on 128 to 12,288 Hygon DCUs with models from 188 billion to 1.04 trillion parameters, the system achieved average throughput gains of 1.14× over Megatron-Kuai and 1.13× over AdaPipe.
Activation-memory spikes caused by uneven expert loads can trigger out-of-memory failures and interrupt distributed MoE training. BaGuaLu-2 addresses this with runtime, operator-level decisions and overlap scheduling, while its released components provide engineers with code and examples for integrating these techniques into existing training pipelines.
机器之心 2026-10-28 12:03 北京
运行时自适应化解显存尖峰，无需修改配置，代码已开源
机器之心发布
大模型训练已经跑到万卡集群规模，显存始终是那道隐形的门槛。
#BaGuaLu-2#BaGuaLu#Tsinghua University#Megatron-Kuai#AdaPipe#Hygon DCU#Hygon BW1000