Prime Intellect扩展Mega MoE训练能力
Prime Intellect 称，其为 DeepSeek 原本仅支持前向计算的 Mega MoE 内核增加了重叠反向传播，使这一融合式 MoE 实现可用于分布式训练。厂商测试称，Prime Mega MoE 内核最高可比默认 PyTorch 实现快 3.11 倍，端到端训练吞吐达到 Prime-RL 基线的 1.34 倍。测试使用 H200 GPU，并在 FP8 配置下运行 Qwen3-30B-A3B 和 Qwen3-235B-A22B。
Mega MoE 将计算与通信融合，直接针对 Prime Intellect 所述分布式 MoE 训练中的 GPU 空闲和跨 GPU 传输瓶颈。
Prime Mega MoE: Fast Fused MoE Kernels for Training
Mixture-of-experts (MoE) models scale more efficiently by activating only a subset of experts per token, but communication between GPUs can become a major bottleneck during training.
We extended DeepSeek's Mega MoE kernel with a fast, overlapping backward pass, making it usable for training and delivering a significant speedup:
- Up to 3.11× faster MoE kernels compared to the default PyTorch-based implementation.
- Up to 1.34× faster end-to-end training compared to the Prime-RL baseline (34% higher throughput).
Technical details
In an MoE model, each token is routed to only a small subset of the model's experts. Those experts live on different GPUs, so training requires a lot of communication between them. In every layer, tokens are sent to the GPU that holds their expert, the expert computation runs, and the results are sent back. This happens twice per layer on every step: once in the forward pass and once in the backward pass.
#Prime Intellect#DeepSeek#Prime Mega MoE#Mega MoE#DeepGEMM#Prime-RL#PyTorch#Qwen3-30B-A3B#Qwen3-235B-A22B#H200