创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

Hugging Face发布Megatron-LM MoE训练手册

AI 总结

Hugging Face作者发布了一份面向实践的MoE训练手册,介绍如何使用Megatron-LM训练混合专家模型,并同步提供55张图、脚本和汇总推荐参数。内容涵盖硬件测试、镜像构建、稠密基线训练、单节点MoE对比、路由与负载均衡、服务及强化学习。作者称,手册中的经验来自CSCS Alps集群约80万GH200 GPU小时的MoE训练;所供报道未对此数字进行独立核实。

深度见解

这份手册为开发者提供了从稠密模型训练转向Megatron-LM MoE训练的端到端参考,并包含代码、脚本、配置建议,以及基于作者所述训练经验给出的推荐。

This handbook is for people who have trained a dense model and want to train an MoE next, or who are curious how MoE training works. Chapter 2 is a Megatron-LM runbook: test the hardware, build the image, train a dense baseline, then a first MoE on one node, compared against that baseline. Chapter 3 takes the MoE layer apart (router, routing, load balancing, experts, block layout, training, systems). For each part we give the Megatron-LM code, what to watch, our recommendation, and what frontier MoEs chose. Chapter 4 follows the model into serving and RL. Chapters 3 and 4 need no GPU. When a result comes from our runs, we say so. Claims about frontier models cite their paper, config or code, and the toy simulations are labelled as such. The lessons come from about 800k GH200 GPU-hours of MoE training on the Alps cluster at CSCS. Released: the handbook, its 55 figures as a gallery (https://huggingface.co/spaces/huggingface/moe-gallery), the scripts (https://gist.github.com/nmntz/5de952d686f66d7bce5bce8cba9d7139), and every recommended flag in one Megatron-LM block. Happy to answer questions! submitted by /u/lvwerra [link] [comments]

#Hugging Face#Megatron-LM#CSCS#Alps#GH200

原文约 1 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。