创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

OpenEBFM 开源代码、权重与训练栈

AI 总结

OpenEBFM 作者开放了完整训练与评测栈、预训练检查点、训练配方,以及基于 EBFM 的聊天机器人。在作者的数据扩展实验中,EBFM 的验证 bits per byte 低于匹配的 GPT 基线,且差距通常随着训练词元预算增加而扩大。在参数扩展实验中,从约 2.68 亿至 10 亿参数,EBFM 的改进速度快于对应的 GPT 基线。在 DCLM 上,最大的 EBFM 零样本问答平均准确率为 44.71%,而作者设置中的对应 GPT 基线为 39.79%。作者提醒,这些自报结果不能证明能量基础模型普遍优于自回归模型,并指出当前实验规模仍约为 10 亿参数。

深度见解

此次发布提供了代码、权重、训练配方、检查点和聊天机器人,便于他人复现并扩展结果,同时通过扩展实验研究能量学习作为直接建模词元概率之外的一种替代方案。不过,报告的发现仍受限于作者实验设置和当前模型规模。

We’ve been working on OpenEBFM (Energy-Based Foundation Models), and I wanted to share the project here and get feedback from the community. Most language foundation models today are based on directly modeling token probabilities. Our question was: Can energy-based learning scale to general-purpose foundation models? The main idea behind EBFM is to separate generation into two steps: The energy and generative heads share the same Transformer backbone. During pretraining, we also parallelize energy steering across all token positions with redesigned attention masks and positional indices, so that the model can be trained at sequence level rather than steering every position separately. What we found We pretrained EBFMs on FineWeb, DCLM, and ClimbMix, and studied scaling with both training tokens and model parameters. A few results we found interesting: For example, on DCLM, the largest EBFM reaches an average zero-shot QA accuracy of 44.71%, compared with 39.79% for the corresponding GPT baseline in our setup. We don't think these experiments establish that energy-based models are universally better than autoregressive models—the current scale is still around 1B parameters—but they suggest that energy-based learning continues to benefit from increasing data and model scale, which we think is an interesting result given that EBMs have historically been studied mostly in specialized settings. Everything is open source We released the full training and evaluation stack, pretrained checkpoints, training recipes, and an EBFM-based chatbot so that the results can be reproduced and extended. Paper: https://arxiv.org/abs/2610.33005 Code: https://github.com/datamllab/openebfm Models: https://huggingface.co/collections/guan-wang/openebfm Paper with Code: https://paperswithcode.co/paper/2610.33005 Disclosure: I'm one of the authors of this work. We’d especially appreciate feedback on the architecture and scaling results. More broadly, I’m curious what people here think: could energy-guided generation become a practical alternative to conventional autoregressive generation as models scale further? Criticism, reproduction attempts, issues, and contributions are all very welcome! Processing img jv81qyxgijyh1... Processing img 5rbyt7ygijyh1... Processing img jf6s09yqijyh1... Processing img ufjmxdtrijyh1... submitted by /u/Dear_Treat3688 [link] [comments]

#OpenEBFM#FineWeb#DCLM#ClimbMix#GPT

原文约 1 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。