Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

OpenEBFM releases code, weights and training stack

AI summary

The OpenEBFM authors released the full training and evaluation stack, pretrained checkpoints, training recipes and an EBFM-based chatbot. In their experiments, EBFMs achieved lower validation bits per byte than matched GPT baselines during data scaling, with the gap generally increasing as the training-token budget grew. From roughly 268 million to 1 billion parameters, EBFMs improved faster than corresponding GPT baselines during parameter scaling. On DCLM, the largest EBFM reached an average zero-shot question-answering accuracy of 44.71%, compared with 39.79% for the corresponding GPT baseline in the authors’ setup. The authors cautioned that these self-reported results do not establish that energy-based models are universally better than autoregressive models, and noted that the experiments remain at a scale of about 1 billion parameters.

In-depth insight

The release provides code, weights, recipes, checkpoints and a chatbot so others can reproduce and extend the results. It also presents scaling experiments that examine energy-based learning as an alternative to directly modeling token probabilities, while the reported findings remain limited to the authors’ setup and current scale.

We’ve been working on OpenEBFM (Energy-Based Foundation Models), and I wanted to share the project here and get feedback from the community. Most language foundation models today are based on directly modeling token probabilities. Our question was: Can energy-based learning scale to general-purpose foundation models? The main idea behind EBFM is to separate generation into two steps: The energy and generative heads share the same Transformer backbone. During pretraining, we also parallelize energy steering across all token positions with redesigned attention masks and positional indices, so that the model can be trained at sequence level rather than steering every position separately. What we found We pretrained EBFMs on FineWeb, DCLM, and ClimbMix, and studied scaling with both training tokens and model parameters. A few results we found interesting: For example, on DCLM, the largest EBFM reaches an average zero-shot QA accuracy of 44.71%, compared with 39.79% for the corresponding GPT baseline in our setup. We don't think these experiments establish that energy-based models are universally better than autoregressive models—the current scale is still around 1B parameters—but they suggest that energy-based learning continues to benefit from increasing data and model scale, which we think is an interesting result given that EBMs have historically been studied mostly in specialized settings. Everything is open source We released the full training and evaluation stack, pretrained checkpoints, training recipes, and an EBFM-based chatbot so that the results can be reproduced and extended. Paper: https://arxiv.org/abs/2610.33005 Code: https://github.com/datamllab/openebfm Models: https://huggingface.co/collections/guan-wang/openebfm Paper with Code: https://paperswithcode.co/paper/2610.33005 Disclosure: I'm one of the authors of this work. We’d especially appreciate feedback on the architecture and scaling results. More broadly, I’m curious what people here think: could energy-guided generation become a practical alternative to conventional autoregressive generation as models scale further? Criticism, reproduction attempts, issues, and contributions are all very welcome! Processing img jv81qyxgijyh1... Processing img 5rbyt7ygijyh1... Processing img jf6s09yqijyh1... Processing img ufjmxdtrijyh1... submitted by /u/Dear_Treat3688 [link] [comments]

#OpenEBFM#FineWeb#DCLM#ClimbMix#GPT

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.