创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

百度开源ERNIE 5.5 Flash Base权重

AI 总结

据Hugging Face模型卡和The Decoder报道,百度发布了稀疏MoE模型ERNIE-5.5-Flash-Base,总参数量280B,每个Token激活14B参数。该模型从零完成33.5T Token预训练,训练上下文最高128K,并宣称微调和推理支持1M Token上下文;FP8和BF16权重现已开放,可供开发者继续适配。百度称其在推理、代码和长上下文评测中领先同规模模型,但现有报道未提供独立复现结果。

深度见解

此次发布为领域适配、指令微调等下游训练提供了大型预训练检查点,但百度的长上下文与效率主张仍需独立评估。

Table of Contents

1. Introduction

A base model is valuable both for the capabilities that pre-training builds and for the starting point it offers for further training. Different domains, reasoning tasks and agentic applications call for different data, training objectives and behaviors. Releasing the weights at the end of pre-training lets developers build on the general capabilities already in place and choose for themselves where to take the model next.

Today we are releasing ERNIE-5.5-Flash-Base, a sparse Mixture-of-Experts (MoE) language model with 280B total parameters, 14B of which are activated per token. It was pre-trained from scratch on 33.5T tokens with a training context of up to 128K, and supports a 1M-token context for fine-tuning and inference. The weights are the checkpoint at the end of pre-training, before any mid- or post-training, and can serve as a common starting point for domain adaptation, instruction tuning and reinforcement-learning research.

To be worth building on, a base model has to stay affordable to train and run, so efficiency was a design goal from the outset. Further training samples from the model repeatedly, and agentic applications bring multi-turn interactions and long contexts. Both add to compute and memory costs. ERNIE 5.5 Flash therefore revisits the trade-off between capability and efficiency in every component, from attention and residual connections to the MoE layers and the output head, and weighs activated compute, KV cache and long-context cost together rather than one at a time.

#Baidu#ERNIE-5.5-Flash-Base

原文约 12 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。