Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Baidu releases ERNIE 5.5 Flash Base weights

AI summary

Baidu released ERNIE-5.5-Flash-Base, a sparse Mixture-of-Experts model with 280B total parameters and 14B activated per token, according to the Hugging Face model card and The Decoder. The model was pretrained from scratch on 33.5T tokens, supports up to 128K tokens during training and a claimed 1M-token context for fine-tuning and inference, and its FP8 and BF16 weights are available for further adaptation. Baidu says it leads similarly sized models in reasoning, code, and long-context evaluations, but those benchmark results have not been independently reproduced in the supplied reports.

In-depth insight

The release gives developers a large pretrained checkpoint for domain adaptation, instruction tuning, and other downstream training, while Baidu’s long-context and efficiency claims still require independent evaluation.

Table of Contents

1. Introduction

A base model is valuable both for the capabilities that pre-training builds and for the starting point it offers for further training. Different domains, reasoning tasks and agentic applications call for different data, training objectives and behaviors. Releasing the weights at the end of pre-training lets developers build on the general capabilities already in place and choose for themselves where to take the model next.

Today we are releasing ERNIE-5.5-Flash-Base, a sparse Mixture-of-Experts (MoE) language model with 280B total parameters, 14B of which are activated per token. It was pre-trained from scratch on 33.5T tokens with a training context of up to 128K, and supports a 1M-token context for fine-tuning and inference. The weights are the checkpoint at the end of pre-training, before any mid- or post-training, and can serve as a common starting point for domain adaptation, instruction tuning and reinforcement-learning research.

To be worth building on, a base model has to stay affordable to train and run, so efficiency was a design goal from the outset. Further training samples from the model repeatedly, and agentic applications bring multi-turn interactions and long contexts. Both add to compute and memory costs. ERNIE 5.5 Flash therefore revisits the trade-off between capability and efficiency in every component, from attention and residual connections to the MoE layers and the output head, and weighs activated compute, KV cache and long-context cost together rather than one at a time.

#Baidu#ERNIE-5.5-Flash-Base

Original · 12 min readRead the original
Group chat

No comments yet. Start the conversation.