Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

tinybrain opens Chinese model training code and 60.26B-token dataset

AI summary

The tinybrain project author open-sourced code and a cleaned Chinese pretraining dataset assembled from 26 public sources, with the full tier containing 60.26 billion tokens. The project documents model designs including mHC residual streams, Engram lookup memory, sliding-window attention, MoE, and a post-training workflow spanning SFT, distillation, and reinforcement learning. The author says usable model weights are not yet complete, while the reported loss and throughput figures are preliminary and self-reported.

In-depth insight

The release provides a detailed, practical reference for reproducing a Chinese-focused training pipeline on a single 16GB GPU, although its performance claims remain preliminary and its model weights are unfinished.

先把丑话说在前面:这篇文章不是来宣布「我训出了一个吊打某某的强力小模型」的。实际上,我现在连能拿出来直接用的权重都还没训完。

发这篇出来,主要是想聊聊我把一个中文小模型从洗数据、搭架构一直到强化学习(RL)完整手搓一遍的过程。中间每一步是怎么设计的、为什么要这么选,以及目前在显卡上跑出来的一些阶段性数据。代码和洗好的数据我都开源了,家里有一张 16GB 显存的显卡就能直接跟着跑。

为什么要自己重造轮子?

之前看 MiniMind 和 nanochat,它们其实已经证明了「一个人从零手搓小模型」这条路完全走得通,代码写得也很清爽。我在做 tinybrain 的时候,就想在这个基础上再往前折腾几步:

  • 数据自己从头洗,不直接拿别人现成洗好的数据集;
  • 模型结构跟进 2025–2026 年的新架构,不再局限于标准的 GPT-2 或 LLaMA 结构;
  • 后训练(Post-training)把全套流程走通:SFT、领域专家、在线蒸馏、强化学习全塞进去;
  • 以中文为核心,而且我按个人喜好,专门塞了不少古文和诗词进去。

#tinybrain#mHC#Engram#DeepSeek#DeepSeek-V3#Qwen3.8-27B#ModelScope

Original · 10 min readRead the original
Group chat

No comments yet. Start the conversation.