Meta研究字节级小模型蒸馏
AI 总结
r/LocalLLaMA 概述了 Meta 的一篇研究论文,该研究分别使用词元或原始字节训练约10亿参数模型,部分字节模型还通过基于词元的 Llama 3 8B 教师进行蒸馏。实际训练中词元模型领先,而字节模型据称会随训练延长继续提升,并以约六分之一的文本量达到相同性能水平。论文外推曲线预测字节模型将超过 Llama 3.2 1B 和 Gemma 3 1B,但这一结果并未在实际训练中出现;字节处理还需要约五倍的序列步骤。
深度见解
这些实验考察了字节级输入能否在保持实测性能的同时减少小模型的文本需求,但预测优势尚未在实际训练范围之外得到验证。
| https://arxiv.org/pdf/2609.12303 this paper suggests a model that skips models reading tokens that and has the model read raw bytes, basically one letter at a time. what they did: what they found: the catch (and there’s a few): still pretty cool imo. for small models on phones/laptops, squeezing more quality out of the same size matters a lot. and getting rid of the giant word list might fix the dumb stuff like miscounting letters in a word anyone know if there’s work on making byte models run as fast as token ones? submitted by /u/DotaMate [link] [comments] |