Meta studies byte-based small models trained with Llama 3 distillation
r/LocalLLaMA summarized a Meta research paper that trained roughly 1B-parameter models on either tokens or raw bytes, with some byte-model training distilled from a token-based Llama 3 8B teacher. Token-based models led in the completed runs, while byte-based models reportedly kept improving with longer training and used about six times less text to reach the same performance level. The paper’s extrapolated curves project a byte model above Llama 3.2 1B and Gemma 3 1B, but that result was not observed in the trained runs; byte processing also required roughly five times more sequence steps.
The reported experiments show whether byte-level inputs can reduce text requirements for small models without sacrificing measured performance, while the projected advantage remains unverified beyond the completed runs.
| https://arxiv.org/pdf/2609.12303 this paper suggests a model that skips models reading tokens that and has the model read raw bytes, basically one letter at a time. what they did: what they found: the catch (and there’s a few): still pretty cool imo. for small models on phones/laptops, squeezing more quality out of the same size matters a lot. and getting rid of the giant word list might fix the dumb stuff like miscounting letters in a word anyone know if there’s work on making byte models run as fast as token ones? submitted by /u/DotaMate [link] [comments] |