Llama 3 8B
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 2
- Meta studies byte-based small models trained with Llama 3 distillation
r/LocalLLaMA summarized a Meta research paper that trained roughly 1B-parameter models on either tokens or raw bytes, with some byte-model training distilled from a token-based Llama 3 8B teacher. Token-based models led in the completed runs, while byte-based models reportedly kept improving with longer training and used about six times less text to reach the same performance level. The paper’s extrapolated curves project a byte model above Llama 3.2 1B and Gemma 3 1B, but that result was not observed in the trained runs; byte processing also required roughly five times more sequence steps.
r/LocalLLaMA · 🔥 0 - Cola2 proposes continuous semantic compression for language modeling
The authors of the Cola2 paper propose a hierarchical language-modeling framework that combines continuous semantic compression, reconstruction and generation supervision, and co-adaptation between continuous and discrete representations. The system dynamically segments text into semantic spans and uses auxiliary-loss-free negative feedback to control compression, while a block-causal masked diffusion decoder handles generation. The authors report experiments on five downstream tasks using about 210 billion training tokens, but the performance claims have not been independently verified.
Hugging Face · Papers · 🔥 0
Experience and discussion from the community
Share my Llama 3 8B experienceAsk about Llama 3 8B
Nobody has shared their experience with Llama 3 8B yet.