QLNI

Headlines · 1

1. Oct 21
   Paper: frozen 16 MB embedding table can match learned embeddings in small models

   A paper posted to r/LocalLLaMA, titled "Embeddings Are Overpriced?", proposes replacing a small language model's trainable embedding table with a frozen 16 MB table of 512-bit codes that has no trainable parameters, derived once from a pretrained embedding via principal-component projection and sign binarisation. The authors trained the same Gemma-shaped body from scratch on 2.05B tokens with and without the table; they report the learned table buys 0.20 nats of validation loss for 2.6 times the trainable parameters, while the frozen-table model is better or tied on LAMBADA, HellaSwag and PIQA, with only ARC-Easy favouring the learned table. They also report that with a ternary BitNet b1.58 body the whole model is a 46 MB file that ties the float model on LAMBADA and PIQA and generates 615 to 740 tokens per second on a laptop CPU in 62 MB of memory, and that weights, data and evaluation answers are released.

   r/LocalLLaMA · 🔥 0

Experience and discussion from the community

Share my QLNI experienceAsk about QLNI