论文称冻结的16MB嵌入表可媲美小模型可学习嵌入
AI 总结
一篇发布在 r/LocalLLaMA 的论文《Embeddings Are Overpriced?》提出，用小语言模型原本可训练的嵌入表替换为一张冻结的 16MB、512 位编码表，该表不含可训练参数，由预训练嵌入经主成分投影和符号二值化一次性生成。作者用相同的 Gemma 形状主体、在相同的 2.05B token 上从头训练带与不带该表的模型；他们称可学习表以 2.6 倍可训练参数换来 0.20 nats 的验证损失优势，但冻结表模型在 LAMBADA、HellaSwag 和 PIQA 上更优或持平，仅 ARC-Easy 更利于可学习表。作者还称，搭配三元 BitNet b1.58 主体时整个模型是 46MB 文件，在 LAMBADA 和 PIQA 上追平浮点模型，并可在笔记本 CPU 上以 62MB 内存实现每秒 615 至 740 个 token 的生成速度；权重、数据和全部评测答案均已发布。
深度见解
若作者自述的结果成立，这种冻结、无可训练参数的词表有望在保持基准表现的同时，大幅降低小型与三元模型的内存占用和文件体积。
| In a small language model the vocabulary is the single largest part. In Gemma 3 270M the embedding table holds 168M of 268M parameters, and because quantisation methods leave the embedding in 16-bit, it is also most of the file once the rest of the model is compressed. We ask what that table actually buys. We replace it with one frozen table of 512-bit codes, 16 MB, with no trainable parameters, derived once from a pretrained embedding by principal-component projection and sign binarisation, and we train the same Gemma-shaped body from scratch on the same 2.05B tokens with and without it. The learned table buys 0.20 nats of validation loss for 2.6 times the trainable parameters, but on LAMBADA, HellaSwag and PIQA the frozen-table model is better or tied; only ARC-Easy favours the learned table. With a ternary BitNet b1.58 body the whole model is a 46 MB file, still ties the float model on LAMBADA and PIQA, and on a laptop CPU generates 615 to 740 tokens per second in 62 MB of memory, five to six times the speed of the float model's 8-bit GGUF in this http URL at a seventh of the memory. The frozen table wins by 1.7 to 2.3 nats on rare words, accepts 10,000 new words after training with a 3% cost where the learned table collapses by 43%, and all of it is released with the weights, the data and every evaluation answer. submitted by /u/pmttyji [link] [comments] |