LookAtMyQuant 2 压缩 Qwen 与 Inkling 模型
LookAtMyQuant 2 是一款开源量化编译器,用于测量模型存储部分在压缩后对输出的影响,并为这些部分选择编码和比特预算。其方法结合后训练量化、trellis 编码、基于 Fisher 信息的逐行比特分配和蒸馏。项目发布了 1.23 GB 的 Qwen3.5-4B 文件,并报告将 Inkling-2-Mini 的 18.89 GB 文本权重压缩至 5.38 GB 和 8.87 GB。方法和 MIT 许可代码可通过项目网站获取。项目方称,在相同文件大小下,其 Gemma 4 E2B GGUF 的 KL 散度比 Q4_K_M 低 12 倍,但所提供报告没有独立佐证。
此次发布提供了有文档说明的量化流程、MIT 许可代码,以及更小的 Qwen 和 Inkling 模型文件。报告同时指出,项目未公开密集权重、逐题结果以及生成网站数据的脚本,相关比较也使用项目自身的评测设置。
LookAtMyQuant is a quantization compiler. It measures how much each stored piece of a model hurts the output when compressed. It then picks a code for each piece so the whole model fits a size budget with the least damage.
On Gemma 4 E2B, its GGUF has 12× lower KL than Q4_K_M at the same file size. The method and every result are on the site.
The code is under the MIT license.
LookAtMyQuant 2: Qwen3.5-4B in 1.2 GB
LookAtMyQuant 2 takes Qwen3.5-4B to 1.23 GB, 2.3 bits per weight. The file solves 330 of 540 exact-answer math problems. Prism ML's Bonsai 2 Ternary 4B solves 328 at 1.20 GB, and the original solves 359. Its KL from the original on held-out text is 0.45 nats, against 0.59 for Bonsai.
The recipe is post-training quantization with trellis codes, calibrated on the model's own worked math solutions with digit tokens weighted 10×, and with bits allocated per row by Fisher information. 4,000 steps of distillation follow, in which every layer is re-quantized once from the original weights. The page has the findings, the full recipe and its cost, about $34 of GPU time.
#LookAtMyQuant 2#Inkling-2-Mini#Qwen3.5-4B#Qwen3.8-27B#Gemma 4 E2B#Qwen3.8-27B-DSpine-b16#Unsloth#Prism ML#Bonsai 2 Ternary 4B#Thinking Machines Lab#Tencent#Apple