Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

LookAtMyQuant 2 Compresses Qwen and Inkling Models

AI summary

LookAtMyQuant 2 is an open-source quantization compiler that measures how compression affects a model’s output and assigns codes and bit budgets to stored model components. Its recipe uses post-training quantization, trellis codes, Fisher information for per-row bit allocation, and distillation. The project released a 1.23 GB Qwen3.5-4B artifact and reports compressing Inkling-2-Mini from 18.89 GB of text weights to 5.38 GB and 8.87 GB. The method and MIT-licensed code are available through the project site. The project claims its Gemma 4 E2B GGUF has 12× lower KL divergence than Q4_K_M at the same file size, but the supplied report contains no independent corroboration.

In-depth insight

The release provides a documented quantization workflow, MIT-licensed code, and smaller Qwen and Inkling model files. The report also notes that the dense weights, per-problem results, and script used to generate the website data are not published, while the reported comparisons use the project’s own evaluation setup.

LookAtMyQuant is a quantization compiler. It measures how much each stored piece of a model hurts the output when compressed. It then picks a code for each piece so the whole model fits a size budget with the least damage.

On Gemma 4 E2B, its GGUF has 12× lower KL than Q4_K_M at the same file size. The method and every result are on the site.

The code is under the MIT license.

LookAtMyQuant 2: Qwen3.5-4B in 1.2 GB

LookAtMyQuant 2 takes Qwen3.5-4B to 1.23 GB, 2.3 bits per weight. The file solves 330 of 540 exact-answer math problems. Prism ML's Bonsai 2 Ternary 4B solves 328 at 1.20 GB, and the original solves 359. Its KL from the original on held-out text is 0.45 nats, against 0.59 for Bonsai.

The recipe is post-training quantization with trellis codes, calibrated on the model's own worked math solutions with digit tokens weighted 10×, and with bits allocated per row by Fisher information. 4,000 steps of distillation follow, in which every layer is re-quantized once from the original weights. The page has the findings, the full recipe and its cost, about $34 of GPU time.

#LookAtMyQuant 2#Inkling-2-Mini#Qwen3.5-4B#Qwen3.8-27B#Gemma 4 E2B#Qwen3.8-27B-DSpine-b16#Unsloth#Prism ML#Bonsai 2 Ternary 4B#Thinking Machines Lab#Tencent#Apple

Original · 15 min readRead the original
Group chat

No comments yet. Start the conversation.