2-bit草稿加残差验证的投机解码方案
r/LocalLLaMA 用户 Happy_Guitar3521 提出一种面向显存放不下模型的投机解码方案：在 GPU 显存中保留 2-bit 模型副本，在系统内存中存放 6–8 bit 残差权重，由 GPU 用 2-bit 副本起草 token、CPU 用残差部分验证，只有激活值经过 PCIe。作者称，在 PyTorch 中反量化运行真实权重得到的自测结果显示，2-bit 加 8-bit 残差比 Q8_0 更接近 bf16，KL 散度约为后者一半；6-bit 残差达到 Q8_0 级质量，同时从内存读取的字节数少约 28%；复用验证器 KV cache 的 2-bit 草稿每次检查接受的 token 数比 Google 官方 Gemma 4 31B 草稿模型多 25–30%（5.23 对 4.20）。这些数据均为作者自测、未经独立验证；该帖没有提供 tok/s 数据，作者表示只有在 Q8 无法放入显存时该方案才有意义，并在 Hugging Face 公开了仓库供他人实验。
它为显存受限的本地推理提供了一个公开分享的具体投机解码思路，但性能说法均为作者自测、未经独立验证。
Hi Guys!, Recently I was making a research and development, but I do not have time to go further with this idea (kids, work, etc.). But maybe it will be interesting for someone. I do not think it would be something big, but maybe interesting approach that can be used in some of the cases. I believe that it is not AI slop, you can use those information's and try by yourself using Claude or Astra. The idea was my own and I am using it in my production app, it saved me a lot of money (project is not related to LLM's). Not sure if it is outdated or not, it can work also with MTP. Please, just don’t be unkind. I’m sharing my idea with you, and anyone is welcome to experiment with it.
TLDR: Keep a 2-bit version of the model on the GPU and the rest of the weights (the residual, 6-8 bit) in system RAM. The 2-bit copy drafts tokens, then the full model checks them: the GPU does the 2-bit part, the CPU does the residual part, and only activations go over PCIe. So the draft costs no extra memory, it's literally half of the verifier.
#twobit-residual-spec#Hugging Face#Gemma 4#llama.cpp#PyTorch#SubSpec