2-bit draft plus residual verifier scheme targets models that don't fit VRAM
A Reddit user posting as Happy_Guitar3521 in r/LocalLLaMA proposed keeping a 2-bit copy of a model in GPU VRAM and the 6–8-bit residual weights in system RAM, with the GPU drafting tokens and the CPU verifying them so only activations cross PCIe. The author reports self-measured results from dequantized weights run in PyTorch: a 2-bit plus 8-bit residual setup was closer to bf16 than Q8_0 at about half the KL divergence, a 6-bit residual reached Q8_0-level quality while reading roughly 28% fewer bytes from RAM, and the 2-bit draft reusing the verifier's KV cache accepted 25–30% more tokens per check than Google's official Gemma 4 31B drafter (5.23 vs 4.20). These results are self-reported and not independently verified; the post provides no tok/s figures, says the approach only makes sense when Q8 does not fit in VRAM, and shares a public Hugging Face repository for others to experiment with.
It offers a concrete, publicly shared speculative-decoding idea for VRAM-constrained local inference, but the performance claims are self-reported and unverified.
Hi Guys!, Recently I was making a research and development, but I do not have time to go further with this idea (kids, work, etc.). But maybe it will be interesting for someone. I do not think it would be something big, but maybe interesting approach that can be used in some of the cases. I believe that it is not AI slop, you can use those information's and try by yourself using Claude or Astra. The idea was my own and I am using it in my production app, it saved me a lot of money (project is not related to LLM's). Not sure if it is outdated or not, it can work also with MTP. Please, just don’t be unkind. I’m sharing my idea with you, and anyone is welcome to experiment with it.
TLDR: Keep a 2-bit version of the model on the GPU and the rest of the weights (the residual, 6-8 bit) in system RAM. The 2-bit copy drafts tokens, then the full model checks them: the GPU does the 2-bit part, the CPU does the residual part, and only activations go over PCIe. So the draft costs no extra memory, it's literally half of the verifier.
#twobit-residual-spec#Hugging Face#Gemma 4#llama.cpp#PyTorch#SubSpec