twobit-residual-spec

Headlines · 1

1. Oct 22
   2-bit draft plus residual verifier scheme targets models that don't fit VRAM

   A Reddit user posting as Happy_Guitar3521 in r/LocalLLaMA proposed keeping a 2-bit copy of a model in GPU VRAM and the 6–8-bit residual weights in system RAM, with the GPU drafting tokens and the CPU verifying them so only activations cross PCIe. The author reports self-measured results from dequantized weights run in PyTorch: a 2-bit plus 8-bit residual setup was closer to bf16 than Q8_0 at about half the KL divergence, a 6-bit residual reached Q8_0-level quality while reading roughly 28% fewer bytes from RAM, and the 2-bit draft reusing the verifier's KV cache accepted 25–30% more tokens per check than Google's official Gemma 4 31B drafter (5.23 vs 4.20). These results are self-reported and not independently verified; the post provides no tok/s figures, says the approach only makes sense when Q8 does not fit in VRAM, and shares a public Hugging Face repository for others to experiment with.

   r/LocalLLaMA · 🔥 0

Experience and discussion from the community

Share my twobit-residual-spec experienceAsk about twobit-residual-spec