ISTA-DASLab prefiller recovers Qwen3.8-27B low-quant accuracy via SSD streaming
A r/LocalLLaMA post describes an ISTA-DASLab demo that uses a prefiller model streamed from SSD to recover the accuracy lost by low-bit quantization of Qwen3.8-27B, while keeping VRAM use low and decoding fast. The post says the accompanying graph shows MMLU-Pro accuracy and points to a Hugging Face page, ISTA-DASLab/Qwen3.8-27B-NVFP4-prefiller. The poster notes the demo requires an Nvidia Blackwell GPU and was therefore not tested by them, and that the claim of no speed penalty versus running without the prefiller for contexts longer than about 4k is an untested expectation.
The demo targets a familiar trade-off — low-bit quantization saves VRAM but costs accuracy — with the poster claiming the prefiller regains accuracy while retaining low VRAM use and fast decoding.
| This demo from ISTA-DASLab recovers low quant accuracy while retaining low VRAM usage and fast decoding. It needs a Nvidia Blackwell GPU to run, so I have not tested it. For context longer than ~4k there should be no speed penalty compare to running without the prefiller model. Graph shows MMLU-Pro accuracy. Huggingface🤗: https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-NVFP4-prefiller submitted by /u/magnus-m [link] [comments] |