Community post reports 262K-context Qwen-Flash-Next Coder on 8 GB VRAM
AI summary
A Reddit community post reports running a heavily quantized Qwen-Flash-Next Coder through llama.cpp with 262K context, 8 GB of VRAM and 24 GB of RAM. The setup uses a 3.5-bit-per-weight Coder variant with selective mixture-of-experts offloading, keeping most experts in RAM, smaller experts in VRAM and n-gram data on the fastest SSD. The post reports 70 tokens per second for prompt processing and about 8 tokens per second for generation, with generation limited by PCIe. It also notes that the first run may be slower because of memory mapping.
In-depth insight
The post documents a reported approach to running a long-context mixture-of-experts model locally with 8 GB of VRAM by selectively placing model components across VRAM, RAM and SSD.
| The methodology Qwen-Flash-Next has n-gram, MOE packing above its weight but it's still 6B active params. We exploit this by keeping most MOEs in RAM (Smallest MOEs go in VRAM for speedup) and n-gram on disk - ensure you put the model on your fastest SSD. 8 GB VRAM, 24 GB RAM cost. You can run this fine if you've ~32 GB total memory too, but you'd have to turn down context size. Customize the launch params to best suit your hardware. The model & quant Many people mistake this model for a 1 bit quant so let's get that out of the way. This model here is a 3.5 Bits Per Weight "Coder" variant by ISTA DASLab, featuring GSQ RCO quantization. Half of its experts are removed, so relatively to original param size it has the size of a 1.9 bit quant but the actual weights use 3.5 BPW. see from model page: Averaged over the transformer, this corresponds to 1.89 bits per parameter of the original model. The figure is an effective rate: it amortises the removed experts over the original parameter count, and so expresses the combined effect of pruning and quantization. No individual weight is stored at 1.89 bits. The retained weights are stored at 3.5 bpw, unchanged by pruning, and the halved expert count accounts for the remainder of the reduction. Following the convention of the accompanying releases, the figure is reported over the transformer... Non-coding vocabulary pruned 400~ MB smaller, recommended Official ISTA DASLab quant The numbers With selective moe-offloading (regex) we reach 70 T/s PP and 8~ T/s PG (Limited by PCIE, which I've 3.0 YMMV) whereas if you'd use --n-cpu-moe for same effect it'd been slower due to the effects mentioned here Because of MMAP your first run will be slower, do a second run for accurate T/s results. Launch params are in the comments below. Ensure you use the qwen sharp chat template, it reduces overthinking. submitted by /u/jonathanx37 [link] [comments] |