社区帖子报告在8 GB显存上运行262K上下文的Qwen-Flash-Next Coder
AI 总结
一篇Reddit社区帖子称，通过llama.cpp在8 GB显存、24 GB内存和262K上下文下运行了强量化的Qwen-Flash-Next Coder。该配置采用每权重3.5比特的Coder变体和选择性混合专家卸载，将大多数专家放在内存中、较小的专家放在显存中，并把n-gram数据存放在最快的SSD上。帖子报告提示处理速度为70 token/s，生成速度约为8 token/s，其中生成受PCIe限制。帖子还指出，受内存映射影响，首次运行速度可能较慢。
深度见解
该帖子记录了一种据称仅需8 GB显存即可本地运行长上下文混合专家模型的方法，即根据模型组件特点将其分别部署在显存、内存和SSD中。
| The methodology Qwen-Flash-Next has n-gram, MOE packing above its weight but it's still 6B active params. We exploit this by keeping most MOEs in RAM (Smallest MOEs go in VRAM for speedup) and n-gram on disk - ensure you put the model on your fastest SSD. 8 GB VRAM, 24 GB RAM cost. You can run this fine if you've ~32 GB total memory too, but you'd have to turn down context size. Customize the launch params to best suit your hardware. The model & quant Many people mistake this model for a 1 bit quant so let's get that out of the way. This model here is a 3.5 Bits Per Weight "Coder" variant by ISTA DASLab, featuring GSQ RCO quantization. Half of its experts are removed, so relatively to original param size it has the size of a 1.9 bit quant but the actual weights use 3.5 BPW. see from model page: Averaged over the transformer, this corresponds to 1.89 bits per parameter of the original model. The figure is an effective rate: it amortises the removed experts over the original parameter count, and so expresses the combined effect of pruning and quantization. No individual weight is stored at 1.89 bits. The retained weights are stored at 3.5 bpw, unchanged by pruning, and the halved expert count accounts for the remainder of the reduction. Following the convention of the accompanying releases, the figure is reported over the transformer... Non-coding vocabulary pruned 400~ MB smaller, recommended Official ISTA DASLab quant The numbers With selective moe-offloading (regex) we reach 70 T/s PP and 8~ T/s PG (Limited by PCIE, which I've 3.0 YMMV) whereas if you'd use --n-cpu-moe for same effect it'd been slower due to the effects mentioned here Because of MMAP your first run will be slower, do a second run for accurate T/s results. Launch params are in the comments below. Ensure you use the qwen sharp chat template, it reduces overthinking. submitted by /u/jonathanx37 [link] [comments] |