Lean-QwenFlash以低内存NVMe卸载运行Qwen3.8-Flash-Next
AI 总结
一名开发者在帖子中报告称，Lean-QwenFlash通过直接卸载至NVMe SSD，可在配备16GB显存和8GB已分配系统内存的RTX 4090上运行125B参数的Qwen3.8-Flash-Next。测试采用3.05 bpw配置和Q4 KV缓存：RTX 4090在8K上下文下处理散文的速度为28–33 token/s、处理代码为19–22 token/s，在10万和26.2万上下文下，解码速度分别约为20.8和20.0 token/s。另一台配备16GB显存和32GB内存的RTX 3080 Laptop在10万上下文下达到约18–21 token/s。开发者还发布了设置说明、模型下载器和打包脚本。原帖中的性能数据均为开发者自行报告。
深度见解
该报告提供了明确的低内存NVMe卸载配置、长上下文基准结果，以及用于在本地运行125B参数模型的已发布工具。
| Hey everyone, Most current setups for Qwen3.8-Flash-Next (like Strata) keep all 24k experts pinned in system RAM (~48–64 GB) and compute cache misses on the CPU. That works great on high-spec workstations, but not on normal machines or laptops with limited RAM. A few weeks ago, I noticed that https://github.com/sybil-solutions/glm-flash-lite was running GLM-5.3 with surprisingly low memory footprints considering the model, way below anything I'd seen at those speeds, which made me realized that something about the design was likely really efficient and being slept on. So I adapted that to the much lighter Qwen3.8 Flash Next model, and the results were pretty impressive. Benchmarks (3.05bpw_h5_ng5, Q4 KV cache) Setup instructions, model downloaders, and packing scripts are up on GitHub: https://github.com/MaxDevv/lean-qwenflash Big thanks to turboderp for ExLlamaV3, sybil-solutions for the original GLM offload implementation, and Alibaba for Flash-Next. submitted by /u/MaxDev0 [link] [comments] |
#Lean-QwenFlash#Qwen3.8-Flash-Next#GLM-5.3#ExLlamaV3#NVIDIA RTX 4090#NVIDIA RTX 3080 Laptop