Developer releases Rust and CUDA LLVQ implementation for Qwen3
Developer Pier-Jean Malandrino released an open-source Rust implementation of LLVQ with a custom CUDA kernel, along with quantized Qwen3 4B, 8B, and 14B files at roughly 2.7 bits per parameter. In the author's same-day L40S tests, LLVQ reportedly beat llama.cpp's IQ2_S on MMLU, GSM8K, and WikiText-2 quality at comparable bit rates, but decoded more slowly. The results remain self-reported, with limitations including calibration sensitivity, limited hardware coverage, higher long-prompt memory use, and support only for dense Qwen3 models up to 14B.
The release provides a new low-bit inference path for local Qwen3 models, with encouraging reported quality results, but its practical value depends on independent validation and whether decoding and long-context limitations can be addressed.
Some source content is omitted. Continue reading on the original site.
#LLVQ#Qwen3#Qualcomm#CUDA#NVIDIA#llama.cpp#IQ2_S#Unsloth UD-IQ2_XXS#AWQ