Developer shares fast LoRA kernels for llama.cpp, lifting Strix Halo prompt processing
A developer posting on r/LocalLLaMA as u/woct0rdho shared LoRA kernels for llama.cpp, saying the project currently has no fast LoRA kernels and that running a model with a LoRA on Strix Halo dropped prompt processing from about 1500 to 300 t/s even though the adapter was only 35 MB and touched few modules. The author reports that a general LoRA kernel raised throughput to 1000 t/s and a fused kernel to 1400 t/s. The changes are currently optimized for Strix Halo and have not yet been upstreamed to mainline llama.cpp or strix-llama.cpp, though the author says porting to other GPUs should not be hard and invites others to reuse the code, which is also discussed in a llama.cpp GitHub issue. The post also points to a third-party Qwen3.8-Flash-Next LoRA that produces an uncensored model without re-downloading the full weights, with a tunable LoRA scale.
LoRA adapters let users change a local model without downloading new weights, but the author's measurements indicate llama.cpp currently pays a large prompt-processing penalty for using them.
I'd like to share a few changes that encourage people to create and use LoRA on LLMs. For example, the LoRA https://huggingface.co/chenrm/qwen3.8-flash-next-abliterated-lora can turn the Qwen3.8-Flash-Next model into the uncensored version, whether you're using Q8_0, IQ4_NL, or GSQ-RCO-Q2_0 model, so you don't need to download the whole model again. You can also tune the LoRA scale for a balance between uncensoring and brain damage.
However, currently llama.cpp does not have fast LoRA kernels. When we run the model with the LoRA on Strix Halo, PP drops from like 1500 to 300 t/s, even though the LoRA is as small as 35 MB and only touches a few modules in the model.
I've made a few LoRA kernels. With a general LoRA kernel it's improved to 1000 t/s, and with a fused LoRA kernel it's further improved to 1400 t/s.
You can see my changes at https://github.com/woct0rdho/llama.cpp . Currently it's optimized for Strix Halo, and it should not be too hard to port to other GPUs. I'll find some time to upstream it to the mainline llama.cpp or strix-llama.cpp , but it involves a few architectural changes so it would take some time. If you want to upstream it earlier than me, feel free to reuse or modify my code.
This is also discussed in https://github.com/ggml-org/llama.cpp/issues/27093
submitted by /u/woct0rdho [link] [comments]