llama.cpp 快速 LoRA 内核提升推理速度
在 r/LocalLLaMA 上，用户 u/woct0rdho 分享了自己为 llama.cpp 编写的 LoRA 内核，称该项目目前没有快速 LoRA 内核，在 Strix Halo 上运行带 LoRA 的模型时，提示处理速度从约 1500 t/s 降到 300 t/s，而该 LoRA 仅 35 MB、只作用于模型少数模块。作者称通用 LoRA 内核可把速度提升到 1000 t/s，融合内核进一步提升到 1400 t/s。这些改动目前针对 Strix Halo 优化，尚未合并进 llama.cpp 主线或 strix-llama.cpp，作者表示移植到其他 GPU 应该不难，并欢迎他人复用代码，相关讨论也出现在 llama.cpp 的 GitHub issue 中。帖子还提到一个第三方的 Qwen3.8-Flash-Next LoRA，无需重新下载完整权重即可得到无审查版本，并可调节 LoRA 强度。
LoRA 让用户无需重新下载权重即可改造本地模型，但作者的测量显示，目前 llama.cpp 使用 LoRA 会付出很大的提示处理性能代价。
I'd like to share a few changes that encourage people to create and use LoRA on LLMs. For example, the LoRA https://huggingface.co/chenrm/qwen3.8-flash-next-abliterated-lora can turn the Qwen3.8-Flash-Next model into the uncensored version, whether you're using Q8_0, IQ4_NL, or GSQ-RCO-Q2_0 model, so you don't need to download the whole model again. You can also tune the LoRA scale for a balance between uncensoring and brain damage.
However, currently llama.cpp does not have fast LoRA kernels. When we run the model with the LoRA on Strix Halo, PP drops from like 1500 to 300 t/s, even though the LoRA is as small as 35 MB and only touches a few modules in the model.
I've made a few LoRA kernels. With a general LoRA kernel it's improved to 1000 t/s, and with a fused LoRA kernel it's further improved to 1400 t/s.
You can see my changes at https://github.com/woct0rdho/llama.cpp . Currently it's optimized for Strix Halo, and it should not be too hard to port to other GPUs. I'll find some time to upstream it to the mainline llama.cpp or strix-llama.cpp , but it involves a few architectural changes so it would take some time. If you want to upstream it earlier than me, feel free to reuse or modify my code.
This is also discussed in https://github.com/ggml-org/llama.cpp/issues/27093
submitted by /u/woct0rdho [link] [comments]