llama.cpp CUDA PR fuses FFN operations for reported speed gains
AI summary
llama.cpp pull request #29948 proposes fusing CUDA FFN gate/up matrix multiplications and GLU in MMQ with interleaved gate/up rows. Its llama-bench results report gains of 1.9%–10.7% over master for listed quantizations of Qwen3.8-27B, Qwen3.6-27B, and Llama-3.1-8B on NVIDIA DGX Spark and RTX 5090. Tests used full GPU offload, Flash Attention, a 16,384-token prompt, and five repetitions.
In-depth insight
The benchmark results show higher token throughput for the proposed kernel optimization across the tested models, quantizations, and NVIDIA hardware, indicating a measurable improvement in the reported configurations.
time to download some speedup
DGX Spark (GB10), llama-bench -ngl 999 -fa on -p 16384 -n 0 -b 2048 -ub 2048 -ctk f16 -ctv f16 -r 5, tok/s:
RTX 5090, llama-bench -ngl 999 -fa on -p 16384 -n 0 -b 2048 -ub 2048 -ctk f16 -ctv f16 -r 5, tok/s: submitted by /u/jacek2023
[link] [comments] |
#llama.cpp#NVIDIA DGX Spark#NVIDIA RTX 5090#Qwen3.8-27B#Qwen3.6-27B#Llama-3.1-8B