Developer reports llama.cpp gains on legacy NVIDIA GPUs
A developer reported improvements to llama.cpp for older NVIDIA GPUs, particularly Kepler and other pre-Ampere architectures. The changes include fixes to cuBLAS compute-type selection that let some hardware that previously failed during initialization run, along with CUDA graphs and Windows NCCL integration. In preliminary testing with MTP Qwen3.6-35B-A3B at Q8_0 quantization on four Tesla K40c GPUs, generation speed rose from about 35 to 55 tokens per second after enabling CUDA graphs under the same reported clock configuration. The developer also reported a 5–10% end-to-end improvement with Windows NCCL, throughput above 60 tokens per second, and prompt-processing speeds ranging from 70 to 100 tokens per second. These were preliminary results from the developer’s own system, not universal benchmarks.
The work directly addresses the goal of extending the useful life of older NVIDIA hardware for local model inference. Its reported compatibility and performance gains remain specific to one developer’s setup and require systematic testing across architectures and workloads.
I've been working on improving llama.cpp support and performance on older NVIDIA GPUs, particularly Kepler and other pre-Ampere architectures.
The goal is simple: get more life out of older CUDA hardware instead of treating it as e-waste. Achieved Performance
On MTP Qwen3.6-35B-A3B with Q8_0 quantization, I've observed the following (generation) speeds using 4x Tesla K40c GPUs (Kepler, compute capability 3.5):
These are preliminary results from my own system, not universal benchmarks. More systematic testing across architectures and workloads is still needed.
I'm especially interested in hearing from people running Kepler, Maxwell, Pascal, Volta, and other older NVIDIA GPUs. If you have legacy hardware, I'd love to compare results and test compatibility across different configurations!
Project / source: https://github.com/theIvanR/llama-on-legacy-gpu
Llama.cpp pull request: https://github.com/ggml-org/llama.cpp/pull/30727
Windows NCCL fork: https://github.com/theIvanR/nccl-on-windows
Feedback, reproducible benchmarks, and contributions are very welcome! submitted by /u/TheSpicyBoi123
[link] [comments] |
#llama.cpp#NVIDIA#Qwen3.6-35B-A3B#Tesla K40c#CUDA graphs#Windows NCCL#cuBLAS