开发者报告llama.cpp在旧款NVIDIA GPU上的优化
AI 总结
一名开发者报告了面向旧款NVIDIA GPU、特别是Kepler及其他前Ampere架构的llama.cpp改进。相关改动包括调整cuBLAS计算类型选择，使部分此前在初始化阶段失败的硬件能够运行，同时加入CUDA图和Windows NCCL集成。开发者使用4张Tesla K40c，以Q8_0量化运行MTP Qwen3.6-35B-A3B，报告在相同的时钟配置下启用CUDA图后，生成速度由约35升至55 tokens per秒。开发者还报告称，集成Windows NCCL后端到端性能提升约5%至10%，吞吐量超过60 tokens per秒，提示处理速度为70至100 tokens per秒。这些是开发者自身系统上的初步结果，并非通用基准。
深度见解
这项工作直接面向延长旧款NVIDIA硬件使用寿命、用于本地模型推理的目标。开发者报告的兼容性与性能提升目前仅来自其单一系统配置，仍需在不同架构和工作负载上进行系统性测试。
I've been working on improving llama.cpp support and performance on older NVIDIA GPUs, particularly Kepler and other pre-Ampere architectures.
The goal is simple: get more life out of older CUDA hardware instead of treating it as e-waste. Achieved Performance
On MTP Qwen3.6-35B-A3B with Q8_0 quantization, I've observed the following (generation) speeds using 4x Tesla K40c GPUs (Kepler, compute capability 3.5):
These are preliminary results from my own system, not universal benchmarks. More systematic testing across architectures and workloads is still needed.
I'm especially interested in hearing from people running Kepler, Maxwell, Pascal, Volta, and other older NVIDIA GPUs. If you have legacy hardware, I'd love to compare results and test compatibility across different configurations!
Project / source: https://github.com/theIvanR/llama-on-legacy-gpu
Llama.cpp pull request: https://github.com/ggml-org/llama.cpp/pull/30727
Windows NCCL fork: https://github.com/theIvanR/nccl-on-windows
Feedback, reproducible benchmarks, and contributions are very welcome! submitted by /u/TheSpicyBoi123
[link] [comments] |
#llama.cpp#NVIDIA#Qwen3.6-35B-A3B#Tesla K40c#CUDA graphs#Windows NCCL#cuBLAS