Windows NCCL

Headlines · 1

1. Oct 24
   Developer reports llama.cpp gains on legacy NVIDIA GPUs

   A developer reported improvements to llama.cpp for older NVIDIA GPUs, particularly Kepler and other pre-Ampere architectures. The changes include fixes to cuBLAS compute-type selection that let some hardware that previously failed during initialization run, along with CUDA graphs and Windows NCCL integration. In preliminary testing with MTP Qwen3.6-35B-A3B at Q8_0 quantization on four Tesla K40c GPUs, generation speed rose from about 35 to 55 tokens per second after enabling CUDA graphs under the same reported clock configuration. The developer also reported a 5–10% end-to-end improvement with Windows NCCL, throughput above 60 tokens per second, and prompt-processing speeds ranging from 70 to 100 tokens per second. These were preliminary results from the developer’s own system, not universal benchmarks.

   r/LocalLLaMA · 🔥 0

Experience and discussion from the community

Share my Windows NCCL experienceAsk about Windows NCCL