NVIDIA CUDA

Headlines · 2

1. 10h
   ik_llama.cpp adds native PQ2_0 support for Ternary Bonsai 2

   A community guide reports that ik_llama.cpp now provides native CPU and CUDA support for PrismML’s Ternary Bonsai 2 in PQ2_0 format. It cites merged CUDA kernels, a lossless repacking script for upstream TQ2_0 and TQ1_0 weights, and runtime repacking with `-rtr 1` for faster CPU prompt processing; no independent benchmarks are provided.

   r/LocalLLaMA · 🔥 4
2. Oct 23
   Trainium3 Advances Native PyTorch Training, but CUDA Gap Remains

   SemiAnalysis reports that Anthropic is running production inference and reinforcement-learning rollouts on AWS Trainium2 and Trainium3, while AWS Neuron is moving Trainium toward native PyTorch training with eager mode, torch.compile and standard APIs. Its engineers found the Trainium3 direction promising after testing the TorchTitan and TorchNeuron stack, but encountered unspecified bugs, long compilation times and a platform that remains far less mature than NVIDIA's CUDA ecosystem. AWS says upcoming TorchNeuron releases will use precompiled NKI kernels and a redesigned compile stack to target faster time to first step, though that target has not yet been demonstrated.

   SemiAnalysis · 🔥 0

Experience and discussion from the community

Share my NVIDIA CUDA experienceAsk about NVIDIA CUDA