ik_llama.cpp
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 3
- ik_llama.cpp adds native PQ2_0 support for Ternary Bonsai 2
A community guide reports that ik_llama.cpp now provides native CPU and CUDA support for PrismML’s Ternary Bonsai 2 in PQ2_0 format. It cites merged CUDA kernels, a lossless repacking script for upstream TQ2_0 and TQ1_0 weights, and runtime repacking with `-rtr 1` for faster CPU prompt processing; no independent benchmarks are provided.
r/LocalLLaMA · 🔥 4 - ik_llama.cpp Fork Update Adds RTX 2060 Inference Results
IceFog72 released an updated ik_llama.cpp fork and invited community feedback on r/LocalLLaMA. The developer self-reported about 45–48 tokens per second on short prompts, about 37 tokens per second at a 28K context length, and prefill speeds of roughly 430–700 tokens per second on an RTX 2060 with 6 GB of VRAM. The update retains the core idea of the earlier approach but has a slightly different implementation and still requires cleanup and broader testing.
r/LocalLLaMA · 🔥 0 - CPU low-bit benchmark: bitnet.cpp wrong, Bonsai correct, ik_llama.cpp fastest
A community benchmark posted on r/LocalLLaMA compared low-bit GGUF runtimes against reference outputs on an ARM Neoverse-N1 server and x86 AVX2 VMs. It reports that bitnet.cpp main (commit 0b341e5) is incorrect for BitNet b1.58 2B4T: x86 text looks fine but perplexity is about 6× the HF transformers reference on the same tokens, while ARM prints ????; the author says fixes from issue #588 and an ARM kernel, bundled in PR #642's third version dated Oct 15, are correct on both. Ternary-Bonsai 1.7B files gave correct output wherever they loaded in llama.cpp, the Prism fork and ik_llama.cpp, with prompt-processing speeds varying by file, runtime and CPU — 71 t/s for PQ2_0 in ik_llama.cpp with -rtr versus 28 in Prism — and the Q2_0 file marked recommended on the model card did not load in any of the three runtimes.
r/LocalLLaMA · 🔥 0
Experience and discussion from the community
Share my ik_llama.cpp experienceAsk about ik_llama.cpp
Nobody has shared their experience with ik_llama.cpp yet.