BitNet b1.58 2B4T

Headlines · 1

1. Oct 19
   CPU low-bit benchmark: bitnet.cpp wrong, Bonsai correct, ik_llama.cpp fastest

   A community benchmark posted on r/LocalLLaMA compared low-bit GGUF runtimes against reference outputs on an ARM Neoverse-N1 server and x86 AVX2 VMs. It reports that bitnet.cpp main (commit 0b341e5) is incorrect for BitNet b1.58 2B4T: x86 text looks fine but perplexity is about 6× the HF transformers reference on the same tokens, while ARM prints ????; the author says fixes from issue #588 and an ARM kernel, bundled in PR #642's third version dated Oct 15, are correct on both. Ternary-Bonsai 1.7B files gave correct output wherever they loaded in llama.cpp, the Prism fork and ik_llama.cpp, with prompt-processing speeds varying by file, runtime and CPU — 71 t/s for PQ2_0 in ik_llama.cpp with -rtr versus 28 in Prism — and the Q2_0 file marked recommended on the model card did not load in any of the three runtimes.

   r/LocalLLaMA · 🔥 0

Experience and discussion from the community

Share my BitNet b1.58 2B4T experienceAsk about BitNet b1.58 2B4T