CPU low-bit benchmark: bitnet.cpp wrong, Bonsai correct, ik_llama.cpp fastest
A community benchmark posted on r/LocalLLaMA compared low-bit GGUF runtimes against reference outputs on an ARM Neoverse-N1 server and x86 AVX2 VMs. It reports that bitnet.cpp main (commit 0b341e5) is incorrect for BitNet b1.58 2B4T: x86 text looks fine but perplexity is about 6× the HF transformers reference on the same tokens, while ARM prints ????; the author says fixes from issue #588 and an ARM kernel, bundled in PR #642's third version dated Oct 15, are correct on both. Ternary-Bonsai 1.7B files gave correct output wherever they loaded in llama.cpp, the Prism fork and ik_llama.cpp, with prompt-processing speeds varying by file, runtime and CPU — 71 t/s for PQ2_0 in ik_llama.cpp with -rtr versus 28 in Prism — and the Q2_0 file marked recommended on the model card did not load in any of the three runtimes.
The author argues low-bit CPU deployments need reference-based checks, because x86 output can look fine while perplexity is roughly six times the HF transformers reference.
| I wanted to know which low-bit (1-bit / ternary / 2-bit) GGUF files actually give correct output on CPU, and how fast they run, so I compared each runtime against a reference instead of judging by how the text looks. Machines: an ARM server (Neoverse-N1) and x86 VMs with AVX2 (Intel Xeon, AMD EPYC). BitNet b1.58 2B4T Bonsai / Ternary-Bonsai (PrismML) Every number has the runtime commit, the file sha256 and the flags, and one script reruns the main measurements. A notebook runs a subset on a free Kaggle or Colab VM (~30 min to 1 h) and compares it with the published numbers. I only have Neoverse-N1 and AVX2 x86. Results from AVX-512 / Zen 4, Graviton 3/4 or Raspberry Pi 5 would fill the biggest gaps; Apple Silicon too, but the scripts are Linux-only so far. Report: https://github.com/matheus-c-feitosa/lowbit-atlas Corrections welcome. I already fixed one: an earlier "~6× slower" for the 27B PTQ1_0 file came from rough timings; llama-bench says 4.4×. (English isn't my first language. I used Claude Code to help with the analysis and the wording; the runs were done on my ARM server and on free Kaggle/Colab VMs, and I checked the numbers.) submitted by /u/matheus-c-feitosa [link] [comments] |
#BitNet b1.58 2B4T#bitnet.cpp#Ternary-Bonsai 1.7B#Bonsai#PrismML#llama.cpp#ik_llama.cpp#lowbit-atlas#Hugging Face Transformers