低比特 CPU 基准：bitnet.cpp 输出错误，Bonsai 正确
AI 总结
r/LocalLLaMA 上的一项社区基准测试在 ARM Neoverse-N1 服务器和 x86 AVX2 虚拟机上，将低比特 GGUF 运行时与参考输出进行对比。作者称 bitnet.cpp main（提交 0b341e5）运行 BitNet b1.58 2B4T 时结果错误：x86 上文本看似正常，但困惑度约为同 token 上 HF transformers 参考的 6 倍，ARM 上则输出 ????；来自 issue #588 的修复和 ARM 内核（打包在 10 月 15 日第三版 PR #642 中）在两种 CPU 上均正确。Ternary-Bonsai 1.7B 文件在 llama.cpp、Prism 分支和 ik_llama.cpp 中能加载的地方均给出正确输出，提示处理速度随文件、运行时和 CPU 而异——ik_llama.cpp 加 -rtr 运行 PQ2_0 为 71 t/s，Prism 为 28——而模型卡上标为“推荐”的 Q2_0 文件在三种运行时中都无法加载。
深度见解
作者认为低比特 CPU 部署需要用参考输出校验，因为 x86 上文本看似正常时，困惑度仍可能达到 HF transformers 参考的约 6 倍。
| I wanted to know which low-bit (1-bit / ternary / 2-bit) GGUF files actually give correct output on CPU, and how fast they run, so I compared each runtime against a reference instead of judging by how the text looks. Machines: an ARM server (Neoverse-N1) and x86 VMs with AVX2 (Intel Xeon, AMD EPYC). BitNet b1.58 2B4T Bonsai / Ternary-Bonsai (PrismML) Every number has the runtime commit, the file sha256 and the flags, and one script reruns the main measurements. A notebook runs a subset on a free Kaggle or Colab VM (~30 min to 1 h) and compares it with the published numbers. I only have Neoverse-N1 and AVX2 x86. Results from AVX-512 / Zen 4, Graviton 3/4 or Raspberry Pi 5 would fill the biggest gaps; Apple Silicon too, but the scripts are Linux-only so far. Report: https://github.com/matheus-c-feitosa/lowbit-atlas Corrections welcome. I already fixed one: an earlier "~6× slower" for the 27B PTQ1_0 file came from rough timings; llama-bench says 4.4×. (English isn't my first language. I used Claude Code to help with the analysis and the wording; the runs were done on my ARM server and on free Kaggle/Colab VMs, and I checked the numbers.) submitted by /u/matheus-c-feitosa [link] [comments] |
#BitNet b1.58 2B4T#bitnet.cpp#Ternary-Bonsai 1.7B#Bonsai#PrismML#llama.cpp#ik_llama.cpp#lowbit-atlas#Hugging Face Transformers