Bonsai

Headlines · 3

1. 5d
   Reddit author argues KLD cannot rank quantized models

   A Reddit author argues that KL divergence should not be treated as a universal measure of quantized language-model quality. The author notes that distillation may be followed by SFT and RL, during which KLD can worsen even as task performance improves, and recommends testing candidates against representative private workloads. KLD is presented as a useful initial filter when calculated on suitable data, rather than a complete ranking metric.

   r/LocalLLaMA · 🔥 0
2. 6d
   Ternary Nemotron 30B-A3B GGUF model repository shared

   A Hugging Face repository identified in an r/LocalLLaMA post presents a GGUF model called Ternary Nemotron 30B-A3B. The submitter describes it as a ternary, 1.58-bit mixture-of-experts model that runs on a 12 GB GPU, supports a 368K-token context, and reaches 172 tokens per second decoding and more than 7,000 tokens per second prefill. The post says one file works with stock llama.cpp and the PrismML/Bonsai fork, listing specific builds where each was validated.

   r/LocalLLaMA · 🔥 0
3. Oct 19
   CPU low-bit benchmark: bitnet.cpp wrong, Bonsai correct, ik_llama.cpp fastest

   A community benchmark posted on r/LocalLLaMA compared low-bit GGUF runtimes against reference outputs on an ARM Neoverse-N1 server and x86 AVX2 VMs. It reports that bitnet.cpp main (commit 0b341e5) is incorrect for BitNet b1.58 2B4T: x86 text looks fine but perplexity is about 6× the HF transformers reference on the same tokens, while ARM prints ????; the author says fixes from issue #588 and an ARM kernel, bundled in PR #642's third version dated Oct 15, are correct on both. Ternary-Bonsai 1.7B files gave correct output wherever they loaded in llama.cpp, the Prism fork and ik_llama.cpp, with prompt-processing speeds varying by file, runtime and CPU — 71 t/s for PQ2_0 in ik_llama.cpp with -rtr versus 28 in Prism — and the Q2_0 file marked recommended on the model card did not load in any of the three runtimes.

   r/LocalLLaMA · 🔥 0

Experience and discussion from the community

Share my Bonsai experienceAsk about Bonsai