Ternary Nemotron 30B-A3B GGUF model repository shared
AI summary
A Hugging Face repository identified in an r/LocalLLaMA post presents a GGUF model called Ternary Nemotron 30B-A3B. The submitter describes it as a ternary, 1.58-bit mixture-of-experts model that runs on a 12 GB GPU, supports a 368K-token context, and reaches 172 tokens per second decoding and more than 7,000 tokens per second prefill. The post says one file works with stock llama.cpp and the PrismML/Bonsai fork, listing specific builds where each was validated.
In-depth insight
The repository combines a 30B-A3B mixture-of-experts design with ternary quantization and compatibility claims for two local-inference engine options.
| Ternary Nemotron 30B-A3B — 368K Context on 12 GB GPU (GGUF) A ternary (1.58-bit) 30B-A3B MoE model that runs on a 12 GB GPU with 368K context, 172 tok/s decode and 7000+ tok/s prefill — in both Stock llama.cpp and the PrismML fork. One file, both engines. submitted by /u/pmttyji [link] [comments] |
#Ternary Nemotron 3.5 Lightning 30B-A3B GGUF#boro64#llama.cpp#PrismML#Bonsai