New IQ2_K–IQ6_K quantization types proposed for llama.cpp
A r/LocalLLaMA post highlights five new quantization types proposed in a llama.cpp pull request — IQ2_K, IQ3_K, IQ4_K, IQ5_K and IQ6_K — and links to test models on Hugging Face. According to the post, the files are the same size as Q-quants, KLD improves substantially over Q-quants, inference performance suffers slightly, and creating IQ quants is somewhat slow. The poster cautions that the numbers presented cover only a single model.
The post reports substantially better KLD than Q-quants at the same file size, though the results cover only one model.
Looks quite interesting. Lots of numbers presented, but for a single model only.
Highlights:
- File sizes are the same as Q-quants.
- KLD improves substantially over Q-quants.
- Performance suffers slightly
- Creating IQ quants is somewhat slow.
Asterisks apply...
https://github.com/ggml-org/llama.cpp/pull/30715
https://huggingface.co/eaddario/IQ2_K-IQ6_K-Test-Models
Appologies to Mr. Addario if I mangled anything.
submitted by /u/ethertype [link] [comments]