IQ4_K
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 1
- New IQ2_K–IQ6_K quantization types proposed for llama.cpp
A r/LocalLLaMA post highlights five new quantization types proposed in a llama.cpp pull request — IQ2_K, IQ3_K, IQ4_K, IQ5_K and IQ6_K — and links to test models on Hugging Face. According to the post, the files are the same size as Q-quants, KLD improves substantially over Q-quants, inference performance suffers slightly, and creating IQ quants is somewhat slow. The poster cautions that the numbers presented cover only a single model.
r/LocalLLaMA · 🔥 0
Experience and discussion from the community
Share my IQ4_K experienceAsk about IQ4_K
Nobody has shared their experience with IQ4_K yet.