llama.cpp 拟增 IQ2_K–IQ6_K 量化类型
AI 总结
r/LocalLLaMA 上一则帖子介绍了 llama.cpp 拉取请求中提出的 IQ2_K、IQ3_K、IQ4_K、IQ5_K 和 IQ6_K 五种新量化类型，并链接到 Hugging Face 上的测试模型。帖子称，这些量化类型的文件大小与 Q 量化相同，KLD 明显优于 Q 量化，但推理性能略有下降，且生成 IQ 量化文件的速度偏慢。发帖者同时提醒，展示的数据仅来自单一模型，相关细节仍待核实。
深度见解
若这些结果能在更多模型上复现，IQ 量化有望在相同文件大小下提供更低的 KLD，不过目前数据仅来自单一模型。
Looks quite interesting. Lots of numbers presented, but for a single model only.
Highlights:
- File sizes are the same as Q-quants.
- KLD improves substantially over Q-quants.
- Performance suffers slightly
- Creating IQ quants is somewhat slow.
Asterisks apply...
https://github.com/ggml-org/llama.cpp/pull/30715
https://huggingface.co/eaddario/IQ2_K-IQ6_K-Test-Models
Appologies to Mr. Addario if I mangled anything.
submitted by /u/ethertype [link] [comments]