Reddit author argues KLD cannot rank quantized models
A Reddit author argues that KL divergence should not be treated as a universal measure of quantized language-model quality. The author notes that distillation may be followed by SFT and RL, during which KLD can worsen even as task performance improves, and recommends testing candidates against representative private workloads. KLD is presented as a useful initial filter when calculated on suitable data, rather than a complete ranking metric.
The post gives practitioners a practical warning against selecting quantized models solely by KLD and emphasizes task-specific evaluation, though its claims come from one Reddit author.
Following my previous post on how KLD is misunderstood, there were a few claims that KLD is the best metric you can use and some other comments were asking, "If we stop using KLD, what should we use instead?" I'm gonna explain both a bit more here.
Assume you're a lab that does distillation. In the distillation process, you choose a teacher model and a smaller student model that you want to learn from the teacher. This learning starts with optimizing a variant of KL loss (it could be KL itself, reverse KL, or JS loss) between the outputs of the student model and the teacher model. After the optimization ends, according to Reddit, you must be done. You have a model that has the best possible KLD against the teacher model, and what could be better than that?
But no, the training can still continue. After that, there can be stages of SFT and RL to improve the performance of the model. These stages improve what really matters, and that's the ability of the model to solve different tasks. And during these stages, the KLD you had can become even worse, but the student model actually becomes better.