Following my previous post on how KLD is misunderstood, there were a few claims that KLD is the best metric you can use and some other comments were asking, "If we stop using KLD, what should we use instead?" I'm gonna explain both a bit more here.

Assume you're a lab that does distillation. In the distillation process, you choose a teacher model and a smaller student model that you want to learn from the teacher. This learning starts with optimizing a variant of KL loss (it could be KL itself, reverse KL, or JS loss) between the outputs of the student model and the teacher model. After the optimization ends, according to Reddit, you must be done. You have a model that has the best possible KLD against the teacher model, and what could be better than that?

But no, the training can still continue. After that, there can be stages of SFT and RL to improve the performance of the model. These stages improve what really matters, and that's the ability of the model to solve different tasks. And during these stages, the KLD you had can become even worse, but the student model actually becomes better.