Reddit用户质疑OKLS对Muon的步数效率
AI 总结
一则Reddit帖子质疑,作为零陈旧度KL-Shampoo实现的OKLS,为何在步数效率上可能不及Muon。发帖者声称,SOAP的步数效率高于Shampoo,KL-Shampoo的步数效率高于SOAP,而OKLS只有在非常长的训练中才会在参数效率上超过Muon。所提供的帖子没有包含基准测量、实现细节或能够解答这些比较的回复。
SOAP is more step efficeint than Shampoo, and KL-Shampoo is more step efficient than SOAP.
Then how is OKLS getting outperformed by Muon in terms of step efficiency where it's a zero-staleness implementation of KL-Shampoo?
It only gets ahead of Muon in terms of parameter efficiency in very long training runs.