Optax

Headlines · 1

1. 7d
   Researchers fix precision bugs in Softmax and cross-entropy

   A paper titled “A Fix for Softmax and Cross-Entropy” reports two floating-point defects in standard Softmax and cross-entropy implementations used by PyTorch, JAX, TensorFlow and other systems. The issues can prematurely zero the correct-class gradient in confident classifiers and leave residual gradients when a student model exactly matches a teacher during knowledge distillation. The proposed changes preserve the original mathematical definitions and avoid requiring higher floating-point precision, while experiments reported smoother training and improved stability in modular-arithmetic and distillation tasks.

   机器之心 · 🔥 0

Experience and discussion from the community

Share my Optax experienceAsk about Optax