Researchers fix precision bugs in Softmax and cross-entropy
A paper titled “A Fix for Softmax and Cross-Entropy” reports two floating-point defects in standard Softmax and cross-entropy implementations used by PyTorch, JAX, TensorFlow and other systems. The issues can prematurely zero the correct-class gradient in confident classifiers and leave residual gradients when a student model exactly matches a teacher during knowledge distillation. The proposed changes preserve the original mathematical definitions and avoid requiring higher floating-point precision, while experiments reported smoother training and improved stability in modular-arithmetic and distillation tasks.
These defects affect foundational AI training code and can silently change gradients under specific numerical conditions, although the supplied evidence does not establish widespread impact on large-model training.
机器之心 2026-10-25 13:28 北京
Gemini发现Bug、包揽大量实验。
机器之心编辑部
深度学习中最基础、使用最广泛的两个函数,竟然藏着两个隐蔽的 Bug。
#Google DeepMind#PyTorch#JAX#TensorFlow#Optax#StableMax#Gemini Flash 3.7#Gemini Flash 3.8#Gemini 4 Argon#A Fix for Softmax and Cross-Entropy