Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Researchers fix precision bugs in Softmax and cross-entropy

AI summary

A paper titled “A Fix for Softmax and Cross-Entropy” reports two floating-point defects in standard Softmax and cross-entropy implementations used by PyTorch, JAX, TensorFlow and other systems. The issues can prematurely zero the correct-class gradient in confident classifiers and leave residual gradients when a student model exactly matches a teacher during knowledge distillation. The proposed changes preserve the original mathematical definitions and avoid requiring higher floating-point precision, while experiments reported smoother training and improved stability in modular-arithmetic and distillation tasks.

In-depth insight

These defects affect foundational AI training code and can silently change gradients under specific numerical conditions, although the supplied evidence does not establish widespread impact on large-model training.

机器之心 2026-10-25 13:28 北京

Gemini发现Bug、包揽大量实验。

机器之心编辑部

深度学习中最基础、使用最广泛的两个函数,竟然藏着两个隐蔽的 Bug。

#Google DeepMind#PyTorch#JAX#TensorFlow#Optax#StableMax#Gemini Flash 3.7#Gemini Flash 3.8#Gemini 4 Argon#A Fix for Softmax and Cross-Entropy

Original · 11 min readRead the original
Group chat

No comments yet. Start the conversation.