Softmax and cross-entropy are staples of neural-network training, but a new paper says common implementations can suffer from two numerical bugs with knock-on effects for learning. Its proposed fixes use log-space operations and, the researchers report, improve several training behaviours, from distillation to grokking.
Watch Desk analysis
What happened
In A Fix for Softmax and Cross-Entropy, researchers describe Softmax Collapse, where gradients for dominant logits can be zeroed during high-confidence predictions, and Softmax Cancellation, where precision is lost as student and teacher logits converge during distillation.
They propose alternative implementations using operations including softplus, expm1 and log1p, intended to preserve accurate, zero-sum gradients across floating-point precisions. The paper reports improvements in several training settings:
Key findings
- Less gradient loss at high confidence
The proposed softmax and cross-entropy implementations aim to prevent dominant-logit gradients being zeroed. - More stable knowledge distillation
The researchers say their changes prevent student networks drifting away from their teachers as logits converge. - Fewer training collapses
The paper reports preventing accuracy collapse without weight decay and eliminating the Slingshot phenomenon, which involves loss spikes during training. - Faster grokking in one setting
The researchers report that their fixes can speed up grokking when weight decay is used.
Why it matters
These functions sit deep in the training stack, so numerical quirks can shape what a model learns, not merely how tidy the code looks. If the reported fixes hold up across implementations and workloads, they could offer practitioners a relatively targeted way to avoid training instability without changing the model architecture.
Our read
This is a useful technical result, not a universal guarantee that swapping one implementation will improve every training run. The appealing part is the connection between specific numerical errors and recognisable failures in training. The next test is whether other teams can reproduce the gains in varied models and precision settings; maths deserves replication as much as any benchmark does.
What to watch
- Independent reproductions across model families and floating-point formats.
- Whether the fixes are adopted in widely used training libraries.
- The conditions under which the reported gains in grokking and stability hold.
Discussion spark: Should core machine-learning libraries replace standard softmax and cross-entropy implementations when a more numerically stable option is available, even if the gains vary by workload?
Sources and evidence
- A Fix for Softmax and Cross-Entropy (23 October 2026, 02:31 UTC)
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.