A paper from Google DeepMind researcher Jon Barron describes fixes for two numerical errors in softmax and cross-entropy calculations, and 36Kr says Gemini models helped uncover and develop the work. The reported tests suggest the changes can prevent lost training signals and spurious gradients, a small numerical detail with consequences for how models learn.
Google DeepMind Watch analysis
What happened
The errors arise when floating-point arithmetic rounds very small values away. According to 36Kr’s account of the paper, this can make a training gradient disappear in some conditions, while a separate calculation used in knowledge distillation can leave a false residual gradient even when a student model matches its teacher.
The paper proposes calculating cross-entropy in a way that avoids adding a tiny number to one, and using expm1 and log1p for the distillation calculation. 36Kr says the paper’s modular-addition test reached 100% accuracy with the fix in bfloat16, compared with 93.3% using the standard code. It also reports that an independent engineer reproduced both issues in PyTorch.
36Kr attributes the initial discovery and fix to Gemini Flash 3.7, says Flash 3.8 was used for most experiments, and says Gemini 4 Argon drafted the paper and charts. Barron, the report says, reviewed and refined the work.
Why it matters
These are not claims that every model has been quietly learning incorrectly. They describe specific numerical failure modes and tests. But when the arithmetic sits inside widely used training operations, a reliable fix matters: disappearing or false gradients can distort optimisation, and apparently erratic training may have a less mysterious explanation than researchers thought.
Our read
The useful story is both the numerical repair and the reported role of AI in finding it. Gemini’s contribution is a claim in 36Kr’s account, not proof that the system independently authored or validated the research. The paper’s proposed calculations and test results are the part practitioners can scrutinise directly. If you work on model training, this is worth checking against your framework’s implementation rather than filing under “someone else’s rounding problem”.
What to watch
- Whether PyTorch, JAX or other frameworks adopt the proposed calculations.
- Whether follow-up tests find the same effects beyond the reported tasks and precisions.
- How the paper describes and documents Gemini’s contribution.
Discussion spark: Should AI assistance in research be judged mainly by whether people can reproduce the result, or should papers set a higher bar for disclosing what the model contributed?
Sources and evidence
not affiliated with, endorsed by, or operated by Google or Google DeepMind