PyTorch错了，JAX错了。连教科书里的标准公式，也错了。谷歌DeepMind的主任研究科学家 Jon Barron披露了这一重磅炸弹！本周，Google DeepMind 展示了一个问题：在 PyTorch、JAX 以及各类教科书中，该算法的每一个标准版本都一直在悄悄把某个数字舍入为零。模型越自信，正确的训练信号反而越容易凭空消失。优化器因此失控，训练曲线剧烈震荡。但最让人毛骨悚然，也最让人兴奋的是：这个隐藏在全人类眼皮底下的底层逻辑漏洞，是AI（Gemini）自己抓出来的。
这个Bug藏在AI最基础的运算里：Softmax和交叉熵。Softmax负责把模型输出的原始得分（logits）转化成概率。交叉熵负责计算误差，告诉模型错得有多离谱。所有的分类器，所有的大语言模型，每天都要大量调用这对函数，数以亿计。为了防止计算溢出，现有的标准框架（无论是PyTorch还是JAX）都会执行一个标准动作：这一减，分母就会变成一个类似「1 + 极小数字」的结构。模型越自信，正确答案的得分比其他答案高得越多，这个数字就越小。但这里藏着一个关键细节：数学上的等价，不代表计算机浮点运算中的结果完全等价。在计算机的浮点数世界里，如果你把一个足够小的数字加到1.0上，计算机会觉得「这太小了，没必要算」，直接把结果四舍五入成精确的1.0。在bfloat16格式下，这个悬崖出现在数值5.5左右。在fp32下，出现在16.6左右。模型明明还在学习，但关于正确答案的反馈信号，被计算机的舍入机制直接抹杀了。DeepMind的论文将此称为Softmax崩溃（Softmax Collapse）。主流优化器Adam的原理，是把每一步更新除以近期梯度的平均值。当梯度因为Bug长时间变成零，这个平均值也会无限趋近于零。等到真实的梯度终于再次出现时，Adam会用这个梯度除以一个极小的数字。结果就是模型向前猛冲了一大步，损失函数（Loss）瞬间飙升。学术界以前观察到这种现象，还给它起了个好听的名字叫「弹弓效应（Slingshots）」，以为这是模型在「顿悟（Grokking）」过程中的某种高级特征。除了崩溃，还有Softmax抵消（Softmax Cancellation）。在知识蒸馏时，如果学生模型完全复制了老师，理论上梯度应该是零。但因为标准代码是分别计算概率再相减，舍入误差会留下一个「幽灵梯度」。学生模型会莫名其妙地偏离正确答案。DeepMind的Jonathan T. Barron团队在arXiv上提交了一篇论文：《Softmax和交叉熵的修复》。
事实证明，所有现有的Softmax和交叉熵实现（包括JAX、PyTorch、教科书等）都有一个微妙的Bug——其实是两个。
简单的重写就能修复它们，并消除顿悟时的「弹弓效应」。
对于交叉熵（修复崩溃）： 把目标从总和里拿出来，直接用log-sum-exp计算边距，然后返回负边距的softplus。核心原则：永远不要把一个极小的数字加到1上。对于蒸馏（修复抵消）： 在对数空间里使用expm1和log1p进行计算。这样一来，当学生完美匹配老师时，梯度就是完美的绝对零，没有任何幽灵残留。在模加法（a + b mod 31）任务中，使用标准代码，bfloat16精度下的测试准确率只有93.3%（而且极其不稳定）。换上修复后的代码，准确率直接飙升到100.0%，稳如泰山。而在蒸馏测试中，修复后的代码让学生模型与老师模型的散度保持为零，准确率维持完美的100%。但真正让整个科技圈炸锅的，是Barron在评论区补充的一句话：
这个Bug最初是由Gemini Flash 3.7发现并修复的。论文里的大部分实验是由Flash 3.8完成的。Gemini 4 Argon写了论文和图表的初稿。
我只是极其苛刻地挑刺，直到内容扎实。
10月24日，独立工程师Satyajit Ghana在PyTorch中进行了全面复现。他在CPU上用打印出来的值复现了这两个Bug，移植了修复程序，并证实了论文中fp32精度下的结果。他得出的结论掷地有声：「不管其他细节如何，机制毫无保留地被证实了。」
数字被悄悄丢掉，这不是第一次。每次发生，代价都极其惨痛。1965年，多伦多大学的William Kahan教授就在《ACM通讯》上发出警告。他指出，当计算机把成千上万个数字累加时，每一次加法都会产生舍入误差。加一百万个数，你可能会丢掉6位有效数字。他的解决方案极其简单：多加一个变量，把上次丢失的误差存起来，下次再加回去。这就是著名的「Kahan求和算法」。Kahan教授在文章里留下一句充满预言意味的嘲讽：「双精度很快就会被普遍接受，用来代替解决数值问题时的聪明才智。」61年后的今天，AI为了追求更快的芯片速度，一路降级到了16位的bfloat16格式。精度变低了，而聪明才智必须重新端上桌。海湾战争期间，一枚美国「爱国者」导弹未能拦截来袭的「飞毛腿」导弹。导弹击中兵营，28名美国士兵丧生。美国会计总署（GAO）随后的调查报告揭示了惨剧的原因。系统内部时钟以十分之一秒为单位计时。但将这个时间转化为实数时，寄存器只有24位长。导弹连已经连续运行了100多个小时。运行8小时，误差0.02秒，距离偏差55米。运行100小时，误差0.34秒，距离偏差高达687米。目标早已飞出了雷达设定的跟踪范围。一次看不见的四舍五入，四天后变成了687米的致命偏差。1994年10月，林奇堡学院的数学教授Thomas Nicely发现，他的两台电脑算出的结果不一样。他在计算孪生素数的倒数和。同样的程序，奔腾芯片和老旧的486算出了分歧。几天后，他锁定了罪魁祸首：当计算 1 除以 824,633,702,441 时，奔腾处理器的浮点运算单元算错了。原因是，奔腾的除法器使用了一个查找表，而表里漏掉了5个条目。发现问题的，不仅是Nicely教授的两台电脑比对，更是严谨的交叉验证。最终，英特尔被迫同意无条件为所有用户更换芯片。1995年1月，英特尔为此计提了4.75亿美元的巨额损失。这是AI底层数学最糟糕的时刻，但它刚刚变得好了一点。而最令人振奋的是：这一次，是机器找到了自身的缺陷，人类负责了最终的把关。
参考资料：
https://arxiv.org/pdf/2610.31642
https://x.com/BrianRoemmele/status/2114362944339448314
编辑：大卫
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
PyTorch was wrong, Jax was wrong. Even the standard formula in the textbooks is wrong. The director of DeepMind's research scientist, Jon Barron, revealed this heavy bomb! This week, Google DeepMind shows a problem: in PyTorch, JAX and various textbooks, every standard version of the algorithm has been quietly rounding a number to zero. The more confident the model is, the more easily the correct training signal disappears. The optimizer is thus out of control and the training curve is convulsed. But the most creepy and exciting thing: this bottom-up logical loophole, hidden under the eyes of all human beings, was captured by AI.
This bug is hidden in the basic AI calculations: Softmax and crossbow. Softmax is responsible for translating the original score (logits) of model output into probability. The crossbow calculates errors and tells the models how far they go. All the sorters, all the large-language models, call these functions in large numbers every day, in hundreds of millions. In order to prevent the calculation of spills, the existing standard framework (whether PyTorch or JAX) implements a standard action: This drop turns the denominator into a structure like "1 + very small numbers". The more confident the model is, the more the right answer scores out of the other answers, the smaller the number. But there's a key detail here: the mathematical equivalent does not mean that the result is exactly the same. In the world of computer floats, if you add a small enough number to 1.0, the computer will think, "It's too small to count," and round the results to the exact 1.0. In the bflaut16 format, this cliff appears at around 5.5. Under fp32, in about 16.6. The model is still learning, but the feedback on the correct answer is directly erased by computer rounding mechanisms. The paper by DeepMind refers to it as the Softmax Collapse. The rationale for the mainstream optimizer, Adam, is to divide each step up by the average of the nearest gradient. When the gradient becomes zero as a result of Bug's prolonged transformation, the average is boundless to zero. And when the real gradient finally appears again, Adam will divide it by a very small number. And the result is that the model is a big leap forward, and the loss function is skyrocketing. The academic community has previously observed this phenomenon and has given it a very good name, Slingshots, as some of the advanced features of the model in the "Grockking" process. In addition to the collapse, there are softmax offsets. When knowledge is distilled, the theoretical gradient should be zero if the student model fully replicates the teacher. But since the standard code is to calculate the probability separately and then decrease, the rounding error leaves a "ghost gradient". The student model somehow deviates from the right answer. Jonathan T. Barron's team in DeepMind presented a paper on arXiv, " The restoration of Softmax and the crossbow ".
All the existing softmax and cross-breeding (including JAX, PyTorch, textbooks, etc.) proved to have a delicate Bug — actually two.
A simple rewrite can fix them and eliminate the slingshot effect of awareness.
For cross-breathing (recovering crash): remove the target from the sum, calculate the margin directly using log-sum-exp, and then return the softplus at negative margins. Core principle: Never add a very small number to 1. For distillation (remediation offset): Calculate using expm1 and log1p in logarithmic space. That way, when students match their teachers perfectly, the gradient is perfect absolute zero, with no ghost remnants. For the task of emulation (a + b mod 31), only 93.3 per cent (and extremely unstable) of the accuracy of tests at bflaut 16 is used. Changed to the repaired code, the accuracy rate went up directly to 100.0 per cent, as stable as Tarzan. In the distillation test, the restored code kept the student model to zero and the teacher model to 100 per cent. But what really blew the entire tech world up was a quote from Barron's comment section:
This Bug was initially discovered and repaired by Gemini Flash 3.7. Most of the experiments in the paper were completed by Flash 3.8. Gemini 4 Argon wrote a first draft of the paper and chart.
I'm just stinging it very hard until it's solid.
On 24 October, the independent engineer Satyajit Ghana fully recovered in PyTorch. He reproduced the two Bugs with printed values on the CPU, transplanted the repair program and confirmed the result of the fp32 accuracy in the paper. The conclusion he reached was pitched: “No matter what the other details, the mechanism has been confirmed without reservation.”
This is not the first time that numbers have been quietly thrown away. Every time it happens, the cost is enormous. In 1965, Professor William Kahan of Toronto University issued a warning on the ACM Newsletter. He noted that when computers added thousands of numbers, each addition would result in rounding errors. Plus a million, you might lose six figures. His solution was extremely simple: one more variable would be added and the last error would be saved and the next added. This is the famous Kahan peaceful algorithm. In his post, Professor Kahan left a premonitionful sarcasm: "The double precision will soon be widely accepted as a substitute for the wisdom of solving numerical problems. Sixty-one years later, in pursuit of faster chips, AI has been downgraded to 16 bits of bfloat16. Precision has gone down, and intelligence has to get back to the table. During the Gulf War, an American "patriot" missile failed to intercept the incoming Scud missile. Missiles hit the barracks, killing 28 United States soldiers. Subsequent investigations by GAO revealed the cause of the tragedy. The clock inside the system is measured in one tenth of a second. But when this time is converted to real, the register is only 24 bits long. The missile company has been operating for over 100 hours. Eight hours running, 0.02 seconds error, 55 metres deviation. 100 hours running, 0.34 seconds with a distance deviation of 687 m. The target has already flown out of radar-set tracking range. An invisible rounding turned four days later to a fatal deviation of 687 metres. In October 1994, Thomas Nicely, a math professor at Lynchburg College, discovered that his two computers had calculated different results. He is calculating the countdown of the twin numbers. The same procedure, the Benzen chip and the old 486 had been divided. A few days later, he's identified the culprit: When calculating 1 divided by 824,633,702,441, the floating point unit of the Benzen processor was miscalculated. The reason for this was that the Benzen decomposer used a search table, which omitted five entries. The problem was not only identified by Professor Nicely's two computers, but also a rigorous cross-check. Ultimately, Intel was forced to agree to replace the chip unconditionally for all users. In January 1995, Intel incurred a significant loss of $475 million. It's the worst time in AI math, but it just got a little better. And the most exciting thing is that this time, the machine found its own flaw and humans took responsibility for the final clearance.
References:
https://arxiv.org/pdf/2610.31642
https://x.com/BrianRoemmele/status/2114362944339448314
Editor: David
Auto-fetched and translated by AI Atlas. All rights belong to the original author.