快速的LLM开发超过了旨在解释的研究，研究发现
斯坦福研究者欧文·奎恩(Owen Queen)和詹姆斯·邹(James Zou)测试了50个被广泛引用的关于大语言模型的声称与20个模型，发现44%已经"过期". 在估计到期日的22项调查结果中，平均寿命仅为15.7个月。
两人在2024年7月至2026年9月期间由8个不同的开发者使用发布模型对声称进行了检查。 过期调查结果背后的原始影响已基本消失，另有34%明显减弱。 只有16%的人还坚持着，而6%的人实际上变得更强。 一年之后，16%的人已经过期。 两年后，这个数字上升到39%。
什么模型"不能做" 过期最快
根据研究，声称关于什么模型不能做的过期最快。 9项推理相关调查结果中有5项被认为已过期，没有任何一项得到维持或变得更加强大。 它们包括一些众所周知的弱点，如在"strawberry"中计算字母，在不熟悉的数字系统中算术，在长上下文中间丢失信息，在用户推回时放弃正确的答案。 研究者怀疑这些错误有"正确"的结果，给开发者一个明确的训练目标。
关于偏好和倾向的声称证明是更长久的。 11项发现中，有3项涉及LLM作为评价者的行为。 他们更喜欢自己的答案，同样地接受和拒绝论文，并在医学摘要中受到语言旋转的影响。 还有三个涉及对用户的节奏和协议。 此外，还有左倾的政治偏见，对开放式问题几乎相同的答案，以及生成随机数字时的非统一分布。
幻觉参考率从55%降至零，但并非到处都有
作为案例研究，王后和邹 looked研究了科学引用的捏造。 在沃尔特斯和怀尔德最初的2023年的研究中，GPT-3.5占了其参考文献的55%. 斯坦福的研究人员向目前的模型发出了同样的提示。 GPT-6.1 Sol制造了0%,Claude Sonnet 5.5以1%的速度进入。 但Qwen3.7-Flash，也是2026型号，仍然在同一测试中编造了69%的参考文献。
研究警告反对读取这些结果为全清。 在所谓的双跳问题上，当一个模型必须结合两个已知的事实时，Claude Sonnet 5.5仍然在大约10%的情况下失败，它正确回答了两个单个问题。 原论文报告ChatGPT为43%. "逆转诅咒"和对配对时间相关问题的自相矛盾的答案也有所削弱，但它们仍然可以被察觉。
研究人员认为，社会偏见是一个混合情况。 左倾的政治偏见依然存在，而性别陈规定型观念和隐含的结社偏见则削弱了。 对非洲裔美国英语使用者的偏见实际上改变了趋势数据的方向。
即使在同一代人中，模型对模型的差距仍然很大
不同模型的结果差异很大，即使在同一个发布窗口内也是如此。 在2026年7月至9月的发布型机中，过期的发现数量从DeepSeek V4 Pro的11个到GPT-6.1 Sol的30个不等。 无论是发布日期，提供者，还是模型大小都无法可靠地预测模式。
后与邹氏作大论。 关于LLMs的实证结论应当作为特定时间点的特定模型的观察，而不是作为一般的真理。 文件应记录模型、版本和日期，并公布提示和分析代码，以便对未来模型再次测试结果。
研究者指出Andrej Karpathy的一篇文章，描述对LLMs实际行为的认识差距如何影响对AI的信任，以及人们使用它的次数。 Karpathy,OpenAI联合创始人，从此加入了Anthropic，最近也加入了称，目前可用的模型只是对未来能力的预览。
AI News Without the Hype – 由人类治愈
订阅THE DECODER免费阅读，每周AI通讯，我们独家的"AI雷达"前沿报告每年6次，全面存档访问，访问我们的评论部分。
现在订阅
读取全照。
订阅无韵报道。
- 全面查阅THE DECODER的每篇文章
- 没有广告
- 加入评论和社区讨论
- 每周通过邮件转载AI新闻
- 6x/年："AI雷达"——对AI最重要的话题进行深度潜水。
- 每日AI新闻，总是最新的
- 我们完整的十年档案
- 由一个在AI工作10年以上的团队负责
订阅到 The Decoder
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Rapid LLM development is outrunning the research meant to explain it, study finds
Stanford researchers Owen Queen and James Zou tested 50 widely cited claims about large language models against 20 models and found that 44 percent have already "expired." Among the 22 findings with an estimated expiration date, the average lifespan was just 15.7 months.
The two examined the claims using models released between July 2024 and September 2026 by eight different developers. The original effect behind the expired findings has largely disappeared, while another 34 percent have weakened significantly. Only 16 percent still hold, and 6 percent have actually gotten stronger. After one year, 16 percent had expired. After two years, that number climbed to 39 percent.
What models "can't do" expires fastest
Claims about what models can't do expire fastest, according to the study. Five of nine reasoning-related findings are considered expired, and none have held up or gotten stronger. They include well-known weaknesses like counting letters in "strawberry," arithmetic in unfamiliar number systems, losing information in the middle of long contexts, and giving up correct answers when users push back. The researchers suspect these kinds of errors have a "correct" outcome, giving developers a clear target to train toward.
Claims about preferences and tendencies prove longer-lived. Of the 11 findings that have held or gotten stronger, three involve LLM behavior as evaluators. They prefer their own answers, rate accepted and rejected papers similarly, and are swayed by linguistic spin in medical abstracts. Three more involve sycophancy and agreement toward users. Moreover, there's a left-leaning political bias, near-identical answers to open-ended questions, and a non-uniform distribution when generating random numbers.
Hallucinated references dropped from 55 percent to zero, but not everywhere
As a case study, Queen and Zou looked at the fabrication of scientific citations. In the original 2023 study by Walters and Wilder, GPT-3.5 made up 55 percent of its references. The Stanford researchers sent the same prompts to current models. GPT-6.1 Sol fabricated 0 percent, and Claude Sonnet 5.5 came in at 1 percent. But Qwen3.7-Flash, also a 2026 model, still fabricated 69 percent of its references in the same test.
The study warns against reading these results as an all-clear. On so-called two-hop questions, where a model has to combine two known facts, Claude Sonnet 5.5 still fails in about 10 percent of cases where it correctly answers both individual questions. The original paper reported 43 percent for ChatGPT. The "Reversal Curse" and contradictory answers to paired time-related questions have weakened too, but they're still detectable.
Social biases are a mixed case, according to the researchers. The left-leaning political bias persists, while gender stereotypes and implicit association biases have weakened. Biases against speakers of African American English have actually flipped direction in the trend data.
Model-to-model gaps remain wide even within the same generation
Results vary considerably across models, even within the same release window. Among models released between July and September 2026, the number of expired findings ranges from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol. Neither release date, provider, nor model size reliably predicts the pattern.
Queen and Zou draw a broad conclusion. Empirical findings about LLMs should be presented as observations about specific models at a specific point, not as general truths. Papers should document the model, version, and date, and publish prompts and analysis code so findings can be re-tested on future models.
The researchers point to a post by Andrej Karpathy that describes how perception gaps about LLMs' actual behavior affect trust in AI and how often people use it. Karpathy, an OpenAI co-founder who has since joined Anthropic, also recently said that currently available models are just a preview of future capabilities.
AI News Without the Hype – Curated by Humans
Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.
Subscribe now
Read on for the full picture.
Subscribe for hype-free coverage.
- Full access to every article on THE DECODER
- No ads
- Join the comments and community discussions
- A weekly AI news recap via mail
- 6x/year: "AI Radar" — deep dives on the AI topics that matter most
- Daily AI news, always up to date
- Our full ten-year archive
- Covered by a team with 10+ years in AI
Subscribe to The Decoder
Auto-fetched and translated by AI Atlas. All rights belong to the original author.