文 | 字母AI
AI现在连审论文，都比人类专家更厉害了。
10月22日，哈佛大学的两位研究人员发表了一篇题为《AI审稿意见全面胜过高水平人类同行评审》的论文，对比了134组AI与人类撰写的审稿意见。结果，AI在七个评价维度上全面领先。
研究人员还请来三位曾经担任重要学术期刊主编的专家进行盲评。30次比较中，他们有26次认为AI的审稿意见更好。
在接受核查的具体论断中，AI审稿意见的事实错误比例只有5.5%，人类却高达15.9%。
这对最近忙着跟AI公司较劲的数学家们来说，可能不是什么令人愉快的消息。
就在上个月，陶哲轩等25位菲尔兹奖得主还联名批评AI公司急于宣布数学突破，强调数学研究需要严谨的验证和审查。
数学家们还在要求AI接受严格的学术检验，另一边，人类可能连审稿这份工作都要被AI抢走了。
人类一败涂地
惨败的审稿人们可不是什么随便招来的志愿者。
研究使用了124篇学术论文，以及哈佛大学定量社会科学研究所旗下rapidPeer服务提供的134份人类审稿意见——这是一项付费的专业审稿服务，审稿人由领域专家挑选，来自不同的研究机构，而且报酬不低。
他们的表现原本也很不错：在接受调查的74位论文作者中，有88%认为收到的审稿意见质量很高，其中60%给出了最高档评价。
研究人员使用Anthropic的Claude Opus 4.8生成审稿意见，再请OpenAI的GPT-5.5担任AI评委，在不知道意见由谁撰写的情况下，比较人类与AI的审稿质量。
结果是，人类输得非常彻底。
即使不允许AI联网，只给它和人类一样的审稿指南，在134组对比中，GPT-5.5也全部选择了AI。
换用专门优化的提示词、开放联网后，134组比较中有132组明确判AI获胜，剩下两组结果不确定，人类依然没有取得任何胜利。
具体来看，在补充参考文献、发现论文弱点和研究方法三个维度上，AI的胜率达到100%；理论创新、论述框架、表达质量等方面，胜率也都超过90%。
研究人员甚至检查了AI引用的651个带有DOI或arXiv编号的参考文献，发现全部真实存在，没有任何一条是编造的。
这时候可能会有人说，AI审稿AI评，这不公平，谁知道它会不会偏袒同类？指不定就在审稿意见里藏了什么暗度陈仓的线索。
于是他们请来了开头提到的三位前学术期刊主编，分别进行了10次盲评。结果，在30次比较中，人类评委有26次选择了AI。
这下，就连人类评委都站到了AI那边。
研究做到这里，作者们甚至开始认真讨论：学术期刊或许可以取消常规的人类同行评审，让AI承担大部分审查工作，只留下人类编辑负责最终决定，必要时再请专家介入。
10月23日，该论文发布后，斯坦福大学政治学副教授徐轶青公开表示，学术界应该重新考虑人类专家在审稿中的角色。在他看来，人类专家仍然需要参与，但已经没必要继续亲自撰写动辄1500字的长篇审稿意见了。
就连那些禁止使用AI审稿的期刊，也到了该重新审视自身规定的时候。
看来，人类审稿人的工作，已经有人想要交给AI了。
AI审稿，认真到烦人
那么问题来了：AI到底是怎么把人类审稿人比下去的？
答案可能非常简单——它压根就不会累。
审稿是一件相当费神的工作。审稿人需要通读论文，检查研究方法、数据、图表和论证，还得寻找作者可能遗漏的相关文献。
论文作者甚至在介绍实验时，顺手吐槽了一些传统期刊审稿人：有些人论文才看到第三页，就开始寻找足以否定论文的致命缺陷，好让自己不用继续读下去。
相比之下，AI显然没有急着下班的需求。
这次实验中，研究人员将Claude Opus 4.8的推理强度调到最高，还专门设计了一套审稿提示词，允许它联网查找资料。
结果发现，AI平均每审一篇论文，就要进行12.4次联网搜索。
不过，认真的效果也相当明显。在不联网、使用普通审稿指南的情况下，AI在补充参考文献方面的胜率为62%；换用专门优化的提示词并开放联网后，胜率直接提高到了100%。
今时不同往日。
今年4月，学术期刊《Organization Science》的编辑团队还发表过一项研究，分析了过去几年收到的一万多份审稿意见，发现大量使用AI的审稿报告反而更难读，讨论的问题也更加狭窄。
当时研究人员甚至认为，AI生成的审稿意见还没有达到人类审稿的水平。结果不到半年，哈佛就拿出了另一份成绩单。
但AI太认真，偶尔也会让人崩溃。
论文作者就举例，AI有时候甚至会吹毛求疵到荒唐的程度：明明知道2+2=4是正确的，却还要指出作者没有定义加号是什么意思。
人类审稿人可能嫌论文毛病太多，AI倒可能“找茬成瘾”，没毛病也给挑点错出来。
何况AI的能力还在不断进步。
以前是论文作者需要向人类审稿人证明，自己的研究值得发表；以后可能就轮到人类审稿人向学术期刊证明，自己为什么还值得花钱请了。
数学家们可能有点尴尬了
最近，数学家和AI公司的战场，已经从验证数学证明一路打到了同行评审。
9月11日，陶哲轩等首批25位菲尔兹奖得主联名发声，批评AI公司把破解数学难题当成模型能力竞赛，急着宣布重大突破，却没有给数学界留下足够的时间验证、理解和讨论。
9月29日，数学与AI咨询小组（AGMAI）又专门发布指南，强调已经被人类数学家充分理解的AI研究成果，应该按照传统学术规范发布论文，接受期刊同行评审。
数学研究有一套延续多年的规矩。证明写出来了，还得经过专家检查、同行讨论，最后才能逐渐成为学术界公认的知识。
AI公司可以兴高采烈地宣布自己又攻克了一道世界难题，但数学界认不认可，就是另一码事了。
还有一部分数学家，甚至连审稿人的椅子都不想让AI碰。
10月7日，一个名叫“人类数学协会”（AHM）的组织公开批评OpenAI一次性发布700多份数学研究文件，认为这更像是在展示力量，而非认真做学术。
该组织甚至明确要求成员不得发表AI生成的数学论文、评论，连审稿报告都不行。
就在数学家们努力给AI立规矩的时候，哈佛的两位研究人员却把人类和AI审稿人一起送上了考场。
结果我们看到，AI不仅开始挑战人类写论文的能力，现在连挑论文毛病，都比不少专业审稿人厉害了。
当然，哈佛这项研究涉及的主要是社会科学论文，尚且无法证明AI已经能够独立审查复杂的数学证明。数学家要求严格验证，本身也没什么错。
但事情发展到这里，多少显得有些荒诞。
数学家们还在强调，AI不能绕过人类同行评审这道门槛。
AI却已经开始研究，能不能把守门的人也换掉了。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Letter AI
AI is now even better than human experts.
On 22 October, two researchers from Harvard University presented a paper entitled " Comprehensive Review of the AI Review: Overrated Human Peer Review ", comparing 134 sets of AI-Human Reviews. As a result, AI has taken the lead across seven evaluation dimensions.
The fellows also invited three experts who had served as editors-in-chief of major academic journals to make a blind review. In 26 of the 30 comparisons, they considered AI ' s review to be better.
Among the specific findings subject to verification, the percentage of factual errors in the AI review was only 5.5 per cent, while humans were as high as 15.9 per cent.
This may not be pleasant news for the mathematicians who have been busy with AI lately.
Just last month, the 25 Philtzian winners, among others, criticized AI for its eagerness to announce a mathematical breakthrough, emphasizing the need for rigorous validation and review of mathematical research.
Mathematicians are also demanding that AI undergo a rigorous academic test, and on the other hand, humans may have been robbed by AI of even the reading.
Humans have lost everything.
The losers are not just volunteers.
The study used 124 academic papers and 134 human reviews provided by the Rapid Peer Service of the Harvard Institute of Quantitative Social Science - This is a professional revision service for paid, with reviewers selected by field experts, from different research institutions and not underpaid.
They also performed well: 88 per cent of the 74 authors of the papers surveyed considered the review opinions received to be of high quality, 60 per cent of whom gave the highest level of evaluation.
Researchers use Claude Opus 4.8 of Anthropic to generate review opinions, and Open AI GPT-5.5 to serve as an AI jury to compare the quality of human and AI reviews without knowing who wrote the comments.
As a result, humanity has lost very thoroughly.
Even if AI is not allowed to be connected, it is given only the same review guide as humans, and in the 134 group comparison, the GPT-5.5 chose AI.
With the replacement of prompt, which is specially optimized, and open access, 132 of the 134 comparisons have clearly ruled out AI, with the remaining two sets of results uncertain, and humanity still has not won.
In particular, AI won 100 per cent of the three dimensions of additional reference literature, discovery of paper weaknesses and research methodologies, and over 90 per cent of theoretical innovations, discussion frameworks and quality of expression.
The researchers even examined 651 references with DOI or arXiv numbers cited by AI and found that all were real and none were made up.
At this point, it may be said that it's not fair for AI to review, and who knows if it favours the same kind? I don't know what's hidden in the review opinion.
They then brought together the three leading editors of the first-mentioned former academic journal, who made 10 blind reviews. As a result, in 26 of the 30 comparisons, the Human Review Board chose AI.
And now even the Human Board is on the side of AI.
This was done, and the authors even began to discuss it carefully: academic journals could perhaps eliminate conventional human peer reviews and leave AI with most of the reviews, leaving human editors in charge of final decisions and, if necessary, experts.
On 23 October, after the paper publish, Xu Zhou Qing, Associate Professor of Political Science at Stanford University, publicly stated that the academic community should reconsider the role of human experts in the review. In his view, human experts still needed to be involved, but it was no longer necessary to continue working on a lengthy version of 1,500 words.
It is time to review its own regulations, even those that prohibit the use of AI.
It appears that the work of the human reviewer has been given to AI.
AIL.E.L.D., seriously.
So the question is: How does AI compare human readers?
The answer may be very simple — it will not be tired at all.
The revision was a very demanding job. Readers need to read papers, examine research methods, data, graphs and justifications, and look for relevant literature that the author may have missed.
Even when the author introduced the experiment, some reviewers of traditional periodicals threw up: It was only on the third page that some of the papers began to look for fatal defects that would negate them so that they would not have to continue reading.
By contrast, it is clear that AI is not in a hurry to leave work.
During the experiment, the researchers moved Claude Opus 4.8 ' s reasoning to its highest strength and also designed a set of revision prompt to allow it to network for information.
As a result, AI conducted 12.4 web searches on average for one paper.
However, the serious effect is also quite clear. In the absence of a network and the use of the generic revision guide, AI won 62 per cent in the additional reference literature; the success rate increased directly to 100 per cent when the specially optimized prompt was used and the network opened.
It's different now.
In April this year, the editorial team of the academic journal Organization Science also published a study analysing the more than 10,000 reviews received over the past few years, which found that the extensive use of AI's review reports was more difficult to read and the issues discussed more narrowly.
At that time, researchers were even of the view that the comments generated by AI were not as good as those produced by humans. In less than six months, Harvard produced another report card.
But it's too serious. Sometimes it's a breakdown.
The author of the paper, for example, sometimes even bragged to the absurd point: knowing that 2+2=4 was right, he also pointed out that the author did not define what the additional sign meant.
Human reviewers may feel that thesis is too bad, but AI may be “addictionful” and pick out a few mistakes.
And the ability of AI continues to improve.
Formerly the author of the paper had to prove to human readers that his research was worth publishing; it might be the turn of human readers to prove to academic journals why they were worth paying for.
Mathematicians may be a little embarrassed.
Recently, mathematicians and AI have gone from proof of mathematics to peer review.
On 11 September, Tao Chul-hyun and others, among the first 25 Fields Prize winners, made a joint statement criticizing AI for treating the problem of decrypting mathematics as a model capacity competition, rushing to announce a major breakthrough without leaving enough time for the mathematics community to validate, understand and discuss.
On 29 September, the Advisory Group on Mathematics and AI (AGMAI) devoted another publish guide to highlighting the findings of AI, which have been well understood by human mathematicians, and which should be peer-reviewed in periodicals in accordance with the traditional academic norm publish.
Math research has a set of rules that last for years. The certificates are written, subject to expert scrutiny and peer-to-peer discussion, before they become recognized knowledge in academia.
It's another thing that AI can excitely declare itself to have taken over another world dilemma.
Some mathematicians don't even want to touch AI's chair.
On 7 October, an organization called the Human Mathematics Association (AHM) publicly criticized Open AI ' s one-time publish 700 mathematical research documents as more like a demonstration of power than a serious academic exercise.
The organization even explicitly requires members not to publish AI-generated mathematical papers, comments or review reports.
As mathematicians worked hard to set the rules for AI, two Harvard researchers put humans and AI readers together.
As a result, not only did AI begin to challenge the ability of humans to write their papers, but even the dissertation problems are now better than many professional writers.
Of course, the Harvard study, which involves mainly social science papers, does not prove that AI has been able to independently review complex mathematical certificates. Mathematicians are demanding rigorous validation, and there's nothing wrong with it.
But it's kind of ridiculous to get here.
Mathematicians are also emphasizing that AI cannot bypass the threshold of human peer review.
AI has begun to study whether the doorman can be replaced.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.