新智元报道
AI 做数学题、写代码，已经能赢过所有人类。
可同一批模型拿到工程师每天用的 CAD 和仿真软件，每道题最多给 24 小时，去设计曲轴、给内存布线、算涡轮叶片散热，最好的 GPT-6 Astra 也只有 37% 能过。
其余模型在这个评测中，更是几乎全员挂零。
Xplore AI 在 10 月 27 日发布了 RWE-Bench，全称 Real World Engineering Bench。
一共 20 道题，机械 9 道，航空航天 6 道，电子 5 道。
每道题都由团队里做这一行的工程师出，出题人还要用同一套工具自己做出一份参考答案。
判分既不靠 AI，也不靠人工。
程序会调用物理求解器逐项检查交上来的设计，只要有一项不达标就算失败。
拿机械题来说，题目是设计三缸柴油机的曲轴。
技术简报里列着质量上限、应力上限、疲劳系数和曲柄销变形量，每一项都有具体数值。
配重加多了会超重，加少了转起来不平衡，还要在 2400 转满负荷下撑过几百万次循环。
模型手里只有 CadQuery、Gmsh、CalculiX 和 Python，建模、划网格、做有限元分析都得自己完成。
电子题是设计在一块量产 FPGA 板上补 DDR3 的走线。
板上近 300 个元件已经摆好，其他线也都布完了，一律不许动。
剩下七十多条网络要在 8 层板的空隙里走通，时序偏差和阻抗都有硬性限制，只要一条超标，整块板就不算数。
航空题要算高压涡轮导向叶片的传热，从建流道一直做到 CFD 收敛，最后交出换热系数和总热负荷。
在工程公司里，这些活要交给有几年经验、熟悉对应软件的工程师来做，门槛相当之高。
1.先看排名。
GPT-6 Astra 37%，Claude Opus 5.5 29%，GPT-6.1 Sol 25%，Gemini 4 Argon 12%，Claude Fable 5.1 7%，其余模型在这个评测中，更是几乎全员挂零。
2.除此以外，答题时间比分数更能说明问题。
每道题最多能用 24 小时，可各家模型平均两三个小时就交卷了，用时最长的也只有 4.6 小时左右。
人类工程师遇到这种题，光是反复改模型、跑仿真就不止这点时间。
模型很可能以为自己做完了，其实做错了...
3.花钱多也不一定分高。
GPT-6.1 Sol 跑一次只要 10 美元出头，Astra 和 Opus 5.5 都在 58 美元上下，分数却只高 12 分和 4 分。
Claude Fable 5.1 一次将近 98 美元，全场最贵，通过率 7%。
4.如果每道题允许试 5 次、过一次就算，Opus 5.5 的分数能到 70%，比 Astra 的 60% 还高。
1.题目只有 20 道，样本容量很小。
官方给出的 95% 置信区间里，Astra 是 30.7% 到 43.8%，Opus 5.5 是 21.9% 到 37.6%，两段有重叠，第一和第二谁更强其实还不能下定论。
2.页面上说所有模型都开了最高推理强度，但数据文件里 Gemini 4 Argon 标的是 high，不是 max。
当然，刚发布的旗舰模型 Gemini 4 Carbon 没来得及出现在本次评测中，我们也很期待 Xplore AI 对它的补测。
3.还有一点，Xplore 本身就是卖 RL 训练环境的。
官网写的是，他们可以帮模型公司做评测、搭训练环境，只是榜单里的题不授权拿去训练。
榜单上的分数越低，这门生意就越好推销。
这不等于数据有问题，题目和判分方式都公开了，但对于题目难度的控制，是关乎他们的生意好坏的。
4.有 4 道所有模型都没做出来的题，具体是哪几道，官方没有公开。
Xplore 下一步要加入生物力学、土木和材料工程的题，目前也在招工程师和研究员。
SWE-bench 在 2023 年刚发布时，最好的模型只解决了不到 2% 的问题，后来分数涨成什么样大家都看到了。
敬请期待新模型们在 RWE-Bench 上的分数飙升吧。
参考资料：
编辑：马可
不喜欢
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
New Ji-won.
AI, doing maths, writing codes, has won all humans.
And the same models get CAD and simulation software that engineers use every day, each for a maximum of 24 hours, to design curves, heat out memory wires, calculate turboleas, and the best GPT-6 Astra is only 37%.
In this assessment, the remaining model is almost entirely dead.
Xplore AI took RWE-Bench, full name Real World Engineering Bench, on October 27th.
A total of 20 questions, mechanical 9, aerospace 6, electronic 5.
Each question is made by a team of engineers who are in the business, and the issuer also uses the same set of tools to make his own reference answer.
Rating is neither AI nor manual.
The program calls for a physical solver to check the submitted design item by item, so long as there is a failure.
To take the mechanical title, the title is to design a three-cylinder diesel engine.
The technical briefs contain quality caps, stress caps, fatigue coefficients and warp deformities, each with specific values.
The extra weight will be overweight, add up to the imbalance, and over a million cycles under 2400.
Only CadQuery, Gmsh, Calculix and Python are in the hands of the models, modelling, gridging and limited meta-analysis.
Electronics are designed to fill the DDR3 line on a mass production FPGA board.
Close to 300 elements on the plate have been set and all the other lines have been laid.
The remaining 70 networks have to walk through the 8 floors, with rigid time-series deviations and resistance, and the entire plate does not count if one exceeds the standard.
The aviation problem is to calculate the heat transfer of the high-voltage turbine-directed blades, so that from the construction of the stream, the CFC is constricted, and finally the heat exchange factor and total heat load is handed over.
In engineering companies, these tasks are left to engineers with several years of experience and familiarity with the corresponding software, and the threshold is quite high.
1. First look at the ranking.
GPT-6 Astra 37 per cent, Claude Opus 5.529 per cent, GPT-6.1 Sol 25 per cent, Gemini 4 Argon 12 per cent, Claude Fable 5.1.7 per cent, and the rest of the model in this evaluation is almost complete zero.
In addition, the response time is more illustrative than the score.
A maximum of 24 hours is available for each Question, but each model has been submitted for an average of two to three hours, with the longest being in the range of 4.6 hours.
Human engineers encounter the problem of changing models over and over again.
The models probably think they're finished, but they're wrong...
3. Cost is not necessarily high.
GPT-6.1 Sol runs at a rate of $10, and Astra and Opus 5.5 are down to $58, but the scores are only 12 and 4 points higher.
Claude Fable 5.1 nearly $98 at once, the most expensive, 7% pass.
If five or more attempts are allowed for each question, the score of Opus 5.5 can reach 70 per cent, 60 per cent higher than that of Astra.
1. The title is only 20 channels, with a small capacity of sample.
Officially, 95% of the confidence compartments, Astra is 30.7% to 43.8%, Opus 5.5 is 21.9% to 37.6%, overlaps between two segments, and the first and second are not conclusive.
2. It is stated on the page that all models have the highest degree of reasoning, but Gemini 4 Argon is marked as high, not max, in the data file.
Of course, Gemini 4 Carbon, the flagship model of just publish, did not appear in time for this evaluation, and we are looking forward to Xplore AI's retrospect.
3. There is also the point that Xplore itself sells RL training environments.
The network wrote that they could help model companies to evaluate and build an environment for training, but that the list did not authorize training.
The lower the score on the list, the better the business goes.
This does not mean that there is a problem with the data, that the title and the way in which the rating is given is made public, but control over the difficulty of the subject is about their business.
4. There are four questions that have not been addressed by all models, specifically which ones are not officially available.
The subject of biomechanical, civil and material engineering is being added to Xplore next step and engineers and researchers are being recruited.
SWE-bench, when it was publish in 2023, the best model solved less than 2% of the problem, and then the score went up to what everyone saw.
Please wait for the new models to have their scores on RWE-Bench soar.
References:
Editor: Marco
I don't like it.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.