新智元报道
如今Agent已经能熟练调用工具、修改文件、运行测试，但面对一个长期演进的大型软件系统，它能否从一份新需求出发，交付完整功能？
产品经理提出新需求，开发者拿到的往往不是文件清单或函数签名。需要先理解约束，追踪模块依赖，再判断改哪里、怎么改、怎样证明改对了。这些分析影响后续交付。
复旦大学、香港城市大学、香港中文大学等机构提出LoLBench，把Agent放进平均约240万行源代码的软件系统，评估从功能提案到模块交付的完整过程，检验把需求落到系统中的感知能力，以及完成跨模块修改的实现能力。
论文链接：https://arxiv.org/abs/2609.37143
数据集：https://huggingface.co/datasets/lolbench26/LoLBench
实验暴露出关键断点：Agent可能改对局部，却漏掉其他模块中必须同步的改动。即使补充实现指引，多数任务仍未解决，从理解需求到完成交付仍有很长距离。
什么样的长程代码任务，才有实际工程价值？
「长程」不能只看耗时、轮数或Token消耗。例如让Agent每分钟输出一个数字，从1数到1000，虽然耗时很久，却只需简单循环。SWE-bench类局部Bug修复的任务，也不能覆盖从需求分析到模块交付的挑战。
LoLBench关注的长程代码任务，是面向复杂系统的模块开发。接到新需求后，开发者会反复分析：哪些行为必须实现？已有架构能否支持？哪些依赖会受影响？怎样验证？LoLBench把这部分藏在开发者脑子里的分析也纳入评估，让Agent自行完成从用户意图到实现方案的推导。
这类任务同时具有感知复杂性和实现复杂性：前者是读懂提案、识别隐含约束和跨模块依赖、推导修改范围与实现规范（specification）的难度；后者是将规范转成正确的跨文件修改，并通过测试和修复避免已有功能被破坏的难度。
图1 感知复杂性与实现复杂性的比较。
LoLBench从社区功能提案（Enhancement Proposal）及对应的已合并PR中选取100项任务，覆盖29个系统、8种语言和5个领域。
Agent仅获提案与实现前代码仓，提案平均约5000个英文单词。评测在断网的Docker环境中开展，防止Agent检索上游实现。
例如，PEP 617［1］提议用PEG替换CPython的LL(1)解析器，实现时需贯通语法定义、解析器生成、抽象语法树与错误报告。Agent必须识别整条实现链路。
感知复杂性方面，LoLBench参考实现的目标类名在提案中的平均提及率仅7.3%，不到SWE-bench Verified（15.9%）的一半，也不到FeatureBench（38.7%）的五分之一。Agent须自行推测需求与代码的联系。
实现复杂性方面，LoLBench仓库平均约240万行源代码，约为SWE-bench Verified的9.4倍、FeatureBench的4.9倍；实现平均改动5499行，后两者分别为38行和2050行。每项任务平均改动64.9个文件、40.1个类和195.9个函数；文件数分别为两者的25倍和4.2倍。跨模块协同与验证更难。
表1 LoLBench与现有代码任务评测集的对比。
LoLBench通过两类测试验证交付正确性：新增功能测试（F2P）要求实现前失败、实现后通过，F2P通过公开接口检验行为，允许不同内部实现。回归测试（P2P）要求前后均通过。
在项目原有的测试基础上，研究团队构造遗漏功能、篡改需求等错误实现作为变异体，补充测试用例识别漏检错误，并提高参考补丁的测试覆盖率。
增强后，1688个变异体的检出率从约61%升至96%，F2P按任务平均的行覆盖率从约60%升至79%。
研究团队评估6个前沿模型与5个脚手架的28个可用组合，单任务限时6小时。Opus 5搭配Claude Code或mini-SWE-agent均以14%的解决率并列最高，全部组合的中位数仅2%。
表2 论文中的主要实验结果
同用Opus 5，mini-SWE-agent与Claude Code解决率相同，前者耗时为后者的3.2倍、生成Token为5倍，更多消耗未换来更高解决率。
从Token消耗来看，按Agent平均，定位阶段占非缓存输入Token的52.4%、生成Token的45.4%。从Agent交互轮数来看，定位阶段的交互轮数也显著多于其他阶段。
图2 各阶段的平均交互轮数。
对照参考实现修改的既有文件和函数，全部28组Agent的文件修改精确率为38.0%至80.4%，召回率为30.3%至65.0%；函数修改精确率为54.2%至65.6%，召回率为20.8%至53.4%。代码定位准确率低，覆盖仍不完整。
研究团队保留原功能提案，额外提供参考实现导出的修改文件树与API实现规范，帮助Agent确定修改范围。Opus 5+Claude Code的解决率从14%升至34%，GPT-5.6 Sol+Codex从3%升至25%，DeepSeek V4 Flash+OpenCode从1%升至17%。
表3 补充修改文件树与API实现规范后的实验结果
补充修改指引能缓解感知负担，但最佳组合仍有66%的任务未解决。Agent仍需协调模块、实现接口并通过验证，这提示大型系统的实现复杂性同样很高。
这些结果为Agent研发提供了可检验的方向：仓库检索能否覆盖关键依赖，编码前的实现规范是否完整，验证能否及时发现跨模块遗漏。
软件工程的价值，在于用户需求能否被可靠兑现。Agent需要理解变更为何发生、如何影响系统，并把变化完整实现。LoLBench希望推动Agent能力评估从生成代码走向支持软件持续演进，让交付结果成为检验Agent能力的尺度。
参考资料：
PEP 617：https://peps.python.org/pep-0617/
编辑：LRST
不喜欢
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
New Ji-won.
Today, Agent is skilled in accessing tools, modifying files, running tests, but can it deliver full functionality from a new demand in the face of a large software system that has evolved over time?
Product managers raise new needs and developers are often given lists of documents or functional signatures. There is a need to understand constraints, to track module dependence, to judge where, how, and how. These analyses affect subsequent delivery.
The University of Jordan, the City University of Hong Kong and the Chinese University of Hong Kong have proposed LoLBench to place Agent in a software system with an average of 2.4 million lines of source code, to evaluate the complete process from functional proposal to module delivery, to test the sensory ability to bring demand into the system, and the ability to deliver across modules.
Paper link: https://arxiv.org/abs/2609.37143
Data set: https://huggingface.co/datasets/lolbench26/LoLBench
The experiment revealed key breakpoints: Agent may change to local, but missing changes in other modules that have to be synchronized. Even if it were to complement the implementation guidelines, most tasks remained unresolved, and there was still a long way from understanding needs to completing delivery.
What kind of long-range code mission would have practical engineering value?
"Long-range" cannot be viewed only as time-consuming, wheeled or Token consumption. For example, it takes Agent a minute to output a number, from 1 to 1,000, which, while taking a long time, simply circulates. The SWE-Bench-type partial Bug repair mission cannot cover the challenges from needs analysis to module delivery.
The long-range code task of concern to LoLBench is modular development for complex systems. When new demands are received, developers will repeatedly analyse what must be done? Can structures exist to support them? Which dependencies will be affected? How? LoLBench also included this part of the analysis hidden in the developer's head, allowing Agent to carry out its own inference from user intent to achievement.
Such tasks are both sensitive and complex: the former are difficult to read proposals, identify implied constraints and cross-module dependence, extrapolate the scope of changes and achieve norms; and the latter are difficult to translate norms into correct cross-documentation and avoid the destruction of existing functions through testing and repair.
Figure 1
LoLBench selects 100 tasks from the Community Production proposal and its corresponding consolidated PR covering 29 systems, 8 languages and 5 areas.
Agent received only the proposal and the pre-achievement code warehouse, with an average of about 5,000 words in English. The assessment was conducted in the off-grid Docker environment to prevent the upstream realization of Agent search.
For example, PEP 617 [1] proposes to replace the CPython's LL(1) solver with PEG, which will be achieved through the use of syntax definitions, solver generation, abstract syntax tree and error reporting. Agent must identify the whole realization link.
In terms of sensory complexity, the average reference rate in the proposal for the target category referred to by LoLBench was only 7.3 per cent, less than half of the SWE-bench Verified (15.9 per cent) and less than one fifth of the FeatureBench (38.7 per cent). Agent has to speculate on the link between demand and code.
In terms of achieving complexity, the LoLBech warehouse averaged about 2.4 million lines of source code, about 9.4 times that of SWE-bench Verified and 4.9 times that of FeatureBench; average changes were achieved in 5499 lines, the latter 38 lines and 2050 lines, respectively. For each mission, an average of 64.9 documents, 40.1 classes and 195.9 functions were changed; the number of documents was 25 and 4.2 times, respectively. Cross-module synergy and validation is more difficult.
Table 1
LoLBech validates delivery accuracy through two types of testing: the additional functional test (F2P) requires pre-facing, post-facing, and F2P allows for different internal performances through an open interface. The re-entry test (P2P) requirements are passed before and after.
On the basis of the original tests of the project, errors such as missing functions in the construction of the team, tampering with needs, etc., were realized as variants, supplemented by cases of testing to identify errors and increased test coverage of reference patches.
The increase resulted in an increase in the detection rate of 1688 variants from approximately 61 per cent to 96 per cent, and in the average F2P coverage by mission from about 60 per cent to 79 per cent.
The research team assessed the six forward models and 28 available combinations of five scaffolds for a single task of limited-time6 hours. Opus 5 combined with Claude Code or Mini-SWE-agent were the highest at a 14 per cent resolution rate, with a median of only 2 per cent for all combinations.
Table 2 Main experimental results in papers
Opus 5, Mini-SWE-agent has the same resolution rate as Claude Code, which takes 3.2 times more time than the latter, generates Token five times more, and more consumption is not replaced by higher resolution rates.
According to Token consumption, the positioning phase accounted for 52.4 per cent of the non-cache input to Token and 45.4 per cent of the generation of Token, on average, by Agent. The number of interactive rounds at the positioning stage is also significantly higher than at the other stages in terms of the Agent interactive rotation.
Figure 2 Average number of interactive rounds at each stage.
For all 28 groups of Agent, the accuracy of document modification rates ranged from 38.0 per cent to 80.4 per cent and recall rates from 30.3 per cent to 65.0 per cent, compared to existing documents and functions where changes were made by reference; the accuracy of function modification rates ranged from 54.2 per cent to 65.6 per cent and recall rates from 20.8 per cent to 53.4 per cent. Code positioning is low and coverage remains incomplete.
The research team retained the original functional proposal and provided additional references to achieve the output of the modified document tree and the API to standardize and help Agent define the scope of the changes. The resolution rate for Opus 5+Claude rose from 14% to 34%, for GPT-5.6 Sol+Codex from 3% to 25% and for DeepSeek V4 Flash+OpenCode from 1% to 17%.
Table 3 Additional results of experiments following the modification of document tree and API
Additional modifications to the guidelines would ease the perception burden, but 66 per cent of the best combination remained unresolved. Agent still needs to harmonize modules, interface and validate, suggesting that the complexity of large systems is equally high.
These results provide a verifiable direction for Agent ' s R & D: whether warehouse searches cover critical dependencies, whether pre-coding realization norms are complete and whether cross-module omissions can be detected in a timely manner.
The value of software engineering lies in the reliability of user needs. Agent needs to understand why changes occur, how they affect the system, and to complete the changes. LoLBench wishes to facilitate the evolution of the Agent capacity assessment from the generation of codes to supporting software, so that delivery results become a measure of Agent's ability.
References:
PEP 617：https://peps.python.org/pep-0617/
Edit: LRST
I don't like it.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.