任何与计算机编程有微小关联的人都知道，现代AI编码助理和agents可以令人难以置信地高效地生成大量功能代码。 但是，使用这些工具的编码员也比相信该代码的准确性更清楚，这意味着需要花费大量精力来审查AI产生的任何产出。
最近对数百家发现公司实际编码做法的研究显示，人代码审查对AI编码工具的总体效率构成了一个重大的“瓶颈”，从而产生了“公司通过使用软件增加软件输出或减少就业的微小证据”。 研究作者认为，在实际编码阶段，任何效率的提高都“被生产过程中的下游限制所吸收”；因为“编码审查过程的长度大大增加，拉动请求更有可能需要修改，审查人员留下更多的评论。”
切一次，量两次
为了得出这些结论，哈佛大学研究人员Fiona Chen和James Stratton利用了来自Jellyfish的汇总分析数据，这些数据测量了工程团队的颗粒输出。 从2021年到2026年3月，该数据涵盖700,000多名相关软件开发公司雇员的3亿个人“工作活动”(例如，承诺和拉动请求)和发布管理软件数据。
为了评估AI工具对这些公司的影响，研究人员利用直接测量的AI使用和分析GitHub活动的组合来确定每个公司何时开始在其工作流程中引入AI编码助理(这可以帮助主要由人类编写的自动完成代码)和/或AI编码agents(主要根据提示自动编写和提交代码). 然后，研究人员进行一些复杂的数学，以确定在不同组织采用这些工具的不同时间点之前和之后，关键变量的“差异”回归。
在原始代码方面，结果既明确又严酷。 在一家公司引入AI编码agents导致生成的代码总行数增加30%，总承诺数增加20%，研究者撰写的拉力请求平均增加23%. 但所有额外的代码并没有直接转化为公司层面改进的软件输出。 相反，由Jira等工具跟踪的问题和Epics(即批发软件特征)的解析率在AI工具为推出之后并没有在统计学上发生显著变化(研究人员还发现AI导言中这些Jira跟踪的问题的规模和复杂性没有"结构转变").
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Anyone who has even tangentially associated with computer programming knows that modern AI coding assistants and agents can be incredibly efficient at generating huge amounts of functional code. But coders making use of those tools also know better than to trust the accuracy of that code, meaning substantial effort needs to be spent reviewing any AI-generated output.
A recent study of actual coding practices across hundreds of firms finds that human code review forms a significant “bottleneck” for the overall efficiency of AI coding tools, resulting in “little evidence that firms increase software output or reduce employment” by using them. Any efficiency increased during the actual coding phase, the study authors find, is “absorbed by downstream constraints in the production process”; as “the code review process significantly increases in length, pull requests are more likely to require revisions, and reviewers leave more comments.”
Cut once, measure twice
To come to these conclusions, Harvard University researchers Fiona Chen and James Stratton made use of aggregated analytics data from Jellyfish, which measures the granular output of engineering teams. That data encompasses 300 million individual “work events” (e.g., commits and pull requests) and issue management software data across more than 700,000 employees at over 700 relevant software development firms from 2021 through March of 2026.
To assess the impact of AI tools on these firms, the researchers used a mix of directly measured AI usage and analyses of GitHub activity to determine when each company started introducing either AI coding assistants (which can help auto-complete code primarily authored by humans) and/or AI coding agents (which primarily write and submit code autonomously based on prompts) into their workflows. The researchers then perform some complicated math to determine a “difference of differences” regression on key variables both before and after the introduction of these tools at different points in time across different organizations.
In terms of raw code being produced, the results are clear and stark. The introduction of AI coding agents at a firm leads to a 30 percent increase in total lines of code generated, a 20 percent rise in the number of total commits, and a 23 percent increase in pull requests on average, the researchers write. But all that extra code doesn’t translate directly into improved software output on the firm level. On the contrary, the resolution rate for Issues and Epics (i.e. wholesale software features) tracked by tools like Jira did not change in a statistically significant way after AI tools were introduced (the researchers also found no “compositional shift” in the size or complexity of those Jira-tracked issues across the AI introduction).
Auto-fetched and translated by AI Atlas. All rights belong to the original author.