JavaScript 框架的人工智能成本

JavaScript 框架的人工智能成本

我对网络开发界的一个宠物小说是，我们所做的许多决定都是基于氛围的。 工具会“加速”你的开发，正确的抽象将有利于您和您的用户(DX和UX都在其中！所有这些无气息的自夸通常都是以零证据提供的。 Baldur Bjarnason将这个“前卫失落的十年”称为流行文化

长期以来，我们社区对证据的过敏性似乎是出于对更大的DX的渴望。 换句话说，devs只是喜欢在React中输入JSX的感觉，而推广React或相关工具的论据(“虚拟DOM是快的！”)主要是在浩劫后进行合理化，以证明这些最初的直觉感是合理的。 (Jonathan Haidt可能同意这一评估。 )

但在代理时代，当开发者不再手动打入代码，因此受到DX的制约较少时，我仍然看到气氛支配着这一天(而不只是"vibe-coding"的意思). 前端被重写，因为有些或另一种框架“对agents更好”。 (有人想问agents吗？由此产生的代码是“较快”,agents犯了“错误”，等等。 再一次，这些声称往往是在没有证据的情况下制造的。 每当我读到其中一个账户时，我都会想：“基准测试在哪里？”

嗯，我好奇，所以我写了一个基准测试。 这个基准测试的结构相当直截了当：

- 使用PokéAPI设计一个小型但非三角的Pokémon应用程序(那些跟踪我作品的人不会被惊叹)

- 指示 agents 以特定的 JavaScript 框架/ meta 框架构建应用程序(例如 Next.js, Nuxt, SvelteKit 等)

- 以Spec符合套件测试输出(以避免agent作弊)

- 在灯塔的开发时间、花费的纪念品、代码线、性能和无障碍性评分等方面评价成果

这个应用程序相当简单：您有一张Pokémon列表，您可以搜索、过滤等：

首先，我应该提出一些告诫：这不是一个完美的实验。 这个应用只代表一个holotype – 它是一个相对基于内容的网站，几乎没有交互作用，比Figma,GMail等更接近维基百科。 称，我不得不选择一些应用程序来构建，这个软件至少在高水平上容易理解，并且相对直截了当地进行检测。

为了尝试甚至竞技场，我让每个框架/meta-框架都使用官方的MCP或该工具的技能。 我想避免有人指责“这个框架在培训数据中代表性不足，我也没有指出该应用应该尽量发挥效果。 我只是以极大的努力给Opus 5.5 提供了这个规格，并告诉它使用给定框架的最新版本。 每个agent运行在一个沙盒中，并不知道会在灯塔进行性能和无障碍测试，虽然它得到了符合性测试套件。

除了避免使用Tailwind、Bootstrap或任何其他组件框架，以及基于一组光谱符号的设计，而不是JS框架，CSR vs SSR, SPA vs MPA等。 所生成的应用软件是用一个基于无障碍树而不是原始的DOM的兼容套件进行测试的，以避免强制使用任何特定的HTML结构 — — 一个框架使用依赖影子DOM的网络组件，所以需要这样做。

从这个实验中我学到的一件事是，我所测量的每件事物都非常有条理。 LLMs是随机字的生成器，所以有时你会得到一个低骰子卷，agent会在一些怪异的切换上浪费一串代币，没有通过框架或其工具的故障。 灯塔本身也可以产生不一致的结果：最大的内容绘画(LCP)可能决定一个图像是“最大的”元素，然后在下一次运行时选择另一个。 很难有一个完全公平的实验，但运行多个迭代确实有帮助

让我们看看结果。

结果

我将用多个维度来分解，对于每个框架/meta框架，我取了5个运行的中位数：换句话说，我在同一框架中构建了5次相同的应用。 (是的，这是那种赞美的事情你可以做编码agents现在。)

例外的是Angular，它两次没有创建工作应用，所以我接受了三个迭代。 (下文对此作了解释。 )

费用

且并不代表我花了多少钱。 所以这只是我每月100美元计划的一部分。

请注意差异：React路由器的中位数大约比Hono+htmx差1美元，但最便宜的React路由器(2.82美元)仍然比最昂贵的Hono+htmx(3.19美元)便宜。 然而，确实出现了一些可靠的模式。

最明显的是，SvelteKit,Gracile+Lit，以及Angular比其他框架更昂贵。 在SvelteKit中，这可能是由于agents在每次运行中都读作SvelteKit v3迁移指南，这可能是因为Opus 5.5的训练数据太新了。 对Gracile来说，agents也不太了解这个框架，对于Angular来说，它似乎是总代码大小的函数(见下文).

我还要指出，并非每个agent都使用MCP/skill/Claude Code插件，每个框架也没有一个：

点击查看AI 工具的使用

例如，TanStack Start的agent只在2/5运行中发现了其技能文件(它们被运入项目的节点单元)。 对于Nuxt,MCP从未被agents使用。 对于Next.js,agents只读作1/5运行中的技能文件，AGENTS.md则读作2/5运行中的技能文件。 这是一系列广泛的结果，这可以追溯到LLMs的浮沉性。 我仍然认为这是一个有效的实验，因为我怀疑许多气温编码器在全高喊着他们的agents，以使用MCP(而且很多可能完全跳过它).

若你想知道我是怎么找到Hono+htmx的：基本上我想用htmx作为代表，关于格拉西尔+ Lit，我想要通过他们的标准持有者框架(Lit)来代表网络组件，如果有更好的候选人，那么这些候选人对于后续试验来说将是非常有益的。

对于“Vanilla”，请注意agent被告知使用“vanilla JavaScript或带有Vite的TypeScript，没有UI框架”。 令人惊讶的是，这导致一些框架表现不佳，这似乎表明，有些框架可能比无更好(或者说比原始维特更好)。 Vite之所以被选中，是因为它使许多框架处于下层，并且代表了相当标准和不可知的JavaScript基础设施。

总的时间和标志相当接近成本：

顺便说一句，这些纪念物是由Claude Code报告的。 持续时间只是总的墙时钟时间。

质量

当然，因为我认为结果很重要，我还用灯塔测试了agents。 作为软件开发者，我认为重要的是我们不仅建造廉价的东西，而且建造得优美和方便。

我首先要提到两个关于我的基准测试的警告：

Pokémon应用程序很小

- 灯塔和核心Web Virtuals总体上偏向于SPA，因此MPA框架几乎可以保证在Total Blocking Time(TBT),JavaScript字节等度量衡上获胜。 我考虑增加一些应用中的性能衡量标准，但因它增加了基准测试的复杂性和可变性而退缩，这些将是我发明的衡量标准而不是行业标准。

我只向主页汇报得分，还有其他几页，但是它们都得分相当相似，或者它们在像LCP这样的度量中有狂野的摇摆，所以把它们包含进来只会使事情复杂化。

简言之，灯塔的性能、无障碍性和CLS(Cumulative Production Shift)得分几乎饱和

因此，为了获得一些更有趣的数字，我们需要看看其他的衡量标准。 最可靠的是发送的JavaScript字节，它主要跟踪客户端框架大小：

Total Blocking Time (TBT) 很有趣，因为它并非纯粹JavaScript字节的功能；有些应用在主线上花费的时间比其他应用多。 称的数字都很小：

最大的内容涂料(LCP)由于上述问题及其计量标准本身而变化很大。 如所料，它广泛惩罚最高人民会议并奖励海洋保护区：

请注意，Next.js数字的野性差异可以用一个应用避免懒惰的图像来解释，而另外四个则没有。 Astro和Vanilla也有类似的摇摆，这似乎是由优先排序或LCP突起引起的。

但总而言之，衡量标准往往会奖励使用HTML快速涂装的MPA，而非客户端JavaScript运行后涂装的SPA(同时可能还会有多个节奏来抓取Pokémon数据)，忽略差异，这里的LCP主要是JavaScript捆绑大小的函数。 (Qwik在这里是一个显著的例外：从技术上讲，它是一个使用客户端路由的SPA，但它在MPA框架中得分).

请注意，有些框架在是否更注重客户端或服务器，甚至是否使用客户端框架方面各不相同。 例如，Gracile agent甚至没有在两次运行中使用Lit. 对于“Vanilla”,agent拥有完全的纬度(除了被告知使用Vite之外)，但仍选择全部5次建造SPA。

SSR vs CSR也是Angular问题的部分来源：在2跑中，agent制造了SSRapp，拒绝非本地主机(尽管明确指示称，该app将会在反向代理背后). 我认为这是正当的未能遵守这个规格，并决定不再重运行两个失败的迭代。

可维持性

有很多方法可以衡量可维持性(包括诸如百科全书的复杂性之类的奇特事物)，但我决定保持简单，并报告一些经过尝试和真实的事物：代码的总行。

对我来说，这是更有趣的图表之一。 从某种意义上讲，一个框架的整个要点是减少你需要维护的代码数量。 用Rich Harris难忘的话来说：

“框架不是组织你们代码的工具，而是组织你们思想的工具。”

- 里奇·哈里斯，“反思反应”

在代理时代，这个达到更加困难，因为代码大小最终会影响符号的数量，包括在未来的编辑中，这意味着它会影响随着时间推移而维持一个代码库的字面成本。 这并不意味着简洁是唯一重要的价值，但看到Hono+htmx可以和Angular一样

结论

有很多方法可以改进这个基准测试。 例如，您可以授权 agent 使用 Service Workers 进行离线缓存，或者优化 Pokémon 图像文件，或者使用懒惰加载，获取优先级，Speculation Rules API 等。 虽然我知道很多这些表演黑客

本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

The AI cost of JavaScript frameworks

One of my pet peeves with the web development community is that so many of the decisions we make are vibes-based. Frameworks are “blazing fast,” tools will “supercharge” your development, the right abstraction will benefit both you and your users (DX and UX all in one!). All of this breathless boasting is usually offered with zero evidence. Alex Russell has called this “frontend’s lost decade”, Baldur Bjarnason has called programming a pop culture, and I’ve tried to counter this trend in my blog with facts and benchmarks (at my best, anyway).

For a long time it seemed like our community’s allergy to evidence was driven by the desire for greater DX. In other words, devs just liked the way it felt to type JSX in React, and the arguments promoting React or related tools (“the virtual DOM is fast!”) were largely post-hoc rationalizations to justify those initial gut feelings. (Jonathan Haidt would likely agree with this assessment.)

But in the agentic era, when developers are no longer typing code by hand and are thus less constrained by DX, I still see vibes ruling the day (and not just in the sense of “vibe-coding”). Frontends are rewritten because some framework or another is “better for agents.” (Did anyone think to ask the agents?) The resulting code is “faster,” the agents make “fewer mistakes,” etc. Once again, these claims are often made without evidence. Whenever I read one of these accounts, I think to myself, “Where are the benchmarks?”

Well I got curious, so I wrote a benchmark. The structure of this benchmark is pretty straightforward:

- Design a small but non-trivial Pokémon app using the PokéAPI (those who have followed my work will be unsurprised)

- Instruct agents to build the app with a particular JavaScript framework/meta-framework (e.g. Next.js, Nuxt, SvelteKit, etc.)

- Test the output against a spec conformance suite (to avoid agent cheating)

- Evaluate the results in terms of development time, tokens spent, lines of code, and performance and accessibility scores in Lighthouse

The app is fairly simple: you’ve got a list of Pokémon with their types, moves, and data from various games. You can search, filter, etc.:

I should start off with some caveats: this is not a perfect experiment. This app represents only one holotype – it’s a relatively content-based site with little interactivity, closer to a Wikipedia than a Figma, GMail, etc. That said, I had to choose some app to build, and this one is at least easy to understand at a high level, and relatively straightforward to spec out and test.

To try to even the playing field, I had each framework/meta-framework use the official MCP or skills for that tool. I wanted to avoid accusations of “this framework is not well-represented in the training data, that’s why we built our MCP, we expect people to use it,” etc. I also didn’t over-specify the exact architecture to use (e.g. SSR vs CSR), nor did I specify that the app should be as performant as possible. I simply gave the spec to Opus 5.5 with high effort and told it to use the latest version of the given framework. Each agent ran in a sandbox and without any knowledge that it would be tested in Lighthouse on performance and accessibility, although it was given the conformance test suite.

Note I also didn’t mandate any CSS methodology, except to avoid using Tailwind, Bootstrap, or any other component framework, and to design based on a set of spec tokens to enforce some visual consistency. I didn’t want to make this test all about CSS, but instead about JS frameworks, CSR vs SSR, SPAs vs MPAs, etc. The resulting app is tested against a conformance suite based on the accessibility tree rather than the raw DOM to avoid mandating any particular HTML structure – one framework uses web components which rely on shadow DOM, so this was required.

One thing I learned from this experiment is that everything I’m measuring is very stochastic. LLMs are random-word generators, so sometimes you get a low dice-roll and an agent goes off on some weird tangent wasting a bunch of tokens, through no fault of the framework or its tools. Lighthouse itself can give inconsistent results as well: Largest Contentful Paint (LCP) may decide that one image is the “largest” element, then pick another one on the next run. It’s hard to have a perfectly fair experiment, but running multiple iterations did help, and I’ve tried to be honest about the variance.

With all that throat-clearing out of the way, let’s get to the results.

Results

I’m going to break this down by multiple dimensions: cost to build, quality, and maintainability. For each framework/meta-framework, I took the median of 5 runs: in other words, I built the same app 5 times in the same framework. (Yes, this is the kind of zany thing you can do with coding agents these days.)

The exception was Angular, which failed to create a working app twice, so I took 3 iterations instead. (This is explained below.)

Cost

Note that this quotes the API prices directly from Claude Code and doesn’t actually represent how much money I spent. I’m on a Max plan, so this was just part of my overall $100/month.

Note also the variance: React Router’s median is roughly $1 worse than that of Hono+htmx, but the cheapest version of React Router ($2.82) is still cheaper than the most expensive version of Hono+htmx ($3.19). However some reliable patterns do emerge.

Most clearly, SvelteKit, Gracile+Lit, and Angular are more expensive to build than the other frameworks. For SvelteKit, this may be due to the fact that the agents read the SvelteKit v3 migration guide in every run, likely because it’s too new to be in Opus 5.5’s training data. Similarly for Gracile, the agents didn’t know the framework very well and either cloned the Gracile repo or read its compiled source from node_modules. For Angular it seems to be a function of the total code size (see below).

I should also mention that not every agent made use of an MCP/skill/Claude Code plugin, nor did every framework have one:

Click to see AI tool usage

For example, TanStack Start’s agent only discovered its skill files in 2/5 runs (they’re shipped in the node_modules of the project). For Nuxt, the MCP was never used by the agents. For Next.js, agents only read the skill file in 1/5 runs and the AGENTS.md in 2/5 runs. This is a wide range of results, which points back to the stochastic nature of LLMs. I still consider this a valid experiment, though, because I doubt many vibe-coders are shouting at their agents in all-caps to use the MCP (and plenty are probably skipping it entirely).

Also, if you’re wondering how I came up with Hono+htmx: essentially, I wanted htmx represented somehow, and Hono seemed like a fine pairing (i.e. as a JavaScript alternative to Rails/Django/etc.). As for Gracile+Lit, I wanted web components represented via their standard-bearer framework (Lit), and was only aware of Gracile as a meta-framework. If there are better candidates, then those would be great for a follow-up experiment.

For “Vanilla,” note that the agent was told to use “vanilla JavaScript or TypeScript with Vite, without a UI framework.” Surprisingly, this underperforms several frameworks, which seems to demonstrate that having some framework can be better than nothing (or better than raw Vite, anyway). Vite was chosen because it undergirds many frameworks and represents fairly standard and agnostic JavaScript infrastructure.

The total time and tokens track pretty closely with the cost:

The tokens, by the way, were reported by Claude Code. The duration is just the total wall-clock time.

Quality

Of course, because I think that outcomes matter, I also tested the agents with Lighthouse. As software developers, I think it’s important that we not only build things cheaply but also performantly and accessibly.

I should first note two caveats about my benchmark:

- The Pokémon app is small and doesn’t represent a large, complex real-world application.

- Lighthouse and the Core Web Vitals in general are biased against SPAs, so the MPA frameworks are almost guaranteed to win on metrics like Total Blocking Time (TBT), JavaScript bytes, etc. I considered adding some in-app performance metrics but backed off because it added complexity and variability to the benchmark, and these would be metrics I invented as opposed to industry standards.

For simplicity’s sake, I’m only reporting the scores for the home page. There are other pages, but they all score fairly similarly or they have wild swings in metrics like LCP, so it just complicates things to include them.

In short: the Lighthouse scores for Performance, Accessibility, and CLS (Cumulative Layout Shift) are nearly saturated and don’t vary much by framework:

So to get some more interesting numbers, we need to look at other metrics. The most reliable one is the JavaScript bytes sent, which largely tracks the client-side framework size:

The Total Blocking Time (TBT) is interesting in that it’s not purely a function of the JavaScript bytes; some of the apps spend more time on the main thread than others. That said, the numbers are all quite small:

Largest Contentful Paint (LCP) is highly variable due to the aforementioned issues with the metric itself. As expected, though, it does broadly punish SPAs and reward MPAs:

Note that the wild variance in the Next.js numbers can be explained by the fact that one app avoided lazy-loading images and the other four didn’t. Astro and Vanilla have similar swings that appear to be caused by prioritization or LCP quirks.

In general though, it’s unsurprising that the metric tends to reward MPAs that paint quickly with HTML rather than SPAs that paint after client-side JavaScript runs (along with possibly multiple throttled round-trips to grab the Pokémon data). Ignoring variance, LCP here is mostly a function of JavaScript bundle size. (Qwik is a notable exception here: technically it’s an SPA that uses client-side routing, but it scores among the MPA frameworks.)

Note though that some of the frameworks varied in whether they focused more on the client or the server, or even whether they used a client-side framework. For example, the Gracile agent didn’t even use Lit in two runs. For “Vanilla,” the agent had full latitude (aside from being told to use Vite), but still chose to build an SPA all 5 times.

SSR vs CSR is also partly where the trouble with Angular came from: in 2 runs, the agent built an SSR app that rejected non-localhost (even though the instructions explicitly said that the app would be behind a reverse proxy). I considered this a valid failure to adhere to the spec and decided not to re-run the 2 failed iterations.

Maintainability

There are lots of ways to measure maintainability (including fancy things like cyclomatic complexity), but I decided to keep it simple and report something tried-and-true: total lines of code.

To me this is one of the more interesting charts. In a sense, the whole point of a framework is to reduce the amount of code you need to maintain. In Rich Harris’s memorable words:

“Frameworks are not tools for organizing your code; they are tools for organizing your mind.”

– Rich Harris, “Rethinking reactivity”

In the agentic era, this hits even harder because the code size ultimately impacts the number of tokens, including in future edits, meaning it affects the literal cost to maintain a codebase over time. This doesn’t mean that succinctness is the only value that matters, but it’s certainly interesting to see that Hono+htmx can do roughly the same thing as Angular with ~70% of the code (and faster, to boot).

Conclusion

There are many ways you could improve this benchmark. For example, you could mandate that the agent use Service Workers for offline caching, or optimize the Pokémon image files, or use lazy-loading, fetchpriority, the Speculation Rules API, etc. Even though I know a lot of these performance hacks, I deliberately didn’t steer the agents toward any of them,

Auto-fetched and translated by AI Atlas. All rights belong to the original author.