扩散语言模型开始向更大规模迈进
大语言模型一定要从左到右、逐个生成 Token 吗？
过去几年，以 GPT、Claude、Qwen 为代表的自回归语言模型一直占据主流。随着模型能力不断提升，逐 Token 解码带来的串行计算开销，也成为影响推理速度的重要因素之一。
扩散语言模型（Diffusion Large Language Model，dLLM）正在探索另一条技术路线。它不必严格按照从左到右的顺序生成文本，而是通过迭代去噪、并行更新多个 Token 完成生成，有望在特定场景下获得更高的解码效率。
但这种路线也面临一个问题：当模型规模不断扩大、任务复杂度持续提高时，能否在保持速度优势的同时，达到主流自回归模型的能力水平？
10 月 28 日，深圳 AI 创业公司扩散智能（DiffuSpace）发布并开源 Clover-35B-A3B-Preview，尝试回答这一问题。
DiffuSpace 由香港大学 NLP 实验室联合主管孔令鹏教授及其博士生龚珊三、叶佳成创立，团队此前曾推出 DiffuSeq、Dream 7B 等扩散语言模型相关研究成果。近期，公司连续完成两轮融资，总金额达数亿元人民币。
此次发布的 Clover-35B-A3B-Preview 采用混合专家（MoE）架构，总参数量为 350 亿，每次推理激活约 30 亿参数。相比上一代 Clover 模型，其总参数规模扩大至约 5 倍。
根据 DiffuSpace 公布的内部测试结果，新模型的整体生成速度达到 680 tokens/s。在部分 Agent 任务和通用能力评测中，其表现达到或超过了同等规模的自回归模型。
对于扩散语言模型而言，这次更新值得关注的地方，在于它试图同时解决两个问题：一是如何在更大参数规模下维持并行解码的效率，二是如何将速度优势转化为真实任务中的执行能力。
680 tokens/s，速度优势能否转化为任务效率？
DiffuSpace 此次公布的测试数据，主要涉及模型生成速度、Agent 任务能力以及端到端执行时间。
在官方展示的一项无人机空域管理 Demo 中，模型需要读取 20 条空管报文，从中提取地点、半径等信息，并在三维地图上设置相应禁飞区。Clover 完成该任务耗时 7.32 秒，对照使用的 Qwen3.5 模型耗时 62.3 秒，两者相差约 8.5 倍。
这一案例展示了扩散模型在结构化信息处理任务中的潜在优势。不过，Demo 中的执行时间也受到模型配置、硬件环境、输出长度和工具调用流程等因素影响。
在 WindTunnel 的一项包含 49 个任务的测试中，Clover-35B-A3B-Preview 的中位执行时间为 1.52 秒，Jev 与 Mercury 2.5 的组合为 3.21 秒，GPT-5.6 Luna 为 5.74 秒，Gemini 3.6 Flash 为 7.22 秒。按这组结果计算，Clover 的中位任务耗时约为 Jev 与 Mercury 2.5 组合的 47%。
不过，不同模型组合的推理部署方式、任务处理流程以及计费规则可能存在差异，仅凭中位执行时间，还不足以判断其在实际生产环境中的综合效率。
对于 Agent 应用来说，这一点尤其重要。模型生成 Token 的速度，并不等同于 Agent 完成任务的速度。一个实际任务可能涉及多轮规划、工具调用、外部服务等待和结果校验。即使模型生成速度提高数倍，如果主要耗时集中在外部工具或任务执行环节，最终收益也可能有限。
因此，相比单纯比较 tokens/s，Clover 在完整 Agent 工作流中的表现更值得关注。
Agent 编程能力达到什么水平？
扩散语言模型过去的一项主要挑战，是在提高生成效率的同时，维持复杂推理、代码生成和指令遵循等能力。DiffuSpace 此次公布的评测结果显示，Clover-35B-A3B-Preview 在多项 Agent 基准测试中取得了较高分数。其中，SWE-bench Verified 得分为 73.6，Terminal-Bench 2.1 为 51.7，τ³ Overall 为 65.3。
根据团队提供的对比数据，Clover 在上述评测中的表现超过了 Mercury 2.5，以及同等参数规模的 Qwen3.6-35B-A3B。这几项测试分别涉及真实软件问题修复、终端环境中的任务执行，以及带有工具交互的复杂任务处理。相比传统知识问答或代码补全，它们更接近 Agent 需要完成的多步骤工作。
在通用能力方面，Clover 还接受了知识、数学、代码、逻辑推理、指令遵循和长上下文理解等测试。从团队公布的数据来看，Clover 在部分指标上超过了同尺寸自回归模型，但并非所有项目都占优。例如，在部分数学和代码评测中，其他模型仍然保持领先。
这也说明，扩散语言模型的能力提升并不是全面、均匀发生的。不同任务对于生成顺序、上下文建模和推理过程的要求不同，模型架构的优势也会随之变化。
重构 GDN：扩散模型如何兼顾长上下文与并行解码？
相比参数规模和测试分数，Clover 此次更值得关注的技术变化，是对底层网络架构的调整。Clover 并非从零训练，而是基于 Qwen3.6-35B-A3B 构建，在保留原有混合 GDN/Attention MoE 骨干架构的基础上，引入块级扩散生成机制，并对门控 Delta 网络（Gated Delta Network，GDN）进行双向化改造，以适应并行去噪过程。
传统自回归模型通常依赖因果注意力机制，当前 Token 只能访问已经生成的历史信息。而扩散语言模型可以在一个文本块内部同时处理多个 Token，并通过迭代更新逐步形成最终输出。
这种生成方式带来了不同的架构需求。一方面，模型需要保留足够的历史上下文，以支持长文本理解和多轮任务；另一方面，又需要允许同一生成块内的 Token 充分交换信息，避免并行生成过程中出现语义不一致。Clover 的设计尝试将这两类需求结合起来。
根据团队介绍，模型通过递归状态对历史上下文进行压缩，减少重复处理历史信息的计算开销；同时，在当前生成块内引入双向注意力，使不同位置的 Token 可以进行更细粒度的信息交互。
在此基础上，模型还结合了可学习的块大小校准机制，以及双向局部特征提取，以适应不同的生成需求。这一设计的核心，是让模型在处理历史信息和当前生成块时采用不同机制：历史信息通过压缩状态保留，当前块内部则允许双向交互与并行更新。
对于扩散语言模型来说，这种设计具有一定的针对性。如果完全依赖标准注意力机制，随着上下文长度和模型规模增加，计算与存储成本可能持续上升；如果过度压缩历史信息，又可能损失复杂任务所需的上下文细节。Clover 试图通过混合结构在两者之间取得平衡。
目前 Clover 展示了一种适用于大规模扩散语言模型的架构设计思路，但其效率优势能否在不同模型规模和任务场景中稳定延续还需要进一步验证。
从文本生成到 Agent，扩散模型还有多远？
Clover-35B-A3B-Preview 的发布，反映出扩散语言模型正在尝试从高速文本生成，向更复杂的任务执行场景扩展。过去，dLLM 最容易展示的优势是生成速度。通过并行更新多个 Token，它有机会减少部分文本生成任务所需的串行解码步骤。
但在 Agent 场景中，模型需要完成的工作远不止生成文本。它需要理解任务目标、规划执行步骤、调用外部工具，并根据执行结果持续调整后续行为。部分任务还要求模型输出严格符合特定格式，或者在多个相关判断之间保持逻辑一致。这些需求也意味着，生成速度只是 Agent 效率的一部分。
DiffuSpace 透露，接下来计划基于 Clover-35B-A3B-Preview 推出决策模型和多模态模型。其中，决策模型将面向快速、结构化的判断任务，计划支持在一次生成过程中联合处理最多 8 个关联问题。按照团队的设想，这种方式可以让模型在多个判断之间共享上下文，减少逐项独立决策可能带来的逻辑冲突。不过，这一能力目前仍属于后续产品规划，实际效果尚未可知。
从更广泛的行业发展来看，扩散语言模型已经不再只是实验室中的研究方向。Google 曾推出 Gemini Diffusion，Inception 持续迭代 Mercury 系列模型，LLaDA 等开源研究也在探索扩散式文本生成的能力边界。不同团队的技术路线并不完全一致，但都在尝试突破传统自回归解码的效率限制。
与此同时，自回归模型也在通过推测解码、模型结构优化、推理引擎改进等方式持续提高生成效率。因此，扩散模型的速度优势并不意味着它一定会取代自回归架构。未来两类技术路线可能在不同任务和部署条件下形成各自的优势。
对开发者而言，更有价值的问题是：在相同硬件、相近模型能力和可接受的质量约束下，哪种方案能够以更低的成本完成任务？这也是扩散语言模型从研究成果走向生产环境时必须回答的问题。
目前，Clover-35B-A3B-Preview 已正式开源，模型权重和相关代码分别发布在 Hugging Face、GitHub 和 ModelScope。模型采用 Apache-2.0 许可证，并提供 SGLang、llama.cpp 等推理后端的相关支持。
随着更多大规模扩散语言模型开源，行业将有机会在统一的硬件和任务条件下，进一步比较两种生成范式的实际效率。对于 Clover 而言，35B 参数规模和 680 tokens/s 的速度数据只是起点。能否将这种速度优势稳定转化为复杂 Agent 任务中的成本和效率收益，才是下一阶段更重要的考验。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
The proliferation of language models is beginning to move towards a larger scale.
Do big language models have to be left to right, token by one?
The self-repatriation language model, represented by GPT, Claude, and Qwen, has been dominant over the past few years. As model capabilities continue to grow, the serial costs associated with the token decode are also one of the important factors influencing the speed of reasoning.
Another technical route is being explored by the Diffusion Large Language Model, d LLM. It does not have to produce text in a strict left-to-right order, but does so through inverted noises, multiple token updates in parallel, and is expected to achieve higher decode efficiency in a given scenario.
But this route also faces a problem: When the size of the model is growing and the complexity of the task continues to grow, can the capacity level of the mainstream self-regression model be achieved while maintaining the speed advantage?
On 28 October, Shenzhen AI Enterprise tried to answer the question by spreading DiffuSpace publish and opening Clover-35B-A3B-Preview.
DiffuSpace was created by Prof. Koon-Peng and his Ph.D. Dr. Xian San and Ye Jia Sung, co-director of the NLP Laboratories of the University of Hong Kong. The team has previously produced research related to proliferation language models such as DiffuSeq and Dream 7B. Recently, companies had completed two consecutive rounds of financing, amounting to hundreds of millions of yuan.
Clover-35B-A3B-Preview of publish this time uses a hybrid specialist (MoE) structure totalling parameter count 35 billion and activates approximately 3 billion parameters per reason. Compared to the previous Clover model, its total parameter size has increased about five times.
According to the internal test results published by DiffuSpace, the new model is generated at a total rate of 680 tokens/s. In some Agent missions and generic capability assessments, their performance reached or exceeded a self-return model of equal size.
For the proliferation of language models, this update is a matter of concern because it seeks to address two issues at the same time: how to maintain the efficiency of parallel decodes on a larger scale of parameters, and how to convert speed advantage into implementation capacity in real task.
680 tokens/s, can speed advantages translate into mission efficiency?
DiffuSpace released test data on model production speed, Agent task capacity and end-to-end execution time.
In Demo, an officially displayed UAV domain management, the model needs to read 20 airway messages, extract from them information on location, radius, etc., and set up a corresponding no-fly zone on a three-dimensional map. Clover takes 7.32 seconds to complete this task, which is about 8.5 times different from the Qwen3.5 model used in 62.3 seconds.
This case demonstrates the potential advantages of proliferation models in structured information processing missions. However, the implementation time in Demo was also affected by modelling, hardware environment, output length and tool call processes.
In a WindTunnel test with 49 tasks, Clover-35B-A3B-Preview had a median execution time of 1.52 seconds, Jev had a combination of 3.21 seconds, GPT-5.6 Luna had 5.74 seconds and Gemini 3.6 Flash had 7.22 seconds. Based on this set, the median task for Clover is approximately 47% of Jev's combination with Mercury 2.5.
However, there may be differences between the reasoning deployments of different models, mission processing processes and costing rules, which are not sufficient to judge their combined efficiency in the actual production environment on the basis of the medium implementation time.
This is particularly important for Agent applications. The speed of the model generation token is not equal to the speed at which Agent has completed the task. One practical task may involve multiple rounds of planning, tool call, external service waiting and results validation. Even if the speed of model generation increases several times, the final benefits may be limited if the main time-consuming focus is on external tools or task execution.
Therefore, the performance of Clover in complete Age workflow deserves more attention than a simple comparison of tokens/s.
What level of programming capacity is Argentina?
One of the main challenges in the past in spreading language models has been to maintain complex reasoning, code generation and command compliance while increasing efficiency in generation. DiffuSpace's published assessment shows that Clover-35B-A3B-Preview achieved higher scores in multiple Agent baseline tests. Of these, SWE-Bench Verified score 73.6, Terminal-Bench 2.1 51.7 and Overall 65.3.
According to comparative data provided by the team, Clover performed more than Mercury 2.5 and Qwen 3.6-35B-A3B of the same size of parameters. The tests concerned, respectively, the restoration of real software problems, task execution in the end environment and complex task handling with interactive tools. They are closer to the multi-step work that Agent needs to complete than traditional knowledge questions and answers or code completion.
In terms of universal capability, Clover has also been tested for knowledge, mathematics, code, logical reasoning, command compliance and long context understanding. From the data published by the team, Clover exceeded the same size self-regression model on some indicators, but not all projects. For example, other models remain in the lead in some mathematical and code assessments.
This also indicates that the upgrading of the capability to diffuse language models does not occur comprehensively and evenly. The advantages of the model structure vary depending on the requirements of the different tasks for the sequence of generation, context modelling and reasoning processes.
Reconstruct GDN: How can proliferation models reconcile long context and parallel decode?
Clover, this more notable technological change than the size of the parameters and test scores, is an adjustment to the bottom network structure. Clover is not based on zero training, but is based on Qwen3.6-35B-A3B, introducing block proliferation generation mechanisms based on the retention of the original mixed GDN/Attention MoE backbone structure, and a two-way re-engineering of the Delta Network (Gated Delta Network, GDN) to adapt to the parallel noise process.
Traditional self-regression models often rely on causal attention mechanisms, and currently Token can access only historical information that has been generated. The diffusion language model, on the other hand, allows multiple tokens to be processed in one text block at the same time and to evolve the final output through an iterative update.
This generation brings with it different structural needs. On the one hand, the model would need to retain sufficient historical context to support long text understanding and multiple assignments; on the other hand, there would need to be a need to allow full exchange of information in Token, within the same generator block, to avoid semantic inconsistencies in parallel processes. The Clover design attempts to combine these two types of needs.
According to the team presentation, the model compresses historical context through a state of regression to reduce the computational costs of duplicate processing of historical information; at the same time, a two-way focus is introduced within the current generation block so that Token, at different locations, can interact with more finer particles.
On this basis, the model also combines learning-based block calibration mechanisms, as well as two-way local characterization extraction, to accommodate different generation needs. The core of this design is to allow models to use different mechanisms when dealing with historical information and currently generating blocks: historical information is kept in a compressed state, while the current interior allows for two-way interaction and parallel updates.
For the proliferation of language models, the design has some relevance. The calculation and storage costs may continue to rise with the increase in the length of context and the size of the model, and the details of context required for complex missions may be lost if historical information is over-compressed. Clover tried to balance the two through hybrid structures.
Clover currently demonstrates a structure design approach applicable to mass proliferation language models, but its efficiency advantages need to be further tested for sustainability in different model scales and mission scenarios.
How far is the proliferation model from text generation to Agent?
Clover-35B-A3B-Preview ' s publish reflects the proliferation language model that is trying to generate high-speed text to expand to more complex task execution scenarios. In the past, the most easily demonstrated advantage of dlm is the speed of generation. By updating multiple Tokens in parallel, it has the opportunity to reduce the serial decoder steps required for some text generation tasks.
In the Agent scene, however, the work the model has to do is much more than generate text. It needs to understand mission objectives, plan implementation steps, mobilize external tools and continuously adapt follow-up behaviour to implementation results. Some tasks also require that model outputs conform strictly to a particular format or that logical consistency be maintained between multiple relevant judgements. These demands also mean that the speed of generation is only part of the Agent efficiency.
DiffuSpace revealed that the next plan was to introduce decision models and multi-modular models based on Clover-35B-A3B-Preview. Of these, the decision-making model will be oriented towards quick, structured judgement tasks, and it is planned to support joint treatment of up to eight interrelated issues during a single generation process. This approach, as envisaged by the team, allows models to share context across multiple judgements and reduces the potential for logical conflicts of individual decision-making. However, this capacity is still part of the successor product planning and the actual results are not yet known.
In terms of broader industrial developments, diffuse language models are no longer merely the direction of research in laboratories. Google has launched the Gemini Diffusion, Inception Ongoing Mercury Series model, and open-source studies such as LaDA are exploring the power boundaries of proliferation text generation. The technical routes of the different teams were not entirely consistent, but attempts were being made to break the efficiency limits of the traditional self-return codes.
At the same time, self-regression models continue to improve their efficiency of generation through decodering, optimization of model structures and improved reasoning engines. Therefore, the speed advantage of the proliferation model does not mean that it will necessarily replace the self-return architecture. The next two types of technical routes may create their respective advantages under different mission and deployment conditions.
For developers, the more valuable question is: which options can accomplish the task at a lower cost under the same hardware, comparable model capabilities and acceptable quality constraints? This is also a question that must be answered when spreading language models from research to the production environment.
At present, Clover-35B-A3B-Preview is officially open, with model weights and associated codes publish in Hugging Face, GitHub and ModelScope, respectively. The model uses the Apache-2.0 license and provides related support to the back end of reasoning such as SGLang, llama.cpp.
With more large-scale proliferation of language model openings, industry will have the opportunity to further compare the practical efficiency of the two production paradigms under unified hardware and mission conditions. For Clover, 35B parameter size and 680 tokens/s speed data are only starting points. The next more important test is whether this speed advantage can be stabilized and translated into cost and efficiency gains in the complex Agent mission.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.