在 Agent 推理负载快速增长的背景下，传统面向聊天场景的推理引擎正面临吞吐优化目标失效、缓存利用率不足、硬件适配滞后等一系列结构性挑战。
本文整理自 TokenSpeed 独立贡献者尚旭春在 AICon 2026 深圳站的分享《构建面向 Agentic 推理的下一代 LLM 推理引擎》。他系统介绍了该项目从立项到版本迭代的完整技术路径，围绕 Agent 场景下的工作负载特征、分层解耦的系统架构、自动并行编译系统、KV Cache 演进路线、Kernel 优化策略以及跨平台抽象设计展开，并给出在 Blackwell 平台上的阶段性性能数据。
以下是演讲实录（经 InfoQ 进行不改变原意的编辑整理）。
为什么 Agentic 推理需要新推理引擎
TokenSpeed 立项之初，我们首先要回答的问题是：为什么 Agent 场景需要一套全新的推理引擎设计。
传统 Chat 类负载的优化目标相对单一，用户发起一次请求，模型一次性生成一段完整回复。在这种模式下，推理系统只需要尽可能将 batch 打满、最大化吞吐，KV Cache 在一次加载之后就被释放，利用率天然偏低。但 Agent 场景的负载特征与 Chat 有本质区别。以 coding agent 为例，模型需要持续地思考、调用工具、获取结果、再基于结果继续思考，整个过程由多轮短请求循环构成。第一次请求可能携带几十 K 的前缀上下文，但后续每次循环的输出往往只有几百个 token。这意味着推理引擎必须重新审视缓存复用策略、请求调度机制以及整个系统的优化目标函数。
在这种负载形态下，用户对延迟的敏感度显著提升。如果一个 Agent 工具在推理环节出现显著滞后，用户会直观地感受到“跟手性”不足。因此，Agent 场景下真正需要优化的是延迟约束下的吞吐。我们内部定义了一个性能指标：每用户 TPS 不低于 70，在此约束下尽可能提升每个 GPU 的吞吐能力。以 TensorRT-LLM 为对比基准，TokenSpeed 在 batch size 等于 1 的最小延迟场景下，端到端延迟比 TensorRT-LLM 快 9%；在每用户 100 TPS 的典型工作点上，每个 GPU 的吞吐高出 11%。
KV Cache 复用率是决定推理成本的关键变量。以 DeepSeek 为例，其缓存复用率据称在 99% 以上，而业界多数系统目前只能达到 70% 到 80%。在 Agent 场景下，同一段几十 K 的上下文如果能够被反复命中，意味着每次命中都能完全省去 prefill 计算，系统的整体成本会大幅下降。我们在测试中验证了 TokenSpeed 在上下文长度从 128K 扩展到 1M 时的表现，解码吞吐率仅下降 16%。此外，随着越来越多的模型采用线性注意力机制，包括千问的 Gated DeltaNet 以及 Kimi K3 所使用的 KDA，KV Cache 的管理模式也在发生变化。线性注意力模型不再维持传统的 token 到 KV 的一一对应关系，而是通过一个持续迭代的状态来表示上下文，其中包括 SSM 状态和卷积状态。如何在推理引擎中高效地更新、保存和恢复这些状态，是一个新的系统设计命题。
模型架构与硬件特性之间的耦合也在同步加深。MOE 架构自 DeepSeek 发布以来已经成为主流选择，但 MOE 模型的优化难度并不低。以 Kimi K3 为例，该模型有 896 个专家，虽然采用了 Latent MOE 技术将写成本大幅降低，激活专家的比例提升到了 16%，但路由和通信的复杂度显著增加。与此同时，Blackwell 这一代硬件引入了 Tensor Memory、新一代 Tensor Core 等一系列新特性，AMD 的 CDNA 架构也有其独特的硬件能力。如果推理引擎无法充分利用这些硬件特性，Blackwell 相对 Hopper 的性能优势就无法真正释放。基于这些观察，TokenSpeed 在 Kernel 层面针对 Blackwell 做了深度优化，确保硬件潜力被充分挖掘。
Runtime 与 Kernel 的协同设计同样不可忽视。调度器设计得再好，如果底层 Kernel 存在性能瓶颈或者耦合过深，系统整体仍然无法达到理想状态；反过来，Kernel 优化得再极致，如果调度器产生大量 bubble，吞吐也无法打满。TokenSpeed 在架构设计上吸取了这方面的经验，将调度、执行和底层算子做了系统性的解耦与重构。
Runtime：构建可扩展的推理引擎架构
TokenSpeed 的整体架构分为三个层次：控制、Runtime 和 Kernel。控制面完全使用 C++ 编写，Runtime 使用 Python 实现，Kernel 则独立为一个子包。这种分层设计的核心目的是保持模块边界的清晰性。控制面只负责调度决策，不涉及任何数据面的执行；Runtime 只负责执行控制面下发的执行计划，包括缓存计划和前向计算计划；Kernel 层则专注于算子实现。三者各司其职，避免了过去推理引擎中调度逻辑、执行逻辑和算子实现相互渗透导致的维护困境。
控制面选择 C++ 而非 Rust，有实际的历史原因。团队成员大多来自上古时代，对 Rust 的熟悉程度有限，C++ 在性能和表达力上完全满足需求，因此最终确定使用 C++ 进行控制面的程序设计。Runtime 保留 Python，则是因为 Python 生态在深度学习领域已经非常成熟，PyTorch、CUDA 等库的集成度高，模型适配层的开发效率远高于其他语言。Kernel 子包独立出来后，NVIDIA 和 AMD 的专家可以针对各自硬件特性做深度优化，而不需要理解调度器或 Runtime 的内部实现。
这种分层架构带来的另一个收益是并行编译的自动化。在过去的推理引擎开发中，模型并行切分的代码往往散落在各个模型适配模块内部，与注意力实现、MOE 实现、通信逻辑等深度耦合。随着支持的模型数量增加，这些散落的并行代码变得越来越难以维护，任何一个新模型的接入都可能牵动多个模块的改动。TokenSpeed 的做法是将并行切分逻辑从模型适配代码中完全抽离出来，形成一个独立的 Placement 系统。模型开发者只需要声明每个模块的边界、输入输出类型，以及该模块在并行维度的属性，是 replicate、shard 还是 partial，系统就会自动处理后续的切分和通信。
具体来说，每个模块可以挂载到不同的并行组上。注意力模块挂载到 Attention Tensor Parallel 组上，MOE 模块挂载到 MOE 并行组上，各模块独立选择并行策略。基于这些 Placement 定义，TokenSpeed 的类型编译系统会遍历整个解码层的执行计划，比较相邻模块之间的 Placement 差异。当发现不一致时，系统自动插入相应的通信原语：从 partial 到 replicate 需要 all-reduce，从 shard 到 replicate 需要 all-gather，从 shard 到 partial 需要 reduce-scatter。整个编译流程的输出是一个统一的 compiled_decoder_layer 执行计划，在运行时分为三个阶段接入：prefill 阶段做前置通信对齐，然后执行模型算子，最后执行通信输出。这种自动化的并行编译不仅大幅降低了模型适配的维护成本，还因为类型系统提供了前置信息，使得相邻算子之间的 fusion 成为可能，减少了 kernel launch 的开销，在性能上也有正向收益。
控制面的核心是一个覆盖单个请求全生命周期的状态机。一个推理请求从进入到完成，不可避免地要经历几个关键状态：submitted、prefilling、decoding、retracting/timeout、finished。每个状态本身是一个资源容器，持有该状态所需的 KV 页引用、SSM/MLA 的 slot 列表等资源。当状态发生迁移时，这些资源的持有权同步传递给下一个状态。整个控制面就是一个持续流转的状态机系统。
状态迁移通过事件驱动机制完成。每个事件承载着对应的 C++ operator 操作符，在状态转移过程中，资源的所有权通过 C++ 的 unique_ptr move 语义进行传递。这种设计的巧妙之处在于将运行时的资源管理问题提前到了编译期。当代码编译通过时，编译器实际上已经验证了资源泄漏和资源分配错误的问题。因为所有的资源转移路径都在类型系统中被显式地定义和检查。资源本身采用 RAII（Resource Acquisition Is Initialization）方式管理，无论是 Cache Block 的引用计数，还是 Device Node 或 Host Side 的引用计数，在数据搬移或操作完成之前，对应的资源都不会被提前释放。
控制面的状态机设计与执行面完全解耦。调度器只需要关注何时将请求从 prefilling 状态推进到 decoding 状态、何时需要触发 retracting，而不需要关心底层 Kernel 如何执行具体的计算。这种解耦使得调度策略可以独立迭代，不必因为底层算子的调整而频繁修改。Runtime 层接收来自控制面的执行计划，按照计划中定义的顺序和资源配置执行计算，执行完毕后将结果返回给控制面，驱动状态机继续流转。
在实际工程中，这种状态机设计显著降低了并发请求调度中的竞态和资源泄漏问题。过去在 Python 实现中常见的资源管理 bug，比如某个请求被取消后 KV 页未能及时回收、或者引用计数在异常路径下未被正确递减，在 C++ 实现中通过编译期检查和 RAII 机制得到了系统性的解决。这也是我们将控制面完全用 C++ 重写的核心动机之一。
KV Cache 管理是 TokenSpeed 架构设计中投入精力最大的模块之一。Agent 场景下的前缀复用对缓存系统的要求远高于传统 Chat 场景。同一个 Agent 请求在一次完整的工作流程中会反复携带相同的前缀上下文，如果每次循环都需要重新计算 prefill，推理成本会急剧膨胀。
我们在第一版实现中借鉴了 SGLang 的思路，使用 Radix Tree 作为前缀缓存的管理结构。Radix Tree 的设计出发点是维护一棵从根到叶的树形结构，树上的每个节点对应一段 token 序列，所有请求的前缀匹配都在这棵树上进行。这种设计在纯 attention 场景下工作良好，但当我们开始适配 Kimi K3 时发现，Radix Tree 的语义与某些新型注意力模式存在冲突。Mamba 类模型和 Sliding Window Attention 天然地破坏了 Radix Tree 所依赖的层级匹配假设，它们的缓存访问模式不是简单的从前缀到后缀的线性延伸。
这一经验促使我们重构了 KV Cache 管理系统。在支持 Kimi K3 的过程中，我们将原本的 Radix Tree 全部迁移到 Flat KV 设计。新的缓存系统不再感知缓存的注意力类型，只要求上层将不同类型的缓存按照统一的格式上报给系统，声明各自的类型和占用空间。底层将所有缓存统一打包成相同尺寸的 block，做平面的存储管理。K3 第一版使用逻辑 block 和物理 block 相等的方式，以 1536 个 token 为一个 block 进行管理。但这个粒度在长上下文场景下导致前缀复用效率和页内碎片问题都比较突出。后续我们借鉴了 vLLM 的做法，引入两层 block 设计：第一层是逻辑块，块大小统一为 128；第二层是物理块，块大小为 1536。128 大小的逻辑块用于前缀复用和更细粒度的缓存匹配，物理层则保持不变。这种双层设计在后来的测试中取得了不错的优化效果。
缓存抢占与恢复是另一个关键能力。TokenSpeed 目前完全接入 Mooncake 的 KV Store 系统。当显存不足时，请求的整个 KV 状态被卸载到 KV Store 中做持久化，待请求重新被调度回来时，通过 lazy 化的 reload 机制重新加载并继续计算。PD 分离（Prefill 与 Decode 分离）方面，我们最初实现的是逐层传输的方式。但逐层传输会导致 RDMA 的带宽利用率非常低，因为每一层的传输都需要独立的网络往返开销。后续我们做了优化，允许用户指定将若干层打包成一个 block 进行传输，即一种可定制的分层传输策略。这种灵活性使得用户可以根据具体的网络拓扑和模型规模来调节传输粒度。
Kernel：让 MLA 真正跑满 GPU
Kernel 层的优化工作在 TokenSpeed 中主要由团队中的 NVIDIA 和 AMD 专家推动。目前 TokenSpeed 的 Kernel 在 Blackwell 上已经达到 state-of-the-art 的性能水平，并且被 vLLM 社区集成为 Blackwell 上的默认 Kernel 包。
MLA（Multi-head Latent Attention）是当前国内开源模型广泛采用的注意力机制。MLA 虽然通过将 KV 压缩为单一的 latent 状态来节省写成本，但它的计算模式对硬件并不天然友好。在 decode 阶段，query 与 latent key 的第一次矩阵乘法中，query 的维度是 1，整个矩阵乘法实际上退化成了向量乘法。这意味着 Tensor Core 的计算资源在 Blackwell 上被严重浪费。第二个瓶颈在于带宽。Agent 场景下前缀长度常常达到几十 K 甚至几百 K，即便使用了 INT8 量化，带宽数据加载仍然是限制因素。第三个问题是形状不匹配：MLA 的 head 数和训练长度组合往往导致矩阵乘法呈现瘦高形态，对 SM 的利用率和 launch 开销都产生负面影响。
针对这些问题，我们在 Blackwell 上重新设计了 Kernel。首先，充分利用 Tensor Memory 这一新引入的片上内存资源。我们专门编写了 Tensor Memory 的分配器，让 Tensor Core 直接读写 Tensor Memory，避免了通过寄存器进行中间数据搬移的开销。其次，在矩阵乘法层面引入了一个折叠因子（fold factor）的优化技术。由于 decode 阶段 query 维度为 1，Tensor Core 利用率极低。我们发现当 head 数小于 128 时，可以将 query token 折叠到 head 维度中。经过折叠后，有效的 head 维度乘以折叠因子，矩阵维度变高，Tensor Core 便可以进行更高效的计算。此外，我们还设计了 Split KV 机制，将计算拆分为两个阶段：第一阶段计算 Split KV 的部分和，第二阶段做归约操作。Split 参数和 workspace 的大小在运行时自动计算并缓存。跨 SM 的协同执行减少了 MMA barrier 的开销，数据写回时也做了带宽友好的优化。这些优化组合在一起，使得 TokenSpeed 的 Kernel 在 Blackwell 上达到了 leading 水平的性能。
投机解码是当前提升推理吞吐的重要特性。EAGLE、DeepSeek 等算法的出现都带来了显著的吞吐提升。TokenSpeed 的仓库目前已经支持三种投机解码算法：MTP、EAGLE3 和 DeepSeek。同时我们也在使用 TokenSpeed 去支持 DeepSeek 发布的推理引擎中提出的投机解码算法，后续会为开源模型专门训练一批对应的投机模型权重，放入 TokenSpeed 仓库中供社区使用。在 Inkling 975B 这个开源模型上，我们在四张 B200 上做了测试：MTP 步数为 3 时每步 TPS 为 317，步数为 8 时为 355，相对于关闭 MTP 的基线，整体性能提升了 2.08 到 2.33 倍。在千问 3.5 上，MTP 带来的性能收益在 100% 到 159% 之间。
对于线性注意力模型，TokenSpeed 也做了针对性优化。线性模型中 MLA 状态随 token 的生成不断向前更新，当达到某个截止时间点时需要将状态写回，这个写回操作会带来一定的开销。我们的做法不是搬移数据本身，而是操作指针，将数据搬移的开销降到最低。每一步的状态直接写入该请求的私有物理行中，用一个索引标记状态所在的位置。当需要截止时，只更新索引指向，而不是搬移整个状态块。在 prefill 场景中，Agent 负载的前缀命中率通常在 90% 以上，此时只需要追加几百个 token 的计算。这意味着 request 的 shape 是变长的，Q 和 KV 的长度也不相等。TokenSpeed 的 MLA Kernel 支持无 padding 的变长批处理，默认实现使用 QDLS 实现，性能表现良好，同时也提供了预编译的 binary 通过 TMA 接入。
跨平台 Kernel 设计
随着推理硬件的多样化，跨平台能力成为推理引擎必须面对的问题。NVIDIA、AMD、华为、平头哥等厂商的加速器各有不同的硬件架构和编程模型。如果 Kernel 代码直接绑定在特定的硬件接口上，每支持一种新硬件都需要大规模重构。TokenSpeed 在设计 Kernel 层时参考了 Linux 内核的抽象思路，建立了 register 和 selector 的机制。
模型适配代码完全不感知底层运行的具体硬件接口。它只调用公共的 AI API 或者公共的 prefix API。这些 API 在设计时经过仔细定义，确保后续不轻易变动。API 的描述类型涵盖张量格式、执行约束等必要参数。底层厂商在实现各自的 Kernel 时，通过 register 机制将自己的内核注册到系统中，声明所支持的操作类型、capability、signature、优先级以及特定的 Tensor Core 特性。运行时系统通过 select kernel 动态地根据平台能力和格式进行过滤，选择最合适且最稳定的 Kernel 完成模型的前向计算过程。
这种设计为不同阶段的开发者提供了分层的接入路径。新模型接入时，可以先用 reference 代码（例如用 PyTorch 编写的实现）把整个模型跑通，建立正确性基线。然后再将 reference 代码替换为 specialized 的定制 Kernel，逐步提升性能。优先级分层机制允许同时存在 reference 实现、performance 实现和 specialized plugin 实现，系统根据注册时的优先级自动选择。TokenSpeed 的 Kernel 包有独立的 AMD Build，AMD 团队基于这套系统完成了对 AMD GPU 的独立适配。
在 AMD 的 MI355X 机器上，使用其 CDNA4 架构的 DSL 能力，开发者可以精细管控流水线和内存布局。在注意力层面，采用 persistent prefill Kernel 实现跨 CDA 的均衡调度。测试结果显示，在 15 种典型的 prefill 形状中有 14 种快于 TensorRT，领先幅度为 1.4 到 2.3 倍；在 MOE 的小 batch rop decode 场景中，比 TensorRT 快 1.7 到 2.1 倍。端到端测试在所有测量点上均实现性能优势。
工程实践与性能收益
经过几个月的迭代，TokenSpeed 在 Blackwell 平台上取得了一系列可量化的性能成果。在发布版本中，我们以 TensorRT-LLM 作为对比基准。TensorRT-LLM 作为 NVIDIA 官方的推理引擎，在 Blackwell 上本身已经具备相当高的优化水平。测试环境基于单机 Blackwell、非 PD 分离架构、使用 EvalScope 这个一键 benchmark 进行流量测评。结果表明，在延迟约束下的吞吐优化这个核心维度上，TokenSpeed 在所有测试场景中都不弱于 TensorRT-LLM。在 batch size 等于 1 的最小延迟场景下快 9%，在每用户 100 TPS 的工作点上每 GPU 吞吐提升 11%。在千问 3.5 397B 模型上，B200 八卡、batch size 等于 1 的配置下跑到了 580 TPS，这在当时是一个比较突出的水平。
MLA Kernel 的优化效果同样显著。在五种典型的 coding agent prefill 负载上，TokenSpeed 全部优于 TensorRT-LLM。在投机解码的 batch size 等于 1 场景下，延迟相对于 TensorRT-LLM 几乎减半。跨模型支持方面，千问 3.5 从 128K 上下文扩展到 1M 时，解码吞吐仅下降 16%，展示了较强的扩展性。在 Kimi 2.5 上，每用户 TPS 达到 423。Kimi K3 当时处于 Day-0 支持状态，优化工作仍在持续迭代中。
回顾整个项目，可以提炼出几条核心的设计原则。第一，在 Agent 场景下，推理引擎的优化目标必须从单纯追求吞吐转向延迟约束下的吞吐最大化，以 Pareto 前沿作为评价推理引擎性能的核心框架。第二，系统分层必须清晰，控制面只负责调度决策，不触碰数据面；Runtime 只负责执行；Kernel 只负责计算。第三，并行切分逻辑必须从模型适配链路中抽离出来，通过 Placement 系统和自动并行编译实现声明式的模型部署。第四，Kernel 优化必须充分发掘新硬件特性，同时通过抽象层为多硬件生态提供友好的接入路径。TokenSpeed 作为一个开源数月的年轻项目，仍在快速迭代之中，我们期待社区的反馈和贡献。
热度延续：12 月 AICon 北京站见
12 月 18–19 日，AICon 全球人工智能开发与应用大会将在北京举办。本届大会以探索 A 应用边界为主题，设置 13 大技术专题，覆盖大模型后训练、推理优化、Agent 工程、知识工程、智能体评测与安全、自进化系统、AI 软件交付、原生架构及组织变革等核心方向，汇聚一线技术专家，深入分享 AI 技术突破与企业规模化落地的真实实践。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Against the backdrop of the rapid growth of the Agent reasoning load, the traditional reasoning engine for chat scenes is facing a series of structural challenges, such as the failure of the optimized goal of swallowing, the underutilization of cache and the lag in hardware.
This post is based on the sharing of the TokenSpeed Independent contributor, Sang Xuchun, at the AICon 2026 Shenzhen Station, Building the Next Generation LLM Logic Engine for Agenic reasoning. His system described the complete technical path of the project from the creation of the item to the version of the version, and the work load features around the Agent scene, the architecture of the system for layering, the automated parallel compilation system, the KV Cache evolution route, the Kernel optimization strategy and the cross-platform abstract design, and gave phase performance data on the Blackwell platform.
The following is a transcript of the presentation (a non-changed editorial by InfoQ).
Why, Agentic, the reasoning needs a new reasoning engine.
At the beginning of the TokenSpeed project, the first question we have to answer is: Why does the Agent scene need a completely new engine of reasoning?
Traditional Chat loads have relatively single optimization objectives, with one request from the user and one full response from the model. In this model, the reasoning system simply needs to fill the bat as much as possible and maximize the vomiting, and KV Cache is released after a load, with a natural low utilization. But the load features of the Agent scene are fundamentally different from Chat. For example, the model needs to think on a continuous basis, call tools, capture results, and continue to think on the basis of results. The process consists of multiple cycles of short requests. The first request may carry dozens of K prefixes context, but there are often only a few hundred tokens for each subsequent cycle. This means that the reasoning engine has to revisit the cache reuse strategy, the requested dispatch mechanism and the system-wide optimization target function.
Under this load pattern, the user ' s sensitivity to delays is significantly increased. If an Agent tool lags significantly behind the reasoning chain, users can visualize a “handy” deficiency. Therefore, what really needs to be optimized in the Agent scenario is throughput under delayed constraints. We have an internal definition of performance indicator: no less than 70 per user of TPS, which increases the capacity of each GPU as much as possible. In comparison with TensorRT-LLLM, TokenSpeed is 9% faster than TensorRT-LLLM in the minimum delay scenario of catch size equal to 1; at typical work point per 100 TPS, each GPU was 11% higher.
The KV Cache reuse rate is the key variable determining the cost of reasoning. In DeepSeek, for example, the reuse rate for Z96Z is said to be over 99 per cent, while most industry systems currently reach between 70 and 80 per cent. In the Agent scenario, the total cost of the system would be significantly reduced if dozens of K-sized contexts were hit repeatedly, meaning that each hit could be completely saved from prefilling. We tested TokenSpeed's performance when context length expanded from 128K to 1M, and decoded throughput rates dropped by only 16%. In addition, KV Cache ' s management model is changing as more and more models use linear attention mechanisms, including the thousands of questions from Gated DeltaNet and Kimi K3. Linear Attention Models no longer maintain a traditional token-KV correspondence, but represent context, including SSM status and volume status, through a continuous iterative state. The efficient updating, preservation and restoration of these states in the reasoning engine is a new system design proposition.
The interface between model architecture and hardware characteristics is also deepening simultaneously. The MOE architecture has become a mainstream option since DeepSeek publish, but the MoE model is not less difficult to optimize. In the case of Kimi K3, for example, the model has 896 experts, although the use of the Latin MOE technology has significantly reduced writing costs and increased the activation of experts to 16 per cent, the complexity of route and communications has increased significantly. At the same time, Blackwell introduced a range of new features, such as Tensor Memoory, a new generation of Tensor Core, and the AMD's CD DNA architecture has its unique hardware capability. If the reasoning engine does not take full advantage of these hardware features, Blackwell ' s relative Hopper performance advantage cannot be truly released. On the basis of these observations, TokenSpeed has made an in-depth optimization of Blackwell at the Kernel level to ensure that the hardware potential is fully exploited.
Runtme and Kernel's co-design can't be ignored. The scheduler was designed so well that if the bottom Kernel had performance bottlenecks or too deep a coupling, the system as a whole would still not be in the desired state; in turn, Kernel was perfected, and could not be filled if the dispatcher produced a large amount of bubble. TokenSpeed drew on this experience in the design of the architecture by systematically decoupling and reconfiguring movement control, execution and bottom algorithms.
Runtme: Build a scalable engine structure
TokenSpeed's overall structure is divided into three levels: control, Runtume and Kernel. The control face is fully written using C++, Runtme is performed using Python and Kernel is independent of a sub-pack. The core purpose of this stratification is to maintain the clarity of the modular boundary. Control is responsible only for movement decision-making and does not involve the execution of any data surface; Runtme is responsible only for the implementation of the plan for the release of control, including the cache plan and forward calculation plan; Kernel floor is focused on the realization of algorithms. The responsibilities of the three parties have avoided the maintenance dilemma caused by the logic of movement, the logic of execution and the mutual penetration of algorithms in the previous reasoning engine.
The control side selection C++ instead of Rust has real historical reasons. The majority of team members come from the ancient era, have limited familiarity with Rust, and C++ fully meets needs in terms of performance and expression, so it is finally determined that C++ will be used to design the control face. Runtime retains Python because the Python ecology is very mature in the field of in-depth learning, and the libraries such as PyTorch and CUDA are highly integrated and modelable layers are developed much more efficiently than other languages. When the Kernel sub-pack is independent, NVIDIA and AMD experts can optimise their hardware characteristics in depth without understanding the internal realization of the scheduler or Runtme.
Another benefit of this layer structure is the automation of parallel compilations. In the past, in the development of the reasoning engine, the code of parallel severing models tends to be scattered within the model-adaptation modules, in depths such as focus realization, MOE realization, communication logic, etc. As the number of supported models increases, these scattered parallel codes become more and more difficult to maintain, and access to any new model can trigger multiple module changes. TokenSpeed's approach is to pull the parallel split logic out of the model fit code and form an independent Placement system. Model developers need only state the boundaries of each module, the type of input output, and the properties of the module in parallel dimensions, whether replicate, shard or partial, and the system automatically handles subsequent splits and communications.
Specifically, each module can be mounted on different parallel groups. The attention module is mounted on the Attension Tensor Parallel group, the MOE module is mounted on the MOE parallel group, and each module independently selects a parallel policy. Based on these Placement definitions, the TokenSpeed type compiler system runs through the entire decoding layer of the implementation plan, comparing the Placement differences between adjacent modules. When inconsistencies are found, the system automatically inserts the corresponding original communication language: all-reduce, all-gather, from party to replicate. The output of the entire compilation process is a unified command decoder layer implementation plan, which is operated in three phases: pre-communication alignment in the prefill phase, then model algorithms, and finally communication output. This automated parallel compilation not only significantly reduces the maintenance costs of the suitability of the model, but also because the type of system provides pre-positioned information that makes it possible to create a fusion between adjacent algorithms, which reduces the cost of Kernel launch and has positive performance benefits.
The core of the control surface is a state machine that covers the entire life cycle of individual requests. From entry to completion, a request for reasoning inevitably goes through several critical states: submitted, prefilling, decoding, retracing/timeout, finished. Each state is itself a resource container that holds resources such as the KV page reference for that state and the slot list for SSM/MLA. The ownership of these resources is transmitted simultaneously to the next state when the state moves. The entire control side is a continuous flowing state system.
The migration is done through the event-driven mechanism. Each event carries the corresponding C++ operator, and during the state transfer the ownership of the resource is passed through the unique ptr move syntax of C++. The cleverness of this design is to advance the management of running resources to the translation period. When the code was compiled, the compiler had in fact validated the issue of leakage and misallocation of resources. Because all resource transfer paths are clearly defined and checked in the type system. The resource itself is managed in a RAII (Resource Assessment Is Internationalization) manner, with neither Cache Block ' s citation nor Divice Node or Host Side ' s citation count being released in advance until data are moved or operated.
The control side's status machine design is completely decoupled from the executive side. The scheduler only needs to focus on when to move the request from prefilling to decoding, when to trigger retracing, and does not need to look at how the bottom Kernel performs the specific calculation. This decomposition allows the movement strategy to be independently iterative and does not require frequent modifications due to the adjustment of the base algorithm. The Runtme layer receives an implementation plan from the control side, which is calculated in the order defined in the plan and the resource allocation, returns the results to the control side after implementation and the driver continues to flow.
In the actual works, the design of this state machine has significantly reduced the competitive and resource leakage issues in the movement of requests. Resource management bugs that are common to past Python implementations, such as the failure to recover a KV page in a timely manner after a request has been cancelled, or the failure to correctly reduce the citation count under an abnormal path, are systematically addressed in C++ implementation through the compilation period check and RAII mechanism. And that's one of the core motivations that we're going to rewrite the control completely with C++.
KV Cache management is one of the most energy-intensive modules in the design of the TokenSpeed architecture. The pre-fix re-use requirements for the cache system in the Agent scene are much higher than the traditional Chat scene. The same Agent requests that the same prefix context be carried over and over again during a full workflow process, and if recalculation is required for each cycle, the reasoning costs will expand sharply.
And in the first edition, we used the idea of SSGLang, using Radix Tree as the management structure for the prefix cache. Radix Tree is designed to maintain a tree structure from root to leaf, each node of the tree corresponds to a token sequence on which all requested prefix matches are made. This design works well in a pure attention scenario, but when we start to fit Kimi K3, it turns out that the syntax of Radix Tree is in conflict with some of the new forms of attention. Mamba-type models and Sliding Windows Act naturally destroy the hierarchy matching assumption on which Radix Tree relies, and their cache access mode is not simply a linear extension of the pre-suffled suffix.
This experience has led to the re-engineering of the KV Cache management system. In support of Kimi K3, we migrate the original Radix Tree to the Flat KV design. The new cache system is no longer aware of the type of attention given to cache, but only requires that the upper layer report the different types of cache to the system in a uniform format, stating their respective types and occupancy space. On the ground floor, all caches are packaged in a single block of the same size for storage management of the plane. The first version of K3 is managed by 1536 token for a block using logical and physical block equivalents. However, this particle mass is more pronounced in the long context scene, leading to pre-suffocative efficiency and the problem of internal debris. Following up on the v LLM approach, we introduced two layers of block design: the first level is a logical block with a single size of 128; the second level is a physical block with a block size of 1536. The size of the logic block is used to match cache for prefixing and more finer grains, while the physical layer remains unchanged. This two-tiered design achieved good optimization in subsequent tests.
The seizure and recovery of cache is another key capability. TokenSpeed is now fully connected to the Mooncake KV Store system. When VRAM is insufficient, the entire KV status of the request is unloaded into KV Store to be sustained and reloaded and continued to calculate through the lazy-modified reload mechanism when the request is rerouted. In terms of PD separation, what we started with was a layer by layer transfer. However, layer-by-layer transmission will result in very low bandwidth utilization of RDMA, as each layer of transmission requires independent network round-trip costs. We then optimized it by allowing users to assign several layers to be packaged into a block for transmission, a custom tiered transfer strategy. This flexibility allows users to adjust the transfer particle size to specific network sizes and model sizes.
Kernel: Make MLA really full of GPU
The optimization of the Kernel layer is largely driven by NVIDIA and AMD experts in the team in TokenSpeed. Kernel of TokenSpeed currently has state-of-the-art performance levels on Blackwell and is used as the default Kernel package on Blackwell by the vLM community collection.
MLA (Multi-head Late Action) is the focus mechanism widely used in the current open-source model. MLA, while saving writing costs by compressing KV into a single latent state, does not have a naturally friendly calculation pattern for hardware. In the decode phase, the first array multiplication between query and the last key, the dimension of query is one, and the entire matrix multiplication actually degenerates into a vector multiplication. This means that Tensor Core's computing resources are severely wasted on Blackwell. The second bottleneck is bandwidth. Agent often has several dozen Ks or even hundreds of Ks in front of the scene, and even using INT8 quantization, bandwidth data loading remains a constraint. The third problem is the mismatch of shapes: the combination of MLA ' s head numbers and length of training tends to lead to a lean matrix multiplication that negatively affects both SM utilization and launch spending.
In response to these problems, we redesigned Kernel on Blackwell. First, make full use of Tensor Memoory's newly introduced film memory resources. We created the Tensor Memoory's dispenser, which allowed Tensor Core to read and write directly, and avoided the cost of moving intermediate data through the register. Second, an optimisation technique has been introduced at the matrix multiplication level. The use of Tensor Core is extremely low owing to the decode phase query dimension 1. We found that when the number of head is less than 128, you can fold Query token into the head dimension. After folding, the valid head dimension multiplied by the folding factor, the matrix dimension became higher, and Tensor Core was able to calculate more efficiently. In addition, we have designed the Split KV mechanism to calculate split in two phases: phase I to calculate the sum of Split KV and phase II to contract. The size of the Split parameter and the workspace is automatically calculated on running time and cache. Coordinated implementation across SM reduced MMA barrier costs and the data was optimized with bandwidth. These optimized combinations enabled TokenSpeed's Kernel to achieve a level of performance on Blackwell.
Speculation decodes are an important feature of increasing reasoning today. The emergence of algorithms such as EAGLE and DeepSeek has led to a significant increase in throughput. TokenSpeed's warehouse now supports three speculative decoded algorithms: MTP, EAGLE3 and DeepSeek. At the same time, we are using TokenSpeed to support the speculative decoding algorithms proposed in the DeepSeek publish reasoning engine, and the follow-up session will be dedicated to training a set of corresponding speculative model weights for open source models, to be placed in TokenSpeed warehouses for community use. In Inkling 975B, this open source model, we tested on four B200s: the MTP step was 317 at 3 and the step was 355 at 8; the overall performance was 2.08 to 2.33 times higher than the baseline for closing MTP. On a thousand questions 3.5, MTP brings performance returns of between 100% and 159%.
TokenSpeed also has targeted optimization for linear attention models. The MLA status in a linear model is constantly updated with the generation of token, and needs to be written back when a certain cut-off point is reached. This write-back will entail a certain cost. Our approach is not to move the data per se, but to operate the compass to minimize the cost of moving the data. The state of each step is written directly into the private physics line of the request and the location of the state is marked with an index. When the deadline is required, the index is only updated and not moved to the whole status block. In the prefill scenario, the prefix rate for the Agent load is usually over 90%, at which point it will only require several hundred additional token calculations. This means that the share is longer, and the length of Q and KV is not equal. TokenSpeed 's MLA Kernel supports long batch processing without padding, defaults the use of QDLS, performances are good and provides pre-compiled binary access via TMA.
Cross Platform Kernel Design
With the diversification of reasoning hardware, cross-platform capability becomes a problem for the reasoning engine. There are different hardware structures and programming models for NVIDIA, AMD, Huacheng and Hiranco accelerators. If the Kernel code is attached directly to a particular hardware interface, a large-scale re-engineering is required for each new hardware supported. TokenSpeed created a mechanism for register and selfer by drawing on the abstract idea of the Linux kernel when designing Kernel.
The model appliance code is completely unsure of the specific hardware interface that runs at the bottom. It only calls public AI API or public prefix API. These APIs are carefully defined in design to ensure that follow-up does not change easily. The description type for API covers the necessary parameters such as the format of the scale, and the binding of implementation. In achieving their own Kernel, lower-end manufacturers register their kernels in the system through the register mechanism, declaring the type of operation they support, capacity, priority, and specific Tensor Core characteristics. The running-time system is dynamically filtered through the platform 's capabilities and formats to select the most appropriate and stable Kernel to complete the precursor calculation of the model.
This design provides a layered access path for developers at different stages. When the new model is accessed, the entire model can be run through with a reference code (e.g. by PyTorch) to establish a correct baseline. And then replace the reference code with the customized Kernel, which will gradually enhance the performance. The priority hierarchy allows for the simultaneous availability of reference, performance and specialized plugin, with the system automatically selected according to the priority at the time of registration. TokenSpeed's Kernel package has an independent AMD Build, and the AMD team has completed an independent fit for AMD GPU based on this system.
On AMD's MI355X machine, using its DSL capability for the CDNA4 architecture, developers can finely control current lines and memory layouts. At the level of attention, the balanced movement across the CDA is achieved by using the Persistent prefill Kernel. The test results show that 14 of the 15 typical prefill shapes are faster than TensorRT, with a lead range of 1.4 to 2.3 times; in MoE ' s small watch rop decode scenario, 1.7 to 2.1 times faster than TensorRT. End-to-end tests achieve performance advantages at all measurement points.
Engineering practices and performance gains
Over the course of several months, TokenSpeed achieved a series of performances on the Blackwell platform that could be quantization. In publish, we use TensorRT-LLLM as a benchmark for comparison. TensorRT-LLLM, the official reasoning engine of NVIDIA, has itself achieved a high level of optimization on Blackwell. The test environment is based on single machine Blackwell, non-PD separation structure, and traffic measurement using EvalScope, a key benchmark. The results show that TokenSpeed is not weaker than TensorRT-LLLM in all test scenarios for optimizing the core dimensions of the paralysing under the delayed constraint. Faster than 9 per cent in the minimum delay scenario of a watch size equal to 1 and higher than 11 per cent per GPU per 100 TPS work point. On the thousands question 3.5 397B model, B200 8-card card, Batch size ran down to 580 TPS, a relatively prominent level at the time.
The MLA Kernel optimizes are equally significant. TokenSpeed is all better than TensorRT-LLM on five typical coding agent load. The delay is almost halved relative to TensorRT-LLLM in a scenario where the catch size is equal to 1. On cross-model support, 3.5% of the requests were extended from 128K context to 1M, decoded throughputs fell by only 16%, showing greater expansiveness. On Kimi 2.5, the TPS per user reached 423. Kimi K3 was in Day-0 support at the time, and optimization was still ongoing.
Looking back at the project as a whole, several core design principles could be distilled. First, in the Agent scenario, the optimal objective of the reasoning engine must be to maximize it from mere pursuit of ingestion to a delayed constraint, using the Paretto front as the central framework for evaluating the performance of the reasoning engine. Second, the stratification of the system must be clear, and the control side must be responsible only for movement control decisions without touching the data surface; Runtme must be responsible only for implementation; Kernel must be responsible only for computing. Thirdly, the parallel split logic must be removed from the model adaptor chain to achieve the deployment of the declaration model through the Placement system and automatic parallel compilation. Fourth, Kernel optimization must fully exploit the new hardware characteristics, while providing friendly access paths to multi-hardware ecology through abstract layers. TokenSpeed, as an open-source youth project for several months, is still in rapid succession, and we look forward to community feedback and contributions.
Thermal continuity: December AICon Beijing station
On December 18-19, the Aicon Global Congress on Artificial Intelligence Development and Applications will be held in Beijing. The Congress, which focused on exploring the A application boundary, set up 13 major technical topics covering the core directions of LLM post-training, reasoning optimization, Agent engineering, intellectual engineering, AI agent assessment and security, self-evolution systems, AI software delivery, native architecture and organizational change, bringing together first-line technical experts to share in depth the real practices of AI technological breakthroughs and enterprise size.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.