OCT. 20, 2026 (英语).
作为我们今天对PyTorch的更广泛宣布的一部分，我们对发布本地PyTorch的VLLM(与Inferact合作开发)和SGLang(与RadixArk合作开发)的后端感到兴奋。
多年来，TPU以高性能和效率为大型AI工作量提供动力。 随着代理和多式联运工作量的增加，高效服务比以往任何时候都更加重要。 这两个引擎已经通过基于JAX的后端运行在TPU上，但我们不断从开发者那里听到他们想要自定义和调试他们在PyTorch的服务堆栈。
因此，我们认真开始使用火炬TPU，它运行PyTorch 本地在TPU。 然后，我们建造了新的VLLM和SGLang引擎在上面。
我们的目标是PyTorch社区将模型带入TPU，并快速使用工具开发者已经熟悉的工具开发者。 发动机带有优化的，生产准备好的服务功能和定制的帕拉斯内核。
规模化服务需要跨越技术堆栈的每一层共同设计，从硅到编译器到管弦乐。 今天新推论引擎的发布处于一个集成的开放的TPU生态系统的中心部分。
火炬TPU使TPU成为本土的PyTorch设备。 tensors作为标准PyTorch tenors在tpu设备上管理，没有包装器类型，代码急切运行或通过火炬编译。 使用 XLA 编译，集体通过 jill.dispended.
因为每个设备都运行着自己的PyTorch进程，所以两个引擎都保留着每个框架的工人模型，调度器，KV缓存管理器，API服务器等，下面都有TPU的实现。
大多数模型在两种引擎中都采用标准上游PyTorch模型定义，在内核和层执行中都有TPU特有的优化。 在Pallas中写入的自定义内核直接从PyTorch调用，作为编译图中的自定义运算符。 这种平台-插件设计在让TPU优化以自己的速度移动的同时，尽可能多地重复使用每个框架。
这两台发动机都是为各种生产工作量而建造的。
以下是支持的特性列表：
- Agentic & Multi-Turn：自动前缀缓存，包括用于混合型号；KV缓存卸载以主机内存，并通过TPU Sync；工具调用和结构化输出管理Lustre
- 互动和低延迟：与MTP、EAGLE-3、DFlash、DSpark和n-g起草器进行分光解码
- 高通量批量：与专家平行性并行注意数据
- 长上下文：用于预填和解码的上下文并行；FP8 KV缓存；主机 KV 卸载
- 分类服务：在 TPU Sync 上，包括在不同的平行布局之间，对单独的主机进行预填和解码
- 多式联运：带有图像和视频输入的远景语言模型；嵌入模型
- 非常大型的模型：混合式和稀疏式结构、抗热器、数据、专家、管道和不同东道主的背景平行；FP8、FP4(MXFP4、NVFP4)、INT8和INT4检查站
代理工作量尤其需要同时具备其中的若干特点。 因为agents反复重播扩展多回合对话，为它们高效服务需要找到缓存前缀而不是重编。 两个引擎都在HBM中保持可重复使用的前缀，随着主机卸载的开启，将旧的前缀自动移动到主机内存作为第二层。
其中两个特性将 KV 缓存从计算它的设备中移走。 分类服务在单独的TPU主机上运行预填和解码，因此每个请求的KV缓存必须从预填主机前往解码主机。 分级的KV卸载将KV缓存中不再适合HBM的部分移动到主机的DRAM和二次存储中，使用Google Cloud管理Lustre，带来在后来请求重用前缀时返回。 无论哪种方式，转移都必须与计算重叠，或者它新增直接到延迟。
两个引擎都为此使用TPU同步。 它是用于TPU上的KV缓存传输和管理的开源 C++库。 它直接读取PyTorch 百分位后的设备缓冲器，因此在主机上不需要额外的中继副本，它在Python外的自身线程上运行传输，同时预填和解码继续。 vLLM-TorchTPU通过vLLM的KV连接器接口将其曝光。 SGLang-TorchTPU将其用作分解传输后端和HiCache的I/O后端。
两台发动机在生产中已经以预览客户的规模运行。 为了为我们的表现提供透明、可复制的证据，我们基准测试通过社区标准半分析推论项目公开。
我国工程团队在铁木公司TPU(TPU 7x)上首次提交Qwen 3.5的基础上，进一步优化了堆栈，使整个百万令牌成本最低，行业领先2.01分。 除了这些改进之外，我们还向AgentX公布了新的最先进(SOTA)绩效结果，这是该项目用于评估Agent长篇和多回合工作量的专用轨道。 铁杉TPU领先于帕雷托边境的大部分地区。 我们期待着在给领导板带来更多模型和配置时不断更新我们的公开业绩简介。
我们没有在真空中建立这些后端。 通过拥抱PyTorch并直接与开源项目维护者合作，我们确保这些引擎保持上游对齐和社区第一：
这种伙伴关系正在使SGLang的星座团队为TPU建设，提供一台已经在Meta生产中运行的发动机，并在要求很高的现实世界情景中进行规模的战斗测试。
" SGLang-TorchTPU在今天的Meta进行生产运行，获得Meta,Google，以及SGLang团队之间的紧密三向协作，使堆积生产做好了准备。 它给了我们一个非常成熟的PyTorch本地服务堆，将无心地整合到我们现有的AI基础设施中，具有性能、可靠性和灵活性，" 称宗韦周，PyTorch-TPU领先于Meta.
"在RadixArk，火炬TPU在我们推出各种硅的SGLang支持时，超速加载了我们的开发速度。 称 Banghua Zhu, CTO和RadixArk的共同创始人。
我们一起整合了2.8T参数Kimi K3模型的高性能Pallas巨核，将整个解码步骤作为一个内核运行，每秒为铁木头TPU上的一个单一用户服务700多个令牌。 在这项工作的基础上，Inferact最近领导了我们第一次在vLLM-TorchTPU堆上提交Kimi K3的初步AgentX. 这个最初的里程碑已经表明，在超高互动工作量中，每个总公司的业绩都不同寻常，TPU上的 vLLM 和 PyTorch 生态系统的成熟度迅速提高，因为它采用了新的前沿开放源码软件模型，并设定了后续提交材料，通过更大的预填/解码分解解解解解解提高效率。
"我们在Inferact的任务始终是使高性能的推论在整个AI生态系统中可以被获取，可靠，成本效益高。 我们很高兴在火炬TPU的发布上合作，并将vLLM的一流支持带给TPU. 通过将 PyTorch 优化的TPU 能力直接连接到 vLLM，开发者可以立即运行 TPU 的生产工作量，高吞吐量，低潜伏度，和内存效率
为了促进持续的开源贡献，vLLM也是发布，这是它最先的社群贡献 TPU 容量池，为积极贡献 vLLM 引擎的开发者提供免费的 TPU 计算。
应用到 vLLM TPU 贡献器池
随着模型的扩大，我们正在演化两个引擎，以使大规模的TPU推论部署能发挥效能和效率。 生产级代理工作量的主要特点已经得到支持。 更多性能优化和先进功能正在进行中 – 打开Github问题，请求在您身上合作！.
在两台引擎上，大多数受欢迎的开放型号家庭都得到了支持，包括Qwen,DeepSeek,GLM,Kimi,GPT-OSS,Gemma等。 团队正在投资提供更多模式在TPU上本土运行，包括Day 0支持新的开放模式。 我们还想向社区提供工具和TPU能力，以调谐自身工作量和配置的性能。
vLLM-JAX和SGLang-JAX仍然是重要项目。 两者将继续得到维护、支持和提供，供社区使用和适应。
Github项目：
- vLLM-TorchTPU:https://github.com/vllm-project/vllm-torchtpu 互联网档案馆的存檔，存档日期2013-12-21.
- SGLang-TorchTPU:https://github.com/sgl-project/sglang-torchtpu 互联网档案馆的存檔，存档日期2014-03-02.
- 火炬TPU:https://github.com/google-pytorch/torch tpu.
有关通知：
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
OCT. 20, 2026
As part of our broader announcements for PyTorch today, we are excited to launch native PyTorch backends for vLLM (developed in partnership with Inferact) and SGLang (developed in partnership with RadixArk).
For years, TPUs have powered large-scale AI workloads with high performance and efficiency. With the rise of agentic and multimodal workloads, serving efficiently is more important than ever. Both engines already run on TPU through JAX-based backends, but we kept hearing from developers that they wanted to customize and debug their serving stack in PyTorch.
So we started in earnest on TorchTPU, which runs PyTorch natively on TPU. We then built new vLLM and SGLang engines on top of it.
Our goal is for the PyTorch community to bring models to TPU and make them fast using the tools developers are already familiar with. The engines come with optimized, production-ready serving features and custom Pallas kernels.
Serving at scale requires co-design across every layer of the technology stack, from silicon to compilers to orchestration. Today's launch of new inference engines is at the central part of an integrated, open TPU ecosystem.
TorchTPU makes the TPU a native PyTorch device. Tensors are managed as standard PyTorch tensors on a tpu device with no wrapper types, code runs eagerly or compiles via torch.compile using XLA, and collectives go through torch.distributed.
Because each device runs its own PyTorch process, both engines keep each framework’s worker model, scheduler, KV cache manager, API server and so forth, with TPU implementations underneath.
Most models use the standard upstream PyTorch model definitions in both engines, with TPU-specific optimizations in kernels and layer implementations. Custom kernels written in Pallas are called directly from PyTorch as custom operators inside the compiled graph. This platform-plugin design reuses as much of each framework as possible while letting TPU optimizations move at their own pace.
Both engines are built for the full range of production workloads.
Here’s the list of supported features:
- Agentic & Multi-Turn: Automatic prefix caching, including for hybrid models; KV cache offloading to host memory and Managed Lustre through TPU Sync; tool calling and structured output
- Interactive & Low Latency: Speculative decoding with MTP, EAGLE-3, DFlash, DSpark and n-gram drafters
- High-throughput Batch: Data-parallel attention with expert parallelism
- Long Context: Context parallelism for prefill and decode; FP8 KV cache; host KV offloading
- Disaggregated Serving: Prefill and decode on separate hosts over TPU Sync, including between different parallel layouts
- Multimodal: Vision-language models with image and video input; embedding models
- Very Large Models: Hybrid and sparse-attention architectures, tensor, data, expert, pipeline and context parallelism across hosts; FP8, FP4 (MXFP4, NVFP4), INT8 and INT4 checkpoints
Agentic workloads in particular demand several of these features simultaneously. Because agents repeatedly resend expanding multi-turn conversations, serving them efficiently requires finding the cached prefix instead of recomputing it. Both engines keep reusable prefixes in HBM and, with host offload turned on, automatically move older prefixes to host memory as a second tier.
Two of these features move the KV cache away from the device that computed it. Disaggregated serving runs prefill and decode on separate TPU hosts, so each request's KV cache has to travel from the prefill host to the decode host.Tiered KV offload moves the part of the KV cache that no longer fits in HBM to the host's DRAM and secondary storage with Google Cloud Managed Lustre, and brings it back when a later request reuses the prefix. Either way, the transfer has to overlap computation or it adds directly to latency.
Both engines use TPU Sync for this. It is an open-source C++ library for KV cache transfer and management on TPU. It reads the device buffers behind PyTorch tensors directly, so a transfer needs no extra staging copy on the host, and it runs transfers on its own threads outside Python while prefill and decode continue. vLLM-TorchTPU exposes it through vLLM's KV connector interface. SGLang-TorchTPU uses it as a disaggregation transfer backend and as the I/O backend for HiCache.
Both engines are already running at scale in production with preview customers. To provide transparent, reproducible proof of our performance, we benchmark in public through the community-standard SemiAnalysis InferenceX project.
Building on our initial submission of Qwen 3.5 on Ironwood TPU (TPU 7x), our engineering teams have further optimized the stack, bringing the lowest cost per million tokens overall to an industry leading 2.01 cents. Alongside these improvements, we have also published new state-of-the-art (SOTA) performance results to AgentX, the project's dedicated track for evaluating agentic long-context and multi-turn workloads. Ironwood TPU leads across most of the Pareto frontier. We look forward to continuously updating our public performance profiles as we bring more models and configurations to the leaderboard.
We did not build these backends in a vacuum. By embracing PyTorch and working directly with open-source project maintainers, we ensure these engines remain upstream-aligned and community-first:
This partnership is bringing SGLang's stellar team to build for TPU, delivering an engine that is already running in production at Meta and battle-tested at scale across demanding real-world scenarios.
"SGLang-TorchTPU is running in production at Meta today, and getting there was a close three-way collaboration between Meta, Google, and the SGLang team to make the stack production-ready. It gives us a remarkably mature, PyTorch-native serving stack that integrates effortlessly into our existing AI infrastructure, with the performance, reliability, and flexibility we need to run frontier workloads efficiently on TPUs. " said Zongwei Zhou, PyTorch-TPU lead at Meta.
"At RadixArk, TorchTPU has supercharged our development velocity as we roll out SGLang support across diverse silicon. It’s given us a painless path to native TPU acceleration, passing massive efficiency gains and flexible compute choices directly down to our customers," said Banghua Zhu, CTO and co-founder of RadixArk.
Together, we integrated a high-performance Pallas megakernel for the 2.8T-parameter Kimi K3 model that runs the entire decode step as one kernel, serving more than 700 tokens per second to a single user on Ironwood TPU. Building on this work, Inferact recently led our first preliminary AgentX submission for Kimi K3 on the vLLM-TorchTPU stack. This initial milestone already demonstrates exceptional performance per TCO in ultra-high-interactivity workloads, and the rapid increase in the maturity of the vLLM and PyTorch ecosystem on TPU for doing new frontier OSS models bring-up, with follow-up submissions set to unlock even greater efficiency through larger scale prefill/decode disaggregation.
"Our mission at Inferact has always been to make high-performance inference accessible, reliable, and cost-effective across the entire AI ecosystem. We are thrilled to partner on the launch of TorchTPU and bring first-class support for vLLM to TPUs. By bridging PyTorch’s optimized TPU capabilities directly into vLLM, developers can immediately run production workloads on TPUs with the high throughput, low latency, and memory efficiency they expect from our stack," said Simon Mo, CEO and co-founder of Inferact.
To foster continuous open-source contribution, vLLM is also launching a first-of-its-kind community contribution TPU capacity pool to provide free TPU compute to developers who actively contribute to the vLLM engine.
Apply for the vLLM TPU Contributor Pool
As the models get larger, we are evolving both engines to make large-scale TPU inference deployments performant and efficient. The major features for production-grade agentic workloads are already supported. More performance optimizations and advanced features are under way – open a Github issue to request collaboration on yours!
Across the two engines, most of the popular open model families are supported, including Qwen, DeepSeek, GLM, Kimi, GPT-OSS, Gemma and more. The team is investing in making more models available to run natively on TPUs, including Day 0 support for new open models. We also want to give the community the tools and TPU capacity to tune performance for their own workloads and configurations.
vLLM-JAX and SGLang-JAX remain important projects. Both will continue to be maintained, supported, and available for the community to use and adapt.
Github projects:
- vLLM-TorchTPU: https://github.com/vllm-project/vllm-torchtpu
- SGLang-TorchTPU: https://github.com/sgl-project/sglang-torchtpu
- TorchTPU: https://github.com/google-pytorch/torch_tpu
Related announcements:
Auto-fetched and translated by AI Atlas. All rights belong to the original author.