GLM-5.3工人AI模型中的交叉请求 KV缓存曝光(155份答复)

GLM-5.3工人AI模型中的交叉请求 KV缓存曝光，以及我们与SGLang的补救

2026年9月8日，来自克诺斯蒂克的安全研究者加迪·埃夫龙(Gadi Evron)警告我们，一个影响GLM-5.3工人AI模型(@cf/zai-org/glm-5.3)的租户隔离问题从2026年8月27日开始。 我们已经纠正了这个问题，并于9月9日恢复了模型。 我们确认，在工人大赦国际上主持的其他模式没有受到影响。

在8月27日至9月9日期间，少数使用GLM 5.3工人AI模型的客户收到了包含其他客户在他们之前不久向同一模型提出的请求中的数据的答复。 总共155个答复受到影响，导致超过681 280个令牌。

在排除了我们自己的测试和研究人员的测试后，我们确定的请求在时间和数量上与普通的长文使用一致，而不是与反复利用bug的人一致。 我们没有发现任何蓄意剥削的证据。

今天，我们正与我们使用的开源推论引擎SGLang的一篇文章一起发表这份报告，作为协调披露。 本文描述造成这个问题的缺陷、我们的反应时间，以及我们随后在SGLang的制作部署上进行的硬化。 我们还详细介绍了我们与SGLang的上游维护者为解决这一问题和类似错误而持续作出的努力。

发生了什么事

GLM是一个来自Z.ai的文本生成模型，为代理编码工作流程所建；它是通过Workers AI提供的许多模型之一。 Gadi Evron于9月8日通过我们的HackerOne程序提交了一份报告，Cloudflare在第二天早上转载了该刊，并在宣布事件发生后40分钟内将模型下线。 然后我们部署一个修正的配置和一个补丁的推论引擎，并在当天恢复服务。

关于交叉请求的KV缓存暴露的具体报告是两个重叠问题的结果。 第一个是开源推论引擎SGLang的错配置，由Cloudflare使用。 我们部署了GLM 5.3，有一个1,048,576的上下文窗口，但是我们预填GPU上的KV缓存池只有681,280个令牌的空间。 在发布之前我们意识到了这种不匹配。 我们期望引擎返回一个错误，以获得任何它无法处理的瞬间，而我们的长文测试返回了成功的答复，因此我们认为这不是一个问题。

第二是SGLang如何在分类预填/解码模式下处理所产生的超长请求的缺陷。 一个不匹配池的提示被内部标记为拒绝并替换为一个单键的占位符，但是不是中止请求并返回客户端错误，而是预先填充生成了占位符的 KV 缓存，从而被传送到解码节点。 解码节点收到一个"传输完整"的消息，并假定预先填充从之前的请求中过度刷新了 KV 缓存，同时对此请求有完全的及时性，而事实上它使得之前状态的大部分都完好无损。 解码节点随后在浏览仍然保存来自先前请求的数据的 Stale 缓存页面时生成了响应。 因此，对超过681,280个令牌的提示的答复可能包含来自其他用户GLM 5.3会议的数据。

立即补救

在使模式失效后，我们采取了以下行动来纠正这种脆弱性，然后恢复公众使用这一模式的机会：

为了修复我们的配置，我们让游泳池保持了完整的环境。 我们调整了我们的GLM 5.3配置，以匹配我们的GLM 5.2配置，它释放了足够的GPU内存，预填KV池现在持有约204万个令牌，大约是1,048 576个上下文长度的两倍。 一个适合上下文窗口的提示现在总是适合集合，因此拒绝路径永远无法为有效的请求到达。 仅此一项就阻止了虫子的锻炼。

为了修复引擎，我们还使拒绝路径安全。 仅依靠池大小就很脆弱：任何变换到量子化，拉伸平行化，CUDA图形大小，或者未来的模型版本会以容易错过的方式改变池大小。 此外，还可能存在预填调度器可以拒绝请求并触发同一错误的其他情况。 为了防止这种情况，我们还修补了引擎，这样，一个载有待决拒绝的请求在分解录用队列中退役，并加上其原始错误，在任何靴子陷阱，分配或KV传输发生之前。 我们提出这个上游是#38704，它也让解码节点验证所传输的符号数量与它保留的数量相符。 修复工程已经在36988号的SGLang主机上进行，9月6日因相关故障路径而合并，但尚未完成发布。 维护者们还按照我们的做法，合并了38935号，拒绝这些要求。

在恢复模型之前我们验证了研究者复制脚本和我们自己的金丝雀标记测试

背景：我们如何服务GLM 5.3

工人AI在分类配置中用SGLang运行GLM 5.3：一个预填GPU的集合会处理该提示并产生模型的KV缓存，然后将缓存在远程直接内存访问(RDMA)上，通过Mooncake，到生成响应的解码GPU集合。 我们曾经在高性能的LLMs和较小的，更快的，更安全的模型上写过这个架构。

GPU内存中的KV缓存是一个共享的固定大小页面集合。 每个请求在被接受时被指定页，发布在完成时被指定页。 目前，在请求之间没有将页面零化；获得页面的下一个请求将直接覆盖。 这种方法在推论引擎中是常见的，也是实现最佳性能所必需的，但这意味着推论引擎负责跟踪这些内存页的所有权。 KV缓存页面所有权和寿命跟踪不仅必须处理节点内的共享所有权(KV缓存卸载和令牌生成均以不同的寿命访问内存)，而且还要处理技术上与远程预填节点共享的所有权(它持有一个同时可变的页面引用).

此外，这个池的大小不是一个配置值。 SGLang在启动时从任何GPU内存中装入模型重量后留下的大小。 该模型的广告上下文长度是一个单独的配置值。 没有任何东西迫使双方同意，我们的职责是检查他们是否同意。

第一问题：配置错误

在我们的GLM 5.3预填节点与我们的配置，游泳池来到681,280个令牌而模型的上下文长度是1,048,576个令牌。 选定的 KV 池大小在启动时由 SGLang 打印在日志中，并作为度量符曝光。 我们意识到这种大小不匹配；然而，在发布前测试中，我们看到HTTP状态代码和一个在基准设定过程中没有错误框架的流体。 我们假设引擎能够优雅地处理 KV 缓存池和请求的上下文窗口之间的差额，通过流或某种其他机制。

这是我们测试中的一个关键缺口：我们只在长时间内验证HTTP响应代码和引擎性能，而没有验证模型输出和一致性。 我们现在从运行的引擎直接对照上下文长度来检查KV池大小，然后进行分门别类的部署改变船，而不是从反应码中推断出来。

第二期：SGLang 错误

在v0.5.21中，SGLang团队现在处理了这个错误，它处理提示的时间比KV池大小长。 以下是之前发生的事情。

当一个超过KV游泳池的快速到达时，SGLang的调度员拒绝了。 拒绝路径(Req.set finish with bort)记录了要返回客户端的错误，并交换了单键占位符的提示，这样请求就可以在不分配真实内存的情况下通过正常的批量机械排空。 在单一节点部署上，此操作有效：占位符运行，批量完成，客户端收到错误。

在像我们运行这样的分类模式中，预填和解码上的录取队列检查请求是否完成，而不是是否被拒绝。 一个被拒绝的请求自称是活的。 因此：

- 鞋带。 路由器告诉解码节点的快速长度(图中的8页). 解码器为请求保留了8 KV页。

- 预填录用在req. finished( ) 为 False 的情况下，预填录取队列接受任何请求。 一个被拒绝的请求将其拒绝存储到完成，而非完成理由，因此完成( ) 仍然返回 False。 被拒绝的请求被接受。

- 预填前传拒绝已被一个单键占位符取代。 Prefill 运行了一个微不足道的前进传递一个标志。

-KV转帐 Prefill发送了KV的一页解码，并信号传输完成。

- 破解密码解码的录取队列有相同的完成()检查，所以接受了交接。

- 解码注意。 解码器开始生成，注意阅读所有8个保留页面。 这一请求只写了P0页。 第P1页至第P7页仍然保留着它们中较早的请求：其他用户的提示和部分代号。

- 回应。 客户端得到了HTTP 200而不是HTTP 400. prools tokens 匹配长的即时且回复一致。 它的到来比8页的预填要快，包含来自其他用户的KV页面的内容。

每一次超过门槛的请求都这样这个bug在2026年7月6日被另一个用户报告为#30233号问题，Cloudflare在调查这一事件时注意到了这个错误。 我们自v0.5.10起就一直与SGLang一起标榜这类问题，这次安全事件加速了Cloudflare与SGLang团队合作已经开始的工作。 这个问题的补丁在SGLang v0.5.20中为发布，在v0.5.21和v0.5.22中为后续，并可供所有其他依赖于SGLang的操作者使用。 我们高度推荐所有运营商将SGLang依赖性更新为v0.5.22或以后。 关于具体PR的更多详情，见下文。

GLM 5.3是唯一受到这个bug影响的模型。 关于工人AI的所有其他分类部署，在预填和解码时，KV集合都大于其上下文长度，因此任何请求都达不到拒绝路径。 我们把发动机固定和下面描述的硬化应用到我们为SGLang服务的每一个模型上，我们的部署检查要求它们用于每次新的部署。

时间线

所有时间UTC.

正在进行的硬化努力

这个bug属于Cloudflare已经工作了几个月的KV缓存存储安全bug的类。 在每一种情况中，一个页面被解禁、重新使用或读取，而其他的网页却仍然有权利主张。 我们看到两个共同的原因。

第一个是单节点内页寿命。 解码节点上的几个任务可以同时持有同一页：令牌生成，卸载到主机内存，KV 传输接收器，以及中止和错误路径。 如果其中之一在另一个仍在使用时释放页面，则可以在仍在读或写时给出新的页面请求。 我们先前对解码卸载(#24226,#30899)和对语法错误(#30937)的双重释放的修饰就是此类。

第二场是横跨节点的比赛。 当一个请求在预填时仍然写入其 KV 缓存以解码 RDMA 时，已经发送的写入无法被召回。 如果解码立即解码这些页面，这些页面可以提供给另一个请求，而后期写到请求的KV缓存中。 Cloudflare通过提议的修补(#31357,#38961)和复制脚本(#32564)为修补这场比赛的努力做出了贡献。 在这两种情况下，结果都是损坏输出，或者一个用户的数据在另一个用户的响应中，没有

本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang

On September 8, 2026, we were alerted by a security researcher, Gadi Evron from Knostic, to a tenant isolation issue affecting the GLM-5.3 Workers AI model (@cf/zai-org/glm-5.3) starting on August 27, 2026. We have remediated the issue, and we restored the model on September 9. We have confirmed that no other model hosted on Workers AI was affected.

Between August 27 and September 9, a small number of customers using the GLM 5.3 Workers AI model received responses that contained data from other customers' requests made to the same model shortly before theirs. In total, 155 responses were affected, all to prompts longer than 681,280 tokens.

After excluding our own testing and the researcher's, the requests we identified are consistent in timing and volume with ordinary long-context use, not with someone repeatedly exploiting the bug. We found no evidence of deliberate exploitation.

Today we are publishing this report alongside a post from SGLang, the open-source inference engine that we use, as a coordinated disclosure. This post describes the bug that caused this issue, our response timeline, and the subsequent hardening we have conducted on our production SGLang deployments. We also detail our continued efforts with the upstream maintainers of SGLang to address this and similar bugs.

What happened

GLM is a text generation model from Z.ai that is built for agentic coding workflows; it is one of the many models available through Workers AI. Gadi Evron submitted a report on September 8 through our HackerOne program stating that very long prompts sent to @cf/zai-org/glm-5.3 on Workers AI sometimes returned text that belonged to other users' sessions. Cloudflare reproduced the issue the next morning and took the model offline within 40 minutes of declaring an incident. We then deployed a corrected configuration and a patched inference engine, and restored service the same day.

This specific report of cross-request KV cache exposure was the result of two overlapping issues. The first was a misconfiguration of the open-source inference engine, SGLang, used by Cloudflare. We deployed GLM 5.3 with a 1,048,576-token context window, but the KV cache pool on our prefill GPUs only had room for 681,280 tokens. We were aware of this mismatch before launch. We expected the engine to return an error for any prompt it could not process, and our long-context tests returned successful responses, so we did not believe this to be an issue.

The second was a bug in how SGLang handled the resulting over-length requests in disaggregated prefill/decode mode. A prompt that did not fit the pool was internally marked as rejected and replaced with a one-token placeholder, but instead of aborting the request and returning a client error, prefill generated the KV cache for the placeholder prompt and transferred it to the decode node. The decode node received a "transfer complete" message and assumed that prefill overwrote the KV cache from previous requests with the full prompt for this request, when in fact it left the majority of the previous state intact. The decode node subsequently generated a response while attending over stale cache pages that still held data from earlier requests. As a result, responses to prompts longer than 681,280 tokens could contain data from other users' GLM 5.3 sessions.

Immediate remediation

After disabling the model, we took the following actions to correct this vulnerability before restoring public access to it:

To fix our configuration, we made the pool hold the full context. We adjusted our GLM 5.3 configuration to match our GLM 5.2 configuration, which freed enough GPU memory that the prefill KV pool now holds about 2.04 million tokens, roughly twice the 1,048,576-token context length. A prompt that fits the context window now always fits the pool, so the rejection path is never reached for a valid request. This alone prevents the bug from being exercised.

To fix the engine, we also made the rejection path safe. Relying on pool size alone is fragile: any change to quantization, tensor parallelism, CUDA graph sizes, or a future model version changes the pool size in a way that's easy to miss. Furthermore, there may be other situations in which the prefill scheduler can reject a request and trigger the same bug. To prevent this, we also patched the engine so that a request carrying a pending rejection is retired at the disaggregation admission queue, with its original error, before any bootstrap, allocation, or KV transfer happens. We proposed this upstream as #38704, which also had the decode node verify that the number of tokens transferred matches the number it reserved. The fix was already on SGLang main in #36988, merged September 6 for a related failure path but not yet released. The maintainers also merged #38935, which rejects these requests earlier, following our approach.

We validated both changes against the researcher's reproduction script and our own canary-marker tests before restoring the model.

Background: how we serve GLM 5.3

Workers AI runs GLM 5.3 with SGLang in a disaggregated configuration: a pool of prefill GPUs processes the prompt and produces the model's KV cache for it, then hands that cache over remote direct memory access (RDMA), via Mooncake, to a pool of decode GPUs that generate the response. We have written about this architecture before in our posts on high-performance LLMs and smaller, faster, safer models.

The KV cache in GPU memory is a shared pool of fixed-size pages. Each request is assigned pages when it is admitted and releases them when it finishes. Pages are not currently zeroed between requests; the next request that gets a page simply overwrites it. This approach is common among inference engines and is necessary to achieve the best performance, but it means that the inference engine is responsible for tracking ownership of these memory pages. KV cache page ownership and lifetime tracking has to deal with not just shared ownership within a node (KV cache offload and token generation both access the memory with different lifetimes), but also ownership that is technically shared with the remote prefill node (which holds a simultaneous mutable reference to the page).

Furthermore, the size of this pool is not a configuration value. SGLang sizes it at startup from whatever GPU memory remains after loading the model weights. The model's advertised context length is a separate configuration value. Nothing forces the two to agree, and it is our job to check that they do.

Issue one: misconfiguration

On our GLM 5.3 prefill nodes with our configuration, the pool came to 681,280 tokens while the model's context length is 1,048,576 tokens. The selected KV pool size is printed by SGLang in logs during startup and exposed as a metric. We were aware of this size mismatch; however, during pre-launch testing we observed successful HTTP status codes and a streaming body with no error frames during benchmarking. We assumed that the engine was capable of gracefully handling the difference between the KV cache pool and requested context window via streaming or some other mechanism.

This was one critical gap in our testing: we only validated HTTP response codes and engine performance at long contexts, without validating model output and coherence. We now check KV pool size against context length directly, from the running engine, before any disaggregated deployment change ships, rather than inferring it from response codes.

Issue two: SGLang bug

In v0.5.21, the SGLang team has now addressed this bug in how it was handling prompts longer than the KV pool size. Here’s what was happening before.

When a prompt arrived that was longer than the KV pool could hold, SGLang's scheduler rejected it. The rejection path (Req.set_finish_with_abort) recorded the error to be returned to the client and swapped the prompt for a one-token placeholder so that the request could drain through the normal batch machinery without allocating real memory. On a single-node deployment this worked: the placeholder ran, the batch finished, and the client received an error.

In disaggregated mode such as we operate, the admission queues on both prefill and decode checked whether a request was finished, not whether it was rejected. A rejected request reported itself as live. So:

- Bootstrap. The router told the decode node the prompt length (8 pages in the diagram). Decode reserved 8 KV pages for the request.

- Prefill admission. The prefill admission queue admitted any request where req.finished() was False. A rejected request stored its rejection in to_finish, not finished_reason, so finished() still returned False. The rejected request was admitted.

- Prefill forward pass. Rejection had already replaced the prompt with a one-token placeholder. Prefill ran a trivial forward pass on that one token.

- KV transfer. Prefill sent one page of KV to decode and signaled that the transfer was complete.

- Decode admission. Decode's admission queue had the same finished() check, so it accepted the handoff.

- Decode attention. Decode started generating and attention read all 8 reserved pages. Only page P0 had been written by this request. Pages P1 to P7 still held what earlier requests left in them: other users' prompts and partial generations.

- Response. The client got HTTP 200 instead of HTTP 400. prompt_tokens matched the long prompt and the reply was coherent. It arrived faster than an 8-page prefill should take, and it contained content from other users' KV pages.

Every request over the threshold behaved this way. This bug had been reported to SGLang by another user as issue #30233 on July 6, 2026, which Cloudflare noticed while investigating this incident. We had been flagging these types of issues with SGLang since v0.5.10 and this security incident sped up work that Cloudflare had already started in collaboration with the SGLang team. The patch for this issue was released in SGLang v0.5.20, with follow-ups in v0.5.21 and v0.5.22, and is available for all other operators dependent on SGLang to use. We highly recommend all operators update SGLang dependencies to v0.5.22 or later. For more details on specific PRs, read on below.

GLM 5.3 was the only model affected by this bug. Every other disaggregated deployment on Workers AI has KV pools larger than its context length on both prefill and decode, so no request could reach the rejection path. We have applied the engine fix and the hardening described below to every model we serve with SGLang, and our deployment checks require them for every new deployment.

Timeline

All times UTC.

Ongoing hardening effort

This bug belongs to a class of KV cache memory-safety bugs that Cloudflare has been working to address for several months. In each case, a page is freed, reused, or read while something else still has a claim on it. We see two common causes.

The first is page lifetime within a single node. Several tasks on a decode node can hold the same page at once: token generation, offload to host memory, the KV transfer receiver, and the abort and error paths. If one of them frees the page while another is still using it, a new request can be given that page while it is still being read or written. Our earlier fixes for decode offload (#24226, #30899) and double release on grammar errors (#30937) were of this kind.

The second is a race across nodes. When a request is aborted while prefill is still writing its KV cache to decode over RDMA, the writes already sent cannot be recalled. If decode frees those pages right away, those pages can be given to another request, and the late writes land in that request's KV cache. Cloudflare has contributed to the effort to fix this race through proposed fixes (#31357, #38961) and a reproduction script (#32564). In both cases the result is corrupted output, or one user's data in another user's response, with no

Auto-fetched and translated by AI Atlas. All rights belong to the original author.