清华大学与CUHK研究人员开源 AKV，是KV缓存工程在LLM agents的服务系统。 agents不总结旧历史，而是放下选定的KV缓存条目，继续重用其余的。 3个型号和4个agent 基准测试的平均成本下降了27.7%。 平均任务分提高1.7分。 代码为Apache-2.0许可并基于SGLang.
TL；数据
服务系统加政策图书馆，没有重量。 测试过GPT-OSS-120B(5.1B活动),MiniMax-M2.7(共230B，共10B活动)和MiroThinker-1.7-Mini(共30B，共3B活动).
- 运行在：NVIDIA GPU通过SGLang. 纸张在BF16中使用了8个A800-80GB GPU. 快速启动服务于2台GPU上的GPT-OSS-120B(频率平行2).
- 业绩：平均成本降低27.7%，加1.7分，平均得分比门槛总和低。
- 最佳：GPT-OSS-120B在SWE-bench验证：每任务成本20.0至9.7美分，得分40.0至43.2.
- 最差：浏览器Comp-Plus上的MiniMax-M2.7得分58.9，文本工程的62.3以下3.4分。
什么是缓存工程，与上下文工程有什么不同？
AKV是一个清华服务系统，它让LLM agents放下并重新定位自己的KV缓存。
背景工程修剪或总结文本历史。 编辑此前缀后每 KV 状态都会失效。 保留后的后缀必须重新进行预装。 因此，利用方式总结了迟到和激烈的情况，这可能会失去中回合的细节。
缓存工程使文本轨迹未受影响。 它只会改变 KV 条目的可见度。 留存的条目无需重新计算即可重复使用。 他们仍然可以携带从投放的信息。 KV缓存占API总成本的82.4%。
AKV是怎样工作的？
AKV定义了3个缓存操作。 附录是新令牌的标准更新。 将选中的条目从未来的注意力中删除而不改变位置。 使用 RoPE 旋转方式，将重置键保留到位置 0 到 r-1。 价值保持不变，因此没有隐藏状态被重算。 在缩写中，加入Repos 融资 GPT-OSS-120B的浏览器Comp-Plus通过率从55%到62%.
难的部分是服务。 同樣的符號可以在不同的操作历史下产生不同的缓存状态。 普通前缀匹配可能会重用错误状态。 AKV扩展SGLang的光圈树，它只在没有主动请求仍然需要时才会释放KV. 片段批注在1个预填通道中恢复被拆迁状态。
AKV API是什么样子的？
该接口为无国籍和OpenAI兼容。 客户端发送完整消息历史和两个可选字段。 drop message 映射要降下的信件的触发消息索引。 重新定位后列出各种指数，然后将职位压缩。
缓存政策通过触发器、选择器和重新定位规则构建这些字段。 内建政策采用回合制，象征性制或模式制。 CacheTooll 让模型调用调用本身 context。
AKV如何在agent 基准测试上表现？.
主要政策是工具响应滚动下降，保持最新的12个工具响应。 标准基线总结为96K令牌(米罗廷克192K). Text Engineering assume at 48K. 基准测试 is browscomp-Plus, GAIA-Text, SWE-bench Verifed and xbench-DeepSearch-2510. (原始内容存档于2018-09-31).
4个基准测试的平均通过率(%)和成本(每任务美元). 资料来源：文件表1。
AKV也耗资21.9%与文本工程公司。 准确性差距长期打开。 在96K令牌及以上的浏览器Comp-Plus上，GPT-OSS-120B与AKV的分数为37%. 标准分数24%，文本工程分数32%，每个项目页。
费用是从公布的费率中得出的API等值估计数。 重新定位的代币按缓存-打击率收费，这是故意保守的选择。
AKV在服务基准测试时有多快？.
服务测试在80项长文SWE-bench验证任务上运行了GPT-OSS-120B. 在每种方法的峰值设定中，AKV提供1.18x输出吞吐量和1.53x输入吞吐量。 P95请求延迟速度快1.51x.
以第16号货币计算，标准号的P95耐用达到 229.90秒对AKV的30.63秒。 KV压力将标准推向重复的重排。 按解码步骤，全程注意时间从7.082米降至2.849米。
在GPU预算下的100个SWE-bench验证任务中，AKV达到了约47%的通过率。 标准达到40.7%，文本工程达到32.3%.
agents能调谐自己的缓存政策吗？
MiniMax-M2.7改进了20项浏览Comp-Plus任务的政策，然后执行100项搁置任务。 重复6政策削减了17.8%的费用，而基准准确性为标准。 抽象的报告显示，在专家设计的政策之外，节省了7.1%。
将缓存控制显示为工具也有助于准确性。 GPT-OSS-120B在一个100任务子集上从52.1上升到61.0. MiniMax-M2. 7的成本从5.4美分上升到5.9美分。
AKV与其他 KV 缓存方法
主要数字来自每个项目自己的纸张，并使用不同的设置。 它们并不具有直接可比性。
关键外卖
- AKV让agents放下并重新定位KV缓存，而不是重写历史。
- 平均成本下降27.7%，在3种模式中得分上升1.7分。
- Repos将浏览器Comp-Plus上的GPT-OSS-120B从55%升至62%.
- 以第16号货币计算，P95延迟从229.90秒降至30.63秒。
- 支持今天只覆盖Qwen,GPT-OSS和MiniMax-M2文本模型。
请检查access-date=中的日期值 (帮助) 页面存档备份，存于互联网档案馆页面存档备份，存于互联网档案馆 GitHub Repo and Project Page. 并加入我们的通讯。 等等你在电报里吗现在你也可以加入我们的电报。
需要和我们合作推广您的 GitHub Repo OR Hugging Face Page OR Product Release OR Webinar 等吗？联系我们
Asif Razzaq是Marktechpost AI Media Inc.的首席执行官。 作为一个有远见的企业家和工程师，阿西夫致力于利用人工智能的潜力促进社会福祉。 他最近的努力是人工智能媒体平台的发布，该平台以深入报道机器学习和深层学习新闻而著称，这些新闻在技术上既可靠又为广大观众所容易理解。 该平台每月有超过200万次观看，表明其受观众欢迎。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Tsinghua University and CUHK researchers open-sourced AKV, a serving system for KV cache engineering in LLM agents. Instead of summarizing old history, agents drop selected KV cache entries and keep reusing the rest. Across 3 models and 4 agent benchmarks, average cost falls 27.7%. Average task score rises 1.7 points. The code is Apache-2.0 licensed and builds on SGLang.
TL;DR
- What it is: Serving system plus policy library, no weights. Tested on GPT-OSS-120B (5.1B active), MiniMax-M2.7 (230B total, 10B active) and MiroThinker-1.7-Mini (30B total, 3B active).
- Runs on: NVIDIA GPUs via SGLang. Paper used 8 A800-80GB GPUs in BF16. Quick start serves GPT-OSS-120B on 2 GPUs (tensor parallel 2).
- Performance: 27.7% lower average cost and +1.7 points average score versus threshold summarization.
- Best: GPT-OSS-120B on SWE-bench Verified: cost per task 20.0 to 9.7 cents, score 40.0 to 43.2.
- Worst: MiniMax-M2.7 on BrowseComp-Plus scores 58.9, 3.4 points below Text Engineering’s 62.3.
What is cache engineering, and how is it different from context engineering?
AKV is a Tsinghua serving system that lets LLM agents drop and reposition their own KV cache.
Context engineering trims or summarizes the text history. Editing that prefix invalidates every later KV state. The retained suffix must then be re-prefilled. Harnesses therefore summarize late and aggressively, which can lose details from middle rounds.
Cache engineering leaves the text trajectory untouched. It only changes which KV entries stay visible to attention. Retained entries are reused without recomputation. They can still carry information from dropped messages. In the research paper’s cost analysis, KV cache reads made up 82.4% of total API cost.
How does AKV work?
AKV defines 3 cache operations. Append is the standard update for new tokens. Drop removes selected entries from future attention without changing positions. Repos rewrites retained keys to positions 0 through r-1 using a RoPE rotation. Values stay unchanged, so no hidden states are recomputed. In ablations, adding Repos raised GPT-OSS-120B’s BrowseComp-Plus pass rate from 55% to 62%.
The hard part is serving. Identical tokens can produce different cache states under different operation histories. Plain prefix matching could reuse the wrong states. AKV extends SGLang’s radix tree with drop and reposition nodes. It frees dropped KV only when no active request still needs it. Segment-batch attention restores evicted states in 1 prefill pass.
What does the AKV API look like?
The interface is stateless and OpenAI-compatible. Clients send the full message history and two optional fields. drop_message maps a trigger message index to the messages to drop. reposition lists the indices after which positions get compacted.
A CachePolicy builds those fields from a trigger, a selector and a reposition rule. Built-in policies are turn-based, token-based or model-based. A CacheTool lets the model call drop_context itself.
How does AKV perform on agent benchmarks?
The main policy is a tool-response rolling drop that keeps the latest 12 tool responses. The Standard baseline summarizes at 96K tokens (192K for MiroThinker). Text Engineering summarizes at 48K. Benchmarks are BrowseComp-Plus, GAIA-Text, SWE-bench Verified and xbench-DeepSearch-2510.
Average pass rate (%) and cost (US cents per task) across 4 benchmarks. Source: Table 1 of the paper.
AKV also cuts cost 21.9% versus Text Engineering. The accuracy gap opens on long runs. On BrowseComp-Plus at 96K tokens and above, GPT-OSS-120B scores 37% with AKV. Standard scores 24% and Text Engineering 32%, per the project page.
Costs are API-equivalent estimates from published rates. Repositioned tokens are charged at the cache-hit rate, a deliberately conservative choice.
How fast is AKV in serving benchmarks?
Serving tests ran GPT-OSS-120B on 80 long-context SWE-bench Verified tasks. At each method’s peak setting, AKV delivers 1.18x output throughput and 1.53x input throughput. P95 request latency is 1.51x faster.
At concurrency 16, Standard’s P95 latency hits 229.90 seconds versus 30.63 for AKV. KV pressure pushes Standard into repeated re-prefill. Per decode step, full-attention time falls from 7.082 to 2.849 ms.
On 100 SWE-bench Verified tasks under a GPU budget, AKV reached about 47% pass rate. Standard reached 40.7% and Text Engineering 32.3%.
Can agents tune their own cache policy?
MiniMax-M2.7 refined policies on 20 BrowseComp-Plus tasks, then ran 100 held-out tasks. The iteration-6 policy cut cost 17.8% versus Standard at baseline accuracy. The abstract reports 7.1% savings beyond the expert-designed policy.
Exposing cache control as a tool also helped accuracy. GPT-OSS-120B rose from 52.1 to 61.0 on a 100-task subset. MiniMax-M2.7’s cost rose, though, from 5.4 to 5.9 cents.
AKV vs other KV cache approaches
The main numbers come from each project’s own paper and use different setups. They are not directly comparable.
Key Takeaways
- AKV lets agents drop and reposition KV cache instead of rewriting history.
- Average cost falls 27.7% and score rises 1.7 points across 3 models.
- Repos lifts GPT-OSS-120B on BrowseComp-Plus from 55% to 62%.
- At concurrency 16, P95 latency drops from 229.90 to 30.63 seconds.
- Support covers only Qwen, GPT-OSS and MiniMax-M2 text models today.
Check out the Paper, GitHub Repo and Project Page. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
Asif Razzaq is the CEO of Marktechpost AI Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.