大模型每生成一个 token，都要从头开始计算吗？
为什么加了一块小小的缓存，生成速度就能快几十倍？
这篇文章用最通俗的方式，把 KV Cache 一次讲透。
本文收录于微信公众号：愤怒的代码 AngryCode
0x00、大模型是如何推理的
在聊 KV Cache 之前，得先搞清楚一件容易被想当然的事：
大模型不是一次性把整句话想好了再说出来的，而是一个字接一个字"蹦"出来的。简单的说大语言模型是做文字接龙的游戏。
这里的"字"，专业术语叫 Token（词元）。它可能是一个完整的词、半个词，甚至一个标点符号。为方便理解，你可以先把它当成"一个字"。
举个例子，你输入：
深圳今天
模型看着这四个字，预测出下一个最可能的字是"天"：
深圳今天天
然后它把"天"接到后面，再看"深圳今天天"这五个字，预测下一个字"气"：
深圳今天天气
如此循环，直到模型觉得该停了为止。这个过程叫自回归生成（Autoregressive Generation）。
关键点来了：模型每预测一个新字，都必须"看"一眼前面所有的字，才能决定下一个字写什么。
这句话，是理解 KV Cache 一切的起点。
0x01、注意力机制：Query、Key、Value 三兄弟
那"看一眼前面的字"具体是怎么做的？
大模型内部有个核心组件叫注意力层（Attention Layer），是整个 Transformer 架构的心脏。它的职责是：判断前面的字里，哪些对预测下一个字更重要。
比如预测"深圳今天天"后面的字时，"今天""天"显然比"深圳"更重要。注意力的作用，就是自动算出这种"重要程度"。
在注意力层里，每个字都会被转化成三样东西：
- Query（查询，Q）：这个字"想找什么"。
- Key（键，K）：这个字"身上有什么标签"。
- Value（值，V）：这个字"实际携带的信息"。
打个比方，你去图书馆想找讲 Transformer 的书：
- Query 是你的需求："我想找讲 Transformer 的书"。
- Key 是每本书侧边贴的标签："深度学习""机器学习""川菜菜谱"……
- Value 是书本身的正文内容。
过程是：拿你的 Query 去和所有书的 Key 比对相似度，算出每本书有多"相关"，然后按相关度，把最相关那几本书的 Value（正文）拿过来用。
翻译成数学公式，就是那行著名的：
Attention(Q, K, V) = softmax(Q · Kᵀ / √d) · V
- Q · Kᵀ：拿 Query 和所有 Key 做点积，算出"相关度分数"。
- softmax：把分数归一化成"注意力权重"，加起来等于 1。
- · V：用权重对所有 Value 加权求和，得到最终的输出。
注意一个细节：每个字的 K 和 V，只由这个字自己决定，和后面来了什么字无关。这一点极其重要，它正是 KV Cache 能成立的根本原因（后面会讲）。
0x02、问题：大量重复计算
现在回到生成过程，看看哪里浪费了。
没有缓存时，模型每预测一个新字，都要把整句话所有字重新过一遍网络——它们的 Q、K、V 要重算，前馈层（FFN）要重跑，注意力分数矩阵也要重新算。而这一轮算出来的东西，除了最后一个位置，其余全部丢掉。
所以浪费的算力远不止"Q、K、V 重算"这一项。真正的大头其实是那些旧 token 的前馈层和投影计算，只是它们算完就被扔了，谁也没用上。
以上面的例子走一遍：
发现问题了吗？
"深"的 K 和 V，在第 1 步算过一遍，第 2 步又算一遍，第 3 步再算一遍。每个已经见过的字，都在被反复计算。
句子越写越长，浪费越严重：如果已经生成了 1000 个字，那么预测第 1001 个字时，就要把前面 1000 个字的 K、V 全部重算一遍，只为算出 1 个新字。
这就是自回归生成最致命的性能瓶颈——计算量随序列长度呈平方级增长。
↑ 同样的「深圳今天天气」，左边每一步都在重算整句，右边只算新 token
0x03、KV Cache 的核心思想：算一次，存起来，反复用
解决思路也很简单：
每个字的 K 和 V，只算一次，存进一块内存里，以后每次直接用，不再重算。
这块内存，就是 KV Cache。
再看一遍刚才的例子，这次有缓存：
第 1 步：输入"深圳今天"
- 计算 4 个字的 K、V，存进缓存。
- 预测出"天"。
- 缓存里现在有：深、圳、今、天 的 K 和 V。
第 2 步：这次只需要输入新字"天"
- 前面的 K、V 直接从缓存里读，不重算。
- 只为新字"天"算一次 K、V、Q，算完也存进缓存。
- 预测出"气"。
- 缓存里现在有：深、圳、今、天、天 的 K 和 V。
第 3 步：只需输入新字"气"
- 前面 5 个字的 K、V 全部从缓存读。
- 只为"气"算一次 K、V、Q，存进缓存。
- 预测出下一个字……
看出来了吧：每一步只算 1 个新字，剩下的全部复用缓存。
0x04、为什么只缓存 K 和 V，不缓存 Q？
这是几乎每个人都会问的问题。答案一句话：
Q 是一次性的，K 和 V 是要反复用的。
展开说：
- Query 只在"当前这个字被预测"的那一刻用一次。当前字拿自己的 Q 去和前面所有字的 K 比对，比对完，这个 Q 的使命就结束了，再也不会有人用到它。
- K 和 V 则要在未来每一个字生成时都被用到。因为每一个新字都要"看"前面所有字，而"看"的动作，本质上就是用新字的 Q 去查所有旧字的 K，再取旧字的 V。
所以缓存里只存 K 和 V，不存 Q——这也是它叫 KV Cache 而不是 QKV Cache 的原因。
0x05、到底快了多少？算一笔账
假设从 1 个字开始，生成到 100 个字：
没有 KV Cache，每个字都要对整句算一遍 K、V，总计算量是：
1 + 2 + 3 + ... + 100 = 5,050 次
有 KV Cache，第 1 步算 1 次，之后每步只算新字 1 次：
1 + 1 + 1 + ... + 1 = 100 次
约 50 倍的差距。而且序列越长，差距越大——这是典型的"把 O(n²) 降成 O(n)"。
需要说明的是，上面这个计数是个简化模型：它只数了 K、V 的投影次数，没有把 FFN、注意力分数这些开销算进来。但即便如此，实际的加速比通常还要更可观。
这就是为什么 KV Cache 是 LLM 推理加速里最基础、也最重要的一个技巧。
0x06、天下没有免费的午餐：速度换内存
KV Cache 用速度换来了一个代价：内存。
缓存里要存下"到目前为止每一个字的 K 和 V"，所以：
序列越长，缓存越大。
缓存大小可以用一个公式精确算出来：
KV Cache 大小 = 2 × 层数 × KV头数 × 头维度 × 序列长度 × 每元素字节数
- 2：因为 K 和 V 各一份。
- 每元素字节数：FP16 精度下是 2 字节。
代入几个真实模型（FP16、单个 token）：
看着每 token 才 0.5 MB 不吓人？算上序列长度就惊人了：
7B 模型，2048 个 token 的上下文，单个请求就要 ~1 GB 的 KV 缓存。
如果同时服务 32 个用户（batch = 32），光 KV 缓存就要 32 GB 显存。这还没算模型本身的权重。
这就是为什么"并发服务很多用户"时，KV Cache 的内存会成为真正的瓶颈 (可以理解今天内存那么昂贵了吧T_T)。
0x07、进阶：当缓存本身成为瓶颈怎么办
理解了上面这些，你就能看懂业界一系列优化是在干什么了。它们本质上都在回答同一个问题：怎么让这块缓存更小、更省、更好管？
1. GQA / MQA：从源头减少缓存
- MQA（多查询注意力，Multi-Query Attention）：让所有注意力头共享同一份 K 和 V，缓存直接砍到原来的 1/头数。
- GQA（分组查询注意力，Grouped-Query Attention）：折中方案，几个头共享一份 K、V。上表里 Llama 2 70B 的 KV 头数只有 8，就是用 GQA 把缓存压了下来。
（顺便提一下老一代的多头注意力 MHA（Multi-Head Attention，多头注意力）：每个头都有自己独立的 K 和 V，缓存最大。MQA 和 GQA 本质上都是往 MHA 这个方向做"减法"。）
2. KV Cache 量化
缓存不用存 FP16，改成 8-bit 甚至 4-bit，内存占用再砍一半以上，而精度损失通常很小。（可以翻阅前面量化的数学原理的文章）
3. PagedAttention（分页注意力）
这是 vLLM 提出的经典方案：把 KV 缓存像操作系统管理内存一样，切成固定大小的"页"来管理，按需分配、灵活回收，彻底解决"预留一块连续大内存"造成的碎片和浪费，大幅提升并发能力。
4. 滑动窗口 / 注意力汇聚（Attention Sink）
对于无限长的对话，只保留最近一段窗口，加上开头几个"注意力锚点"，让缓存固定在一个大小，不再无限膨胀。
这些名字你现在听来可能陌生，但只要记住一句话就够了：它们全部是冲着 KV Cache 的内存开销去的。
0x08、总结
把这篇文章浓缩成几句话：
- 大模型逐字生成，每预测一个字都要参考前面所有字。
- 回头看前面那些字，靠的是 Q、K、V：Q 查、K 匹配、V 取值。
- 如果不缓存，每个旧字的 K、V 会被反复重算，计算量呈平方级增长。
- KV Cache 的核心：每个字的 K、V 只算一次，存起来反复用，把计算量从 O(n²) 降到 O(n)。
- 只缓存 K、V 不缓存 Q，因为 Q 是一次性的，K、V 要反复用。
- 代价是内存——序列越长、并发越高，缓存越吃显存。
- 后续的 GQA/MQA、量化、PagedAttention 等优化，都是在跟这块缓存的内存开销较劲。
KV Cache 是大模型推理优化的第一块基石。理解了它，你就拿到了进入 LLM 推理、vLLM、量化压缩这些话题的入场券。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Does LLM have to start from scratch for every token generated?
Why do you add a little cache, so it's a few dozen times faster?
This article talks about KV Cache in the most common way.
This is WeChatWeChat Official Account: AngryCode
How does 0x00 and LLM work?
Before we talk about KV Cache, we need to figure out something that is easy to think about:
LLM is not a one-time, one-on-one "jump" out of the whole phrase. A simple large-language model is a text relay game.
Here's the word, the professional term is Token. It could be a complete word, half a word, or even a punctual symbol. For ease of understanding, you can use it as a word first.
For example, you enter:
Shenzhen today
The model looks at these four words and predicts that the next possible word is "day":
Shenzhen today
And then it puts the sky in the back, and then it looks at the words "Today in Shenzhen," and predicts the next word:
Shenzhen today
That's how it goes until the model thinks it should stop. The process is called Autogressive Generation.
The key point is here: for every new word the model predicts, you have to look at all the words before you can decide what to write.
That's the starting point for understanding KV Cache.
0x01 Attention: Query, Key, Value
So, what exactly does the word "a glance" do?
Within LLM there is a core component called Attention Layer, the heart of the entire Transformer architecture. Its function is to determine which of the preceding words is more important for predicting the next word.
For example, when predicting the words after Shenzhen today, "Today" is clearly more important than Shenzhen. The effect of attention is to automatically figure out what's important.
Each word is translated into three things in the mind:
- Query (Query): "What do you want?"
- Key: "What's the label on your body?"
-Value (value, V): The word "information actually carried".
For example, you went to the library to find a book about Transformer:
- Query is your need: "I want to talk about Transformer's book."
- Key is the label on each book: "Deep Learning" "Mechanic Learning" "Kawakawa"
- Value is the text of the book itself.
The process is: to compare your Query to all the books' Key, to figure out how "relevant" each book is, and then, to the extent relevant, to bring back the Value of the most relevant books.
Translation of the mathematical formula is the famous line:
Attention(Q, K, V) = softmax(Q · Kᵀ / √d) · V
- Q.K.T.: Take Query and all Key do a little bit to calculate the "relevance score."
-Softmax: Putting the score together into "the weight of attention" and adding up to one.
- V: Weighting of all Value to achieve final output.
One detail: the K and V for each word, it's only for the word to decide, not about what comes in the back. This is extremely important, and it is precisely the underlying cause of KV Cache ' s existence (discussed later).
0x02, problem: extensive double counting
Now go back to the generation and see where it was wasted.
In the absence of cache, each new word is projected by the model to re-examine the entire sentence -- their Q, K, V, re-calculate, the front feed (FFN) reruns, and the focus fraction matrix is recalculated. And this round of counting things, except for the last one, all of them.
So there's more to waste than "Q, K, V recalculating." The real big heads are the old token's front layers and projection calculations, but they're just thrown out and nobody's going to work.
Take the example above:
Got a problem?
"Deep" K and V, count again at step 1, step 2 and step 3. Every word that has been seen is being counted over and over again.
The longer the sentence is written, the more wasteful it is: if 1,000 words have been generated, the 1001 word is projected to be fully recalculated with the 1000 words K, V, and only one new word.
This is the most lethal performance bottleneck of self-regression - calculated volume increases at a horizontal level with the length of the sequence.
The same Shenzhen weather today. Every step on the left is recasting the whole sentence.
The core idea of 0x03, KV Cache: once, save, repeat.
The solution is simple:
The K and V of each word are counted once, stored in a single memory, and then used directly and not recalculated.
This memory is KV Cache.
Look at the example. This time there's cache:
Step 1: Enter Shenzhen today
- Calculating four words of K, V, stored in cache.
- Forecast the day.
- cache now has deep, Shenzhen, today, day K and V.
Step 2: This time just enter the new word "day."
- The K and V in front read directly from cache, not recalculated.
- Just once for the new word "day" K, V, Q, and then in cache.
- Prognose "breath."
- cache now has the deep, Shenzhen, today, day, day K and V.
Step 3: Just enter the new word "gir"
- The first five words of K and V are all read from cache.
- Just for gas, K, V, Q, in cache.
- Forecast the next word...
See: one new word at a time, and all that's left is cache.
0x04, why only cacheK and V, not cacheQ?
This is a question that almost everyone will ask. The answer is:
Q is one-off, K and V are to be used repeatedly.
Opens up and says:
- Query's only used once at the moment the word is projected. The current word takes its own Q to match all the words in front with the K, and this Q mission is over, and no one will ever use it again.
- K and V are to be used for every word in the future. Because every new word has to be "see" all the words before it, and the "see" move is essentially using the Q of the new word to check the K of all the old words and then take the V of the old words.
So cache has only K and V and no Q -- that's why it's called KV Cache instead of QKV Cache.
0x05. How much is it? Make a deal.
Assume from 1 word to 100 words:
No KV Cache, every word counts the whole sentence.
1 + 2 + 3 +... + 100 = 5,050 times
KV Cache, 1st step count, followed by 1 new word:
1 + 1 + 1 +... + 1 = 100 times
About 50 times the gap. And the longer the sequence, the bigger the gap -- it's typical of "thing O(n) down to O(n)".
And it's a simple model: it counts the K-V projection, and it doesn't count the FNN, the focus points. But even so, the real acceleration is more significant than usual.
That's why KV Cache is the most fundamental and most important technique in the acceleration of LLM reasoning.
0x06, no free lunch: speed change memory
KV Cache traded speed for a price: memory.
"K and V for every word so far", so:
The longer the sequence, the bigger the cache.
The size of cache can be calculated precisely by a formula:
KV Cache size = 2 x layers x KV head count x head dimension x serial length x Z113Z number per element
- 2: Because one for K and one for V.
- Number of bytes per element: FP16 is 2 byte at precision.
Add several real models (FP16, single token):
Watching every token is 0.5 MB. The length of the sequence is amazing:
7B model, context for 2048 token, KV cache for ~1 GB.
If 32 users (batch = 32) are served at the same time, 32 GB Z VRAM is required for KV cache alone. It's not the weight of the model itself.
That's why KV Cache's memory becomes a real bottleneck when it's "many users at the same time."
0x07, Progress: What happens when cache itself becomes a bottleneck?
To understand this, you can see what the industry is doing with a set of optimizations. In essence, they answer the same question: How can this cache be smaller, more economical and better managed?
1. GQA / MQA: Reduction from source cache
- MQA (More Query Attention): Share the same K and V, cache directly to the original 1/head count.
- GQA (grouped for attention, Grouped-Query Attention): compromise option, several heads shared one K, V. The first number of KVs for Llama 2 70B in the above table is only 8, which means that cache was pressed down with GQA.
Each head has its own K and V, cache. MQA and GQA are essentially doing "deductive" in this direction to MHA.
2. KV Cache quantization
cache does not have to save FP16 and is replaced by 8-bit, or even 4-bit, with more than half of the memory occupancy, which is usually a small loss. You can look at the mathematical principles of quantization.
PagedAttention
This is a classic scheme proposed by vLM: to manage KV cache as an operating system management memory, cutting into a fixed-sized "page" that distributes, recycles with flexibility on demand, completely resolves the debris and waste caused by the "retention of a continuous large memory" and significantly increases co-generation capacity.
Slide Window / Focus
For an infinity-long dialogue, only the most recent window will be retained, with the initial focus anchors, so that cache will be fixed to one size and will not expand indefinitely.
These names may sound strange to you now, but just remember one sentence: They're all for KV Cache's memory.
0x08, Summary
This post is a few words:
- LLM is generated word for word, with reference to all the words before each word.
- Look back at the words, Q, K, V:Q check, K match, V value.
- If not cache, each old word of K and V will be recalculated over and over again and calculated at a level of square growth.
- The core of KV Cache: K and V of each word is counted once, stored repeatedly and reduced from O(n2) to O(n).
- Only cache K, V not cache Q, because Q is one-off, K and V are to be used repeatedly.
- The price is memory - the longer the sequence grows higher and the more cache eats VRAM.
- Subsequent GQA/MQA, quantization, PagedAttention, etc., are spending more on this cache memory.
KV Cache is the first cornerstone of LLM ' s reasoning optimization. Understand it, and you get tickets to get into the LLM reasoning, vLM, quantization compression.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.