1. 从逐 Token Decode 说起
大语言模型生成文本时，Decode 阶段具有严格的自回归依赖：只有得到当前 Token，才能计算下一个 Token。KV Cache 可以避免重复计算已经出现的上下文，但没有消除 Token 之间的串行依赖。
Token 1 → Token 2 → Token 3 → Token 4 → …
对于大模型，单个 Decode Step 往往只处理很少的 Token，却要读取大量模型权重，因此常呈现 memory-bound 特征。投机解码的目标，就是在不跳过 Target Model 校验的前提下，让一次 Target Forward 尽可能推进多个 Token。
2. 投机解码：Draft 负责猜，Target 负责判
典型的投机解码包含一个轻量 Drafter 和一个最终负责生成质量的 Target Model。Drafter 先提出最多 K 个候选 Token，Target Model 再用一次批量 Forward 计算这些位置的 logits，并从前向后决定哪些候选可以接受。
假设 Drafter 提出 A B C D，Target 判断前三个候选可接受、第四个不可接受，那么本轮至少可以输出 A B C，并在拒绝位置使用 Target 的结果继续生成。
这里还有一个经常被忽略的细节：验证 K 个 draft token 时，Target 通常还会产生下一个位置的 bonus token。
当每轮最多起草 K 个 Token 时，平均接受长度的范围是 1 到 K+1；其中"1"就是 Target 自己产生的 bonus token。
3. 为什么需要 EAGLE/EAGLE3
普通小型 Draft Model 虽然比 Target Model 便宜，但它与 Target 是两个独立模型。模型越小，起草越快，却可能越不了解 Target 的真实生成倾向，导致接受率下降；模型越大，接受率可能提高，但起草成本也随之增加。
EAGLE 的切入点是：Target Model 在上一轮计算中已经产生了信息密度很高的隐藏状态，与其让 Drafter 只看历史 Token，不如把 Target 的隐藏特征也交给 Drafter。
3.1 EAGLE：在特征空间里做自回归外推
EAGLE 全称为 Extrapolation Algorithm for Greater Language-model Efficiency。它不是简单复制一个缩小版 Target Model，而是训练一个轻量的特征预测网络：输入当前 Token 的 embedding 和 Target Model 已计算出的 hidden state，外推（Extrapolation）出未来位置的隐特征向量，再复用 Target 冻结的 LM 头，把预测出来的特征转成 token 概率 / 候选 token。
"Feature Extrapolation（特征外推）"是理解 EAGLE 的关键。Token 是离散结果，而隐藏特征包含更丰富的上下文信息。Drafter 借助 Target 的特征进行预测，通常比只依赖 Token 序列的小模型更容易贴近 Target，从而争取更高的接受率。
不过，EAGLE 并没有天然消除 Drafter 内部的串行依赖。在 vLLM 默认实现中，如果配置 num_speculative_tokens=K，Drafter 会先生成第一个候选，再循环执行 K-1 次轻量 Draft Forward，逐步得到后续候选。
Target hidden state + 当前 token
↓
Draft token 1
↓
Draft token 2
↓
Draft token 3
↓
Target 一次批量验证
只有模型经过相应训练并显式启用 parallel_drafting 时，才属于并行生成多个 draft token 的路径。不能把"投机解码能一次验证多个 Token"等同于"EAGLE 一次 Forward 就生成多个 Token"。
3.2 EAGLE-3：从单一最终特征走向多层特征融合
EAGLE-3 延续了特征级 Drafter 的思路，但进一步使用 Target Model 的多个辅助隐藏状态。不同深度的 Transformer 层携带的信息侧重点不同：较浅层更接近局部和词法信息，中间层逐步形成语义表示，较深层更贴近最终预测。将多层特征融合后，Drafter 可以获得比单一末层状态更丰富的条件信息。
这里不能简单写成"固定使用最后三层"。在 vLLM 中，辅助层首先由 checkpoint 的 eagle_aux_hidden_state_layer_ids 等配置决定；若 checkpoint 没有指定，通用接口才会按模型深度选择三个默认位置，分别靠近早期、中间和后期。
还需要保留一个实现层面的例外：部分 EAGLE-3 checkpoint 会设置 use_aux_hidden_state=false，退化为只使用最后层输出。因此，"多层特征融合"是 EAGLE-3 的典型设计，而不是所有 checkpoint 都必须满足的硬编码规则。
3.3 EAGLE 与 EAGLE-3 有什么不同
EAGLE-3 的主要改进点是 Drafter 获得了更丰富的 Target 特征，而不是取消 Target Verification，也不是天然把所有候选 Token 改成一次并行生成。
4. 为什么不会牺牲生成正确性
投机解码中，Drafter 只提供候选，最终输出仍由 Target Model 的 logits 和验证算法决定。Drafter 猜错的直接后果是接受长度变短，而不是错误 Token 被无条件写入结果。
在 greedy decoding 下，可以直观理解为逐位置比较 draft token 是否与 Target 的选择一致；一旦不一致，就在该位置使用 Target 的结果。在概率采样下，需要使用与配置相匹配的拒绝采样流程，才能保持目标采样分布。
因此，α=min(1,p(x)/q(x)) 是"Draft 按概率分布 q 采样"场景下的经典接受概率，但不能不加条件地套到所有 EAGLE 配置上。当前 vLLM 的 draft_sample_method 默认值是 greedy，验证侧则由标准 rejection sampler 结合实际采样参数处理。
5. 性能收益取决于什么
EAGLE 的收益来自一个权衡：用较便宜的 Draft 计算，换取更少的 Target 串行 Decode Step。决定最终加速效果的不是某一个指标，而是以下因素共同作用：
- Draft 成本： Drafter 越轻、生成 K 个候选越快，额外开销越小。
- 接受率： 提出的候选中有多少被 Target 接受。
- 平均接受长度： 每轮验证实际输出多少 Token，包含 bonus token。
- 候选 K 的大小： 候选 K 太小，减少的串行步数有限；候选 K 太大，Draft 与验证开销会上升。
因此，"候选越多越快"是不成立的。vLLM 的动态投机解码文档特别指出，验证阶段的有效 batch 可近似理解为 BS × K；并发度升高后，过大的 K 可能让验证从 memory-bound 转向 compute-bound，反而使 TPOT 变差。实际部署应基于目标模型、硬件和业务并发测量 K，而不是照搬固定值。
6. EAGLE 与 KV Cache 的关系
KV Cache 与 EAGLE 优化的是两个不同维度。KV Cache 复用已经计算过的历史 K/V，避免每次 Decode 重算全部上下文；EAGLE 尝试预测未来 Token，让一次 Target Verification 推进多个位置。
实现上，Target 与 Drafter 还可能分别维护自身需要的 KV Cache。EAGLE 并不是 KV Cache 的替代品，而是在 KV Cache 基础上进一步降低 Decode 的串行性。
7. 适用场景与局限
EAGLE 更适合 Decode 占比高、中低 QPS（Queries-per-second，并发）、单步 Decode 受内存带宽限制的场景，例如长文本生成、代码生成、对话和 Agent 长输出。若请求主要耗时在长 Prompt 的 Prefill，或者系统已经处于高并发、高计算利用率状态，端到端收益可能有限，甚至可能因为 Draft 和批量验证开销而出现负收益。
部署时还需要考虑 Drafter checkpoint 与 Target 的模型家族、词表、隐藏维度和辅助层配置是否匹配；Draft 模型会增加权重和 KV Cache 占用；EAGLE-3 还要求 Target 实现能够导出所需的辅助隐藏状态。
8. 一句话理解 EAGLE / EAGLE-3
EAGLE 利用 Target 已计算的隐藏特征，让轻量 Drafter 在特征空间中自回归预测未来 Token；EAGLE-3 进一步融合多个层级的 Target 隐藏特征。候选最终仍由 Target 批量验证，从而在保持 Target 生成语义的前提下，减少串行 Decode 轮数。
KV Cache：过去算过的，不要重复算
EAGLE：未来可能生成的，先低成本地猜
Target Verification：猜得对就批量接受，猜错就由 Target 纠正
9. 参考资料
- vLLM：docs/features/speculative_decoding/eagle.md
- vLLM：docs/features/speculative_decoding/README.md
- vLLM：docs/features/speculative_decoding/acceptance_metrics.md
- vLLM：docs/features/speculative_decoding/dynamic_speculative_decoding.md
- vLLM 实现：vllm/v1/spec_decode/llm_base_proposer.py
- vLLM 实现：vllm/model_executor/models/llama_eagle.py、llama_eagle3.py
- EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
- EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
1. Starting with Token Decode
When large-language models generate text, the Decode phase is heavily self-regressive: the next Token can only be calculated if the current Token is available. KV Cache can avoid double counting the context that has emerged, but does not eliminate the symbiotic dependency between Token.
Token 1 → Token 2 → Token 3 → Token 4 → …
For LLM, single Decode Step tends to deal with only a small number of Tokens, but has to read a large number of model weights, thus often presenting memory-born features. The aim of the speculation decodes is to get Target Forward to push as many Tokens as possible without skipping the Target Model verification.
2. Speculation decoded: Draft for guessing, Target for sentencing
The typical speculative decoder consists of a light Drafter and a Target Model that ultimately produces quality. Drafter first presented the highest number of K candidates, Token, Target Model, and then used Forward to calculate the logits for these positions in bulk, and to decide which candidates were acceptable.
Assuming that Drafter proposes AB C D, Target to judge that the first three candidates are acceptable and the fourth is unacceptable, the current round can at least export AB C and continue to produce results using Target at the rejected location.
Here's another detail that is often overlooked: Target usually produces a next position when checking K draft token.
When a maximum of K Token is drafted per round, the average length of acceptance is 1 to K+1; of which "1" is a bonus token created by Target itself.
3. Why EAGLE/EAGLE3
Ordinary small Draft Model, though cheaper than Target Model, is two independent models with Target. The smaller the model, the faster it is drafted, the less aware it is of the real tendency of Target to produce, the lower the acceptance rate; the larger the model, the higher the acceptance rate, the higher the drafting costs.
The entry point for EAGLE is that Target Model has created a very high information density hidden in the last round of calculations, and instead of leaving Drafter with history Token, the hidden feature of Target is given to Drafter.
3.1 EAGLE: Self-regressive extrapolation in characteristic space
EAGLE is known as Extrapolithm for Greener Language-model Efficacy. Instead of simply replicating a smaller version of Target Model, it trains a light-weight feature prediction network: input the calculated hidden state of the current Token embedding and Target Model, extrapolate the hidden feature vector of the future location, re-use the Target frozen LM head, and convert the projected feature into token probability / candidate token.
"Feature Extrapolation" is the key to understanding EAGLE. Token is the result of fragmentation, and hidden features contain more information about context. Drafter predicts using the characteristics of Target, which is usually closer to Target than a small model relying only on Token sequences, thus securing higher acceptance rates.
However, EAGLE did not naturally eliminate the serial dependence within Drafter. In v LLM default, if Mr. Drafter is configured to be the first candidate to recycle K-1 with a minor Draft Forward and gradually receive a follow-up candidate.
Target hitden state + current token
↓
Draft token 1
↓
Draft token 2
↓
Draft token 3
↓
Target one batch authentication
Only if the model is trained and visiblely enabled is the path to generate multiple draft tokens in parallel. You can't decode "speculation" token at a time equals "EAGLE at a time forward to produce more token."
3.2 EAGLE-3: From a single final feature to multilayered integration
EAGLE-3 continues the feature-level Drafter thinking, but further uses Target Model 's multiple auxiliary hides. Transformer layers, at different depths, carry a different focus of information: the shallower layer is closer to local and syntax information, and the mesmerization of the mesotic layer means that the deeper layer is closer to the final projection. By combining multilayer features, Drafter can obtain more information on conditions than a single terminal state.
It can't be simply "fixed to the last three." In vLM, the auxiliary layer is first configured by eagle aux hidden state layer ids etc. of checkpoint; if checkpoint is not specified, the common interface selects three default positions by model depth, close to the early, intermediate and post-production, respectively.
There is also a need to retain an exception to the level of achievement: part of EAGLE-3 checkpoint will set up the use aux hidden state=false, which is degraded to use only the last layer of output. So multi-layer characterization is a typical design for EAGLE-3, not a hard code that all checkpoints have to meet.
3.3 EAGLE AND EAGLE-3
The main improvement point of EAGLE-3 is that Drafter has acquired a richer Target feature, not the cancellation of Target Verification, or the natural conversion of all candidates to Token into a parallel generation.
Why not sacrifice to create correctness?
In the decoding of speculation, Drafter offers only candidates, and the final output is still determined by Target Model's logits and authentication algorithms. The direct consequence of Drafter's miscalculation is to accept that the length is shorter, not that the error Token is written unconditionally.
Under Gladys working, it is understood intuitively that a place-by-place comparison of draft token is consistent with Target 's selection; if inconsistent, use the results of Target at that location. In the case of probabilistic sampling, a denial process matching the configuration is required to maintain the target sample distribution.
Thus, alpha =min (1,p(x)/q(x)) is the classic acceptance probability in the "Draft by probability distribution q sampling" scenario, but cannot be applied to all EAGLE configurations without conditions. The current vLM default value for draft sample method is greeny and the authentication side is handled by standard rejection sampler in conjunction with actual sampling parameters.
5. What benefits depend on
EAGLE gains come from a trade-off between cheaper Draft calculations in exchange for fewer Target serials of Decode Step. It is not an indicator that determines the ultimate acceleration effect, but rather a combination of the following factors:
- Draft cost: The lighter Drafter, the faster the K candidate, the smaller the extra costs.
- Acceptance rate: How many of the candidates have been accepted by Target.
- Average length of acceptance: token, including bonus token, to verify the actual output per round.
- Size of candidate K: Candidate K is too small, with a limited number of fewer steps; Candidate K is too large, Draft and validation costs will rise.
So, "the more the candidates," it's not valid. The vLM Dynamic Speculation Decoding Document states, inter alia, that the valid watch at the validation stage can be approximated as BS x K; after the increase in hair, too large a K may allow validation to shift from memory-bond to computer-bond, but instead modify TPOT. The actual deployment should be based on target models, hardware and operations combined with measurements of K, rather than replicating fixed values.
6. EAGLE Relationship with KV Cache
KV Cache and EAGLE optimize two different dimensions. KV Cache reuses calculated history K/V to avoid every Decode recalculating all context; EAGLE tries to predict the future Token, allowing Target Verification to advance multiple positions.
In practice, Target and Drafter may also maintain their own KV Cache. EAGLE is not a substitute for KV Cache but a further reduction of Decode 's seriality based on KV Cache.
7. Applicable scenarios and limitations
EAGLE is better suited to the high- and medium-to-low QPS (Queries-per-second, both distributed) and one-step Decode scenarios limited by memory bandwidth, such as long text generation, code generation, dialogue and Agent long output. The end-to-end benefits may be limited if the request is mainly time-consuming in Prompt's Prefil, or if the system is already in a state of high cogeneration and high computing utilization, and may even result in negative returns due to Draft and bulk validation costs.
The deployment also needs to consider whether Drafter checkpoint matches the model family, vocabulary, hidden dimensions and assistive layer configuration of Target; the Draft model increases weights and KV Cache occupancy; and EAGLE-3 also requires Target to achieve the supporting hidden state necessary to export.
EAGLE / EAGLE-3
EAGLE uses hidden features calculated by Target to allow light Drafter to return to predict the future in the feature space; EAGLE-3 further integrates multiple layers of Target hidden features. Candidates are ultimately verified in Target batches, thereby reducing the number of serial Decode rounds, while maintaining the syntax of Target.
KV Cache: Do not double count
EAGLE: Possible future, low-cost guess
Target Verification: Accept the mass and correct the error by Target
References
- vLLM：docs/features/speculative_decoding/eagle.md
- vLLM：docs/features/speculative_decoding/README.md
- vLLM：docs/features/speculative_decoding/acceptance_metrics.md
- vLLM：docs/features/speculative_decoding/dynamic_speculative_decoding.md
- v LLM Achieved: vllm/v1/spec decode/llm base proposer.py
- v LLM Achieved: vllm/model executor/models/llama eagle.py,llama eagle3.py
- EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
- EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test
Auto-fetched and translated by AI Atlas. All rights belong to the original author.