【轻量化复刻大模型】Linear Attention 与 Sparse Attention 梳理
0x00 概要
0.1 新系列
准备开一个新系列：在 CPU（PC）上复刻（微缩）大模型。
为何要开这个系列？是因为：
- Transformer 书上市之后，很多读者问我 ，是否有可以运行的代码来配套书籍进行阅读。
- 有同学问我如何在没有GPU的情况下学习。
- 很多同学都没有GPU或者机器人可以实验（无论是在学校还是在企业）
- 绝大多数同学没有机会训练超大模型，这些只能在屈指可数的几个大公司才可以。
因此我们会对大模型（LLM & 机器人）进行学习，不但有博客，还有github项目 github.com/RossiLH/Mod… 作为配合使用。
对于项目的一些说明：
- 涉及 KIMI K3，DeepSeek V4，PI0.5，PI0.7 等模型。后续可能会跟踪新模型。
- 全 CPU 复刻，不涉及 GPU，也不涉及大规模训练（这是另外一个庞杂深邃的领域）和硬件。
- 本项目模型的组件尽量是原始大模型的缩小版，如果组件没有开源版本，则用近似组件替代。
- 会解释每个组件在解决一个什么问题、这个问题难在哪、为什么只能这么解、解法的原理是什么，以及组件之间如何咬合成一个整体。
- 每个主要组件都有可直接运行的 Gradio demo——无需 GPU。另有若干可视化脚本（plot_*.py），可生成静态图。
- 所有文件行号基于本地源码快照；源码后续变化时，行号可能移动，但函数名和调用关系仍可作为定位入口。
- 重点不是记名词，而是理解一条主线，比如：Prefix Cache 不再只是 "找到相同 token，然后搬 KV"；它变成 "找到共同前缀边界，并恢复一组语义一致、生命周期不同的模型状态"。
- 为了保证每个文章可以尽量独立成文，所以有些内容会在不同文章中都存在，可能在系列层面上存在冗余，还请大家理解。
- 参考项目：在开始这个项目之前，我去github找了一圈，发现一些类似的项目（有用于生产的，也有用于学习的），都非常出色。因此已经有类似项目，我就尽量借鉴。因此，本项目也可以认为是对这些项目的学习和二创。
- KIMI K3 的参考代码为：github.com/TimRots/kim… 为主要参考（拷贝）。另外三个项目被用来核对和补充。
- DeepSeek V4 的参考代码为：github.com/hebo1221/na…
- PI 的参考代码为：github.com/Physical-In…
这个项目和系列博客是我的一个尝试，也是我学习的过程。不确定是否满足读者的需求，也肯定有疏漏和谬误（尤其是demo和测试部分），我自己会反思改进，也希望大家提出宝贵意见。
0.2 注意力
因为会以 KIMI K3， DeepSeek V4 为例来进行，所以我们先来快速浏览其核心。
Linear Attention 的核心
- 用固定状态替代不断增长的历史 KV，把历史读取成本从 "按 token 扫描" 变成 "读取工作集"。
- 它的优势是效率、流式和长期状态；它的代价是容量、可寻址性和 Query 无关压缩。
Sparse Attention 的核心
- 保留可寻址历史，只让当前 Query 读取少量相关 token/block。
- 它的优势是精确检索、prefix reuse 和 Query 条件化；它的代价（举例）是 indexer、gather 和 KV Cache。
Kimi K3 的核心
- 用 KDA 处理大多数长序列状态混合，用 Gated MLA 提供周期性的精确全局交互，再用 MoE、AttnRes、低精度和原生多模态把它们组织成大规模 Agent 系统。
DeepSeek‑V4 的核心
- 用 local、CSA、HCA 组成多分辨率记忆系统：近处精确，中处选择，远处压缩。
0x01 总览：两条路线到底在压缩什么
1.1 Full Attention 的特色
标准 Full Attention 保留完整的 Key‑Value 历史。每来一个 Query，都可以重新决定历史中的每个 token 应该分多少权重：
它最强的地方是读取时仍然拥有选择权，代价是：
- prefill 需要大量 Query‑Key 交互；
- decode 要不断读取越来越长的 KV Cache；
- KV Cache 体积随上下文增长。
1.2 差别
Linear 和 Sparse 的差别可以浓缩成一句话：Linear 压缩"历史怎么存"；Sparse 压缩"这次读哪些历史"。
注：此处也将 CSA/HCA 纳入比对。
更深一层看，这其实是在设计一个 Memory Subsystem:
- Linear 的 state 像芯片里的 register file: 小、快、反复读写，但容量有限；
- Full/Sparse 的 KV 像 cache: 容量更大、地址可寻，但访问成本更高；
- K3 和 DeepSeek‑V4 做的不是单纯换一个公式，而是让模型记忆结构和硬件存储结构协同设计。
Linear 和 Sparse 的真正分歧，不是哪个公式更漂亮，而是要不要在读取时保留 "选择对象" 的权利。
1.3 深度思考：三条压缩轴
1.3.1 token 序列轴
压缩 token 数量:
- block;
- top‑k;
- CSA/HCA;
- 更长跨度的 Linear state。
1.3.2 hidden‑dim 通道轴
压缩每个 token 的表示宽度:
- MLA;
- 低秩 KV;
- 子空间映射。
1.3.3 模型深度轴
用更多层传播、重组和补偿每层丢失的信息。Linear 单层省下了 KV, 但可能把代价转移为:
- 更多层；
- 更多激活参数；
- 更长串行深度；
- 更高 TPOT。
所以应比较 "每层参数效率", 而不是只比较单层 FLOPs。
0x02 Full Softmax Attention：为什么它难以被完全替代
2.1 数学逻辑
对每个 Query，先用点积给 Key 打分：
再用 Softmax 变成一份总和为 1 的预算：
最后用这份预算加权平均 Value：
通俗理解：
- 点积：谁和当前问题相关；
- 缩放：避免维度变大后分数过于极端；
- Softmax: 所有 token 竞争一笔总预算；
- Value 加权：把相关内容合成输出。
2.2 为什么 Full Attention 成为基线
如果任务要求：
- 找到很久以前的某个精确 token；
- 区分多个相似实体；
- 根据不同 Query 对同一历史重新排序；
- 组合多个远程证据；
那么"保存所有 token，读取时再决定"天然可靠。
Linear 或压缩 Sparse 的问题都是：它们在 Query 到来前，已经提前做了一部分信息合并。如果这一步把需要的信息丢掉，后面再聪明的读取函数也无法恢复。
2.3 深度思考
深度思考：为什么固定精度下，任意未来 Query 需要较大的存储？
如果要求对所有未来 Query 都保持通用正确性，就不能提前知道未来会访问哪一个 token。最坏情况下，几乎每个历史 token 都可能成为 needle, 需要保留足够的独立信息和地址。
因此:
- Full/Sparse 需要随 T 增长的可寻址存储；
- Linear 把它压成 state, 必然依赖任务分布假设；
- CSA/HCA 处于二者之间，用 压缩槽换取折中。
0x03 Linear Attention：把历史提前汇总
3.1 Linear 路线
Full Softmax
-> 核化 Linear Attention
-> 卷积 + SiLU, 补局部上下文和非线性
-> Mamba‑2, 加入输入相关记忆寿命
-> DeltaNet, 沿当前 Key 定向改写
-> GDN, 全局遗忘 + 定向改写
-> KDA, 逐通道衰减 + GPU 友好分块
-> KDA + MLA, 固定状态 + 可寻址检索
每一步的目的：
每次改进都没有消灭固定状态，而是在 "如何写、如何擦、如何忘、何时精确检索" 上做补偿。
3.2 数学本质
Linear Attention 将相似度近似为可分解形式：
于是：
令：
历史就变成一个固定大小的状态，Query 到来时只需读取 。
左边看起来像：
每个 Query 都扫描历史
右边变成：
历史先求和 → 得到状态 → Query 读取状态
因此，左式能做到每步近似 读取的唯一原因是：求和被移到了 Query 之前。
这同时意味着一个关键牺牲：你把 "先看 Query 再决定保留什么" 的权利卖掉了，换来固定大小状态。
**深度思考：为什么 Linear 的 O(1) 读取等于放弃 Query 选择权？**因为历史已经在 Query 出现前被求和成状态。之后 Query 只能在状态矩阵的方向中读取，无法回到原始 token 列表重新挑选某个对象。
Sparse 则相反：它保留 token 条目，Query 可以选择对象，只是选择过程要付出 indexer 成本。
3.3 Vanilla Linear Attention
3.3.1 它解决了什么
相对于 Full Attention:
- 历史不再为每个 Query 重复扫描；
- decode 的持久状态不随 T 增长；
- 适合流式输入；
- 可以用固定状态处理超长序列。
3.3.2 它的问题
Vanilla 通常是不断累加：
问题也很直观：
- 不会主动删除旧信息；
- 相互冲突的 Key‑Value 会叠在一起；
- 历史越长，状态越像不断往同一个登记簿上写字；
- 一阶近似可能产生负权重，失去 Softmax 的概率语义；
- 固定状态无法保存无限多个独立事实。
3.3.3 "压通道轴" 和 "压 token 轴"
Linear 主要压的是状态通道：
每个 token 最后还能以多少维信息留下来。
”
Sparse 主要压的是 token 数量：
这次到底保留多少个 token 参加注意力。
”
所以二者并非同一种压缩：
- 压通道轴：改变 "每个东西看得多清"。
- 压 token 轴：改变 "有几个东西可挑"；
深度思考：token 轴和通道轴压缩的区别
- hidden‑dim 轴：每个 token 保留多宽；
- token 轴：留下多少个 token/block;
- 模型深度轴：用多少层传播和补偿信息。
token 轴压缩直接影响 "找不得到谁"; 通道轴压缩更直接影响 "找到后看得多清"。
3.4 DeltaNet：从累加改为纠错
DeltaNet 把状态看作一个在线学习模型。先用当前 Key 查询旧状态：
再用真实 Value 和预测的差值更新：
可以把它理解为：
- 先看旧账本对当前 Key 的预测；
- 把当前方向上的错误擦掉；
- 写入新的 Value。
3.4.1 相对于 Vanilla 的改进
- 从 "只写不擦" 变成 "先纠错再写"；
- 新值可以覆盖旧关联；
- 状态污染减轻；
- 具有在线回归的解释。
3.4.2 局限
- 擦除通常沿当前 Key 方向；
- 两个 Key 很相似时，擦一个可能误伤另一个；
- 写入顺序会影响结果；
- 没有解决固定状态容量；
- 没有让未来 Query 参与历史压缩。
3.5 GDN：加入全局遗忘
Gated DeltaNet 在 DeltaNet 上增加衰减：
它包含三步：
旧状态整体衰减 → 当前 Key 方向擦除 → 写入新的残差
3.5.1 相对于 DeltaNet 的改进
DeltaNet 擅长 "快速改写当前方向"，但不会主动清空全局旧状态。GDN 增加了输入相关的记忆寿命：
- 不再需要等某个新 Key 恰好撞上旧信息；
- 很久不用的信息可以慢慢消失；
- 状态不会无限累积。
3.5.2 局限
GDN 的一个 head 通常只有一个标量 decay：
这相当于假设状态空间各个方向共享同一个遗忘速度。可是实际信息的寿命不同：
- 语法局部信息需要快速更新；
- 主题信息需要长期保留；
- 实体绑定可能要中等时间保留。
一个标量无法同时满足这些需求。
3.6 KDA：逐通道遗忘
KDA 将 GDN 的标量衰减改成逐通道衰减：
简化写法：
每个状态通道有自己的寿命：
- 有些通道接近长期记忆；
- 有些通道快速清空；
- 有些通道负责中短期上下文。
3.6.1 相对于 GDN 的改进
GDN 是 "一整个房间共用一个温控器"；KDA 是 "每个房间有自己的温控器"。
”
数学上， 出现在 Delta 梯度更新之前，意味着后面的纠错不是在同一个统一尺度上发生，而是在不同坐标上使用不同的有效步长。因此：
- GDN：逐 head 标量门，状态各向同性；
- KDA：逐 channel 门，状态可以各向异性；
- KDA 能让不同状态方向有不同学习速度和记忆寿命。
但 KDA 仍然没有解决：
- 固定状态的容量上限；
- Query 无关压缩；
- token 的独立地址；
- 相近 Key 的覆盖；
- 单步只能沿有限方向擦除。
0x04 Sparse Attention：保留 token 地址，减少读取对象
4.1 数学本质
Sparse Attention 先为 Query 选择支持集 ，再只在这个集合中做 Softmax：
它没有改变 Softmax 的竞争方式，只是删掉一部分参赛者。
因此 Sparse 的误差核心不是 “top‑k 这个数字多大”，而是：
被删掉的 token 原本拿走了多少注意力预算。
如果被删掉的概率质量很小，Sparse 误差可能很小；如果被删掉的 token 恰好是唯一的 needle，误差会很大。
”
4.2 Sparse 的发展脉络
Full Attention
-> 静态窗口/全局点
-> DSA 动态 indexer
-> HiLS 可学习块摘要
-> IndexCache 跨层复用
-> CSA 压缩后选择
-> HCA 强压缩后全读
每一步的核心矛盾：
静态模式: 便宜, 但不知道要找谁
动态模式: 知道要找谁, 但 indexer 变贵
块路由: 更规则, 但块内会带噪
压缩路由: 更省, 但信息先被混合
具体而言，每一步都解决前一代暴露出的新瓶颈：
深度思考：Sparse Attention 的完整发展脉络
Full Softmax: 保留全部 token, 精确但昂贵
静态 Sparse: 固定窗口/全局点, 访问便宜但与内容无关
DSA: 低成本 indexer 动态选择, 精度提高但索引变贵
HiLS: 让块摘要和选择直接接受任务损失训练
IndexCache: 跨层复用 indexer 结果, 降低重复选择成本
CSA: 先压缩 block, 再做稀疏选择, 降低对象数和访存随机性
HCA: 进一步压缩后全读, 取消 indexer, 但牺牲块内精度
4.3 DSA
DSA 通过低维、低精度 indexer 先筛选候选，再用较高精度注意力读取候选。
它的价值：
- 保留 token‑level 的动态选择；
- 不必对全部候选做完整高维点积；
- top‑k 让 Softmax 的注意力分布更尖锐。
它的问题：
- 如果 indexer 仍扫描全部历史，二次项只是变便宜，不是消失；
- top‑k 是硬截断，选错后无法补救；
- token gather 可能比 FLOPs 账面显示的更慢。
4.4 HiLS
简单 mean pooling 的块摘要会稀释块内的单个关键 token。HiLS 的方向是：
- 用更合理的可学习摘要；
- 让块质量和熵信息进入摘要；
- 让检索分数直接参与前向；
- 让最终语言模型 loss 训练选择器。
核心变化：
原来: 索引器负责选，主模型负责算
现在: 索引结果也参与主模型的任务损失
因此 “选谁” 不再只是模仿 Full Attention，而是直接由最终任务效果塑形。
4.5 IndexCache
相邻层的 top‑k 支持集通常有较高重叠（只在稳定层 / 对角区成立）。IndexCache 让：
- 少数层计算 indexer；
- 多数层复用已有索引。
它体现了 Amdahl 定律：当主注意力已经很便宜时，原本次要的 indexer 会成为新的瓶颈。
IndexCache 的限制是：局部索引重叠高，不代表最终输出误差一定小。因此索引复用策略应以端到端 loss / 任务指标校准，而不是只看相似度。
0x05 CSA 与 HCA：压缩后再做 Sparse
5.1 CSA 的本质
CSA 的流程是：
token KV
-> 块内可学习压缩
-> 压缩块 indexer
-> top‑k 压缩块
-> 精确注意力 + exact local window
5.1.1 相对于 DSA 的改进
- indexer 不再面对全部 token；
- KV Cache 也变小；
- 选择对象变成连续 block，访存更规则；
- 局部窗口保留近期 token 细节。
5.1.2 CSA 的核心代价
块内压缩发生在 Query 到来之前：
一个块里的多个 token -> 一个压缩槽
之后 Query 只能决定 “读不读这个块”，无法重新决定块内哪个 token 更重要。
所以 CSA 的误差有两部分：
- 选错压缩块；
- 压缩时已经把块内重要 token 混掉。
深度思考：Sparse Attention 的压缩为什么说是学习出来的池化？
CSA/HCA 首先把多个 token 聚合为一个压缩槽。这个槽不是简单固定平均，而是由内容、门控和位置共同决定的加权池化。因此 Sparse 的 "压缩" 不只是删 token, 也是在学习：
哪些 token 应该共同形成一个摘要
5.2 HCA 的本质
HCA 更激进：
大块 token -> 更少的压缩槽 -> 对所有压缩槽做稠密注意力 + exact local window
HCA 不再需要 top‑k indexer，因为压缩后的槽已经很少，全部读取仍然可接受。
5.2.1 相对于 CSA 的改进
- 去掉动态 indexer；
- 读取规则；
- 全局分支更便宜；
- 更适合作为低分辨率全局上下文。
5.2.2 HCA 的核心缺点
HCA 的支持集是全部压缩槽，所以读取阶段基本没有漏选问题；但所有损失都集中在池化阶段：
Query 到来前，L 个 token 已经混成一个槽；如果目标 token 被同窗噪声污染，读取端只剩一个不可再分的对象。
”
因此：
- HCA 的读取端相对完整；
- HCA 的写入端不可逆；
- 更换 Query 不能把被池化抹掉的 token 找回来。
深度思考：为什么 HCA 的损失全在池化阶段？
HCA 会读取全部压缩槽，因此没有 "selected block 漏选" 这一层损失。但 token 在压缩时已经混合。如果关键 token 和噪声进入同一槽，读取端只能得到混合结果，换一个 Query 也不能重新拆分。
深度思考：HCA 为什么容易发生块内污染？
因为 HCA 在 Query 到来前把同一窗口内的多个 token 混成一个槽。假设:
关键 token + 大量无关 token -> 一个 compressed slot
如果关键 token 的池化权重不够大，它的信息就被稀释。后续 Query 只能读取这个混合槽，无法回到原始窗口重新挑出关键 token。
这正是 HCA 用极低成本换取全局概览时付出的主要代价。
5.3 CSA/HCA 的分辨率分工
local window = 高分辨率近期记忆
CSA = 中分辨率选择性全局记忆
HCA = 低分辨率全局概览
DeepSeek‑V4 的精华不是某一个模块单独胜出，而是：
让不同注意力分支保存不同分辨率的记忆。
”
深度思考：Sublinear 压缩为什么要数据感知？
如果不同位置的信息重要性不同，固定比例压缩会浪费容量:
- 背景 token 可以压得很厉害；
- 关键实体和工具调用 token 需要高保真；
- 任务切换附近可能需要更高分辨率。
所以未来的 sublinear compression 更可能由数据和任务驱动，动态决定不同片段保留多少信息。
0x06 Linear 与 Sparse 的系统级比较
设上下文长度为 T，Linear 状态维度为 ，Sparse 选择 个 token/block，压缩比例为 ，局部窗口为 。
6.1 Prefill
6.1.1 Full/Sparse
prefill 有大量 Query，可以通过 tile 复用 KV，FlashAttention 等 kernel 能提高算术强度。
6.1.2 Linear
朴素递推有时间依赖。chunkwise 的作用是：
- 块内把多个递推步骤重新组织成矩阵计算；
- 块间只传递一个状态；
- 用结合律恢复一部分并行度。
6.1.3 CSA/HCA
多了一条压缩流水线：
- 先形成压缩槽；
- 再做索引或压缩槽注意力；
- 未完成的尾块需要临时缓存；
- block 边界必须和缓存布局协调。
6.2 Decode
Decode 常常是 memory‑bound：
- Full: 读取长 KV 流；
- Sparse: 读取 indexer + selected KV；
- Linear: 读取固定状态并写回；
- CSA/HCA: 读取更短、更规整的压缩 KV。
所以 Sparse 的 “少读” 只有在索引扫描和 gather 也足够便宜时才会兑现；Linear 的 “固定状态” 只有在状态能驻留较近缓存层级、并且递推串行不成为瓶颈时才会兑现。
6.3 Prefix Cache
Sparse/Full 的 KV 是 append‑only：
- token 可以按前缀树共享；
- 任意 token 边界都可复用；
- 分支请求只需引用只读 KV。
KDA 更适合保存 prefix 末尾的完整状态快照：
- 状态与前缀长度无关；
- 但快照通常需要复制到新请求；
- 中间任意分支点可能需要重放；
- speculative decoding 需要状态回滚。
所以：Sparse 的 prefix reuse 更像共享文件；Linear 的 prefix reuse 更像复制一个程序快照。
深度思考：Sparse 没有主动遗忘，Linear 是否因此更强？
Sparse 的 KV 是 append‑only, 通常不擦除历史，而是让 indexer 在读取阶段不选择某些 token。
Linear 的确有更强的状态更新和遗忘归纳偏置，可能更适合:
- 流式序列；
- 长期状态；
- 近因任务；
- 持续在线更新。
但 Linear 付出的代价是:
- 固定容量；
- 不可寻址；
- Query 无关压缩；
- prefix 和 speculative state 管理复杂。
因此不是谁绝对更强，而是记忆任务不同。
6.4 "Linear + Sparse" 值得研究
Linear 擅长隐式、长期、低带宽记忆；Sparse 擅长显式、可寻址、按 Query 检索。
一个自然的组合是：
Linear state: 长期趋势、主题、语法和状态
Sparse/MLA: 精确 token 检索
local window: 短程细节
真正需要优化的不是 “某个 attention 公式最优”，而是：
压缩应该插在哪个轴、哪个时间点、哪个层级，才能尽量不剥夺读取端的选择权。
”
深度思考："把压缩插在哪一段" 为什么重要？
压缩放在不同阶段，损失不同:
- 写入前压缩：影响所有未来 Query;
- 读取前压缩：当前 Query 仍可能先选择；
- token 轴压缩：减少可选择对象；
- 通道轴压缩：降低每个对象的分辨率；
- 层间压缩：可能用更多深度补偿。
因此真正的问题是压缩位置和信息选择权，而不是 "压缩率越高越好"。
TransFormer-封面
0xFF 参考
48 小时啃完 Kimi K3：一个模型装得下 22580 个 GPT-2
详解 DeepSeek V4：Infra 巨鲸 “四连击”，百万上下文走进现实
详解 Kimi K3：强到冲击 Anthropic 估值的模型什么样？丨晚点播客
本文使用 markdown.com.cn 排版
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Linear Attention and Sparse Attention combo
0x00 Summary
0.1 New Series
Prepare a new series: Recapture (micro) LLM on CPU(PC).
Why did you open this series?
- After the Transformer book was put on the market, a lot of readers asked me if I could run a code to accompany the book.
- Some kids asked me how to learn without GPU.
- Many students don't have GPU or robots to experiment with.
- The vast majority of students do not have the opportunity to train over LLM, which can only be done in a few large companies.
So we learn from LLM (LLM & Robots), not only blogs, but also Github.com/RossiLH/Mod... as a combination.
Some descriptions of the project:
- Models such as KIMI K3, DeepSeek V4, PI0.5, PI0.7 etc. New models may be followed up.
- All CPUs, not GPUs, not large-scale training (an additional area of depth) and hardware.
- The components of the project model are, as far as possible, a reduced version of the original LLM and, if they do not have an open-source version, are replaced by approximate components.
- Explains how each component solves a problem, where it is difficult, why it can only be so solved, what the rationale for the solution is, and how the components combine into a whole.
- Each major component has a direct running Gradio demo - no GPU. There are also several visual scripts (plot *.py) that generate static maps.
- All file line numbers are based on local source snapshots; if the source code changes, the line numbers may be moved, but the function name and call relationship can still serve as the location entry.
- The focus is not on naming, but on understanding a main line, such as Prefix Cache, which is no longer just, "Find the same token, then move KV"; it becomes, "Find a common prefix border and restore a group of semantic models with different life cycles."
- In order to ensure that each article can be written as independently as possible, some elements will be present in different articles and may be redundant at the level of the series.
- Reference project: Before starting this project, I went to Github to look for a circle of similar projects (both for production and for learning) which were excellent. So there are already similar projects, and I try to draw on them. The project can therefore also be considered a learning exercise and a two-in-one exercise on these projects.
- The reference code for KIMI K3 is: Github.com/TimRots/kim... as the main reference (copy). Three other projects were used to reconcile and supplement.
- DeepSeek V4 reference code: github.com/hebo1221/na...
- PI 's reference code is: Github.com/Physical-In...
This project and series of blogs are an attempt and a learning process. Uncertainty as to whether the needs of the reader are being met, and certainly there are omissions and errors (especially in the demo and test sections), and I myself will reflect on improvements, and I would like to see valuable comments.
0.2 Attention Power
Because it's going to take the example of KIMI K3, DeepSeek V4, so we're going to go through its core quickly.
Linear Attention core
- Replace the growing history with a fixed state of KV, turning the cost of historical reading from a "token scan" to a "read the work set".
- Its advantages are efficiency, fluidity and long-term state; its costs are volume, addressability and Query's non-compression.
Sparse Attention core
- Keep addressable history and allow the current Query to read only a few associated token/blocks.
- Its advantages are precise searching, prefix reuse and Query conditionality; its costs (for example) are indexer, gather and KV Cache.
Kimi K3 core
- Processing of most long-sequence state mixing with KDA, providing accurate global periodic interactions with Gated MLA, and organizing them into large-scale Agent systems with MoE, AttnRes, low precision and primary multimodules.
DeepSeek -V4 core
- Composed of multi-resolution memory systems using local, CSA, HCA: close precision, medium selection, remote compression.
0x01 Overview: What are the two routes compressing?
1.1 FullAttention features
Standard FullAttention preserves the full Key-Value history. Every Query comes to decide again how much weight each token in history should have:
Its strongest is that it still has the right to choose when reading, at the cost of:
- Prefill requires a lot of Query-Key interaction;
- Decode, keep reading the longer KV Cache;
- KV Cache grew with context.
1.2 Difference
The difference between Linear and Sparse can be condensed into one sentence: Linear compresses "how history survives" and Sparke compresses "what history this read."
Note: CSA/HCA is also included in the comparison.
More deeply, it's actually designing a Memoory Subsystem:
- Linear's state is like a register file in a chip: small, fast, repeated, but limited;
- KV for Full/Sparse, like carche: larger capacity, more addressable, but more expensive to access;
- K3 and DeepSeek-V4 do not simply change the formula, but rather coordinate the design of model memory structures and hardware storage structures.
The real difference between Linear and Sparse is not whether the formula is more beautiful, but whether to retain the right to "choice."
1.3 Deep thinking: three compression axes
1.3.1 Token series Axis
Compress token number:
- block;
- top‑k;
- CSA/HCA;
- Longar state.
1.3.2 Hidden-dim channel Axis
Compress the width of each token:
- MLA;
- Low KV;
- Subspace map.
1.3.3 Model depth axes
More layers of information are disseminated, reorganized and compensated for each layer. Linear has saved KV on one level, but may shift the cost to:
- More layers;
- More activation parameters;
- Longer serial depth;
- Higher TPOT.
So we should compare "Efficiencies per layer," not just single layer FLOPS.
0x02 Full Softmax Attention: Why can't it be completely replaced
2.1 Mathematical Logic
For each Query, put a little bit on Key first.
Use Softmax to become a sum of 1 budget:
Finally, using this budget weighted average
Common understanding:
- Point size: who is relevant to the current problem;
- Zooming: to avoid over-extreme scores after a larger dimension;
- Softmax: all tokens compete for a total budget;
- Value weights: synthesizing content.
2.2 Why Full Attention becomes a baseline
If the mandate requires:
- Find some exact token long ago;
- Distinguishing between several similar entities;
- Reordering the same history according to different Query;
- Combining multiple remote evidence;
So, "Save all token, read and decide" is natural.
Linear or Sparse's problems are that they have been able to consolidate some of the information in advance of Query's arrival. If this step throws away the information that is needed, then the smartest reading function can not be restored.
2.3 Deep thinking
Depth thinking: Why do any future in fixed precision Query need a larger storage?
If all future Querys are required to be universally correct, they will not know which token will be visited in the future. At worst, almost every history token can become a need to maintain sufficient independent information and addresses.
Therefore:
- Full/Sparse needs a searchable storage with T growth;
- Linear presses it into state, necessarily relying on task distribution assumptions;
- CSA/HCA is between the two in exchange for a compromise in a compressed slot.
0x03 Linear Attention: an advance summary of history
3.1 Linear Route
Full Softmax
NUCLEAR Attention
- > Volume + SiLU, plus local context and non-linear
- > Mamba2, add relevant memory life
DeltaNet, rewrite the current Key orientation
GDN, Global Forget + Orientation Rewrite
KDA, channel-by-channel decay + GPU friendly segment
->KDA + MLA, fixed state + searchable
The purpose of each step is:
Each improvement does not eliminate a fixed state, but compensates for "how to write, how to rub, how to forget, when to search accurately".
3.2 Mathematical Nature
Linear Attention approximates similarity to decomposition:
So:
Your orders:
History becomes a fixed-size state, and Query only needs to read when it comes.
The left looks like:
Every Query scans history
Right turn to:
History pre-summation get state Query read state
So the only reason left can be close to reading every step is because the sum is moved to Query.
And that means a key sacrifice: you sold the right to look at Query and decide what to keep, in exchange for fixing your size.
** Deep thinking: Why is Linear 's O(1) reading equal to giving up Query 's choice? ** Because history has been demanded before Query appeared. Then Query can only be read in the direction of the status matrix and cannot return to the original token list to re-select an object.
Sparse, on the contrary: it keeps the token entry, and Query can select the object, except that the selection process costs the indexer.
3.3 Vanilla Linear Attention
3.3.1 What has it solved?
Full Act:
- History no longer repeats every Query;
- The persistence of decode does not follow the growth of T;
- Suitable for current input;
- Overlong sequences can be processed in fixed condition.
3.3.2 Its problems
Vanilla usually adds:
The problem is also intuitive:
- Will not proactively delete old information;
- The conflicting Key-Value folds together;
- The longer the history, the more the state is like writing on the same register;
- The proximity of the first step may result in the loss of the probability semantic of Softmax;
- Fixed status does not preserve an unlimited number of independent facts.
3.3.3 "Pressed Channel Axes" and "Pressed Token Axes"
Linear's main pressure is the state channel:
Each token lasts with how much information.
”
Sparke's main pressure is the number of tokens:
How many tokens have been retained this time?
”
So they're not the same compression:
- Thrust axis: Change "everything looks good."
- Token axis: Change "a few things to pick";
Deep thinking: the difference between the token axis and the channel axis compression
- Hidden-dim axis: how wide each token retains;
- Token axis: how many left token/block;
- Model depth axis: how many layers are used to disseminate and compensate information.
Token axle compression directly affects "who can't be found"; and channel axle compression is more directly affected by "how to see when found."
3.4 DeltaNet: from add-up to correct
DeltaNet sees the state as an online learning model. Query old status first with current Key:
Update with real Value and projected margin:
It can be understood as:
- See the current Key forecast of the old books;
- Erase errors in the current direction;
- Write to the new Valle.
3.4.1 Improvements relative to Vanilla
- From writing to writing;
- New values cover old associations;
- Reduction of state pollution;
- Have an explanation for the online return.
3.4.2 Limitations
- Erasing usually follows the current Key direction;
- Two Keys, when they're very similar, rub one with the potential to wrong another;
- The order of writing will affect the outcome;
- Failure to address fixed-state capacity;
- No future Query involved in history compression.
3.5 GDN: Joining the global forgotten
Gated DeltaNet adds decay on DeltaNet:
It consists of three steps:
The old state's overall decay, the current Key direction erases, and write to the new disability
3.5.1 Improvements relative to DeltaNet
DeltaNet is good at "quick rewriting the current direction" but does not proactively clear the old state. GNN increases the memory life associated with input:
- There is no longer a need to wait for a new Key to hit the old information;
- Information that has not been used for a long time can slowly disappear;
- State does not accumulate indefinitely.
3.5.2 Limitations
A head of GDN usually has only one standard:
This is equivalent to sharing the same oblivion speed across the hypothetical space. However, the life expectancy of the actual information varies:
- Local grammatical information needs to be updated quickly;
- The need for long-term retention of thematic information;
- An entity binding may be retained for a medium period.
A target cannot meet these needs simultaneously.
3.6 KDA: Channel by Channel
KDA replaces the GDN target decay with a channel-by-way decay:
Simplified:
Each status channel has its own lifetime:
- Some corridors are close to long-term memory;
- Some corridors are quickly cleared;
- Some corridors are responsible for short- and medium-term context.
3.6.1 Improvements relative to GDN
G.D.N. is "an entire room shares a thermostat"; K.D.A. is "each room has its own thermostat".
”
Mathically, before the Delta gradient is updated, it means that the subsequent error does not occur on the same uniform scale, but uses different effective steps at different coordinates. Therefore:
- GDN: head-by-head marker door, in all states of the same sex;
- KDA: à la channel door, in a state of varying degrees of opposite sex;
- KDA can give different degrees of learning and memory life.
But KDA is still unresolved:
- Maximum capacity in a fixed state;
- Query has nothing to do with compression;
- Independent address of token;
- Close Key coverage;
- One step can only be erased in a limited direction.
0x04 Sparse Attention: retain token address and reduce reading objects
4.1 Mathematical Nature
Sparse Attention first selects the support set for Query, then only does Softmax in this collection:
It doesn't change the way Softmax competes, it just deletes some contestants.
So the core of Sparse's error is not "top-k" but:
The deleted token had taken a lot of attention from the budget.
If the probability of being deleted is of very low quality, Sparse may have a very small error; if the deleted token is precisely the only need, the error will be very large.
”
4.2 Sparse development context
Full Attention
-> Static Window/Global Point
DSA dynamic indexer
- HiLS, learn summary.
IndexCache cross-layer reuse
CSA Compressed Select
- HCA, compress and read it all.
The core contradiction of each step:
Static mode: Cheap, but unknown.
Dynamic mode: Know who to look for, but indexer becomes expensive
Route: More regular, but with noise inside the block
Compressed router: much less, but information is mixed first
Specifically, each step addresses new bottlenecks revealed by the previous generation:
In-depth thinking: Full development of Sparky Attention
Full Softmax: Keep all token, precise but expensive
Static Sparse: Fixed Window/Global Point, cheap access but not content
DSA: Low-cost indexer dynamic selection, increased precision but more expensive index
HiLS: Block summary and selection for direct mission loss training
IndexCache: cross-layer reuse indexer result, reduce the cost of repeated selection
CSA: Compress block first, then make thin choices, lower the number of objects and access randomity
HCA: Read all after further compression, cancel indexer, but sacrifice the inner accuracy of the block
4.3 DSA
DSA screens candidates by low-dimensional, low-precision indexer, then reads them with higher precision.
Its value:
- Retain the dynamic selection of token-level;
- There is no need for a complete high-dimensional build-up of all candidates;
- top-k let Softmax's focus get sharper.
Its question:
- If the indexer still scans the entire history, the secondary item is cheap, not disappearing;
- Top-k is hard-cut and cannot be remedied by mistake;
- Token loader may be slower than FLOPS shows.
4.4 HiLS
Simple means summary dilutes a single key in the block. HiLS' direction is:
- More rational learning of summary;
- Access to summary for mass and entropy information;
- Direct access points forward;
- Let the final language model loss train the chooser.
Core changes:
It's the indexer.
Now: The index results are also involved in the mission loss of the main model.
So "select whom" is no longer just a copy of Full Act, but is directly shaped by the effect of the final mission.
4.5 IndexCache
There is usually a high overlap of top-k support sets in the adjacent layers (established only in the stabilization layer / diagonal area). IndexCache:
- Minority calculations indexer;
- Most layers are re-indexed.
It reflects the law of Amdahl: when the attention of the Lord is already very cheap, the secondary indexer becomes a new bottleneck.
IndexCache limits that local index overlaps and does not mean that the final output error is small. The index reuse policy should therefore be calibrated with end-to-end loss/ task indicators, rather than looking at similarities only.
0x05 CSA and HCA: compression before Sparse
5.1 Nature of CSA
The CSA process is:
token KV
- > Learn to compress
- ♪ Compression block indexer ♪
- ♪ Top-k, compression ♪
- Exact local window.
5.1.1 Improvements relative to DSA
- indexer no longer faces all token;
- KV Cache gets smaller;
- The selection of objects becomes a continuous block, making access more rules;
- Local windows keep recent token details.
5.1.2 Core CSA Costs
The internal compression occurs before Query:
One block of multiple token-- a compression slot.
Then Query can only decide whether to read or not, and not which token is more important.
So the CSA error is two parts:
- Wrong compression blocks;
- When compressed, you've mixed the important token in the block.
Deep thinking: Why is the compression of Sparky Attention a learning pool?
CSA/HCA first aggregates multiple tokens into a compressed slot. The slot is not a simple fixed average, but a weighted pool determined by common content, door control and location. So Sparse's "compression" is not just deleting token, but learning:
Which token should form a summary together?
5.2 Nature of HCA
HCA More Radical:
Large token-- > Less Compressed-- > Keep a sharp eye on all condensed slots + exact local window
The HCA no longer needs top-k indexer because the compressed slots are small and all read is still acceptable.
5.2.1 Improvements relative to CSA
- Remove the dynamic indexer;
- Reading rules;
- The global branch is cheaper;
- better suited as a low resolution global context.
5.2.2 Core shortcomings of HCA
The HCA support set is the full compressor, so there is little leakage at the reading stage; however, all losses are concentrated in the pool phase:
Before Query arrived, the L token had been mixed into a slot; if the target token was contaminated with the same window noise, the reading end was left with only one inseparable object.
”
Therefore:
- The readend of HCA is relatively complete;
- The writing end of HCA is irreversible;
- To replace Query can't get token erased from the pool.
Deep thinking: Why is the loss of the HCA all at the level of the pool?
HCA will read all the compressor slots, so there is no "selected block out" loss of this layer. But token was already mixed at compression. If key token enters the same slot with noise, the reading end only gets mixed results, and one Query cannot reset split.
Deep thinking: Why is HCA vulnerable to internal pollution?
Because HCA mixed multiple tokens in the same window before Query arrived. Assumptions:
Key + mass irrelevant
If critical token's pool weight is not large enough, its information is diluted. The subsequent Query can only read this hybrid slot and cannot return to the original window to re-pick the key token.
That's the main cost of HCA in exchange for a global overview.
5.3 resolution Division of Labour of CSA/HCA
Local window = high resolution recent memory
CSA = resolution Selective Global Memory
HCA = global overview of low resolution
DeepSeek-V4 is not the essence of a single module, but of:
Let the different focus keep different memories of resolution.
”
Deep thinking: Sublinear, why data perception?
If information in different locations is of different importance, a fixed-scale compression can waste capacity:
- The background token can be pressurized;
- The need for high authenticity for key entities and tool call token;
- Mission transition may require higher resolution.
So future sublinear communications are more likely to be data- and task-driven, and dynamically determine how much information is left in different segments.
A system-level comparison between 0x06 Linear and Sparse
Set context length to T, Linear state dimension, Sparke selects a token/ block, compression ratio, local windows.
6.1 Prefill
6.1.1 Full/Sparse
Prefill has a large number of Querys that can increase arithmetical intensity by reusing KV, FlashAttention and so on.
6.1.2 Linear
PARK push has time to depend. The role of chunkwise is:
- Reorganization of multiple step-by-step steps into matrix calculations in blocks;
- Only one state is transmitted between blocks;
- Reactivate part of the parallel with a combination.
6.1.3 CSA/HCA
There's an additional compression line:
- The formation of compressors first;
- Re-index or compress attention;
- Unfinished tails require temporary cache;
- Block border must be coordinated with cache layout.
6.2 Decode
Decode is often memory-born:
- Full: Read long KV streams;
- Sparse: Read indexer + self-contained KV;
- Linear: Read the fixed state and write back;
- CSA/HCA: Read shorter, more structured compression KV.
So Sparse's “low read” will only be realized when index scans and loads are cheap enough; Linear's “fixed state” will only be realized if the state is closer to the cache level and the chain of push is not a bottleneck.
6.3 Prefix Cache
Sparse/Full's KV is append-only:
- Token can be shared by prefixing trees;
- Any token border is reusable;
- The branch request simply refers to KV only.
KDA better save complete state snapshots at the end of prefix:
- Status is not related to prefix length;
- But snapshots usually need to be copied to new requests;
- Any intermediate branch may need to be reopened;
- Speculate decoing needs status roll back.
So: Sparse prefix use more like shared file; Linear prefix use more like copying a program snapshot.
Deep thinking: Was Linear stronger because he didn't voluntarily forget?
Sparse's KV is an appendéonly, usually without erasing history, but instead of choosing some token at the reading stage.
Linear does have a stronger status update and forget to summarize bias, which may be more appropriate:
- Fluid sequences;
- Long-term status;
- Close cause tasks;
- Ongoing online updates.
But Linear paid the price:
- Fixed capacity;
- Non-location;
- Query has nothing to do with compression;
- Prefix and complex state management.
So not who's definitely stronger, but remembering things differently.
6.4 "Linear + Sparke" worth studying
Linear is good at hidden, long-term, low-bandwidth memories; Sparke is good at visible, searchable and searchable by Query.
A natural combination is:
Linear state: Long-term trends, themes, grammar and status
Sparse/MLA: Exact token search
Local window: Short-range details
What really needs to be optimized is not "some attention formula best" but:
Compression should be inserted at what axis, at what point in time and at which level so as not to deprive the reading end of its choice.
”
In-depth thinking: "Where does it fit in?" Why does it matter?
Compression at different stages with different losses:
- Pre-writing compression: affects all future Query;
- Pre-read compression: Query may still be the first option;
- Token axle compression: reduce optional objects;
- Channel axe compression: lower resolution per object;
- Inter-layer compression: may be compensated with more depth.
So the real problem is to compress position and information choices, not "the higher the rate."
TransFormer - Cover
0xFF Reference
48 Hours after eating Kimi K3: A model can fit 22580 GPT-2
DeepSeek V4: Infra Big Whale, Four Strikes, millions of context into reality
Kimi K3: What's the model of the Anthropic valuation? We'll be late.
This article is available in markdown.com.cn
Auto-fetched and translated by AI Atlas. All rights belong to the original author.