【轻量化复刻大模型 / KIMI K3】KDA：按 Channel 控制遗忘
0x00 概要
本文档是复刻 KIMI K3 KDA（Kimi Delta Attention）的第一部分，因为文章实在太长了，所以分为两期发出。
不同于 GDN 的 head-wise 粗粒度遗忘和 vanilla 线性注意力的无选择性累积，KDA 通过 Diag(α_t) channel-wise 有界衰减 + delta-rule 写入，在固定大小状态矩阵 S 上实现了可控记忆、可稠密 chunkwise 并行、且无位置编码也自洽的线性注意力。
注：
- 项目github.com/RossiLH/Mod… 和系列博客是我的一个尝试，也是我学习的过程。不确定是否满足读者的需求，我自己会反思改进，也希望大家提出宝贵意见。
- 涉及 KIMI K3，DeepSeek V4，PI0.5，PI0.7 等模型。后续可能会跟踪新模型。
- 全 CPU 复刻，不涉及 GPU，也不涉及大规模训练（这是另外一个庞杂深邃的领域）和硬件。
- 模型的组件尽量是原始大模型的缩小版，如果组件没有开源版本，则用近似组件替代。
- 会解释每个组件在解决一个什么问题、这个问题难在哪、为什么只能这么解、解法的原理是什么，以及组件之间如何咬合成一个整体。重点不是记名词，而是理解一条主线。
- 每个主要组件都有可直接运行的 Gradio demo——无需 GPU。另有若干可视化脚本（plot_*.py），可生成静态图。
- 所有文件行号基于本地源码快照；源码后续变化时，行号可能移动，但函数名和调用关系仍可作为定位入口。
- 为了保证每个文章可以尽量独立成文，所以有些内容会在不同文章中都存在，可能在系列层面上存在冗余，还请大家理解。
0x01 演进脉络与定位
1.1 前身
- Vanilla 线性注意力（Linear Transformer） ：S = Σ k v^T，矩阵加法可交换。缺陷：固定状态容量受限、表达力弱、无选择性遗忘。
- Gated Linear Attention（GLA） ：channel-wise gating 的代表，但沿用的是线性注意力（外积累积），没有 delta-rule 的"写入时先减去旧贡献"机制。
- Gated DeltaNet（GDN） ：把 delta rule（DeltaNet，Mamba-2 同源）加上 head-wise forget gate。缺陷：gate 是每个 head 一个标量，遗忘粒度太粗，无法对不同 feature 维度做差异化记忆控制。
KDA 的目标是：在 GDN 的 delta rule 基础上，把遗忘粒度细化到 channel 级——这需要在状态更新的转移矩阵中加入对角 gate Diag(α_t)，同时保持 chunkwise 可分块并行（DPLR / Diagonal-Plus-Low-Rank，对角加低秩结构）。
1.2 同期竞品
1.3 KDA 论文
KDA 有两篇直接相关的论文，本文档以技术报告（K3 报告）为主线、Kimi Linear 报告为算法细节补充：
- KDA 首先在 Kimi Linear 技术报告中被完整定义：它扩展 Gated DeltaNet（GDN），把粗粒度 head-wise forget gate 换成 channel-wise 对角 gate Diag(α_t)。
- K3 报告 §2.1.1 进一步给出 KDA 在 K3 中的实现细节（有界衰减 Eq.5、全秩输出门 Eq.6），并在 §5.1 系统阐述了 KDA 的算法-系统协同设计（FlashKDA、KCP、KDA-aware 前缀缓存）。
paper-eq-5
paper-eq-6
1.4 定位
Attention 的抽象不是"必须出现 Softmax"，而是：
根据 Key 决定如何读取 Value
KDA 通过状态矩阵完成：y_t = q_tᵀ S_t。所以它仍然是"Query 根据 Key-Value 关系读取内容"，只是：
- Full Attention: 保存 token 级 KV，读取时计算 Softmax；
- Linear/KDA: 保存压缩状态，读取时用核 / 矩阵乘法；
- KDA 还加入了带遗忘的在线更新。
因此可以说：KDA 是注意力的一种递推、核化、带门控的实现。 注意力的本质是内容寻址，不是 Softmax 这个具体函数。
1.5 KimiDeltaAttention
KimiDeltaAttention 代码如下：
class KimiDeltaAttention(nn.Module):
"""KDA 完整层：投影（q/k/v/beta/decay）+ 核心递推 + 输出门。
参数结构（报告 Eq. 2 / Eq. 5 / Eq. 6）：
- ``q_proj/k_proj/v_proj``：线性投影到每 head 的 key/value 维度。
- ``q_conv/k_conv/v_conv``：因果 depthwise ShortConv（宽度 4），给无位置编码的
模型注入 token 级"位移"信息（报告 §2.1.1；Kimi Linear 论文称其为位置编码替代）。
- ``beta_proj``：delta-rule 写入强度 ``beta = sigmoid(x)``，控制"本次写入覆盖
多少旧记忆"（``beta`` 接近 1 表示完全重写该方向的记忆）。
- ``alpha_down/alpha_up/alpha_bias``：**低秩** decay logits ``z = W_up W_down x + b``
（Eq. 2），逐 channel 的 log 衰减来源。
- ``a_log``：Eq. 5 的逐 head 对数尺度 ``A_h``，论文初始化为 0。
- ``o_norm``：head-wise RMSNorm（Eq. 6 的第一步）。
- ``g_proj``：全秩、依赖输入的输出门 ``sigmoid(W_g x)``（Eq. 6 的第二步）。
"""
def __init__(self, dim: int, cfg: KDAConfig, *, layer_idx: int = 0, eps: float = 1e-6):
super().__init__()
self.dim = dim
self.cfg = cfg
self.layer_idx = layer_idx
h, dk, dv = cfg.num_heads, cfg.head_dim_k, cfg.head_dim_v
self.num_heads, self.head_dim_k, self.head_dim_v = h, dk, dv
k_dim, v_dim = cfg.k_dim, cfg.v_dim # k_dim = H*Dk, v_dim = H*Dv
# --- 线性投影到未分 head 的宽向量，再交给 ShortConv 做局部混合 ----------
self.q_proj = nn.Linear(dim, k_dim, bias=False)
self.k_proj = nn.Linear(dim, k_dim, bias=False)
self.v_proj = nn.Linear(dim, v_dim, bias=False)
self.q_conv = ShortConv(k_dim, cfg.conv_kernel) # 宽度 4 的因果深度卷积
self.k_conv = ShortConv(k_dim, cfg.conv_kernel)
self.v_conv = ShortConv(v_dim, cfg.conv_kernel)
# --- delta-rule 写入强度 beta 与低秩衰减 logits ----------------------
self.beta_proj = nn.Linear(dim, h, bias=False) # -> (B, T, H) 每 head 一个标量
# 低秩 decay logits（Eq. 2）：z = W_up W_down x + b_alpha。
self.alpha_down = nn.Linear(dim, cfg.alpha_rank, bias=False) # 压到 rank
self.alpha_up = nn.Linear(cfg.alpha_rank, k_dim, bias=False) # 展开回 k_dim
self.alpha_bias = nn.Parameter(torch.empty(k_dim)) # 逐 channel 偏置
# Eq. 5 的逐 head log-scale A_h；论文初始化为 0。
self.a_log = nn.Parameter(torch.zeros(h))
# --- 输出门（Eq. 6）：head-wise RMSNorm + 全秩依赖输入的门 ------------
self.o_norm = HeadRMSNorm(dv, eps)
self.g_proj = nn.Linear(dim, v_dim, bias=False) # 全秩输出门
self.o_proj = nn.Linear(v_dim, dim, bias=False)
self.reset_parameters()
def forward(
self,
x: Tensor,
*,
reset_mask: Tensor | None = None,
cache: KDACache | None = None,
initial_state: Tensor | None = None,
) -> tuple[Tensor, Tensor]:
"""``x`` 是 ``(batch, seq, dim)``。返回 ``(output, final_state)``。
形态选择：训练/prefill 走 chunkwise（块内并行、块间串行传递状态）；
自回归 decode（``t == 1`` 且已有状态）走单步 ``kda_step``。两种形态数值上
完全等价（``kda_ref`` 里有 ``kda_recurrent`` 作为 ground truth 校验）。
"""
b, t, _ = x.shape
q, k, v, beta, g, conv_states = self._project(x, cache)
# (B, T, H, D) -> (B, H, T, D)：递推在 T 维进行，把 head 提到中间便于广播。
qh, kh, vh = (z.transpose(1, 2) for z in (q, k, v))
gh = g.transpose(1, 2)
bh = beta.transpose(1, 2)
state_in = initial_state if initial_state is not None else (cache.state if cache else None)
# 中间计算用 fp32（activation 是 bf16/fp16 时），递推状态累积不丢精度。
work_dtype = torch.float32 if x.dtype != torch.float64 else torch.float64
chunk = self.effective_chunk_size(work_dtype)
if t == 1 and state_in is not None:
# decode 形态：单 token 单步更新。注意此分支只在"有先前状态"时启用，
# 保证 decode 的第一 token 也能走 chunkwise（初始状态为 0）。
out, state = kda_step(
qh[:, :, 0].to(work_dtype),
kh[:, :, 0].to(work_dtype),
vh[:, :, 0].to(work_dtype),
gh[:, :, 0].to(work_dtype),
bh[:, :, 0].to(work_dtype),
state_in.to(work_dtype),
)
out = out.unsqueeze(2) # (B,H,1,Dv)
else:
# prefill / 无状态 decode：chunkwise 并行形式（块内并行，块间串行）。
fn = self._kernel()
out, state = fn(
qh,
kh,
vh,
gh,
bh,
chunk_size=chunk,
initial_state=state_in,
reset_mask=reset_mask,
dtype=work_dtype,
)
# Eq. 6 输出门：head-wise RMSNorm -> 全秩依赖输入的门 -> 输出投影。
out = out.transpose(1, 2).reshape(b, t, self.num_heads, self.head_dim_v)
out = self.o_norm(out.to(x.dtype)).reshape(b, t, -1)
out = out * torch.sigmoid(self.g_proj(x))
out = self.o_proj(out)
if cache is not None:
# decode 时把本轮状态写回 cache，供下一 token 使用。
cache.state = state
cache.conv_q, cache.conv_k, cache.conv_v = conv_states
return out, state
0x02 原理讲解
2.1 直觉类比视角
想象你是一位图书管理员，面前只有一个固定大小的"记忆白板"（状态矩阵 S），没有书架（KV cache）。
核心动作是"先按 channel 褪色，再按 delta 写入"。每来一本新书（token），你要做两件事：
- 先遗忘：用每本书每个主题自己的"褪色速度"（channel-wise decay α）把白板上旧内容整体变淡一点。
- 再更新：用新书内容按 delta rule 写上去——但写之前，先把你已经知道的新书内容"抹掉"原有信息（v − S^T k），只写新信息本身。这样不会重复累积。
关键区别：softmax 书架有无限大，能精确记住每本书；KDA 白板永远这么大，但每次都能精确地决定忘多少、更新什么。当上下文超过记忆容量时，KDA 靠与 MLA 全局层混合兜底——"小管家 + 定期大整理"。
2.2 专家视角
输入输出：
- 输入：x ∈ R^{B×T×d}，其中 B=batch、T=序列长、d=隐维度。
- 投影后：query/key q_t, k_t ∈ R^{d_k}（L2 归一化后），value v_t ∈ R^{d_v}，每 head 一个。
- 状态：S_t ∈ R^{d_k × d_v}，固定大小，不随 T 增长。
核心递推（Eq.1，单 head 视角） ：
S_t = (I − β_t k_t k_tᵀ) Diag(α_t) S_{t-1} + β_t k_t v_tᵀ (Eq.1)
- Diag(α_t)：channel-wise 衰减因子，α_t = exp(g_t)，g_t ≤ 0 是 log-decay。每一行（channel）独立遗忘。
- Diag(α_t) S_{t-1}：按 channel 衰减旧状态。
- β_t：delta-rule 写入强度 (0,1)，控制"覆盖旧记忆的方向"。
- (I − β_t k_t k_tᵀ)：秩-1 投影，把状态中与 k_t 对齐的成分按 β_t 削弱。即，沿 k 方向把旧状态中"和新 key 重叠的成分"减去——这就是 delta rule 的"写新前擦旧"。
- + β_t k_t v_tᵀ：写入新记忆。
输出：o_t = S_tᵀ q_t（对每个 head）。
等价改写（数值更稳，实际实现） ：
S_t = S_{t-1} + β_t k_t (v_t − S_{t-1}ᵀ k_t)ᵀ
即先计算"本次要写的新信息" = v_t 减去"旧状态沿 k_t 方向已记住的内容"，只写增量。
挖掘："选择性遗忘 + 在线学习率"的 delta rule 与线性回归的递归最小二乘同构，把 k_t 看作输入、v_t 看作监督、S 看作线性映射权重，S_tᵀ k_t 是预测，v_t − S_tᵀ k_t 是残差，β_t 是学习率。因此 KDA 把注意力建模成"可变学习率的在线最小二乘"，理论上表达力强于纯累积的线性注意力，这让它天然有"压缩即智能"的理论支撑。
2.3 KDA 原理图
先看公式（Eq.1）
S_t = (I - beta_t * k_t k_t^T) Diag(alpha_t) S_{t-1} + beta_t * k_t v_t^T
└─────────── 逐 token 过渡矩阵 ───────────────┘ └── 写入新信息 ──┘
alpha_t = exp(g_t) 逐 channel 保留因子（0 < alpha <= 1）
K3 架构总览，左下为 KDA 模块图
KIMI-K3-Arch
2.4 深入讨论
- 理论上为什么 work：KDA 把状态转移矩阵参数化为 DPLR 特例——Diag(α_t)（对角）+ β_t k_t k_tᵀ（秩-1）——这正是"可 chunkwise 并行"的结构前提。相比一般 DPLR，KDA 的转移矩阵与经典 delta rule 一致（S 的列沿 key 方向衰减），物理语义清晰（每列是"key 记忆的遗忘曲线"）。
- 下界衰减的必要性：Kimi Linear 用无界 softplus g = −exp(A)·softplus(z)，累积衰减乘积下溢、其倒数（K/Γ 的重缩放）上溢，迫使对角 tile 走显式 position-pair 路径；K3 改用 g = g_min·sigmoid(exp(A)z) 把 log-decay 锁在 (g_min, 0)，g_min=−5 时 exp(80)=5.5e34 仍在 BF16 范围内——每个因果 tile（含对角）都能用稠密 Tensor-Core matmul 计算。
- NoPE 自洽性：KDA 的衰减和 delta 更新天然位置感知（recency 由衰减累积编码），因此 MLA 层无需 RoPE 也能全局混合；上下文扩展到 1M 不需要任何位置参数手术。
- 论文未解决的理论问题：固定状态容量对超长程精确检索的根本上限（论文承认线性注意力检索弱，用 MLA 混合弥补）；状态矩阵的容量-精度权衡（state expansion 方向仍在探索）。
2.5 图例
图例如下：
plot_kda_principle
图中 2×2 四格，全部由真实递推（kda_recurrent，全项目 KDA ground truth）跑出：
左上 —— 状态矩阵 S（记忆） ：KDA 的"记忆"就是一个固定大小的 Dk × Dv 矩阵。图中热图是真实跑完 32 个 token 后的最终状态；T=32 个 token 只产生一个 16×16 矩阵——这就是它对比 KV cache 的核心卖点：内存 O(1)，不随序列长度增长。
右上 —— 单步三件事拆解：每来一个 token，状态只做三件事：
- 衰减 Diag(alpha) S：旧记忆先按通道乘一个 alpha=exp(g)<=1 的保留因子（旧信息"忘掉一点"）；
- 检索 S^T k：用当前 key 在记忆里"查一下"已存了什么；
- 写入 beta * k (v - S^T k)^T：只把误差 v - 检索到的记忆 写进状态（delta rule 的核心）。
左下 —— delta rule 的"纠错"本质：人为构造"同一个 key 反复出现、v 逐步变化"的场景，展示 key 检索到的值如何收敛到最新的 v。普通线性注意力（S += k v^T）会"混在一起"，而 delta rule 通过 v - S^T k 把冲突压下去——这就是"记忆能持续更新、不互相污染"的原因。图中三条线是真实 kda_step 跑出的检索值。
右下 —— 有界 log-decay（Eq.5） ：g = g_min * sigmoid(exp(A) z) 把 log-decay 严格锁在 (g_min, 0)，即 g_min=-5 时 g ∈ (-5, 0)。这个"下界"是 Q4/Q5/Q6 的数值稳定性前提。
一句话结论：KDA = 固定大小的记忆矩阵 S（O(1) 内存）+ 每 token 三件事（衰减→检索→只写误差）+ 有界衰减（保证 chunkwise 不溢出）。与标准 MHA 的 KV cache 随 T 线性增长 形成本质对比。
2.6 各种机制对比
这张表的主线是：不同方法主要在两个地方产生差异：
- 旧状态如何衰减；
- 当前 Value 如何写入以及是否使用残差纠错。
线性注意力没有遗忘和纠错；DeltaNet 增加残差写入；GDN 增加按 Head 的统一遗忘；KDA 则把遗忘细化到每个 Key 通道。
0x03 子知识点拆解
KDA（Kimi Delta Attention）
├── 状态递推与 delta rule（核心算子）
├── 有界衰减 log-decay（Eq.5）
├── channel-wise decay 与低秩参数化（Eq.2）
├── ShortConv（无位置编码的局部混合）
├── chunkwise 并行（Eq.3-4，DPLR）
├── 单步递推 kda_step（decode 形态）
├── 输出门（Eq.6）
├── 解码缓存 KDACache 与 clone() 分支
└── KDA 上下文并行（KCP，报告 §5.1.2）
代码对应如下：
use_triton=False 时 _kernel() 返回纯 PyTorch kda_chunkwise。
统一符号约定（本文档所有小节遵循）：
- S_t ∈ R^{d_k × d_v}：递推状态矩阵（每 head 一个）。
- q, k ∈ R^{d_k}（已 L2 归一化）、v ∈ R^{d_v}。
- g_t ∈ R^{d_k}：逐 channel log-decay（≤0）；α_t = exp(g_t) 为 retention 因子。
- β_t ∈ (0,1)：delta-rule 写入强度。
- chunk 尺寸 C（配置 64，tile 16），tile 是数值安全的基本衰减块。
0x04 子知识点深度分析
4.1 状态递推与 delta rule（Eq.1）
KDA 保留了 Delta Rule 的残差覆写机制，但把 GDN 的标量遗忘门扩展成了一个 d_k 维向量。
4.1.1 原理
KDA 的基本假设是：在每一层、每一个 Head 中，每个请求都维护一个固定大小的状态矩阵 S_t。这个矩阵代表当前请求到目前位置为止的全局记忆。它不是为每个历史 Token 单独保留一条记录，而是把所有历史 Token 的上下文信息压缩到同一个矩阵里。
当 KDA 处理当前 Token 时，模型会根据当前 Token 的 Hidden State 和对应的投影参数，计算出：
- q_t：Query，用于从记忆中读取信息；
- k_t：Key，用于定义寻址模式；
- v_t：Value，用于提供要写入的内容；
- α_t：遗忘门，决定各个 Key 通道保留多少旧信息；
- β_t：写门，决定当前残差写入多少；
- γ_t：输出门，控制输出信息的强度。
KDA 的状态更新可以写成：
(1) 先使用
中，遗忘门 决定旧状态中的不同 Key 通道保留多少。因为 α_t 是向量，所以并不是所有列都以相同比例衰减。第 j 个 Key 通道拥有独立的遗忘速度 α_{t,j} ，重要的通道可以保留更多，应该淡出的通道可以衰减得更快。
如果状态矩阵的形状是：。那么右乘 Diag(α_t) 会对状态矩阵的每一列分别缩放。每一列对应一个 Key 通道，因此 KDA 实现的是“按 Head、按 Key Channel”的遗忘。
门的规模也因此从 GDN 的：
num_heads
扩展为：
num_heads × head_dim
如果没有遗忘门，纯线性注意力可能只是：。由于矩阵加法满足交换律，输入顺序 [A,B] 和 [B,A] 会产生相同的最终状态。这意味着简单的累加状态天然缺少位置信息：它知道有哪些关联被写入，却不一定知道这些关联的先后顺序。KDA 中的遗忘可以部分恢复这种顺序信息。距离当前越远的 Token，经历的衰减次数越多，它的信号就可能越弱。因此，遗忘过程可以看成一种隐式的相对位置编码。
(2) 再执行 Delta Rule：。
是当前 Value 的写入，即当前 Token 带来的原始新知识。它把 Value v_t 和寻址模式 k_t 绑定起来，准备写入状态。
这一项会减掉衰减后的旧状态中、与当前 Key 模式匹配的那部分内容。它的作用不是简单地把新内容加进去，而是避免同一个特征在长上下文中被不断重复累加。先删除或修正旧的匹配内容，再写入新的目标 Value，状态因此具备更精确的覆写能力。
与标准做法的公式级对比：
4.1.2 对比基线
改动带来的收益：O(1) 显存、O(T) 时间；代价：检索变为压缩近似。
4.1.3 计算复杂度分析
KDA 是典型"递推序列模型"：训练/prefill 用全序列 chunkwise 前向（每个位置都要输出，用于算 loss）；推理 decode 用单步递推（自回归逐 token 生成，状态传递）。桥梁是 chunkwise（块内并行、块间串行）。
- FLOPs（单 token）：KDA = O(d_k·d_v)（状态更新 + 输出读出）vs softmax = O(T·d_k)（需与全部历史比较）。T 大时 KDA 完胜。
- 内存：KDA 状态 = d_k×d_v/head（固定）；softmax KV = T×(d_k+d_v)/head（线性增长）。
- 参数：KDA 无新增参数在递推本身（投影共享）。
4.1.4 参数敏感性
- head_dim_k（默认 128）：决定状态方阵大小与"key 记忆方向"的容量。调大→状态容量↑但 FLOPs↑；过小→容量不足。
- head_dim_v（默认 128）：value 信息容量。
- 论文经验：d_k=d_v=128 是 Kimi Linear 经过调参后的默认，配合 96 head（d=7168）。
4.1.5 反方辩论
- 最强反对理由：固定矩阵的压缩必然导致信息丢失，超长上下文的精确检索（如"在第 40 万 token 提到过的某个具体数字"）会失败。
- 若失败，最可能错在假设：假设"可压缩的记忆足够表达任务所需信息"。对需要精确逐 token 回溯的任务（复杂 agentic 轨迹、代码库检索），此假设脆弱。
- 胜出条件：记忆可近似、语义密集、长上下文且 token 语义冗余高（对话、长文总结）。失败条件：需要精确位置检索、稀疏的关键信息。
- 更稳健做法：状态扩容（state expansion）、或 KDA+MLA 混合比例调高（3:1 是论文最优，但任务敏感）。
- 受伤方：长程精确检索任务方；受益方：1M 上下文推理吞吐方。
4.1.6 工程实现
伪代码（逐行） ：
for t in 1..T:
state *= exp(g[:, :, t]) # Diag(alpha) S channel 衰减
sk = einsum('bhd,bhdv->bhv', k[:, :, t], state) # S^T k 预测
delta = beta[:, :, t] * (v[:, :, t] - sk) # 残差 × 学习率
state += k[:, :, t] * delta # rank-1 外积更新
out = einsum('bhd,bhdv->bhv', q[:, :, t], state) # 读出
项目真实代码（kda_ref.py 中 kda_recurrent） ：
def kda_recurrent(q, k, v, g, beta, *, initial_state=None,
reset_mask=None, dtype=torch.float64):
"""Eq.
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
KDA: Press Channel to control forgetting
0x00 Summary
This document is the first part of the recapitulation of KIMI K3 KDA (Kimi Delta Act), as the article is too long to be issued in two stages.
Unlike GDN, coarse-wise corrosive oblivion and the non-selective accumulation of linear attention from vanilla, KDA is written through Diag (α t) channel-wise with boundary decay + delta-rule, which achieves controlled memory, dense chunkwise parallel and uncodified linear linear attention on fixed-size matrix S.
Notes:
- Project Github.com/RossiLH/Mod... and series of blogs are an attempt and a learning process. I will myself reflect on improvements and hope that you will make valuable comments, in the absence of certainty as to whether the needs of the reader are met.
- Models such as KIMI K3, DeepSeek V4, PI0.5, PI0.7 etc. New models may be followed up.
- All CPUs, not GPUs, not large-scale training (an additional area of depth) and hardware.
- The components of the model are as narrow as possible from the original LLM and, if there is no open-source version of the components, are replaced by approximate components.
- Explains how each component solves a problem, where it is difficult, why it can only be so solved, what the rationale for the solution is, and how the components combine into a whole. The focus is not on naming, but on understanding a thread.
- Each major component has a direct running Gradio demo - no GPU. There are also several visual scripts (plot *.py) that generate static maps.
- All file line numbers are based on local source snapshots; if the source code changes, the line numbers may be moved, but the function name and call relationship can still serve as the location entry.
- In order to ensure that each article can be written as independently as possible, some elements will be present in different articles and may be redundant at the level of the series.
0x01 integrator and positioning
1.1 Front
- Vanilla Linear Transformer: S = k v^T, matrix added. Deficiencies: Limited fixed-state capacity, weak expression, non-selective memory.
- Gated Linear Attention (GLA): representative of Channel-wise gaing, but following linear attention (extra-cumulative), there is no delta-rule mechanism for writing "less old contributions".
- Gated DeltaNet: add delta rule (DeltaNet, Mamba-2 co-source) to head-wise forget Gate. Deficiencies: Gate is each head of a scale that forgets particles are too thick to allow for differential memory control of different feature dimensions.
The goal of KDA is to fine-tune the amount of forgotten particles to the channel level based on the delta rule of GDDN - This requires the insertion of a diagonal gate Diag (alpha t) in the state-update transfer matrix, while keeping the chunkwise split in parallel (DPLR / Diagonal-Plus-Low-Rank, diagonal low-column structure).
1.2 Concurrent competition
1.3 KDA Papers
KDA has two directly relevant papers, which are supplemented by the technical report (K3 report) and Kimi Linear report (K3 report) for algorithm details:
- KDA was first fully defined in the Kimi Linear technical report: it expands the Gated DeltaNet (GDN) by replacing the coarse particle size with the chanel-wise diagonal gate (alpha t).
- K3 Report § 2.1.1 provides further details of KDA's realization in K3 (with boundary decay Eq.5 and full output door Eq.6) and describes KDA's algorithm-system co-design (FlashKDA, KCP, KDA-aware prefixed cache) in § 5.1.
paper-eq-5
paper-eq-6
1.4 Positioning
The abstract of Attention is not "must appear Softmax" but:
According to Key, decide how to read Valle
KDA through status matrix: y t=q ttS t. So it's still "Query reading from Key-Value relations", just:
- Full Attention: Save token level KV, calculate Softmax when read;
- Linear/KDA: Save compression and use nuclear / matrix multiplication for reading;
- KDA also added a forgotten online update.
So it can be said that KDA is an incremental, nuclear, door-to-door realization of attention. The essence of attention is the location of content, not the specific function of Softmax.
1.5 KimiDeltaAttention
KimiDeltaAttention has the following code:
class KimiDeltaAttention(nn.Module):
""KDA" complete layer: Projection (q/k/v/ beta/decay)+ core push + output door.
Parameter structure (report Eq. 2 / Eq. 5 / Eq. 6):
- `q proj/k proj/v proj`: linear projection to the key/value dimension per head.
- ``q conv/k conv/v conv``: causality and effect ShortConv (width 4) for which no location coded
Models inject token level "diversion" information (report §2.1.1; Kimi Linear paper refers to it as place code replacement).
- '``beta proj ' :delta-rule writing strength '```beta = sigmoid(x)`, controls 'overwrite
How many old memories" (``beta ' close to 1 means fully rewritten memories in that direction).
- `alpha down/alpha up/alpha bias`:** low** decay logits `z = W up W down x + b '
(Eq. 2) The source of log decay by Channel.
- `A log' : Eq. 5 `A h ', initialized as 0.
- `o norm ' :head-wise RMS Norm (Eq.6 first step).
- ' `g proj ' : complete, input-dependent output door ``sigmoid (W g x) ' (Eq.6, second step).
"""
def __init__(self, dim: int, cfg: KDAConfig, *, layer_idx: int = 0, eps: float = 1e-6):
super().__init__()
self.dim = dim
self.cfg = cfg
self.layer_idx = layer_idx
h, dk, dv = cfg.num_heads, cfg.head_dim_k, cfg.head_dim_v
self.num_heads, self.head_dim_k, self.head_dim_v = h, dk, dv
k_dim, v_dim = cfg.k_dim, cfg.v_dim # k_dim = H*Dk, v_dim = H*Dv
# --- Linear projection to unspecified head width, then to ShortConv for local mixing ----
self.q_proj = nn.Linear(dim, k_dim, bias=False)
self.k_proj = nn.Linear(dim, k_dim, bias=False)
self.v_proj = nn.Linear(dim, v_dim, bias=False)
Self.q conv = ShortConv (k dim, cfg.conv kernel) # Genematic depth volume of width 4
self.k_conv = ShortConv(k_dim, cfg.conv_kernel)
self.v_conv = ShortConv(v_dim, cfg.conv_kernel)
# --delta-rule write strength beta and low dyslexia logits -- -- -- -- --
Self. bita proj = nn.Linear (dim, h, bias=False) # - > (B, T, H) per weight
# Low data: z = W up W down x + b alpha.
Self.alpha down = nn.Linear (dim, cfg.alpha rank, bias=False) # Press on rrank
Self.alpha up = nn.Linear
Self.alpha bias = nn. Parameter
#Eq.5's head log-scale A h; initial paper to 0.
self.a_log = nn.Parameter(torch.zeros(h))
# --- Output door (Eq. 6): head-wise RMSNorm + Total import-dependent door - -
self.o_norm = HeadRMSNorm(dv, eps)
Self.g proj = nn.Linear
self.o_proj = nn.Linear(v_dim, dim, bias=False)
self.reset_parameters()
def forward(
self,
x: Tensor,
*,
reset_mask: Tensor | None = None,
cache: KDACache | None = None,
initial_state: Tensor | None = None,
) -> tuple[Tensor, Tensor]:
``x ' is `(batch, seq, dim) '. returns `(output, final state) '.
Form selection: Training/prefill walking chunkwise (parallel, inter-column transfer);
Self-return decode (``t = = 1` and already in status ' ) One step ``kda step '. Two morphological values
Full equivalent price (`kda ref`incl. `kda recurent` ' as ground truth verification).
"""
b, t, _ = x.shape
q, k, v, beta, g, conv_states = self._project(x, cache)
# (B, T, H, D) - (B, H, T, D): Expulsion in T-dimensional, with a head mentioned to be easy to broadcast.
qh, kh, vh = (z.transpose(1, 2) for z in (q, k, v))
gh = g.transpose(1, 2)
bh = beta.transpose(1, 2)
state_in = initial_state if initial_state is not None else (cache.state if cache else None)
# The intermediate calculation is fp32 (activation is bf16/fp16), with no loss of precision accumulated in the push state.
work_dtype = torch.float32 if x.dtype != torch.float64 else torch.float64
chunk = self.effective_chunk_size(work_dtype)
if t == 1 and state_in is not None:
# decode form: single token one step update. Note that this branch is only active when there is a "previous state".
# Ensure that decode's first token can also walk chunkwise (initial state is 0).
out, state = kda_step(
qh[:, :, 0].to(work_dtype),
kh[:, :, 0].to(work_dtype),
vh[:, :, 0].to(work_dtype),
gh[:, :, 0].to(work_dtype),
bh[:, :, 0].to(work_dtype),
state_in.to(work_dtype),
)
out = out.unsqueeze(2) # (B,H,1,Dv)
else:
# prefill / no status decode: chunkwise parallel form (column within block, line between blocks).
fn = self._kernel()
out, state = fn(
qh,
kh,
vh,
gh,
bh,
chunk_size=chunk,
initial_state=state_in,
reset_mask=reset_mask,
dtype=work_dtype,
)
#Eq.6 Output door: head-wise RMSNorom - > All rely on the entered door - > Output projection.
out = out.transpose(1, 2).reshape(b, t, self.num_heads, self.head_dim_v)
out = self.o_norm(out.to(x.dtype)).reshape(b, t, -1)
out = out * torch.sigmoid(self.g_proj(x))
out = self.o_proj(out)
if cache is not None:
# decode write back the current cycle tocche for next token.
cache.state = state
cache.conv_q, cache.conv_k, cache.conv_v = conv_states
return out, state
0x02 Rationale
2.1 Intuitive analogy perspective
Imagine you're a librarian, with only one fixed-sized memory whiteboard (state matrix S), no bookcase (KV Cache).
The core action is "Press channel fade, then delta write." For every new book you have to do two things:
- Forget it first: use each book's own chanel-wise decay alpha to dilute the old part of the whiteboard.
- Update: press delta rule with the new book -- but before writing, erase the original information that you already know, only the new information itself. This does not accumulate over time.
Key difference: Softmax has an infinite shelf that remembers every book precisely; KDA whiteboard is always this big, but it always decides exactly how much to forget and what to update. When context exceeds memory capacity, KDA blends in with the MLA global layer -- "Small butler + Regular Big Pack."
2.2 Expert perspective
Input output:
- Enter: x ∈R{BXTxd}, where B=batch, T=serial length, d=hidden dimensions.
- After projection: query/key q t, k t ^d k} (L2 unified), value v t ^d v}, one per head.
- Status: S t RR{d k xd v}, fixed size, without T-growth.
Core push (Eq.1, single head perspective):
S_t = (I − β_t k_t k_tᵀ) Diag(α_t) S_{t-1} + β_t k_t v_tᵀ (Eq.1)
- Diag(alpha t): channel-wise decay factor, α t = ext(g t), g t ≤ 0 is log-decay. Each line is forgotten.
- Diag(alpha t) S t t m: Press channnel to reduce old state.
- β t: delta-rule writing strength (0,1), controls "overriding the direction of old memories".
- (I − β t t t tT): A thorium-1 projection, which reduces the composition in the state aligned to k t by β t. That's, in the direction of k, the old state "and the new key overlap" is subtracted -- this is delta rule, "Play the old before writing."
- + beta t k t v tt: Write new memories.
Output: o t = S tT q t (for each head).
Equivalent rewrite (stable, actual):
S_t = S_{t-1} + β_t k_t (v_t − S_{t-1}ᵀ k_t)ᵀ
This is to calculate "new message to write this time" = v t minus "remembered in the old state of k t" and write only increments.
Digging: Delta rule, "Selectional Forgetness + Online Learning Rates", is co-mingled with linear regression by a minimum of two times, treating k t as input, v t as monitoring, S as linear map weight, S tT k t as forecast, v t - S tT t t as disability, beta t as learning. So the KDA modeled attention as "the minimum two-fold online learning rate" and the theoretical expression was stronger than pure linear attention, which made it naturally supported by the theory of "compression is intelligence."
2.3 KDA Rationale
First read formula (Eq.1)
S_t = (I - beta_t * k_t k_t^T) Diag(alpha_t) S_{t-1} + beta_t * k_t v_t^T
– Token Transition Matrix – – – – – – Peter – to write new messages – – – – – – – – – – and – – and – – and – – and – and – – and – and – and – and – and – and – and – and – and – and – and — and — and — and — and — and — and — and — and — and — and — and — and — and — and — and — and — and — and — and — and — and — — and — and — and — and — and — — and — and — and — and — — — and — and — and — — and — and — and — — and — — and — and — and — — — and — — — — and — — —
Alpha t = ext(g t) à channel retention factor (0 < alpha < = 1)
K3 Architecture Overview, Bottom Left is KDA Modular Chart
KIMI-K3-Arch
2.4 In-depth discussion
- In theory, work: KDA translates the status transfer matrix parameter into a DDPLR special case - Diag (alpha t) (argument) + β t k t tT (thirty-1) — this is the structural premise of "canchkwise parallel". Compared to the normal DPLR, KDA's transfer matrix corresponds to the classic delta rule (S column decays along the key) and the physical synonym is clear (each column is "key memory oblivion curve".
- Need for decline in the lower boundary: Kimi Linear uses unbound softplus g = -extplus (A) -softplus (z), cumulative decay multiplier drops, and its inverse (K/k) re-scalation, forcing the diagonal tile to use a visible position-pair path; K3 changes to g = g min sigmoid (exp(A)z) to lock log-decay in (g min, 0), g min =5 time exp(80) = 5.5e34 still within the range of BF16 - each tile (with an angle) can be calculated in dense Tensor-Core matmul.
- NoPE self-conformity: the decline of KDA and the updating of natural location awareness by delta, so that the MLA layer does not need RPE to mix globally; the extension of context to 1M does not require any position parameter surgery.
- Theoretical issues left unsolved in the paper: the fundamental upper limit of fixed-state capacity to precise over long distances (the paper recognizes the weakness of linear attention search, offset by MLA mix); the volume-precision balance of the state matrix (state expansion direction still being explored).
2.5 Legend
For example:
plot_kda_principle
In the figure, 2 x 2, four, all by real roll (kda recurrent, full project KDA ground truth):
Top Left - State Matrix S (Memory): KDA 'memory' is a fixed-sized Dk x Dv matrix. The heat map in the figure is the end state after 32 tokens have actually run; T=32 tokens produce only one 16 x 16 matrix - this is the core selling point of KV Cache: memory O(1), which does not increase with the length of the sequence.
Top right -- one step to break three things: every token, state only three things:
Diag (alpha) S: Old memory first multiplied by a alpha =exp(g)< =1 retention factor (old message "forget it");
- Retrieval of S^Tk: "Check what's in your memory" with the current key;
- Write to Beta *k (v - S^Tk)^T: Only the error v - the memory retrieved is written to the state (the core of delta rule).
Bottom left -- delta rule 'The essence of error: man-made constructs' of the same key recurring, v-changing scene showing how the values that Key retrieves are condensed to the latest v. Normal linear attention (S + = k v^T) is "mixed," and delta rule presses the conflict through v-S^Tk -- that's why memory is constantly updated and not contaminated with each other. The three lines in the figure are the real search values from kda step.
Bottom-right - a bounded log-decay (Eq.5): g = g min * sigmoid (exp(A)z) locks log-decay strictly (g min, 0), i.e. g min=5 (-5, 0). This "down" is a numerical stability prerequisite for Q4/Q5/Q6.
One sentence concludes: KDA = fixed size memory matrix S (O(1) memory) + three things per token (depreciation search only errors) + boundary decay (guarantee that chunkwise does not spill). The KV Cache of the standard MHA is fundamentally compared to T linear growth.
2.6 Comparison of mechanisms
The main lines of this table are: the different approaches differ mainly in two places:
- How the old state is declining;
- How is the current Value written and whether or not to use disability to correct errors?
Linear attention is not forgotten and corrected; DeltaNet adds disability to writing; GDN adds one-size-fits-all forgets by Head; and KDA elaborates the forget into each Key channel.
0x03 Subknowledge Point Dismantling
KDA（Kimi Delta Attention）
Ideas - Status is cumulative with delta rule
Log-decay (Eq.5)
Ideas -channel-wise decay and low curvature (Eq.2)
Ideas - ShortConv
Chunkwise Parallel (Eq.3-4, DPLR)
Ideas - Step by Step Kda step (decode)
Eq.6
Ideas - Decoding cache KDACache and Cline() branch
└ - KDA context Parallel (KCP, report §5.1.2)
The code corresponds to the following:
When using triton=False kernel() returns pure PyTorch kda chunkwise.
Uniform symbol agreement (all subsections of this document follow):
- S t ∈R{d k x d v}: A matrix (one per head).
-q, k∈R{d} (L2 integrated), v∈R^d v}.
-g t^R^d k}: channel log-decay (≤0); α t = ext(g t) is retention factor.
- beta t∈ (0,1): delta-rule writing strength.
- chunk size C (configuration 64, tile 16), tile is the basic component of numerical security.
0x04 sub-knowledge point depth analysis
4.1 State increment with delta rule (Eq.1)
KDA retains Delta Rule's residual overwrite mechanism, but expands the GDN marker to a d k-dimensional dimension.
4.1.1 Rationale
KDA is based on the assumption that each request will maintain a fixed-size state matrix in each layer, each Head. This matrix represents the global memory of the current request to its current location. It does not keep a single record for every history Token, but it compresses all historical context information from Token into the same matrix.
When KDA handles the current Token, the model calculates based on the current Token Hidden State and the corresponding projection parameters:
- q t: Query for reading from memory;
- k t:Key, which defines the mode of location;
- v t: Value, which provides the content to be written;
- α t: Forget the door and decide how much old information each Key channel retains;
- Beta t: The writing door, which determines how much of the current disability is written;
- gamma t: The output door, which controls the intensity of the output information.
KDA status update can be written as:
(1) Use first
, forget the door and decide how much the old Key channel retains. Because alpha t is a vector, not all columns decay at the same rate. The j-Key channel has an independent forgotten speed. Alpha t,j}, important channels can be preserved more, and channels that should be diluted can decay faster.
If the shape of the state matrix is: So right times Diag (alpha t) scales each column of the status matrix separately. Each column corresponds to a Key channel, so KDA achieves the "Key Channel" forgetful.
The size of the door is therefore from GDN:
num_heads
Extension to:
num_heads × head_dim
If the door is not forgotten, purely linear attention may simply be: As the matrix plus meets the exchange, the order of input [A, B] and [B, A] produces the same end state. This means that a simple cumulative state naturally lacks location information: it knows what links are written, but it does not necessarily know the sequence of these linkages. Forgotten in KDA can partially restore this sequence of information. The farther away from the present Token, the more decay it experiences, the weaker its signal. Thus, the forgotten process can be seen as a hidden relative location code.
(2) Re-execut Delta Rule:
is the writing of the current Value, the original new knowledge brought by the current Token. It binds the Value v t and the address mode k t, ready for writing.
This reduces the part of the old state that matches the current Key mode. Its function is not simply to add new elements, but to avoid the constant repetition of the same feature in long context. Deletes or amends the old matching content before writing to a new target, Value, and therefore has a more precise overwriting capability.
Formula level comparison with standard practice:
4.1.2 Comparison of baselines
Gains from changes: O(1) VRAM, O(T) Time; Cost: Retrievation to compressed approximation.
4.1.3 Computation complexity analysis
KDA is the typical "Incremental Sequence Model": Training/prefil with a full sequence of chunkwise forwards (outputs for each location to calculate the loss); reasoning decode with a one-step thrust (regression token generation, status transfer). The bridge is chunkwise.
- FLOPS (single token): KDA = O(d k.d v) (state update + output read out) vs softmax = O (T.d k) (need to compare with all history). T-time KDA wins.
- Memory: KDA status = d kxd v/head (fixed); softmax KV = Tx (d k+d v)/head (linear growth).
- Parameters: KDA does not add any additional parameters to the projector itself.
4.1.4 Parameter sensitivity
-head dim k (default 128): determines the size of the state array and the capacity of "key memory direction". Larger but low FLOPS; too small.
-head dim v (default 128): value information capacity.
- Thesis experience: d k=d v=128 is Kimi Linear's default, following a transfer, 96 head(d=7168).
4.1.5 Counterarguments
- The strongest objection: the compression of the fixed matrix inevitably leads to the loss of information, and the precise retrieval of a super-long context (e.g. "a specific number mentioned in 400,000 token") fails.
- If it fails, it is most likely wrong to assume that "compressable memory is enough to express the information required for the mission". This assumption is fragile for missions requiring accurate token retrieval (complex agentic tracks, code library retrieval).
- Winner conditions: similarity of memory, semantic intensity, long context and excessive redundancy of token (dialogue, long summary). Failure condition: A precise location is required to retrieve the missing key information.
- A more robust approach: state expansion, or a higher combination of KDA+MLA (3:1 is the best, but mission sensitive).
- Wounded: a long-range precise search of the mission's party; beneficiary: 1M context reasoned swallower.
4.1.6 Project realization
False code (line by line):
for t in 1..T:
# Diag(alpha)Schannel Decline
sk = einsum ('bhd, bhdv->bhv', k[:,, t], state) #S^Tk
delta = beta[:,, t]* (v[:,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, x,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
state + = k[:,, t]* delta #rank-1 external update
= einsum ('bhd, bhdv->bhv', q[:,, t], state) #
Real project code (kda ref.py):
def kda_recurrent(q, k, v, g, beta, *, initial_state=None,
reset_mask=None, dtype=torch.float64):
"""Eq.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.