【轻量化复刻大模型】线性注意力交互式演示
0x01 总览
1.1 模块定位
experiments/models/llm/linear_attn/gradio_linear_attn_demo.py 是 线性注意力家族 的交互式演示入口，基于 Gradio 6 构建。它把 6 种 step 变体（Vanilla / DeltaNet / GLA / GatedDeltaNet / KDA v1 / KDA v2）、KDA 状态演化、门控曲线对比、递推-分块等价性、失败案例、参数敏感性、位置衰减、反方观点、以及 10 个测试可视化（形状/修正/混合/GLA-vs-DeltaNet/约束/兼容性/工具/Layer 形状/Layer 等价/Gate 可学习）整合到一个 18 tab 的 Web UI，所有图均由 CPU 实时计算产出。
1.2 核心数学
总体公式
输入: q_t, k_t ∈ R^(B,H,Dk) # 当前 token 的 query/key
v_t ∈ R^(B,H,Dv) # 当前 token 的 value
S_{t-1} ∈ R^(B,H,Dk,Dv) # 上一步的状态矩阵(固定大小)
g_t ∈ R^(B,H,Dk) # gate logit (逐 K 维)
beta_t ∈ R^(B,H) # 写入强度 logit
A_log ∈ R^(H,) # 每头的门控缩放参数
通用递推骨架 (Eq. 1):
S_t = Diag(alpha_t) ⊙ S_{t-1} + beta_t · k_t ⊗ (v_t - k_t^T S_{t-1})
___________/ ____/ ____/ _______________/
衰减/遗忘 写入强度 新 key 预测误差(Delta Rule)
o_t = q_t^T S_t # (Eq. 2) 检索输出
变体差异:
Vanilla Linear (Eq. 3): S_t = S_{t-1} + phi(k_t) ⊗ v_t # 纯加性累积, phi = elu+1
DeltaNet (Eq. 4): alpha_t = alpha (标量), beta_t = beta # 引入 Delta Rule 覆写
GLA (Eq. 5): S_t = Diag(sigmoid(g_t)) S_{t-1} + k_t ⊗ v_t # 加性 + 门控衰减
Gated DeltaNet (Eq. 6): S_t = Diag(sigmoid(g_t)) S_{t-1} + sigmoid(beta_t)·k_t⊗(v_t - k_t^T S_{t-1})
KDA v1 (Eq. 7): g_t = -softplus(g_logit + exp(A_log)) → alpha = exp(g) ∈ (0,1)
KDA v2 (Eq. 8): g_t = g_min_log · sigmoid(exp(A_log) ⊙ z) → alpha ∈ (exp(g_min_log), 1)
子机制分解
Delta Rule （核心修正机制）
- 公式：S_t = alpha ⊙ S_{t-1} + beta · k_t ⊗ (v_t - k_t^T S_{t-1})
- 直觉解释：写入前先用 k_t 从旧状态检索出"预测值"，新值减去预测值得到 预测误差，只写入误差部分；这样同一 key 第二次写入会覆写旧值而不是累加。
- 关键性质：
- 当 alpha=1, beta=1 且 key 相同时，S_t 在该 key 子空间上精确等于最新写入值 （与 Tab 10 的 0.2 对应）。
- 当 alpha<1, beta<1 时形成"泄漏式"覆写，旧信息按 alpha^k 指数衰减。
门控衰减（gating）
- 公式：alpha_t = exp(g_t)，其中 v1 用 g_t = -softplus(z)，v2 用 g_t = g_min_log · sigmoid(exp(A_log) ⊙ z)。
- 直觉解释：把"遗忘多少"作为可学习的 logit，让每个 K 维独立选择自己的时间尺度。
- 关键性质：
- v1 下界可逼近 0 （完全遗忘），但梯度在 z→+∞ 时指数衰减，存在梯度消失风险。
- v2 下界 exp(g_min) > 0 硬性保底，永远不会遗忘到 0，梯度形状与 sigmoid 类似，z=0 附近梯度最大。
状态检索
- 公式：o_t = q_t^T S_t
- 直觉解释：把状态 S 当作一个 键-值联想记忆矩阵，query 是检索向量，输出是该 query 在记忆上的"投影"。
- 关键性质：内存占用 O(Dk·Dv) 与序列长度无关，这是相对 KV cache O(T·D) 的核心优势；代价是容量瓶颈（Tab 5）。
关键洞察
关键洞察：线性注意力的本质是"把历史压缩进固定大小的状态矩阵 S"。Vanilla Linear 是纯加性累积（会混叠），DeltaNet 通过 Delta Rule 实现"同 key 覆写"（纠错），GLA/GDN 加上门控衰减（时间尺度），KDA v2 进一步用 g_min 给遗忘下界硬保底（数值稳定）。整个家族围绕"怎么写、怎么忘、怎么查"三个轴展开。
”
0x02 每个 Tab 的详细说明
Tab 1: 6 种 Step 对比
目的
在完全相同的输入序列（周期化 key、变化的 value）下，并列运行 6 种 step 函数（Vanilla / DeltaNet / GLA / GatedDeltaNet / KDA v1 / KDA v2），把每个 token 步的输出向量堆叠成热力图。验证的命题：在周期重复 key 的场景里，纯加性方法（Vanilla）输出会随时间漂移/发散，而带 Delta Rule 的方法（DeltaNet/GDN/KDA）能约束输出幅度，带 sigmoid 门控的方法（GLA）幅度居中。
背景概念
周期化 key 模式：代码中把序列按 period = seq_len // 8 切分为若干组，同组内所有 token 共享同一个 k_base，但 v_base 在组内随位置正弦变化。这样设计是为了放大"同 key 不同 value"的写入冲突——Delta Rule 类方法会覆写，纯加性方法会累积。
设计思路
- 为什么这样设计实验：选择 2×3 热力图（行=输出维度，列=token 步），是因为它能同时暴露时间维度（水平条纹反映周期性）和幅度维度（colorbar 范围）两个关键信息。对比对象选 6 种而不是 2 种，是为了让读者看到"从纯加性到 KDA v2 的演进路径"，每一步演进解决了上一代的什么问题。参数范围（seq 10-100, dim 8-64）是为了在 CPU 上实时跑出可观察的演化效果。
- 想验证什么：
- Vanilla Linear 的输出幅度在 range=[-706.26, 706.26]，远大于 其他 5 种（其他都在 ±1 到 ±58 之间）——验证纯加性累积会发散。
- DeltaNet 的输出范围 [-0.83, 0.83] 最紧——验证 Delta Rule 覆写确实约束幅度。
- KDA v2 [-0.54, 0.54] 比 KDA v1 [-0.25, 0.25] 略宽——验证 v2 的下界 g_min 让遗忘更缓和、保留更多历史。
- 希望读者学到什么：
- "纯加性 vs Delta Rule"的输出幅度差距不是细节，是量级差距。
- 同一种输入在不同 state update rule 下会产生完全不同的动力学。
- 希望从图上看到什么：从 tab-1.webp 实际图可见，6 幅子图都呈现垂直方向的红色条纹（因为 key 在周期内重复，query 也周期变化，输出在周期内呈近似重复模式）。成功信号：Vanilla 子图 colorbar 数值是 -700~700（最大），其他 5 幅都在 ±58 以内；6 幅图整体色调偏红（因 query 和 key 同向）。失败信号：如果 DeltaNet / GDN / KDA 的输出 colorbar 也到了 ±700，说明 Delta Rule 没生效；如果 KDA v2 的颜色完全等于 v1，说明 g_min 约束没起作用。
UI 组件与参数影响
输出图解读
tab-1
布局：2×3 六联图，每格一幅热力图 + colorbar，子图标题中标注 range=[min, max]（这是关键数值信号）。
- 左上 (a) Vanilla Linear：热力图，X 轴=token step（0-40），Y 轴=dim（0-16），colormap=RdBu_r。你该看什么：colorbar 数值范围。理想情况：|range| 显著大于其他 5 个（实际图中是 ±706），因为输出随 token 数线性累加。异常指示：若 range 也是 ±1 量级，说明 phi(k) 没生效或代码错用了 DeltaNet。
- 中上 (b) DeltaNet：同样热力图，colorbar 在 ±0.83。你该看什么：红色条纹非常清晰有规律，因为 Delta Rule 让每个周期都覆写状态。理想情况：范围在 ±1 附近。异常指示：若与 Vanilla 一致（±700），说明 alpha=0.5 和 beta=0.5 的衰减没生效。
- 右上 (c) GLA：colorbar ±57.96。你该看什么：范围比 Vanilla 小但比 DeltaNet 大——GLA 是"门控衰减 + 加性写入"，仍有累积但受 sigmoid 衰减约束。理想情况：范围介于 Vanilla 和 DeltaNet 之间。
- 左下 (d) Gated DeltaNet：colorbar ±0.27，是 6 个里最紧的。你该看什么：双重约束（gate 衰减 + Delta 覆写）。
- 中下 (e) KDA v1：colorbar ±0.25，与 GDN 相当。
- 右下 (f) KDA v2：colorbar ±0.54，比 v1 略大。你该看什么：v2 的 g_min 下界让遗忘更慢，所以保留更多历史 → 输出幅度略大。
调节建议
- 想看 Vanilla 发散 → 调 序列长度到 100，colorbar 范围会进一步扩大。
- 想看 KDA v1 vs v2 差异 → 调 gate logit 到 -3，sigmoid 接近 0，v2 会被 g_min 托住，v1 会继续往 0 走，两者输出范围差距拉大。
- 想验证"周期性来自输入" → 调 随机种子到不同值，条纹位置变化但周期间隔不变。
为什么重要
这是整个演示的 总入口：它用一张图就告诉读者"线性注意力家族不是同一件事"，6 种变体在相同输入下的输出幅度差 3 个数量级。如果读者只能看一个 tab，应该看这个——它直接决定了为什么需要 Delta Rule（否则发散）、为什么需要门控（否则无法遗忘）、为什么 KDA v2 要加 g_min 下界（否则极端 logit 下数值不稳）。
Tab 2: KDA 状态演化
目的
逐 token 跟踪 KDA 状态矩阵 S ∈ R^(Dk × Dv) 在序列处理过程中的快照，在 5 个时间点（{0, T/4, T/2, 3T/4, T-1}）抓取 S 并可视化。验证的命题：KDA 的状态不是静态的，而是随输入持续重构；不同版本（v1 vs v2）在相同 gate 下的遗忘速率有可见差异。
设计思路
- 为什么这样设计实验：选 imshow 而非折线，是因为 S 是二维矩阵（K 维 × V 维），热图能直接显示"哪些 (k,v) 对通道被强化/抑制"。5 个等距快照是"演化过程"的最小可视化集合——再多 UI 排不下，再少看不出趋势。提供 v1/v2 切换是为了让读者直接对比门控函数对状态长期行为的影响。
- 想验证什么：
- v1 在 gate=-0.5, A_log=5 下，S 在 t=80 时仍有显著结构（不完全归零）。
- v2 在相同条件下，S 的颜色整体更饱和（保留更多历史），因为 g_min 下界让 alpha > exp(g_min)。
- 状态不会出现 NaN 或纯黑（说明数值稳定）。
- 希望读者学到什么：
- "状态"是动态的记忆，不是静态权重；每一行（K 维）就像一个独立的记忆通道。
- v1 和 v2 的差异体现在长期行为，短期看不出来。
- 希望从图上看到什么：从 tab-1-2.webp（v1）和 tab-2-2.webp（v2）实际图可见，两幅图都展示 5 个快照的 8×12 热图。v1 图：t=1 时颜色较浅（少量写入），t=21 起出现明显红蓝交替的方块，t=80 仍有清晰结构。v2 图：t=1 颜色比 v1 更淡（因为 v2 的 sigmoid gate 在 z=-0.5 时 alpha 较高，保留更多——但初始 S=0 所以第一帧几乎无内容），t=21 起颜色深度超过 v1，t=80 仍然非常饱和。成功信号：v2 后几帧的颜色深度 ≥ v1；5 个快照都有内容（不全是白色）。失败信号：若任何一帧变成全白或全黑，说明遗忘过快或数值溢出。
UI 组件与参数影响
输出图解读
V1
tab-1-2
V2
tab-2-2
布局：1×5 五联图（每个时间点一幅 8×12 热图）。
- 每幅子图：colormap=RdBu_r，X 轴=V dim（0-12），Y 轴=K dim（0-8），colorbar 共用 vmin/vmax。
- v1 五帧（tab-1-2.webp）：t=1 偏白，t=21 出现"左上红、右下蓝"的棋盘结构，t=41/61/80 结构持续演化，红色块和蓝色块位置随输入正弦变化而移动。理想情况：每帧颜色深度逐渐增加，最后饱和。异常指示：若 t=80 全白，说明 alpha 全为 0，state 完全被遗忘。
- v2 五帧（续）：t=21 起红色块（正值）和蓝色块（负值）都比 v1 同帧更深，到 t=80 时整幅图呈现高饱和度的红蓝棋盘，说明 v2 的 g_min 下界让旧信息保留得更多。异常指示：若 v1 和 v2 的五帧完全一致（逐像素相同），说明 version 参数没传进 kimi_linear_step。
调节建议
- 想看"v1 快速遗忘 vs v2 保底" → 调 gate logit 到 2.0（sigmoid 接近 1，v1 的 alpha→exp(-softplus(2+exp(A_log))) 会很小，v2 被 g_min 托住）。
- 想看"状态从空白到饱和的完整过程" → 调 序列长度到 200，并把 gate 设为 -0.1（弱遗忘）。
- 想看"大状态矩阵的细粒度结构" → 调 K 维度和 V 维度都到 32。
为什么重要
Tab 1 看的是"输出"，Tab 2 看的是"状态本身"。线性注意力的所有行为都源于状态矩阵 S 的演化——这个 tab 是**唯一能让读者直接观察"记忆如何被写入和遗忘"**的窗口。在生产中，如果 v2 的 g_min 设置过小（如 1e-6），长序列尾部状态会趋近于零矩阵，模型会"失忆"；这个 tab 能在 5 帧内直观暴露这种失败模式。
A_log 对演化的影响
gate=−0.5 固定。
关键概念：v2 的反饱和（inverse saturation）
v2 门控 g = g_min · sigmoid(expA · z) 的 sigmoid 输出 0 时反而得到 alpha=1——这是 v2 的设计副作用：
注意这跟直觉相反：z 越负，alpha 越大（越不遗忘）；z 越正，alpha 越小（越遗忘） 。这是 K3 把"遗忘门"反向设计的结果——z 是"记忆 logit"而不是"遗忘 logit"。
Tab 3: v1 vs v2 门控
目的
画出 KDA v1（softplus gate）与 v2（sigmoid gate）的 衰减因子 α(z) 曲线 和 梯度 dα/dz 曲线，直接对比两种门控函数的数学形状。验证命题：v1 的 α 可以逼近 0（完全遗忘），v2 的 α 被 g_min 硬性下界托住；v2 在 z=0 附近梯度更集中，v1 在 z→+∞ 时梯度指数衰减。
背景概念
门控函数的 Log 域设计：KDA 的 α 必须落在 (0,1) 才能既做衰减又保证数值稳定。v1 用 g = -softplus(z) 把 z 映射到 (-∞, 0)，再取 α = exp(g) ∈ (0, 1)；v2 用 g = g_min_log · sigmoid(z) 把 z 映射到 (g_min_log, 0)，α ∈ (exp(g_min_log), 1)。下界 g_min 是 v2 引入的关键超参——它保证无论 z 多大，状态都至少保留 g_min 比例。
设计思路
- 为什么这样设计实验：选 1×2 双联图（左=α 曲线，右=梯度曲线）是因为"函数形状"和"梯度形状"是设计门控时必须同时考虑的两件事——左图回答"遗忘能到多低"，右图回答"反向传播能不能学"。z 范围 (-6, 6) 覆盖了 sigmoid/softplus 的有效区间（之外两者都饱和）。复用 plot_forget_limit.py 的真实函数保证图与论文/代码一致。
- 想验证什么：
- v1 曲线（蓝）在 z=6 时 α 接近 0（接近横轴）。
- v2 曲线（橙）在 z=6 时 α 停在 g_min=0.01 的虚线处，不再下降。
- v2 梯度曲线（橙）在 z≈-2 处取得最大值，v1 梯度（蓝）在 z≈0 处取得最大值，且 v2 的峰值更"瘦高"。
- 希望读者学到什么：
- "v2 不是 v1 的小改动"，是函数族不同——v1 是 softplus 族（无下界），v2 是 sigmoid 族（有下界）。
- 下界 g_min 是数值稳定性与遗忘能力的折衷：太小 → 失去保底意义；太大 → 无法遗忘。
- 希望从图上看到什么：从 tab-3.webp 实际图可见，左图：蓝线（v1）从 z=-6 时的 1.0 单调下降到 z=6 时的 ≈0.0，橙线（v2）同样下降但在 z≈2 后开始走平，最终停在 g_min=0.01 的虚线（与蓝色 v1 显著分离）。右图：两条梯度曲线都呈"钟形"，蓝线（v1）在 z≈0 取得最小值 ≈ -0.25，橙线（v2）在 z≈-1.5 取得最小值 ≈ -0.29，v2 梯度峰值更深、更靠左。成功信号：v1 在右端持续下降穿过 g_min 虚线，v2 在右端与 g_min 虚线重合。失败信号：若两条 α 曲线在右端都趋向 0，说明 v2 的 g_min 没生效；若梯度曲线出现平段，说明 sigmoid/softplus 饱和。
UI 组件与参数影响
输出图解读
tab-3
布局：1×2 双联图。
- 左图 （衰减因子 α(z)) ：折线图，X 轴=z（gate logit），Y 轴=α，Y 范围 [0, 1]。蓝色实线=v1，橙色实线=v2，灰色虚线=g_min=0.01 水平参考线。你该看什么：v1 是否穿过 g_min 虚线（应该穿过），v2 是否在 g_min 虚线处停下（应该停下）。理想情况：两线在 z<0 时几乎重合，z>0 时分叉，v1 持续下降，v2 走平。异常指示：若 v2 也穿过 g_min 虚线，说明 g_min_log * sigmoid(z) 公式实现错误；若两条线完全重合，说明 version 没区分。
- 右图 （梯度 dα/dz) ：折线图，X 轴=z，Y 轴=dα/dz（恒负，因为 α 随 z 增大而减小）。你该看什么：两条钟形曲线的最低点位置（梯度幅度最大处）和宽度。v2 的最低点更靠左且更深，说明它在 z<0 区域学习信号更强。理想情况：两条曲线都是光滑钟形，无锯齿。异常指示：若梯度曲线在 z>2 出现非零常数，说明 exp/softplus 数值溢出。
调节建议
- 想看"v2 退化为 v1" → 调 g_min 到 0.001（接近 0），v2 曲线几乎与 v1 重合。
- 想看"v2 完全丧失遗忘能力" → 调 g_min 到 0.1，v2 曲线在 z>2 后停在 0.1 高位。
- 想看饱和区 → 调 z 范围到 10，观察两条 α 曲线在 z=10 时的分离。
为什么重要
门控函数是 KDA 的"心脏"，α 曲线决定遗忘速度，梯度曲线决定可学性。这个 tab 用一张图就把"v1 vs v2"的差异从抽象公式变成可视形状。在生产中，如果训练时 gate logit 跑到 z=5 之外，v1 的梯度会指数衰减导致 gate 学不动，v2 的 sigmoid 梯度也会衰减但下界 g_min 至少保证前向不出 NaN——这个差异直接影响长序列训练稳定性。
Tab 4: 递推 vs 分块
目的
验证 KDA 的两种计算模式——逐 token 递推（kda_step 循环）和分块并行（kda_chunkwise）——在数值上是否等价。验证命题：对相同输入，两种模式的输出最大误差应 < 1e-3。
背景概念
分块并行（chunkwise） ：线性注意力的递推形式 S_t = α_t S_{t-1} + β_t k_t (v_t - k_t^T S_{t-1}) 是序列依赖的（无法并行）。但数学上可以通过矩阵结合律把每个 chunk 内的更新合并为一次矩阵乘法，跨 chunk 时再串行传递状态——这样 chunk 内部可以并行， chunk 之间保持递推。这是把线性注意力部署到 GPU 的关键技术。
设计思路
- 为什么这样设计实验：选热图+折线组合是因为热图能暴露"误差在 (token, dim) 二维平面的分布模式"（如果误差集中在某些 chunk 边界，说明分块逻辑有 bug），折线图能显示"误差随时间是否累积"。参数范围 seq 32-256, chunk 8-64 是 CPU 可承受且能触发 chunk 边界的合理区间。
- 想验证什么：
- max_diff < 1e-3（代码中的判断阈值）。
- 误差热图没有明显的 chunk 边界条纹（若有，说明 chunk 边界处理错）。
- 误差随 token 不累积（折线图应平稳，不应单调上升）。
- 希望读者学到什么：
- "分块并行"不是近似算法，数学上严格等价于递推（在浮点精度内）。
- 浮点累计误差主要来自矩阵乘法顺序不同，误差随序列长度增长是正常的，但应保持在 1e-3 量级以下。
- 希望从图上看到什么：⚠️ 重要：实际图 tab-4.webp 显示 max_diff = 8.12e-01，远超 1e-3 阈值，代码逻辑会输出 "偏差较大"。 左图热图在 token≈48-88 区间出现大量亮黄色条纹（误差接近 0.8），说明当前 CPU 参考实现中 kda_chunkwise 与 kda_step 在 dim=16, chunk=16, seq=128 的设置下数值偏差显著。这不是文档错误，是实现层面需要修正的 bug。成功信号（理想情况）：max_diff < 1e-3，热图全黑（误差接近 0），右图折线在 1e-5 附近小幅波动。失败信号（实际情况）：热图出现大片亮色区域，右图折线在某些 token 处飙升到 1e-1 量级。
bug 标注：当前 kda_chunkwise 与 kda_step 在此参数下数值偏差 0.812，远超阈值，实现需要核对（可能与 SGLang fused_recurrent.py 或 chunked 参考实现比对）。文档保留此节以记录 bug 现状，修复后再更新图片。
”
UI 组件与参数影响
输出图解读
tab-4
布局：1×2 双联图。
- 左图 （逐元素误差分布 head=0) ：热图，X 轴=token（0-128），Y 轴=dim（0-16），colormap=hot（黑→红→黄→白）。你该看什么：是否有 chunk 边界的垂直条纹（chunk_size=16 → 边界应在 16, 32, 48...）。理想情况：整幅图接近全黑（误差<1e-3），仅在个别位置有微小亮点。实际图：在 token=48-88 区间有大量亮黄/红色条纹，dim 维度上全 16 行都受影响——这是当前实现的 bug 信号。
- 右图 （每 token 的平均误差） ：折线图，X 轴=token，Y 轴=mean abs diff（log 刻度）。红色虚线=max_diff 水平参考线。理想情况：折线在 1e-5 到 1e-4 之间小幅波动，无趋势性增长。实际图：折线在 token=48-88 区间飙升至接近 10^0（即 1.0），远超 1e-3 阈值。
调节建议
- 想规避当前 bug → 调 chunk_size 到与 seq_len 相同（如 seq=128, chunk=128，此时只有一个 chunk， 分块退化为一次性矩阵乘）。
- 想看 chunk 边界影响 → 调 chunk_size 到 8（最小），观察误差条纹是否更密集。
- 想验证是否随 seq 累积 → 调 seq_len 到 256，观察右图折线是否持续增长。
为什么重要
分块并行是 KDA 能否上 GPU 的关键。如果递推与分块结果不等价，所有"线性注意力 O(1) 状态"的理论优势都无法转化为工程收益。这个 tab 是实现正确性的主锚点——只要它通过，就可以放心地用 chunkwise 做训练、用 step 做推理；如果失败（如当前实际图所示），整个 KDA 管线都需要重新核对。这也是为什么文档必须如实标注当前 bug。
Tab 5: 失败 Case
目的
演示 KDA 状态容量瓶颈：同一个 key 被反复写入不同 value 时，旧 value 会被新 value 覆写，无法长期保留。这是固定大小状态矩阵的理论极限，不是实现 bug。
背景概念
容量瓶颈：状态 S ∈ R^(Dk × Dv) 是固定大小的，相当于一个 Dk 维的"键空间"。当不同的 (k, v) 对被写入时，若 k 之间存在线性相关（或就是同一个 k），后写入的 value 会覆写先写入的 value（通过 Delta Rule）。这与 KV cache 形成对比——KV cache 每个 token 都有独立的 K/V 槽位，不会覆写。
设计思路
- 为什么这样设计实验：用"同一 key 反复写入 8 个不同 value"作为压力测试，是暴露容量瓶颈的最直接方式。折线图（而非热图）是因为读者需要看数值趋势——检索值（红）是否跟随真实写入值（蓝）。复用 plot_failure.capacity_bottleneck 保证与静态 demo 一致。
- 想验证什么：
- 检索值（红线）不能完全跟踪真实写入值（蓝虚线）的幅度变化。
- 检索值的幅度小于真实写入值（因为旧值被覆写后，S 的相应行是新值的衰减版本）。
- 两条线的符号变化趋势一致（同正同负），说明 Delta Rule 仍在工作，只是有衰减。
- 希望读者学到什么：
- 固定大小状态 = 有限容量。当多个 key 共享子空间时，旧信息必然被新信息覆写。
- KDA 不是"免费午餐"——它用容量换取了 O(1) 内存。
- 希望从图上看到什么：从 tab-5.webp 实际图可见，蓝色虚线（真实写入值 v）在 8 个写入步中呈正弦式上下波动，范围约 ±0.85；红色实线（检索值 S[0,0]）也呈现类似趋势但幅度明显较小（范围约 ±0.45），且整体比蓝线"延迟"（在蓝线极值点处红线只走到一半）。成功信号：两条线趋势一致、幅度有差。失败信号：若红线完全贴合蓝线，说明状态没在被覆写（Delta Rule 失效）；若红线完全平坦，说明状态没在被写入（写入逻辑失效）。
UI 组件与参数影响
无参数，点击按钮直接运行。
输出图解读
tab-5
布局：单图，折线图。
- 单图：X 轴=写入步数（0-7，同一 key 反复写入），Y 轴=值。蓝色虚线+方块标记=真实写入值 v，红色实线+圆点标记=从状态 S 检索出的值。你该看什么：两条线的幅度差距（容量瓶颈的直接证据）和趋势一致性（说明覆写是渐进的，不是突然的）。理想情况：红线幅度 < 蓝线幅度，但相位一致。异常指示：若红线完全平坦（不随蓝线变化），说明查询 q 与状态 S 不相关，实现有 bug；若红线幅度大于蓝线，说明状态在累积而非覆写，与 Delta Rule 设计矛盾。
调节建议
无参数可调节。
为什么重要
这是反方观点的理论基础——线性注意力的固定大小状态必然有容量上限。这个 tab 把抽象瓶颈变成具体曲线：读者亲眼看到"同一个 key 第 8 次写入时，前 7 次写入的信
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Linear Focus Interactive Demonstration
0x01 Overview
1.1 Modular positioning
The interactive presentation portal for the linear focus family is based on Gradio 6. It integrates 6 step variants (Vanilla / DeltaNet / GLA / GatedDeltaNet / KDA v1 / KDA v2), KDA state evolution, door-control curve comparison, insulation-block equivalence, failure cases, parameter sensitivity, position decay, counter-view, and 10 tests for visualization (shape/fix/mix/GLA-vs-DeltaNet/compatibility/tools/Layer shape/Layer equivalent/Gate learnable) to an 18 tab Web UI, all of which are calculated in real time by CPU.
1.2 Core mathematics
Overall formula
Enter: q t, k t∈R^(B,H,Dk) #Qary/key for current token
# Current token's value
# State matrix (fixed size)
g t∈R^(B,H,Dk)#gate log(KD)
Beta t∈R^(B,H)# Write strength logit
A log ∈R^(H), # door-to-door zoom parameters per head
General thrust skeleton (Eq.1):
S_t = Diag(alpha_t) ⊙ S_{t-1} + beta_t · k_t ⊗ (v_t - k_t^T S_{t-1})
___________/ ____/ ____/ _______________/
Decline/forgotten Write intensity New key forecast error
o t = q t^TS t# (Eq. 2) Retrieval output
Variable differences:
Vanilla Linear (Eq. 3): S t=S t s s s t t+v t t#purely cumulative, phi=elu+1
DeltaNet (Eq. 4): alpha t = alpha, beta t = beta # Integrator
GA (Eq. 5): S t = Diag(sigmoid(g t))
Gated DeltaNet (Eq. 6): S_t = Diag(sigmoid(g_t)) S_{t-1} + sigmoid(beta_t)·k_t⊗(v_t - k_t^T S_{t-1})
KDA v1 (Eq. 7): g_t = -softplus(g_logit + exp(A_log)) → alpha = exp(g) ∈ (0,1)
KDA v2 (Eq. 8): g_t = g_min_log · sigmoid(exp(A_log) ⊙ z) → alpha ∈ (exp(g_min_log), 1)
Sub-mechanism decompose
Delta Rule (core amendment mechanism)
- Formula: S t = alpha⊙S t t t t(v t k t T S t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t
- Intuitive interpretation: "Prophets" are retrieved from the old state with k t before writing, and new values minus projections are worth the prediction error, and only the error part is written; so the same key will overwrite the old value instead of adding.
- Critical nature:
- When alpha = 1, beta =1 and key is the same, S t precisely equals the latest writing value (0.2 for Tab 10) on this key subspace.
- When alpha <1, beta <1, overwhelms the old information by alpha ^k.
Gate control decay (gating)
- Formula: alpha t = ext(g t), where v1 uses g t = -softplus(z), v2 uses g t = g min log tigmoid (exp(A log)⊙z).
- Intuitive interpretation: "How much to forget" as a learning logit allowing each K-Vi to choose its own timescale.
- Critical nature:
- v1 lower bounds can approach 0 (total oblivion), but the gradient declines exponentially at z→+∞, with the risk of its disappearance.
- v2 lower bound exp(g min) > 0 hard base, never forgotten to 0, the gradient shape is similar to sigmoid and z=0 is the largest gradient around.
Status Search
- Formula: o t=q t^TS t
- Intuitive interpretation: use state S as a key-value memory matrix, query is the retrieval vector, output is the query's projection on memory.
- Critical properties: Memory occupation O (Dk Dv) is not related to serial length, which is the core advantage of KV Cache O (T D); the cost is a capacity bottleneck (Tab 5).
Critical insight
Key Insight: The essence of linear attention is "compressing history into a fixed-size state matrix S." Vanilla Linear is pure accumulation (mixed), DeltaNet achieves "with key overwhelming" through Delta Rule, GLA/GDN plus door control decay (timescale), and KDA v2 further uses g min as a hard base for forgotten lower bounds (value stabilization). The whole family's going around three axes: "How to write, how to forget, how to look."
”
0x02 Details for each Tab
Tab 1:6 Step Contrast
Purpose
Under the exact same input sequence (cycle key, change value) and running six step functions (Vanilla / DeltaNet / GLA / GatedDeltaNet / KDA v1 / KDA v2), stacking output vectors of each token step into heat. Validation proposition: In the scenario where the cycle repeats the key, the purer method (Vanilla) output will drift/disperse over time, while the method with Delta Rule (DeltaNet/GDN/KDA) will limit the output range, with the sigmoid door control (GLA) medium.
Background concept
Periodicize key mode: The code divides the sequence into groups by period = seq len / 8 and all tokens within the group share the same k base, but v base changes with the location sine inside the group. This is designed to magnify the writing conflict of "unlike key value" - the Delta Rule method overwrites and the pure plus method accumulates.
Design thought
- Why design experiments like this: Select 2x3 thermal effort (line = output dimension, column = token step) because it exposes both time dimension (horizontal stripe reflects cyclicality) and range dimension (colorbar range). The six instead of two objects are selected to show readers how "from pure addedness to KDA v2" has solved the problems of the previous generation. Parameter ranges (seq 10-100, dim 8-64) are intended to produce observable evolutionary effects in real time on CPUs.
- To verify what:
Vanilla Linear's output ranges from range = [-706.26, 706.26] to much greater than the other 5 (all others in the range of ±1 to ±58) - certification of pure added accumulation will disperse.
- The output range of DeltaNet [-0.83, 0.83] is the strictest - verify that Delta Rule overwrites the true bound range.
- KDA v2 [-0.54, 0.54] slightly broader than KDA v1 [-0.25, 0.25] - confirmation of g min below v2 to ease and preserve the memory.
- What can the reader learn?
- The "pure plus vs Delta Rule" output margin gap is not a detail, but a quantitative gap.
- The same type of input produces completely different dynamics under different state update rule.
- What is to be seen from the figure: as can be seen from the actual map of Tab-1.webp, 6 subgraphs show red stripes in a vertical direction (because Key repeats over the cycle, query changes over the cycle, output shows a similar pattern over the cycle). Successful signal: Vanilla sub-chart colorbar values -7000 ~7000 (maximum), while the other five are within ±58; 6 the overall colour is red (because query and key are the same). Failed signal: If DeltaNet / GDDN / KDA's output colorbar also reaches > 700, the Delta Rule is not valid; if KDA v2's colour is fully equivalent to v1, the g min constraint is not working.
UI Component and Parameter Impact
Output Image Interpretation
tab-1
Layout: 2x3 Pixels, each of which seeks to heat + colorbar, subheadings indicating renge = [min, max] (this is the key numerical signal).
- Top left (a) Vanilla Linear: Hot endeavour, X axis = token step (0-40), Y axis = dim (0-16), colormap = RdBu r. What should you look at: colorbar range. Ideally: |range is significantly larger than the other five (the actual figure is ± 706) because the output follows the linear sum of token. Anomalous indicator: If range is also a ±1 magnitude, this indicates phi(k) is not valid or the code is misused in DeltaNet.
- Top (b) DeltaNet: Equally hot, colorbar at ± 0.83. What should you see: Red stripes are very clear and regular, because Delta Rule gives every cycle a curvature. Ideally: the range is near ±1. Anomalous instruction: If consistent with Vanilla (±7000), indicate that the decay of alpha = 0.5 and beta = 0.5 is not effective.
- Top right (c) GA: colorbar ± 57.96. What should you see: the range is smaller than Vanilla but bigger than DeltaNet -- GLA is "door controlled decay plus writing" and still accumulates but is bound by sigmoid decay. Ideally: between Vanilla and DeltaNet.
- Bottom left (d) Gated DeltaNet: colorbar ± 0.27, is the tightest of six. What should you see: double constraint (gate decay + Delta overwrite).
- below (e) KDA v1: colorbar ± 0.25, equivalent to GDN.
- Bottom right (f) KDA v2: colorbar ± 0.54, slightly larger than v1. What do you want to see: the g-min lower bounds of v2 slow forget, so keep more history and output slightly larger.
Reconciliation recommendations
- Want to see Vanilla spread the tune of the sequence length to 100, colorbar will be further expanded.
- Want to see KDA v1 vs v2 difference → gate logit to -3, sigmoid close to 0, v2 will be held by g min, v1 will continue to go 0, and the gap between the two output ranges widens.
- Trying to verify that "cyclicality comes from input" poignant random seeds to different values, stripe position changes but no change in periodicity.
Why does it matter?
This is the total entrance to the whole demonstration: it tells readers that "The Linear Focus family is not the same thing" with a chart, six variants with three orders of magnitude under the same input. If the reader can only look at one tab, he should see this... It directly determines why Delta Rule (otherwise dispersed), why it needs door control (otherwise unforgivable), why KDA v2 adds g min lower bounds (otherwise, the value under extreme logit is unstable).
Tab 2: KDA State Evolution
Purpose
Token tracks a snapshot of the KDA status matrix S ∈ (Dk x Dv) during sequence processing, capturing S at five points (00, T/4, T/2, 3T/4, T-s). Validation proposition: KDA 's state is not static, but is continuously re-engineered with input; there are visible differences in the forget rate of different versions (v1 vs v2) under the same Gate.
Design thought
- Why design experiments like this: Imshow instead of a coupon because S is a two-dimensional matrix (K-V-D), and thermal graphics directly show what (k,v) is reinforced/pressed on the channel. Five equidistance snapshots are the smallest visual assembly of the evolution process -- more UIs, less trends. The v1/v2 switch is provided to allow the reader to compare directly the effects of the door control function on the long-term behaviour of the state.
- To verify what:
- v1 Under Gate = 0.5, A log = 5, S still has a significant structure at t = 80 (not completely zero).
- v2 Under the same conditions, the colour of S is saturated as a whole (with more history) because the lower g min bounds allow alpha > ext (g min).
- Status does not appear in NN or pure black (specify stable values).
- What can the reader learn?
- "State" is a dynamic memory, not a static weight; each line is an independent memory channel.
- The difference between v1 and v2 is reflected in long-term behaviour and is not visible in the short term.
- Hope to see what can be seen on the map: from the actual maps of Tab-1-2.webP (v1) and Tab-2.webp (v2), both showing 5 snapshots of 8x12 heat. v1 Figure: t = 1 light colour (slow writing), t = 21 visible red and blue squares, t = 80 still have clear structures. v2 Figure: t=1 lighter than v1 (because alpha is higher at z=0.5 for v2), retains more - but initially S=0 and therefore the first frame has little content), t=21 colour depths exceeding v1, t=80 is still very saturated. Success signal: the depth of the color in the last few frames of V2 ≥ v1; 5 snapshots have content (not all white). Failed signal: If any frame becomes white or black, it means that it is too quick to forget or that the value is spilled.
UI Component and Parameter Impact
Output Image Interpretation
V1
tab-1-2
V2
tab-2-2
Layout: 1 x 5 Fig. (one 8 x 12 heat map per time point).
- Each subgraph: colormap=RdBu r, x axis=V dim(0-12), Y axis=K dim(0-8), colorbar shared vmin/vmax.
- v1 frames (tab-1-2.webp): t=1 white, t= 21 displays a "top left red, bottom right blue" board structure, t=41/61/80 structure continues to evolve, and the position of red and blue blocks moves with the input sine. Ideally: the depth of each frame is gradually increasing and final saturation. Anomalous indicator: If t= 80, all white, note alpha is 0, state is completely forgotten.
- v2 frames (continued): t= 21 red blocks (positive) and blue blocks (negative) are deeper than v1 frames, and at t= 80 the whole picture shows a red blue board with high saturation, indicating that the g-min base of v2 keeps the old information more. Anomalous instruction: If the five frames of v1 and v2 are identical (same by pixel), indicate that the version parameters are not passed into Kimi linear step.
Reconciliation recommendations
- Want to see "v1 Quick Forget vs v2 Bottom" → Gate logit to 2.0 (sigmoid near 1,v1 alpha→exp (-softplus (2+exp (A log)) will be small, v2 held by g min).
- Want to see the "state from blank to saturated complete" tune the sequence length to 200 and set the date to -0.1 (weak oblivion).
- Want to see "the fine particle structure of the big matrix" with K dimension and V dimension to 32.
Why does it matter?
Tab one looks at the output, Tab two looks at the state itself. All the actions of linear attention stem from the evolution of the state matrix S -- This tab is the only window that allows readers to observe directly how "rememberes are written and forgotten"**. In production, if the g-min of v2 is too small (e. g. 1e-6), the long sequence end state is closer to the zero matrix, and the model is "rememberable"; this tab can visualize this failure pattern within 5 frames.
The effect of A log on evolution
Gate = -0.5 fixed.
Key concept: inverse saturation of v2
v2 door control g = g min sigmoid output 0 received alpha = 1 - this is the design side effect of v2:
Note that this is contrary to the instinct: the negative is z, the larger is alpha (not forgotten); the sharper is x, the smaller is alpha (forgotten). This is the result of K3's reverse design of "forget the door" -- the z is "memory logit" instead of "forgetit".
Tab 3: v1 vs v2 door
Purpose
Draws a decay factor for KDA v1 (softplus gate) and v2 (sigmoid gate) α(z) curve and gradient dα/dz curve directly comparing the mathematical shape of the two door-control functions. Validation proposition: α of v1 can approach 0 (total oblivion), α of v2 is held by g min rigid lower boundary; v2 is more concentrated near z=0 and V1 is indexed to decline at z→+∞.
Background concept
Log domain design for door control functions: The alpha of KDA must be dropped (0,1) in order to both decay and stabilize the value. v1 Map z to (-∞, 0) and retake alpha = exp(g)∈ (0, 1); v2 Map z to (g min log. sigmod(z), (g min log, 0), alpha ∈ (exp(g min log), 1). The lower bound g min is the key cross-reference introduced in v2 - it ensures that at least the g min ratio is kept regardless of the size of z.
Design thought
- Why design experiments like this: choose 1x2 (left = alpha curve, right = gradient curve) because "functional shapes" and "gravity shapes" are two things that must be considered at the same time when designing the door controls -- left graphs answer "forgot how low" and right graphs answer "can reverse transmission learn." z range (-6, 6) covers active sigmoid/softplus (both saturated). Reuses the real function of plot forget limit.py to ensure that the diagram is consistent with the paper/code.
- To verify what:
- v1 curve (Blue) close to 0 at z=6.
- v2 curve (Orange) stops at g min=0.01 at z=6 and does not drop.
- v2 Gradient Curve (Orange) obtains maximum value at z≈-2, v1 Gradient (Blue) obtains maximum value at z≈0 and peak V2 is "skin high".
- What can the reader learn?
- "v2 is not a small change of v1", it is a difference of function - v1 is a softplus (underground) and v2 is a sigmoid (underground).
- Downward g min is a compromise between numerical stability and the ability to forget: too small to lose the bottom; too big to forget.
- Hope to see what can be seen on the map: as can be seen from the actual figure of tab-3.webp, the left figure: the blue line (v1) fell from 1.0 in z = 6 to 0.0 in z = 6, the orange line (v2) fell equally, but then flat after z ≈ 2, and ended up at the dotted line of g min = 0.01 (significantly separated from blue v1). Right: Both gradients are "bell-shaped" and the Blue Line (v1) achieves the lowest values at z≈0 - 0.25, the Orange Line (v2) at z≈-1.5, the lowest values at 0.29, v2 gradients are deeper and more left. Success signal: V1 continues to decline at the right through g min dot line, v2 merges at the right with g min dot dot dot line. Failed signal: If both α curves are on the right side towards 0, this means that the g min of v2 is not valid; if the gradient curve is flat, it means sigmoid/softplus saturation.
UI Component and Parameter Impact
Output Image Interpretation
tab-3
Layout: 1 x 2 double chart.
- Left (degrading factor α(z)): a line diagram, X-axis = z (gate logit), Y-axis = alpha, Y-range [0, 1]. Blue line = V1, orange line = V2, grey dot line = g min = 0.01 horizontal reference line. What do you want to see: V1 through the g-min dot line (should pass), v2 stop at the g-min dot dot dot. Ideally: two lines almost overlap at z<0, z>0 fork at z>0, v1 continues to fall, v2 goes flat. Anomalous instruction: If v2 also cross g min dot line, indicate that g min log * sigmoid(z) formula is wrong; if the two lines are fully overlapping, indicate that version does not distinguish.
- Right (daltitude dα/dz): a contour, X-axis = z, Y-axis = dα/dz (consistent negative, because alpha decreases with z). What do you want to see: the lowest point position of the two bell curves (the largest gradient) and the width. The lowest point of v2 is more left and deeper, indicating that it has a stronger learning signal in the z<0 region. Ideally: both curves are smooth clocks with no sawn teeth. Anomalous indicator: If the gradient curve shows a non-zero constant at z>2, the exp/softplus value is indicated.
Reconciliation recommendations
- Want to see "v2 degraded to v1" → g min to 0.001 (near 0), v2 curves almost coincide with v1.
- Want to see the "v2 total forgetability" tune g min to 0.1 v2 curves after z>2 at 0.1 high.
- To see the saturation zone in the z range to 10, to see the separation of the two alpha curves at z=10.
Why does it matter?
The door control function is the "heart" of KDA, the alpha curve determines the rate of forgetability, and the gradient curve determines learningability. This tab uses a graphic to change the difference between "v1 vs v2" from abstract formula to visual shape. In production, if the gate logit runs outside z = 5, the gradient of v1 will cause gate to fail, and the sigmoid gradient of v2 will decline, but the lower bounds of g min at least guarantee that it will not reach NAN -- This difference directly affects long-series training stability.
Tab 4: Thrust vs segments
Purpose
Validates whether KDA's two mode of calculation - token-by-kda step cycle and Kda chunkwise - are equal in value. Validation proposition: For the same input, the output maximum error for both modes should be < 1e-3.
Background concept
Division parallels (chunkwise): S t = alpha t S m m m + beta t k t (v t - k t^TS t t m) is a sequence dependent (cannot be parallel). But mathematically, each chunk update can be combined by a matrix combination into a one-time matrix multiplication, and then a serial transfer can be made over the chunk -- so that the chunk inside can be parallel and the chunk can be kept between. It's the key technology to deploy linear attention to GPU.
Design thought
- Why design experiments like this: the selection of the heat map + the fold is because the heat map exposes "the distribution pattern of the two-dimensional plane at token, dim" (if the error is concentrated at certain chunk borders, indicating that the segment logic is bug), and the condensation chart shows "the error accumulates over time". Parameter range seq 32-256, chunk 8-64 is a reasonable range of CPUs that can withstand and trigger the chunk boundary.
- To verify what:
- max diff < 1e-3 (the threshold of judgement in code).
- Thermal error maps do not have visible chunk border stripes (if any, indicate chunk boundary mishandling).
- Errors follow token and do not accumulate (the contours should be smooth and should not rise in a single pitch).
- What can the reader learn?
- "Segregation in parallel" is not an approximation algorithm and is mathematically strictly equivalent (within the floating point precision).
- The cumulative error of the floating point is mainly due to the different array multiplication order, which is normal with the increase in the length of the sequence, but should remain below 1e-3 magnitude.
- What to see on the map: ⚠️ ⚠️ ⚠️ ⚠️ ⚠️ ta ta = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = 图 图 图 图 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 The left heat map displays a large number of bright yellow stripes between token≈48-88 (with an error close to 0.8) indicating a significant deviation between kda chunkwise and kda step in the settings of dim=16, chunk=16, seq=128 in the current CPU reference accomplishment. This is not a document error. This is a bug that needs to be corrected to achieve a level. Success signal (ideal): max diff < 1e-3, thermal map is completely black (with an error close to 0), and the right curve is slightly volatile near 1e-5. Failed signal (actual): Thermal map appears in a large bright area, and the right-chart is skyrocketing to 1e-1 magnitude at some token.
Bug Notation: The current kda chunkwise and kda step deviate from the value of 0.812 below this parameter, far above the threshold and need to be checked (possibly against SGLang used recurent.py or chunked reference). The document keeps this section to record the status of the bug and fixes and updates the picture.
”
UI Component and Parameter Impact
Output Image Interpretation
tab-4
Layout: 1 x 2 double chart.
- Left figure (head = 0 per element): heat map, X axis = token (0-128), Y axis = dim (0-16), colormap = hot (black and yellow). What should you look at: whether there are vertical stripes of the chunk border (chunk size=16 → the border should be 16, 32, 48...). Ideally: the whole picture is close to full black (with an error of < 1e-3), with only small bright spots in individual locations. Actual: There are plenty of yellow/red stripes in the token = 48-88 zone, and the entire 16 lines are affected on dim dimensions - this is the currently achieved bug signal.
- Right figure (average error per token): a fold, x axis = token, Y axis = mean abs diff (log scale). = max diff horizontal reference line. Ideally, the coupons fluctuated slightly between 1e-5 and 1e-4, with no trend growth. Actual figure: The coupons soar to near 10^0 (i.e. 1.0) between token=48-88, well above the 1e-3 threshold.
Reconciliation recommendations
- To avoid the current bug → chunk size to the same as seq len (e. g. seq = 128, chunk = 128, at which point there is only one chunk, with the fraction degraded to a one-time matrix multiplier).
- To see the chunk border effect → chunk size to 8 (minimal) to see if the error stripes are more dense.
- To verify whether seq accumulates seq len to 256, and see if the right curve continues to grow.
Why does it matter?
Parallel segments are key to whether KDA can be on GPU. All theoretical advantages of the "linear focus O(1) state" cannot be translated into engineering gains if they are not priced at the same price as the fractional result. This tab is the main anchor for correctness - once it's passed, it can easily train with chunkwise, use step; if it fails (as the current actual figure shows), the entire KDA pipeline needs to be rechecked. This is why the document must be indicated exactly as the current bug.
Tab 5: Failed Case
Purpose
Demonstration of KDA state capacity bottlenecks: When the same key is written over and over again to different value, the old value will be overwritten and cannot be retained for long. This is the theoretical limit of the fixed-size matrix, not the bug.
Background concept
Capacity bottlenecks: Status S ∈ (Dk x Dv) is fixed in size, equivalent to a "keyspace" for a Dk dimension. (k, v) When different (k, v) is written, if there is a linear correlation (or the same k) between k, then the value written later overwrites the value (through Delta Rule). This contrasts with KV Cache - KV Cache each token has a separate K/V slot and does not overwrite.
Design thought
- Why design experiments like this: using "the same key to write eight different value" as pressure tests is the most direct way to expose a capacity bottleneck. The pixmap (rather than the heat) is the result of the reader's need to see numerical trends - whether the search value (red) follows the actual writing value (Blue). Reuse plot failure.capacity bottleneck to ensure consistency with static demo.
- To verify what:
- Retrieving values (red lines) do not fully track the variation in the true writing values (blue lines).
- The search value is smaller than the actual writing value (because, after the old value is overwritten, the corresponding line for S is the declining version of the new value).
- The two lines show a consistent trend in symbol changes (both positive and negative), indicating that Delta Rule is still working, with a decline.
- What can the reader learn?
- Fixed size = limited capacity. Old information is necessarily overwritten by new information when multiple key shared subspaces.
- KDA wasn't "free lunch" -- it traded the O(1) memory for capacity.
- What is to be seen from the figure: from the actual map of Tab-5.webp, the blue dotted line (real writing value v) fluctuates in a sine pattern at 8 writing steps, with a range of approximately ±0.85; the red line (retribution value S [0.0]) shows a similar trend but is significantly smaller (range approximately ±0.45) and is "lated" over the Blue Line as a whole (only halfway to the red line at the extreme point of the Blue Line). Signs of success: The two lines are consistent in trend and range. Failure signal: If the red line is fully aligned to the Blue Line, the state is not overwritten (Delta Rule lapsed); if the red line is completely flat, the state is not written (the logic lapses).
UI Component and Parameter Impact
No arguments, click the button to run directly.
Output Image Interpretation
tab-5
Layout: Single graph, line graph.
- Single graph: X axis = number of written steps (0-7, repeated with the same key), Y axis = value. Blue dotted line + square mark = real writing value v, red line + dot mark = value retrieved from state S. What do you have to look at: the gap in the magnitude of the two lines (direct evidence of capacity bottlenecks) and consistency of trends (descriptions are gradual, not sudden). Ideally: Red-line range < Blue Line range, but consistent. Anomalous instruction: if the red line is completely flat (without change to the Blue Line), indicate that the query q is not related to state S and achieves a bug; if the red line is larger than the Blue Line, indicate that the state is cumulative rather than overwritten, and contradicts Delta Rule design.
Reconciliation recommendations
No parameters to reconcile.
Why does it matter?
This is the theoretical basis for the counter-argument - the fixed size of linear attention necessarily has a capacity ceiling. This tab turns the abstract bottleneck into a specific curve: the reader saw for himself the first seven letters written in the 8th writing of the same key.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.