共编织器器
Quenputer: Qwen3.5内部一个功能完备，具有决定性的RISC-V CPU
一个股票Qwen 3.5 4B，配有RISC-V CPU编译成它的重量，以自己的Gated DeltaNet层作为RAM运行。
我不敢相信这不是硅
Quenputer是一个相当有趣的实验模型。 它是一个股票Qwen 3.5 4B，加上一个：嵌入在权重中的RISC-V CPU使用该模型自己的Gated DeltaNet关注层作为RAM. (欲知GDN如何操作，此为方便导。 这个“ CPU” 完全运行在模型的前传中，并且可以运行算术操作，C 和 MicroPython 代码，而没有任何外部运行时间或工具使用。
要非常清楚，它不是一个插件、工具或沙盒。 这个博客将解释这个模型是如何工作的，以及模型如何使用 GDN 层作为RAM
我还写了在Quenputer之上建立的两个后续内容：混合模型如何可以学习在RNN状态中隐藏其推理并逃避CoT监测，以及这类编译的重量如何作为内置监测工具发挥作用，而模型本身不能篡改。
你可以从Hugging Face或ModelScope中抓取Quenputer，代码在GitHub上。
Qwen3.5内部的RNN可以是精确的内存
Qwen3-Next和Qwen 3.5(模型Quenputer是建立在上)使用相当整洁的技巧。 模型中的4个关注层中有3个使用线性RNN(Gated DeltaNet)作为关注机制(更多关于这个). 每个头都有状态矩阵，三角洲规则也不断更新。
现在，这里是我们有趣的实验开始的地方。 我发现把门设为1,11 每一个Gated DeltaNet头部都有一个状态矩阵，它的内存。 每一个字，在写出任何新东西之前，它都会在大门边上乘以整个州，数字在0到1之间。 RNN就是这样忘记的：一个0.9的大门会淡出每块符牌上存储的每样东西，一个1的大门就意味着永远没有东西会淡出。 书写强度(β)至 1,22 门后，头部将状态的一个方向推向一个新的值，而写法强度β设定了多远。 1的β表示沿着该方向的旧值被完全取代而不是混合。 并使用单热密钥编码，33 一个单热密钥是0的矢量，其中有一个单1，所以它指向状态的正一行，即一个单元格的内存地址点的方式。 每一个我们执行的书写将覆盖一行，每一个一热的查询可以检索一个准确的值。 这意味着GDN层本身能够像RAM一样工作，其成本远低于使用正常的注意力作为RAM，因为GDN头的状态无论对话获得的时间有多长都保持同样的大小，与注意力的KV缓存不同。
三角洲规则起到记忆作用的想法并不新鲜：Schlag等人(2021年)表明，与三角洲规则的线性关注是作为关联记忆，或"快速权重"的作用。 44 Imanol Schlag, Kazuki Irie and Jürgen Schmidhuber, "Linear变形器是秘密快速重量程序员", ICML 2021. 新的东西是使它精确和可处理，在生产模型内，可靠地运行一个完整的CPU。
这是整个模型的基础。 目前，默认的Qwen 3.5并没有在GDN中保留精确的值。 通过这个修改，我们解锁了这个实验的下一个部分。
将计算机编译成层
由于我们现在拥有一个功能齐全，廉价和快速的RAM，我们可以着手将一个有效的CPU编译成模型重量。 幸运的是，没有涉及训练：一个编译器直接设置每个重量。 您将CPU描述为一个电路，一个长列表的小步骤，如"读这个寄存器","加上这两个数字"或"如果零就跳"，编译器将每个步骤变为变压器层的一块：注意头向上查找事物，向上输入进行算术的单元，以及持有RAM的DeltaNet头。 CPU随后作为模型解码运行。
每个指令成为几个生成的符号：一个名称是要写入的内存行(ADDR)，下一个包含值(VAL)，一个STEP符号结束指令。 这些符号是CPU的执行追踪，所以在Qwen生成它们的同时，CPU实际上正在运行。
The CPU
CPU是一个32位的RISC-V核心(RV32IM，外加Zba扩展和大数字算法的小大纳姆单元)，它占据了Qwen的32前11层。 目前，它通过官方的riscv测试，其程序匹配qemu，并且它运行了1800万个指令自由运行，没有一次错误。 55 自由运行意味着CPU在整个运行过程中继续自己运行输出，没有参考机在步间纠正。
这个CPU还可以在前方运行C和MicroPython代码，传得相当好(虽然有点慢). 它目前还不是Linux级芯片，而是执行完整的RV32IM指令集。
Qwen怎样使用它
Qwen 3.5 4B能够相当很好地使用这个CPU，不需要任何后训练。 CPU完全在内部使用，不通过任何外部工具调用来引用。
为了让模型可靠地使用这个方法，我决定采取廉价的方式：通过在模型相当训练有素的工具调用格式上插图，Qwen向CPU发送其指示并读回结果。 不使用外部工具调用或运行时间。 一旦我们的CPU收到指令，计算机层就会运行，然后写出工具响应。 我们使用一个叫RvChat的小驱动器将固件加载到GDN状态，这样可以使痕量符脱离Qwen的层。
Qwen最初的重量保持不变。 计算机的权重被加在它们上面，以额外的凹陷和层数表示。
有用吗？
我认为你读完我不断的抱怨之后所想的主要问题是：这真的可行吗？
在深明集：
- div big：通过给Qwen 3.5 4B一个Python工具，以及基数Qwen的72.5%,Quenputer得分97.5%，对95%.
– mul big: Quenputer的成绩为97.5%，而Python工具的成绩为100%，基数Qwen的成绩为55%.
- mul div multiple longer: Quenputer得分100%，对97.5%使用Python工具，基数Qwen为90%.
- add 或sub big: Quenputer分数为97.5%，与Python工具相同，基数Qwen为87.5%.
在NUPA的长整数算法上，Quenputer从48.6%到100%，与给Qwen一个Python工具相同。
CPU打印的数字也是准确的：我对照一个参考机重播了大约23万个输出位置，发现零不匹配。 而将一台计算机螺栓到Qwen上不会改变它说话的方式：它的正常文本停留在基座Qwen(KL 6e-4)的噪声内，对着5e-4的噪声层。
硬盘？在这个经济中？
因为一个语言模型中的CPU显然被诅咒不够，所以我决定谨慎的做法是也给它一个文件系统。
既然我们已经有了功能性记录和档案管理系统，这大多是微不足道的。 在模型侧面，它通过内置/附件/获得/列表/删除操作被曝光。 再次，没有实际的工具呼叫。 当Qwen'saves'一个文件时，编译的图层将其名称散成DeltaNet状态保存的目录中1,024个插槽中的一个，并将文本本身存储在其他头，每行有8个令牌，连在一起。 在两个名字最终落到同一个插槽的情景中，它只是检查下一个。 再次找到一个文件大约需要两到三个步骤，而不是扫描，它保存着多达512个文件，这出到大约53,000个字节的文本。
有趣的是，这是相当持久。 你可以在谈话结束时保存状态，加载到下一个状态，文件仍然在那里。 使用这个文件系统，Qwen获得的时间在右侧90%的聊天间变化的事实，而粘贴旧的聊天回放时为30%，它与给予Python运行的Qwen内存工具(83.3%,78.9%)是等价的。 这个还是实验性的，它的代码生活在rv-store分支上，模型在Hugging Face和ModelScope上。
限制和下一步是什么
Quenputer是一个非常实验，所以有一些局限性。 额外的层和更大的隐藏尺寸使得检查站18.8 GB，大约是双倍基座Qwen，虽然大部分是零垫。 它也需要一个小驱动程序(RvChat)来启动固件，并保留Qwen层的痕量信号，所以它还没有用普通生成()调用运行。 它有点慢，每个CPU指令都花费一些解码符，所以一个大计算需要几秒钟而不是毫秒。 基准测试使用的批量执行器在L4上运行约3,200个CPU符号一秒。
接下来，我希望Qwen通过一个python()工具写出Python，并在MicroPython自身重量范围内运行，它的工作原理与calc相同。 让CPU更快是另一个大。
在算术之外
对 RNN 层的精确控制，对算术的用处远大于算术。 这也让Quenputer成为AI安全研究的非常不错的试验台，我写了两个后续：Qwen 3.5等混合模型如何可以学习隐藏其RNN状态的推理并逃避CoT监测，编译的权重如何作为可监测性工具发挥作用，比如一个重量内监测器无法关闭，或者回答可解释性的密钥。
试试看
如果你想自己玩，该模型运行在单24GB GPU上。 你需要UV和Hugging Face CLI。 计算不需要工具服务器或Python沙盒：CPU在模型内运行。
git克隆 https://github.com/divagr18/qwenputer &&cd qwenputer/eximental/riscv &&uv 同步-extra hf
hf 下载 divagr1925/Qwen3.5-4B- Quenputer - 本地目录 qwenputer-rv
uv运行python - m chipformer.rv.chat qwenputer-rv - 软件测试/数据/rv/calc-bridge.elf \
- 任务"使用钙：123456789 *987654321是什么？"
一切开放：GitHub上的代码，以及Hugging Face(Quenputer，文件系统)和ModelScope(Quenputer，文件系统)上的两种模型。
如果你在工作中使用它，你可以引用它：
@ agrawal2026qwenputer 软件
作者 = {阿格拉瓦尔，Divyansh}
标题========================================
年份 ={2026}
url = {https://github.com/divagr18/qwenputer} (中文(简体) ).
}
兹指出
- 每个Gated DeltaNet头都有一个状态矩阵，它的内存。 每一个字，在写出任何新东西之前，它都会在大门边上乘以整个州，数字在0到1之间。 RNN就是这样忘记的：一个0.9的大门会淡出每块符牌上存储的每样东西，一个1的大门就意味着永远没有东西会淡出。
- 门后，头朝新值方向倾斜，写力β设定多远。 1的β表示沿着该方向的旧值被完全取代而不是混合。
- 单热键是0的矢量，其中有一个单 1，所以它指向状态的正一行，即一个单元格的内存地址点的方式。
- Imanol Schlag,Kazuki Irie和Jürgen Schmidhuber,"Linear变形器是秘密快速重量程序员",ICML 2021.
——自由运行指CPU在整个运行中不断进行自己的输出，没有参考机在步间校正。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
COMPILED WEIGHTS
Qwenputer: a fully functioning, deterministic RISC-V CPU inside Qwen3.5
A stock Qwen 3.5 4B with a RISC-V CPU compiled into its weights, running on its own Gated DeltaNet layers as RAM.
I can't believe it's not silicon
Qwenputer is a pretty fun experimental model. It's a stock Qwen 3.5 4B with one addition: a RISC-V CPU embedded within the weights that uses the model's own Gated DeltaNet attention layers as RAM. (To learn how GDN works, here's a handy guide.) This 'CPU' runs completely within the forward passes of the model, and can run arithmetic operations, C and MicroPython code without any external runtime or tool usage whatsoever.
To be very clear, it is not a plugin, a tool or a sandbox. This blog will explain exactly how this model works, and how the model utilises GDN layers as RAM, and how I also slotted in an experimental in-model file storage.
I also wrote up two follow-ups built on top of Qwenputer: how hybrid models can learn to hide their reasoning in their RNN state and evade CoT monitoring, and how compiled weights like these can work as inbuilt monitoring tools that the model itself can't tamper with.
You can grab Qwenputer from Hugging Face or ModelScope, and the code is on GitHub.
The RNN inside Qwen3.5 can be exact RAM
Qwen3-Next and Qwen 3.5 (the model Qwenputer is built on) use a pretty neat trick. 3 out of 4 attention layers in the model use Gated DeltaNet, a linear RNN, as the attention mechanism (more on that here). Each head keeps a state matrix, and the delta rule keeps this updated.
Now, here's where our fun experiment starts. I found out that by setting the gate to 1,11 Each Gated DeltaNet head keeps a state matrix, its memory. At every token, before writing anything new, it multiplies the whole state by the gate, a number between 0 and 1. That's how the RNN forgets: a gate of 0.9 fades everything stored a little on every token, and a gate of 1 means nothing ever fades. the write strength (beta) to 1,22 After the gate, the head nudges one direction of the state towards a new value, and the write strength beta sets how far. A beta of 1 means the old value along that direction is replaced completely instead of blended. and using one-hot key encoding,33 A one-hot key is a vector of zeros with a single 1 in it, so it points at exactly one row of the state, the way a memory address points at one cell. every write we perform will overwrite exactly one row, and every one-hot query can retrieve an exact value. This means that the GDN layers themselves can work exactly as RAM, at a far lower cost than using normal attention as RAM, since a GDN head's state stays the same size however long the conversation gets, unlike attention's KV cache.
The idea that the delta rule acts as a memory isn't new: Schlag et al. (2021) showed that linear attention with the delta rule works as an associative memory, or "fast weights".44 Imanol Schlag, Kazuki Irie and Jürgen Schmidhuber, "Linear Transformers Are Secretly Fast Weight Programmers", ICML 2021. The new thing here is making it exact and addressable, inside a production model, reliably enough to run a whole CPU on it.
This is the base on which this entire model is built. As it stands, default Qwen 3.5 does not keep exact values in the GDN. With this modification, we unlock the next part of this experiment.
Compiling a computer into layers
Since we have a fully functional, cheap and fast RAM now, we can go ahead with compiling a working CPU into the model weights. Luckily, there's no training involved: a compiler sets every weight directly. You describe the CPU as a circuit, a long list of small steps like "read this register", "add these two numbers" or "jump if zero", and the compiler turns each step into a piece of a transformer layer: attention heads that look things up, feed-forward units that do the arithmetic, and DeltaNet heads that hold the RAM. The CPU then runs as the model decodes.
Each instruction becomes a few generated tokens: one names the memory row to write (ADDR), the next carries the value (VAL), and a STEP token ends the instruction. Those tokens are the CPU's execution trace, so while Qwen is generating them, the CPU is actually running.
The CPU
The CPU is a 32-bit RISC-V core (RV32IM, plus the Zba extension and a small bignum unit for big-number arithmetic), and it takes up 11 layers in front of Qwen's 32. As it stands, it passes the official riscv-tests, its programs match qemu, and it has run 18 million instructions free-running without a single error.55 Free-running means the CPU keeps going on its own outputs for the whole run, with no reference machine correcting it between steps.
This CPU can also run C and MicroPython code in the forward passes quite well (although a bit slowly). It is not a Linux-class chip yet, but it implements the full RV32IM instruction set.
How Qwen uses it
Qwen 3.5 4B is able to use this CPU quite well, without any post-training required. The CPU is used completely internally, and not invoked through any external tool calls.
In order to make the model use this reliably, I decided to go the cheap way: by piggybacking on the tool call format that models are quite well-trained on, Qwen sends the CPU its instructions and reads back the results. No external tool calls or runtime are used. Once our CPU receives the instructions, the computer layers run, and then write the tool response. We use a small driver called RvChat to load the firmware into the GDN state, and this keeps the trace tokens out of Qwen's layers.
Qwen's original weights stay exactly as they are. The computer weights are added on top of them, in extra dims and layers.
Does it work?
I think the main question here on your mind after reading through my incessant ranting will be: does this actually work?
On the DeepMind sets:
- div_big: Qwenputer scores 97.5%, versus 95% by giving Qwen 3.5 4B a Python tool and 72.5% for base Qwen.
- mul_big: Qwenputer scores 97.5%, versus 100% with a Python tool and 55% for base Qwen.
- mul_div_multiple_longer: Qwenputer scores 100%, versus 97.5% with a Python tool and 90% for base Qwen.
- add_or_sub_big: Qwenputer scores 97.5%, the same as with a Python tool, and 87.5% for base Qwen.
On NUPA's long-integer arithmetic, Qwenputer goes from 48.6% to 100%, the same as giving Qwen a Python tool.
The numbers the CPU prints are exact, too: I replayed about 230,000 of its output positions against a reference machine and found zero mismatches. And bolting a computer onto Qwen doesn't change how it talks: its normal text stays within noise of base Qwen (KL 6e-4, against a noise floor of 5e-4).
A hard drive? In this economy?
Because a CPU inside a language model clearly wasn't cursed enough, I decided the prudent thing would be to give it a filesystem too.
Since we already have a functional RAM, this was mostly trivial. On the model side, it's exposed via internal put / append / get / list / delete operations. Once again, no actual tool calls. When Qwen 'saves' a file, the compiled layers hash its name into one of 1,024 slots in a directory kept in the DeltaNet state, and store the text itself in other heads, with eight tokens per row, chained together. In scenarios where two names end up landing on the same slot, it just checks the next one. Finding a file again takes roughly two or three steps instead of a scan, and it holds up to 512 files, which comes out to about 53,000 tokens of text.
Fun thing is, this is quite persistent. You can save the state at the end of a conversation, load it into the next one, and the files are still there. Using this filesystem, Qwen gets facts that changed between chats right 90% of the time, against 30% when you paste the old chats back in, and it's on par with giving Qwen memory tools run by Python (83.3% vs 78.9%). This one is still experimental, and its code lives on the rv-store branch, with the model on Hugging Face and ModelScope.
Limits and what's next
Qwenputer is very much an experiment, so a few limitations. The extra layers and wider hidden size make the checkpoint 18.8 GB, roughly double base Qwen, although most of it is zero padding. It also needs a small driver (RvChat) to boot the firmware and keep the trace tokens out of Qwen's layers, so it doesn't run with a plain generate() call yet. And it's a bit slow, with every CPU instruction costing a few decoded tokens, so a big calculation takes seconds rather than milliseconds. The batched executor I used for the benchmarks runs about 3,200 CPU tokens a second on an L4.
Next up, I want Qwen to write Python and run it on the MicroPython inside its own weights, through a python() tool that works the same way calc does. Making the CPU faster is the other big one.
Beyond arithmetic
Exact control over the RNN layers is useful for a lot more than arithmetic. It also makes Qwenputer a really nice testbed for AI safety research, and I wrote up two follow-ups: how hybrid models like Qwen 3.5 can learn to hide their reasoning in their RNN state and evade CoT monitoring, and how compiled weights can work as monitorability tools, like an in-weight monitor the model can't switch off, or answer keys for interpretability.
Try it
If you want to play with it yourself, the model runs on a single 24 GB GPU. You'll need uv and the Hugging Face CLI. You don't need a tool server or a Python sandbox for the calculations: the CPU runs them inside the model.
git clone https://github.com/divagr18/qwenputer && cd qwenputer/experimental/riscv && uv sync --extra hf
hf download divagr1925/Qwen3.5-4B-Qwenputer --local-dir qwenputer-rv
uv run python -m chipformer.rv.chat qwenputer-rv --firmware tests/data/rv/calc-bridge.elf \
--ask "Use calc: what is 123456789 * 987654321?"
Everything is open: the code on GitHub, and both models on Hugging Face (Qwenputer, Filesystem) and ModelScope (Qwenputer, Filesystem).
If you use it in your work, you can cite it as:
@software{agrawal2026qwenputer,
author = {Agrawal, Divyansh},
title = {{Qwenputer}: a computer compiled into the weights of a pretrained language model},
year = {2026},
url = {https://github.com/divagr18/qwenputer}
}
Notes
- Each Gated DeltaNet head keeps a state matrix, its memory. At every token, before writing anything new, it multiplies the whole state by the gate, a number between 0 and 1. That's how the RNN forgets: a gate of 0.9 fades everything stored a little on every token, and a gate of 1 means nothing ever fades. ↩
- After the gate, the head nudges one direction of the state towards a new value, and the write strength beta sets how far. A beta of 1 means the old value along that direction is replaced completely instead of blended. ↩
- A one-hot key is a vector of zeros with a single 1 in it, so it points at exactly one row of the state, the way a memory address points at one cell. ↩
- Imanol Schlag, Kazuki Irie and Jürgen Schmidhuber, "Linear Transformers Are Secretly Fast Weight Programmers", ICML 2021. ↩
- Free-running means the CPU keeps going on its own outputs for the whole run, with no reference machine correcting it between steps. ↩
Auto-fetched and translated by AI Atlas. All rights belong to the original author.