中小企业跑125B 大模型：显存装不下的时候，Strata 能做什么？
作者：吴佳浩 撰稿时间：2026-10-24 测试环境：
- GPU：RTX 5090（32GB VRAM，驱动 580.142，CUDA 13.0，PCIe Gen3 ×16）
- CPU：Intel Xeon Gold 5218 ×2（32 物理核 / 64 线程，2 个 NUMA 节点，无 AVX512-BF16）
- Memory：125.6 GiB RAM
- Storage：/workspace 3.5TB（/dev/sdb1，实测顺序写 435 MB/s）
- OS：Ubuntu 20.04.3 LTS（kernel 5.15.0-58）
- Engine：Strata v0.2.2（engine 0.1.47，CUDA 13.0，官方容器镜像）
- Model：Qwen3.8-Flash-Next Q2_0（125B 级 MoE，48 层 × 512 专家 = 24,576 个专家槽）
引言
面向工程师与研究者的体系化分析。之前我们在个人pc上做了尝试今天待大家从企业视角来来看看 这条线是不是行得通。 阿拉不止讨论「跑起来了没有」，更追问「这件事对本地部署意味着什么、代价又落在哪一层」。 125B 级 MoE 模型过去只活在几百 GB 显存的服务器上。Strata 把「模型装不下」这件事，从一道算术死题，换成了一个带宽工程题。 本文是这个系列的第一篇，只讲一件事：当显存放不下模型时，Strata 到底把什么东西放在了哪里，又是怎么把「显存缺口」翻译成「吞吐」的。
注：全文所有数字均来自本机真实运行的日志与遥测，未经验证的推算会明确标注为推算。因为阿拉好奇，所以拿一块 32GB 的卡，一道来白相相——看看这套 Spill 设计是徒有虚名，还是真的结棍。
目录
一、问题的本质：为什么「显存占用率」是个废指标
1.1 先把算术摆出来
先看这台机器上真正要放进去的东西有多大。用我这套部署里实际落盘的 Q2_0：
而 RTX 5090 的显存是 32,607 MiB，约 31.8 GiB。
66.4 GB 对 31.8 GiB，缺口接近 2.1 倍。 这不是调参能解决的问题，是算术问题。
1.2 MoE 把「算力」和「显存」解耦了
关键在于这个模型是 MoE（混合专家）：48 层，每层 512 个专家，但每个 token 每层只路由 10 个。
稠密模型：算力 ∝ 总参数量
MoE 模型：算力 ∝ 激活参数量（远小于总量）
显存 ∝ 总参数量（一点没少）
这条式子解释了消费级显卡的处境：算力富余，显存贫瘠。我这台机器上，压测期间 GPU 功耗峰值 472 W，而卡的上限是 575 W——算力还有余量；但显存占用稳定在 31,879 / 32,607 MiB，也就是 97.8%，一格都挤不出来了。
1.3 「显存占用率」为什么骗人
这是本文最想先纠正的一个观念。
我做了两轮完全相同的部署，唯一区别是显卡上有没有别的进程在占显存。结果：
两轮显存占用率几乎一模一样（差 0.1 个百分点），速度却差 82%。
所以「显存占用 97%」这个数字本身不携带任何性能信息——因为 Strata 的设计就是把能用的显存全部吃满来放专家缓存。它不占满才是浪费。
本文后面会证明：真正决定速度的，不是显存占了多少，而是「缓存之外的专家占多少」。 前者恒等于 97%，后者才是那个会变的数。
二、Strata 的显存账本：三个区、一条公式
2.1 三个区
启动后的引擎日志把显存分成了清楚的三块。用第二轮（独占）的实测值：
用引擎日志的原话对一遍：
strata generate: 301 native projection matrices, 1376.20 MiB of weights, in 1.8 s
strata mtp: draft layer loaded, 839 MiB of VRAM (experts 675, dense 111)
strata mtp: draft head over 106299 tokens (178.4 MiB)
strata verify: window up to 6 tokens, 55.0 MiB of device buffers
strata generate: expert cache auto: 26.01 GiB free, 700 MiB reserved (+184 MiB for the draft head) -> 19530 slots
strata generate: expert cache 19530 slots, 25.14 GiB of VRAM
strata serve: 410 MiB of VRAM free with everything loaded
2.2 一条可以自己算的公式
从两轮日志里能反推出这套引擎的分配常数，而且两轮完全一致：
- 每个专家槽位 1.318 MiB（7,625 / 5,784 = 1.318；25,747 / 19,530 = 1.318）
- 固定预留 700 MiB（系统）+ 184 MiB（投机解码头）
- 槽位上限 24,576（= 48 层 × 512 专家）
写成一条式子：
\(\text{槽位数} = \min\left(24576,\ \frac{\text{启动时空闲显存} - 884\text{ MiB}}{1.318\text{ MiB}}\right)\)
拿实测回代，三个不同显存条件下的启动全部对得上：
（三行都取自引擎日志原文，误差 0.04% 以内，来自字节对齐取整。）
这条公式的实际价值：你不需要跑基准，只要知道启动时能空出多少显存，就能算出这台机器能覆盖多少比例的专家。第三篇里我把它做成了一张速查表。
2.3 一个反直觉的推论：多给的显存几乎全部变成缓存
把两轮的「引擎总显存」减去「专家缓存」，得到的就是固定开销：
6,045 vs 5,927，差 2%。 也就是说，固定开销基本不随缓存大小变化。
结论很干净：你多给显卡的每一 GB 显存，几乎 1:1 变成了专家缓存。 不存在「显存越大、开销越大」的摊薄陷阱——反而是显存越大，固定开销的占比越被摊薄，显存利用效率越高。
三、一个 token 的显存之旅
前两节是静态的账本，这一节跟一个 token 走一遍完整路径，看清楚「性能」到底花在哪里。
三个细节值得单独说。
3.1 为什么「全部专家」必须待在主机内存里
引擎加载时把完整的 24,576 个专家搬进了主机内存 arena：
strata generate: expert arena: cudaHostRegister PORTABLE ok
strata generate: loaded 31.64 GiB at 0.99 GiB/s
strata generate: expert arena read through the file cache (buffered)
31.64 GiB 常驻内存，加上 PLE 表（--ple-io ram）和进程本身，压测期间系统内存稳定在 96.3 GiB / 125.6 GiB。
这一步是「慢」的来源，也是「能跑」的前提：只有全部专家都在内存里，才能做到「未命中时立刻能搬」，而不是回头去读 SSD。
3.2 KV Cache 便宜得让人意外
我原本以为长上下文会让 KV 吞掉大量显存，实测打了脸：
上下文从 32K 翻到 64K，只多花了 55 MiB 显存。
原因是这个模型的注意力结构是带压缩的（GGUF 元数据里有 qwen4exp.attention.compress_ratios，48 层各一个压缩比），KV 本身就是压缩表示。所以在这套架构里：
上下文长度几乎不花显存，显存预算基本可以全部押在专家缓存上。
这和很多「长上下文很吃显存」的经验不同，是模型结构差异带来的红利。第三篇会用它解释「为什么 64K 上下文几乎不损失速度」。
3.3 MTP 投机解码：让一次搬运服务多个 token
Spill 架构下 decode 是逐 token 的，每个 token 都要搬一次专家。工程上的破解办法是投机解码：用一个小草稿模型先猜接下来几个 token，再让主模型一次并行校验。
我这套配置里 MTP 的实测接受率：
64% 的接受率意味着：大约每 1.6 个 token 就能省下一次专家搬运。这也解释了后面实测里 decode 能到 200+ tok/s——单靠「一次搬一个专家算一个 token」是做不到的。
四、模型是什么格式：GGUF、pack、MTP 三层产物
很多人以为本地推理就是「下载一个模型文件然后加载」。这套东西实际落盘是三层不同格式的产物，搞清楚它们才能理解「哪个文件对应哪块显存」。
4.1 主模型：GGUF v3，而且是混合精度
我直接读了两个分片的文件头：
「Q2_0」只是档位标签，文件其实是混合精度。 按 tensor 统计：
所以它是「2-bit 专家 + 4~6 bit 投影 + BF16 超连接」的混合精度，不是一个统一 2-bit 文件。这也是为什么同样的「Q2_0」在不同仓库里质量表现会不一样——混合策略不同。
4.2 运行时产物：三层落盘
各层的作用：
引擎的启动参数正好把这五块串起来：
strata --serve \
--pack /data/Strata-data/packs/q2_0 \
--native /data/Strata-data/models/Q2_0/...-00001-of-00002.gguf \
--ple-gguf /data/Strata-data/models/Q2_0/...-00002-of-00002.gguf \
--mtp /data/Strata-data/mtp/rt \
--max-context 32768 --kv int8 --ple-io ram
4.3 这套格式带来的两个后果
后果一：不能直接丢给 vLLM / SGLang。 它要的是 GGUF + 专门的运行时；HF safetensors 那套流程吃不了。
后果二：目录结构就是显存账本的物理映射。 之前那 31.64 GiB 的 arena 和 25.7 GiB 的缓存，都能在磁盘上找到对应来源——这也意味着换更小的量化档，省下来的是 arena，而不是缓存。这一点在第三篇选型时会用到。
五、效率是怎么被决定的：五段传导链
到这里，静态结构已经清楚。接下来把「显存」和「tok/s」之间的因果链接起来——这是本文的核心，也是第二篇实测要逐段验证的东西。
五段的性质完全不同：
这条链里真正"非线性"的是第③段和第⑤段。 而第①段那个漂亮的 1:1，恰恰是最容易被误读的地方——很多人看到「显存翻倍只快 1.8 倍」就以为存在「边际递减」，其实是把第③⑤段的非线性算到了第①段头上。
第二篇会把这条链逐段用 2,890 条逐请求遥测拆开验证。
5.1 一个必须先说清的坑：不是「搬得慢」，是「等得久」
按带宽算，未命中的成本不该那么贵：
每 token 480 次专家查表 × 1% 溢出 = 4.8 个专家
× 1.38 MB = 6.6 MB / token
÷ 13.9 GB/s（实测 PCIe Gen3 ×16）= 0.47 ms
而 141 tok/s = 7.1 ms / token
→ 0.47 / 7.1 ≈ 6.6%
理论上只该慢 6.6%，但我实测在 0.5% 溢出时就已经慢了 39%。差了 6 倍。
原因不在带宽，在同步：PCIe 搬运是异步发起的，但主模型的 attention 与 MoE 计算有严格的数据依赖，一批专家没到，后续 kernel 只能干等。等待期间 GPU 表面上是「忙」的，实际在空转：
「GPU 利用率」这个指标在这类架构上是失效的——它只回答「有没有活干」，不回答「干的活有没有产出」。第二篇会用功耗曲线把这个现象坐实。
六、可验证的预测：把「玄学」变成可测量的量
理论讲完，得给出可以被证伪的预测。基于上面五段链，我在跑第二轮之前写下了四条预测：
第二轮实测结果（详见第二篇）：
四条全中，说明这条五段链不是一个「事后编出来的故事」，而是能在跑之前就能预测、跑之后能被数据对上的机制。
七、总结
7.1 本篇的三条结论
7.2 一句话判断
消费级显卡跑 125B MoE，难的不是让它算得动——它算得动，功耗只用掉六成；难的是让热专家待在显存里。
而这件事的上限，被一条简单的式子锁死：(可用显存 − 884 MiB) / 1.318 MiB，最多 24,576 个槽。
7.3 下一篇讲什么
第二篇拿着这套机制去做对照实验：同一台机器、同一份模型、同一套参数，只把显卡上别人的 18 GB 占用停掉，然后用两轮完整基准 + 2,890 条逐请求遥测，把五段传导链逐段量化，给出「每 GiB 显存值多少 tok/s」「多少 PCIe 溢出会掉多少速度」这种可以直接拿去做容量规划的数字。
附录：本篇引用的原始证据
A.1 两轮引擎分配对比
A.2 模型文件校验（可直接复核）
A.3 引擎启动日志关键行（原文摘录）
下面每一行都取自 docker.engine.log 原文。同一个字段在不同次启动中会有小幅浮动（比如 arena 装载速率实测在 0.94 – 2.58 GiB/s 之间，MTP 草稿层占用在 820 – 839 MiB 之间），这是页缓存与内存压力的正常差异，不影响上面的公式。
strata generate: memory guard: limit 125.6 GiB (source: meminfo), current 30.6 GiB,
reclaimable cache credited 0.0 GiB; available 94.9 GiB (MemAvailable 94.9 GiB), floor 2560 MiB
strata generate: 301 native projection matrices, 1376.20 MiB of weights, in 1.8 s
strata generate: PLE table locked in RAM (--ple-io ram) in 3.8 s
strata generate: profile /opt/strata/terminal/data/expert-profile.bin: 24576 ranked pairs, built for 24576 slots
strata mtp: draft layer loaded, 839 MiB of VRAM (experts 675, dense 111), files read in 0.54 s (1442 MiB/s)
strata mtp: draft head over 106299 tokens (178.4 MiB)
strata generate: PCIe probe: 13.9 GB/s host->device (best of 13.8 13.8 13.9 13.9) -> pcie_frac 0.38 (default 0.55)
strata generate: this CPU has no AVX-512: the expert kernels run on AVX-2 (multi-token for the i-quant gate/up rows)
strata generate: expert arena: MAP_HUGETLB unavailable (needed 16201 2 MiB pages, vm.nr_hugepages=0)
strata generate: loaded 31.64 GiB at 0.99 GiB/s
strata generate: expert cache auto: 26.01 GiB free, 700 MiB reserved (+184 MiB for the draft head) -> 19530 slots
strata generate: expert cache 19530 slots, 25.14 GiB of VRAM
strata generate: pre-filled 19530 of 19530 slots from the profile in 2.1 s (12936 MB/s); slot 0 verified
strata serve: prompt chunk auto: 8192 tokens, a 384-slot ring
strata verify: window up to 6 tokens, 55.0 MiB of device buffers
strata serve: 410 MiB of VRAM free with everything loaded
A.4 本篇的测量限制
- 只测了 Q2_0 一档，不能跨量化档外推；IQ3_XXS / IQ3_S 的 arena 更大，槽位公式相同但覆盖率起点不同。
- KV 的 384 MiB 与「激活/CUDA 上下文」的 3,100 MiB 为结构推算，非直接测量，误差约 ±300 MiB。
- 固定开销恒定的结论来自多个数据点（13 次启动中，缓存 4,043 / 5,784 / 17,279 / 18,240 / 19,530 槽均有样本），固定开销始终在 5.9–6.0 GiB 区间，但该结论未在 48 GB 卡上验证。
- PCIe 固定为 Gen3 ×16（实测 13.9 GB/s），换平台会移动第五段「断崖」的位置，但定性结论不变。
- 单卡、单模型、单网络环境；同机另一张 4090 全程被 vLLM 占据，未参与任何测量，但可能对 PCIe/内存带宽有轻微干扰。
邦友们，第一篇到这里就把「为什么」讲完了：显存不够时，Strata 不是把模型变瘦，而是把「装不下」重新定义成「缓存哪些」。
下一篇是这套系列里数字最多的一篇——我会把显卡上那 18 GB 的占用停掉，用同一套配置重跑，然后拿 2,890 条遥测告诉你：这 18 GB 到底值多少 tok/s，又卡在哪个小数点上。
如果本文对你有所帮助，欢迎点赞、收藏、转发，也欢迎关注 「全栈架构师笔记」。下一次，我们继续一起把这些新东西跑明白、测明白，也讲明白。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
125B LLM: What can Strata do when VRAM cannot fit?
Author: Wu Jia Ho Written : 2026-10-24 Test Environment:
- GPU: RTX 5090 (32GB VRAM, driver 580.142, CUDA 13.0, PCIe Gen3 x 16)
- CPU: Intel Xeon Gold 5218 x 2 (32 Physical core / 64 thread, 2 NUMA nodes, no AVX512-BF16)
- Memory：125.6 GiB RAM
-Storage:/workspace 3.5 TB(/dev/sdb1, sequenced 435 MB/s)
- OS：Ubuntu 20.04.3 LTS（kernel 5.15.0-58）
- Engineering: Strata v. 0.2.2 (engine 0.1.47, CUDA 13.0, official container mirror)
- Model: Qwen3.8-Flash-Next Q2 0 (125B Level MoE, 48 Layer x 512 Specialists = 24 576 Teams)
Introduction
Systematic analysis for engineers and researchers. We tried on a personal pc to see if this line worked today from a corporate perspective. Alaa is not only talking about "run or not" but is also asking, "What does this mean for local deployment and at what cost?" The 125B level MoE model used to live only on several hundred GB VRAM servers. Strata replaced the matter of "simulations can't be loaded" with a matter of arithmetic, a matter of bandwidth engineering. This is the first in this series to say only one thing: When VRAM couldn't leave the model, Strata actually put something in it, and how the "VRAM gap" was translated into "swallow."
Note: All figures for the full text are derived from logs and telemetry actually operating on the machine and unverified extrapolations are clearly indicated as extrapolations. Because of Allah's curiosity, take a 32-GB card and look at it in white -- look at this Spill design with a false name or a real stick.
Contents
I. The essence of the problem: Why is the "VRAM occupancy rate" a waste indicator?
1.1. Show me the math first.
Let's see how big this machine really is. Q2 0:
RTX 5090's VRAM is 32,607 MiB, about 31.8 GiB.
66.4 GB against 31.8 GiB, the gap is close to 2.1 times. It's not a question of participation. It's a matter of arithmetic.
MoE untied "calculation" and "VRAM."
The key is that the model is MoE (mixed expert): 48 layers, 512 specialists per layer, but 10 roads per layer per layer.
A dense model: Calculus, Zulu, total parameter count
MoE Model: Calculus ∝ Activate parameter count ( Far less than Total)
VRAM ∝ Total parameter count
This formula explains the situation of the consumer class GPU: the surplus, the infertile VRAM. On my machine, during the compression period, GPU peaked at 472 W, while the upper limit of the card was 575 W — there was a surplus; but the VRAM occupation stabilized at 31,879/32,607 MiB, or 97.8 per cent, which could not be squeezed out.
1.3 "VRAM Occupancy Rate"
This is one of the ideas that this paper wishes to correct first.
I made two identical rounds of deployment, the only difference being that there was no other process on GPU that was in VRAM. Results:
The two rounds of VRAM occupancy were almost identical (0.1 percentage points), but at 82 per cent.
So the figure "VRAM occupies 97%" does not carry any performance information per se -- because Strata was designed to fill up all the available VRAMs for expert cache. It is waste when it is not full.
This post will be followed by evidence that it is not VRAM's share that really determines speed, but what is “the share of experts outside cache”. The former is always 97%, the latter is the variable.
II. VRAM ledger for Strata: three sectors, one formula
2.1 Three sectors
The post-start engine log divided VRAM into three clear pieces. With the second round (exclusive):
I'm going to use the exact words of the engine log:
strata generate: 301 native projection matrices, 1376.20 MiB of weights, in 1.8 s
strata mtp: draft layer loaded, 839 MiB of VRAM (experts 675, dense 111)
strata mtp: draft head over 106299 tokens (178.4 MiB)
strata verify: window up to 6 tokens, 55.0 MiB of device buffers
strata generate: expert cache auto: 26.01 GiB free, 700 MiB reserved (+184 MiB for the draft head) -> 19530 slots
strata generate: expert cache 19530 slots, 25.14 GiB of VRAM
strata serve: 410 MiB of VRAM free with everything loaded
2.2 A formula that can be calculated by itself
The distribution constant of this engine can be reversed from the two rounds of logbook, and the two rounds are identical:
- 1.318 MiB (7,625 / 5,784 = 1.318; 25,747 / 19,530 = 1.318) per expert slot
- Fixed reservation of 700 MiB (system) + 184 MiB (speculation debut)
- Maximum slot 24 576 (=48 floor x 512 experts)
Write it in one form:
\(\text{Trails} = \min\left(24576,\frac{text{time free VRAM} -884\text{MiB}1.318\text{MiB}\right)\
From this point of view, all three different VRAM conditions are set for start-up:
(All three lines are taken from the original engine log, with an error of 0.04%, from byte alignment.
The real value of this formula: you don't need to run the benchmark, you just need to know how much VRAM is left on startup, and you can figure out the proportion of experts the machine can cover. I made it into a quick check sheet in the third.
2.3 An intuitive inference: the extra VRAM becomes almost all cache
Two rounds of "total engine VRAM" minus "specialist cache" receive fixed expenses:
6,045 vs 5,927, 2%. In other words, fixed expenses do not vary with the size of cache.
The conclusion is clean: you give GPU every GB VRAM, almost 1:1 becomes expert cache. There is no "VRAM's Bigger, High Cost" — instead, the larger VRAM's, the lower the percentage of fixed expenses, the more efficient VRAM's use.
III. A VRAM tour of Token
The first two sections are static books, and this section goes through the whole path with a token to see where the performance is spent.
Three details deserve to be elaborated separately.
3.1 Why must all experts remain in the mainframe memory?
24,576 specialists moved into the mainframe memory when the engine was loaded:
strata generate: expert arena: cudaHostRegister PORTABLE ok
strata generate: loaded 31.64 GiB at 0.99 GiB/s
strata generate: expert arena read through the file cache (buffered)
31.64 GiB permanent memory, plus PLE tables (-ple-io Ram) and the process itself, stabilizes the system at 96.3 GiB/125.6 GiB during the control period.
This step is both a source of slowness and a prerequisite for running: only if all the experts are in the memory will they be able to move at once, instead of going back to SSD.
3.2 KV Cache is surprisingly cheap.
I thought long context would let KV swallow up a lot of VRAM and actually hit my face:
context went from 32K to 64K, only spent 55MiB VRAM.
The reason is that the focus structure of the model is compressed (qwen4exp.attention.compress radios, one compression ratio for each 48th floor in the GGF metadata), and KV is itself a compression expression. So in this structure:
context is barely spent on VRAM, and the budget for VRAM is almost entirely available on expert cache.
Unlike many of the "long context eats VRAM" experiences, this is a dividend from differences in model structures. The third will explain "Why 64K context has barely lost speed".
3.3 MTP Speculation decode: multiple tokens for one move service
Spill decode is on a token-by-token basis, and each token moves an expert once. The method of decorating the project was speculative: using a small draft model to guess the next few tokens, and allowing the main model to be checked in parallel.
The actual acceptance rate for MTP in my configuration:
The acceptance rate of 64 per cent means that approximately every 1.6 token can save the next expert move. This explains the fact that decode can reach 200+tk/s -- it's not possible to count one expert at a time.
IV. Format of the model: GGUF, Pack, MTP 3
Many people think that local reasoning is "download a model file and load it." This set of things is actually a product of three different formats to understand which document corresponds to which piece of VRAM.
4.1 Primary model: GGUF v3 and mixed precision
I read directly to the file header of two fractions:
"Q2 0" is a slot label, and the file is actually a mixture of precision. By tensor:
So it is a hybrid precision of "2-bit expert + 4-6 bit projection + BF16 superconnection" and not a uniform 2-bit file. And that's why the same "Q2 0" does not perform the same quality in different warehouses -- different mix strategies.
4.2 Operating time products: three layers of falling disc
Roles at all levels:
The start-up parameters of the engine just put the five pieces together:
strata --serve \
--pack /data/Strata-data/packs/q2_0 \
--native /data/Strata-data/models/Q2_0/...-00001-of-00002.gguf \
--ple-gguf /data/Strata-data/models/Q2_0/...-00002-of-00002.gguf \
--mtp /data/Strata-data/mtp/rt \
--max-context 32768 --kv int8 --ple-io ram
4.3 Two consequences of this format
Consequences one: cannot be thrown directly to vLM / SSGLang. It wants GGUF+ special running time; HF safeensors can't eat that process.
Outcome 2: The catalogue structure is the physical map of the VRAM book. Before that, the 3164 GiB arena and 25.7 GiB's cache found their source on disks - It also means changing the smaller quantization file, saving the ones for the arena, not the cache. This will be used in the third selection.
V. HOW THE EFFECTIVENESS ARE DECIDED: FIVE TRANSPORT CRANTS
Here, the static structure is clear. The next step is to link the causal link between "VRAM" and "tok/s": This is at the heart of this paper and the second piece of reality that needs to be tested by paragraph.
The nature of the five paragraphs is completely different:
The real "non-linear" in this chain is paragraphs 3 and 5. And the beautiful one-to-one, which is the easiest place to be misread — many people think that there is a “marginal decline” when they see “VRAM double only 1.8 times” and actually count the non-linearity of paragraph 35.
The second part of this chain will be split into sections with 2,890 requests for telemetry.
5.1 A pit that must be clear: not "slow" but "slow"
By bandwidth, the cost of a failure should not be that high:
= 4.8 experts
× 1.38 MB = 6.6 MB / token
÷ 13.9 GB/s (projected PCIe Gen3 x 16) = 0.47 ms
141 tok/s = 7.1 ms / token
→ 0.47 / 7.1 ≈ 6.6%
Theoretically, it should be only 6.6% slow, but I'm actually measuring it at 0.5% when it spills, 39% slow. Six times worse.
The reason is not bandwidth, in sync: PCIe handles are initiated by walk, but the main model's attestation and MoE calculations are heavily data-dependent, a group of experts is absent, and the follow-up kennel can only wait. During the waiting period, GPU appears to be "working" and is actually turning empty:
The indicator "GPU utilization" is ineffective in such structures — it answers only "doesn't work" and not "dos work or doesn't work." The second chapter will use a working curve to hold this phenomenon to heart.
VI. Verifiable predictions: transforming "genetic" into measurable Volume
When the theory is finished, you have to give false predictions. Based on the five chains above, I wrote four predictions before the second round:
Results of the second round (detailed in part two):
All four of them indicate that this five-part chain is not an "ex post facto story" but a mechanism that can be predicted before running, that can be matched by data.
Summary
7.1 Three conclusions in this section
7.2 A judgement
The consumer-grade GPU runs 125B MoE, which is hard not to count -- it works, and it uses only 60%; the hard is to keep heat experts in VRAM.
The upper limit of the case is locked to death by a simple sub-column: (available for VRAM - 884 MiB) / 1.318 MiB, with a maximum of 24,576.
What's next?
The second, with this mechanism, was used for comparative experiments: the same machine, the same model, the same set of parameters, with only 18 GBs occupied by others on GPU, and then with two full rounds of baseline + 2,890, requested telemetry, and five sections of the chain of conduct by section quantization, giving the figure "How much tok/s per GiB VRAM value" "how much PCIe spills" that can be taken directly into capacity planning.
Appendix: Original evidence cited in this section
A.1 Comparison of distribution of two engines
A.2 Validation of model documents (direct reviewable)
A.3 Engine Launch Log Key Line (extract from original)
Each line below is taken from the original docker.engine.log. The same field is subject to small floats at different start-ups (e.g. isna load rate is measured between 0.94 – 2.58 GiB/s, and MTP draft layer is occupied between 820 – 839 MiB), which is the normal difference between page cache and memory pressure and does not affect the formula above.
strata generate: memory guard: limit 125.6 GiB (source: meminfo), current 30.6 GiB,
reclaimable cache credited 0.0 GiB; available 94.9 GiB (MemAvailable 94.9 GiB), floor 2560 MiB
strata generate: 301 native projection matrices, 1376.20 MiB of weights, in 1.8 s
strata generate: PLE table locked in RAM (--ple-io ram) in 3.8 s
strata generate: profile /opt/strata/terminal/data/expert-profile.bin: 24576 ranked pairs, built for 24576 slots
strata mtp: draft layer loaded, 839 MiB of VRAM (experts 675, dense 111), files read in 0.54 s (1442 MiB/s)
strata mtp: draft head over 106299 tokens (178.4 MiB)
strata generate: PCIe probe: 13.9 GB/s host->device (best of 13.8 13.8 13.9 13.9) -> pcie_frac 0.38 (default 0.55)
strata generate: this CPU has no AVX-512: the expert kernels run on AVX-2 (multi-token for the i-quant gate/up rows)
strata generate: expert arena: MAP_HUGETLB unavailable (needed 16201 2 MiB pages, vm.nr_hugepages=0)
strata generate: loaded 31.64 GiB at 0.99 GiB/s
strata generate: expert cache auto: 26.01 GiB free, 700 MiB reserved (+184 MiB for the draft head) -> 19530 slots
strata generate: expert cache 19530 slots, 25.14 GiB of VRAM
strata generate: pre-filled 19530 of 19530 slots from the profile in 2.1 s (12936 MB/s); slot 0 verified
strata serve: prompt chunk auto: 8192 tokens, a 384-slot ring
strata verify: window up to 6 tokens, 55.0 MiB of device buffers
strata serve: 410 MiB of VRAM free with everything loaded
A.4 Measurement limits in this section
- Only Q2 0 is measured and cannot be pushed outside quantization; the IQ3 XXS / IQ3 S arena is larger, the slot formula is the same but the coverage threshold is different.
- 384 MiB of KV and 3,100 MiB of "Activating/CUDA context" for structural calculations, indirect measurements, with an error of about ±300 MiB.
- The fixed-cost constant conclusion is derived from multiple data points (13 launches, cache 4,043 / 5,784 / 17,279 / 18,240 / 19,530 slots have sample) and the fixed-expenditure remains in the 5.9-6.0 GiB sector, but the conclusion is not validated on 48 GB cards.
- PCIe is fixed to Gen3 x 16 (practical 13.9 GB/s), and the change of platform will move the fifth segment of the "Crisis Break" but the qualitative conclusion remains unchanged.
- Single card, single model, single network environment; another 4090 co-ming was occupied by v LLM at the end of the flight and did not participate in any measurements, but could cause minor interference with PCIe/RAM bandwidth.
Friends of the Nation, this is the first time you've come here to say "why": when VRAM is not enough, Strata not thinned the model, but redefined it as "cache."
The next one is the largest number in this series -- I'll stop the 18 GB on GPU, rerun it with the same configuration, and take 2,890 telemetrys to tell you what the 18 GB is worth tok/s and which decimal point it is stuck to.
If this is of help to you, welcome the praises, collections, forwardings, and the attention given to the "All-Inner Architects Notes." Next time, we continue to work together to figure out, measure and clarify these new things.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.