中小企业跑125B 大模型：18GB 显存之差，到底值多少吞吐
作者：吴佳浩 撰稿时间：2026-10-24 测试环境：
- GPU：RTX 5090（32GB VRAM，驱动 580.142，CUDA 13.0，PCIe Gen3 ×16，实测 13.9 GB/s）
- CPU：Intel Xeon Gold 5218 ×2（32 物理核 / 64 线程，2 个 NUMA 节点，无 AVX512-BF16）
- Memory：125.6 GiB RAM
- OS：Ubuntu 20.04.3 LTS（kernel 5.15.0-58）
- Engine：Strata v0.2.2（engine 0.1.47，CUDA 13.0，官方容器镜像）
- Model：Qwen3.8-Flash-Next Q2_0（125B 级 MoE，24,576 专家槽）
- 两轮配置完全相同：CONTEXT=32k / KV=int8 / PARALLEL=1 / VISION=cpu / MTP spec4
- 唯一变量：第一轮同机其他容器占了 GPU1 的 18,106 MiB；第二轮停掉它们
引言
面向工程师与研究者的体系化分析。 第一篇讲清了机制。这一篇只做一件事：把机制里的每一段，用实测数据钉死。 我把同一台机器、同一份模型、同一套参数跑了两轮，唯一区别是显卡上有没有别人的 18 GB。然后我从引擎日志里挖出 2,890 条逐请求遥测（每条都带「专家缓存命中率」和「PCIe 溢出比例」），把「显存 → 吞吐」这条链逐段拆开。 你可以把这篇当成一份「容量规划参考表」：给定显存，我告诉你能期待多少 tok/s；给定速度目标，我告诉你需要多少显存。
注：全文所有性能数字均来自本机两轮真实压测（合计 1,507 次请求、1,362 个资源采样点、317 个服务端指标采样点），未经验证的推算会明确标注。因为阿拉好奇，所以停掉别人占的卡，一道来白相相——看看这 18 GB 是"锦上添花"还是"生死攸关"。
目录
一、实验设计：只改一个变量
1.1 为什么这个实验值得做
大多数「本地大模型性能」文章的问题是：所有变量一起变。换了量化档、换了上下文、换了并发，然后报一个 decode 数字——这个数字没法归因。
我这次刻意做成单变量对照：
1.2 停掉的东西，以及怎么回滚
先交代清楚，避免误会成"破坏性操作"：
- 停止方式是 docker stop -t 30（优雅停止，不是 kill）
- 容器没有被删除，docker start ms_reranker7 ms_reranker8 ms_reranker9 即可完整恢复
- 完整容器配置快照已存档（docker inspect 全量 JSON）
- 同机另一张卡（RTX 4090）上的 vLLM 全程未动
停完之后 GPU1 的状态：
index, name, memory.total, memory.used, memory.free
1, NVIDIA GeForce RTX 5090, 32607 MiB, 13743 MiB, 18367 MiB
pid, process_name, used_gpu_memory
2027940, .../strata, 13670 MiB
1.3 重部署后的第一件事：看缓存怎么自动扩
同一个镜像、同一套 -e 环境变量，重启后引擎自己把缓存吃满了：
strata generate: expert cache auto: 8.31 GiB free, 700 MiB reserved (+184 MiB for the draft head) -> 5784 slots # 第一轮
strata generate: expert cache auto: 26.01 GiB free, 700 MiB reserved (+184 MiB for the draft head) -> 19530 slots # 第二轮
5,784 → 19,530，3.38 倍。 这就是第二篇全部故事的起点。
二、第一跳：显存 → 缓存覆盖率（干净的线性）
2.1 实测值
两轮完全吻合，误差在 0.04% 内（来自字节对齐取整）。这是整条链上唯一一段"可以用计算器算出来"的关系，也是本文最有工程价值的一段——它意味着你不需要跑基准就能预估覆盖率。
2.2 固定开销恒定，是这条线性关系的前提
差 118 MiB，2%。固定开销不随缓存变大而变大，所以「多给的显存」几乎全部落进缓存。
三、第二跳：覆盖率 → 命中率（长尾的胜利）
3.1 实测
从引擎日志里抽出每条请求的 decode expert cache hit rate，按轮次统计（取生成 150–260 token 的可比样本）：
覆盖率 +56 个百分点，命中率只 +13.2 个百分点。 看起来像"收益递减"，其实是长尾分布。
3.2 为什么覆盖率 23.5% 就能命中 86.5%
因为专家访问极度倾斜。Strata 有一个 expert-profile.bin（24,576 个排名对），按路由频率决定谁先占缓存：
strata generate: profile /opt/strata/terminal/data/expert-profile.bin: 24576 ranked pairs, built for 24576 slots
strata generate: pre-filled 5784 of 5784 slots from the profile in 0.6 s (12647 MB/s); slot 0 verified
于是第一轮那 23.5% 的槽位，装的恰好是最热的专家：
3.3 关键不在"命中率高低"，在"剩下多少"
这里的直觉陷阱是：86.5% 听起来很高（接近九成），所以会觉得"还行"。
但反过来看才是对的：
第一轮：未命中 13.5%
第二轮：未命中 0.3%
→ 未命中的量差了 45 倍
性能是被"未命中"驱动的，不是被"命中"驱动的。 所以后面所有数据都指向同一个方向：盯着那 0.3% 到 13.5% 的区间，而不是盯着 86.5% 这个整数。
四、第三跳：命中率 → PCIe 溢出（断崖）
4.1 先把 2,890 条遥测摊开
我从引擎日志里把「专家缓存命中率」和「PCIe 溢出比例」两条逐请求记录配对，得到 2,890 条样本（跨两轮，覆盖短问答、长上下文、并发、过载各种负载）。按 PCIe 溢出比例分桶：
这张图是本文最重要的一张。
三个特征：
- 0 → 0.5% 之间是断崖：231.8 掉到 141.4，−39%。
- 0.5% → 3% 几乎是平台：141.4 → 137.7，只掉 2.6%。说明"已经很慢了，再慢一点点也无所谓"。
- 5% 以上二次断崖：141 → 72，再掉一半。这时候系统已经进入"搬运主导"模式。
4.2 命中率分桶交叉验证
同一批样本按命中率分桶，结论一致：
Pearson 相关系数 r = 0.779（n = 2,890）。
注意 95% 处的跳变：90–95% 还在 143，95–99% 直接 280。这不是线性爬坡，是跨过阈值后的换挡。
4.3 分轮次的直接对比
第二轮把 PCIe 溢出打到了 0.00% —— 这是「+82% 吞吐」的直接来源。
五、为什么 1% 的溢出贵 39%
5.1 先按带宽算一遍，发现算不通
每 token 480 次专家查表（48 层 × 10 专家）
× 1% 溢出 = 4.8 个专家
× 1.38 MB / 专家 = 6.6 MB / token
÷ 13.9 GB/s（实测 PCIe Gen3 ×16 带宽）= 0.47 ms / token
而 141 tok/s = 7.1 ms / token
→ 0.47 / 7.1 = 6.6%
理论上只该慢 6.6%，实测慢了 39%。差了 6 倍。
所以代价不是"搬得慢"，是"等得久"。
5.2 真正的成本：流水线气泡
GPU 的计算是高度并行的，但依赖是串行的。一批专家没到，后面的算子只能干等；等的时候 SM 是 idle 的，但驱动仍然报告「occupied」——这就是「利用率 89% 却只跑一半速度」的来源。
5.3 用功耗曲线把"假忙"坐实
利用率几乎一样（89.1% vs 89.8%），功耗却差了 47%。
这是我这轮实验里觉得最值得记住的一条：
GPU 利用率是「有没有活干」，功耗才是「干的活有没有产出」。 两者背离时，瓶颈一定在数据搬运路径上。
六、三层效率：绝对、边际、能源
同样一组数字，用三种不同的分母去看，会得出三种不同的结论。这一节把三种都算出来，因为它们经常被混用。
6.1 绝对效率：每 GiB 显存换多少 tok/s
看到这里，很多人会下结论："边际收益递减"。这个结论是错的，因为它把固定开销也算进了分母。
6.2 边际效率：每多 1 GiB 专家缓存
Δ专家缓存 = 25,747 − 7,625 = 18,122 MiB ≈ 17.7 GiB
Δdecode = 256.2 − 140.6 = 115.6 tok/s
→ 6.53 tok/s per GiB 专家缓存
正确算法是做线性拟合、把截距算出来：
第一轮：7.45 GiB 缓存 → 140.6 tok/s
第二轮：25.14 GiB 缓存 → 256.2 tok/s
斜率 = (256.2 − 140.6) / (25.14 − 7.45) = 6.53 tok/s per GiB
截距 = 140.6 − 7.45 × 6.53 = 91.9 tok/s
也就是说：每加 1 GiB 缓存，线性加 6.53 tok/s。 截距 91.9 的含义是「缓存为 0 时的理论速度」——这是一个外推值，未实测，只用来验证线性拟合的自洽性，不能当成真实性能引用。
这条线性关系是本篇最实用的结论：它让「给定显存预测速度」变成一道乘法题，而不是必须跑基准。
6.3 能源效率：每 token 耗多少焦耳
这是最常被忽略、但对生产部署最重要的一层：
多花 47% 的电，做出 82% 更多的活 → 单位能耗降 20%。 这还是"假忙"的代价：等 PCIe 的时候 GPU 照样耗电，只是不产出。
6.4 三层汇总
七、反例：显存多了，为什么不能开 parallel=4
显存变多之后，最自然的想法是"开更多并发"。我用 4 组配置做了对照，结论否定了这个直觉。
7.1 四组配置实测
关键发现：parallel=4 的聚合吞吐（179.3）反而低于 parallel=1（220.0），而且单流 decode 从 204 掉到 186，4 并发时每客户端只剩 58 tok/s。
7.2 机制：并发不是扩容，是切分
parallel=N 不是「把车加宽」，是「把油分成 N 份」。 每个 slot 的覆盖率降到约 1/N，PCIe 溢出暴涨，于是第四节那个"断崖"同时触发 N 次。
7.3 什么时候该开多槽
在这台 32GB 的卡上，parallel=1 是唯一正确解。
八、硬件资源画像与瓶颈定位
8.1 两轮的资源对比
采样 2 秒一次，两轮合计 1,362 个有效采样点：
8.2 排除法：四个"常见嫌疑"全都不成立
8.3 两个硬约束
约束一：显存上限决定了覆盖率上限。
卡上可用 ≈ 31.8 GiB
− 固定开销 6.0 GiB
= 25.8 GiB 可给缓存
÷ 1.318 MiB/槽 = 19,600 槽
÷ 24,576 = 79.8%
在 32GB 卡上，80% 覆盖就是天花板。 想上 100%，需要约 38.5 GiB 可用显存（= 32.1 GiB 缓存 + 6.0 GiB 开销），也就是 48GB 的卡。
约束二：PCIe Gen3 ×16 = 13.9 GB/s。
strata generate: PCIe probe: 13.9 GB/s host->device (best of 13.8 13.8 13.9 13.9) -> pcie_frac 0.38 (default 0.55)
这是引擎自己测的，也是第 4 节那个断崖的物理基础。RTX 5090 本身支持 Gen5，这张卡却插在 Gen3 ×16 的槽上——链路能力只有卡能力的四分之一。
8.4 瓶颈已经转移
九、稳定性与并发：排队语义的实证
9.1 长稳：两轮都零失败
两轮都是 100% 成功、零漂移、零显存泄漏。
9.2 并发扫描：吞吐平坦，延迟线性
这是「排队语义」的直接证据：
c: 1 2 4 8 16
吞吐: 192 220 216 213 209 ← 平坦（甚至微降）
TTFT: 0.16 0.80 1.94 3.54 7.28 ← 近似线性增长
单张卡的专家搬运带宽是共享的。多客户端不会让它更快，只会让每个请求等更久。 这也解释了为什么第 7 节里 parallel=4 反而更差——它连"真并发"都没换来，只换来了缓存被切碎。
9.3 过载：32 并发时的行为
两轮都没有拒绝请求，全部排队完成。这说明这套部署没有启用准入控制——生产环境该加队列上限，否则延迟会不可控地膨胀。
十、结论与容量规划表
10.1 五段传导链的实测结论
10.2 容量规划速查表
基于实测的两点（固定开销 ≈ 6.0 GiB、1.318 MiB/槽）和线性拟合（截距 92 tok/s、斜率 6.53 tok/s per GiB）：
换算关系：缓存 ≈ 整卡可用 − 6.0 GiB，槽位 = (缓存字节 − 884 MiB) / 1.318 MiB，decode ≈ 92 + 6.53 × 缓存GiB（92 为拟合截距，外推值）。
10.3 三句话收尾
一句关键判断：显存不是"够不够"，是"热专家放不放得下"；而放不下的代价不是线性打折，是断崖。
10.4 这一篇的测量限制
- Decode 跨轮次可比，跨内容不完全可比：MTP 接受率随生成内容波动（实测 63.5%–64.0%，但单请求区间更宽），同轮内不同请求的 decode 天然有 ±15% 波动。
- 命中率与速度是相关性（r=0.779），不是严格因果；两者都由缓存覆盖率驱动，但请求长度、上下文长度是混杂因素。
- 容量规划表的 12 / 18 / 24 / 38.5 GiB 四行为推算，非实测；截距 92 tok/s 也是外推。
- KV 与"激活/CUDA 上下文"为结构推算，误差约 ±300 MiB。
- 只测了 Q2_0 + 32K/64K + parallel 1/2/4，未测 IQ3_XXS 等其他档位、未测多卡。
- PCIe 固定 Gen3 ×16：换到 Gen4/Gen5 后第 4 节的断崖位置会移动，"溢出比例"的门槛会变宽松，但"溢出 → 掉档"的定性结论不变。
- 同机另一张 4090 全程被 vLLM 占据（43,573 MiB，计算利用率 0%），未参与测量，但可能对 PCIe/内存带宽有轻微干扰。
附录 A：两轮基础压测原始数据
A.1 官方标准基准（strata bench run，同协议）
A.2 长上下文对比（冷前缀）
A.3 新解锁的能力
附录 B：2,890 条遥测是怎么来的
引擎每处理完一条请求会写两行日志：
strata serve: prompt 26 tokens = 0 reused + 26 read in 43 ms, 192 generated in 1367 ms (140.5 tok/s), drafts accepted ...
strata serve: decode expert cache hit rate: 86.3% (23414 hits / 27131 lookups); 412 more read by the GPU over PCIe or from another GPU (2.31% of all 17850 routed)
我用脚本按顺序配对这两种行，得到每条请求的三元组 (命中率, PCIe 溢出比例, decode)，跨两轮共 2,890 条。
- 采样覆盖：官方基准（短/4K/16K）、并发扫描（c=1/2/4/8/16）、长稳（c=1 与 c=4）、长上下文（4K/16K/18K/20K）、图片输入、过载（32 并发）
- 配对方式：只接受"生成行在前、命中率行紧随其后"的严格相邻对，避免错配
- 数据集：results-fullvram/vram-efficiency-dataset.json（581 KB）
- 相关系数按 Pearson 计算，n=2,890，r=0.779
邦友们，第二篇的数字就摆在这里了。
回头看，最让我意外的不是"+82%"这个结果，而是它其实是可以算出来的：一条 1.318 MiB/槽的公式，一个 6.53 tok/s per GiB 的斜率，一个 95% 附近的阈值。跑之前我就写下了四条预测，跑完之后四条全中——这比任何单个数字都更说明这套机制是"可理解"的，而不是"玄学"。
下一篇是这套系列的最后一篇，也是最能直接照做的一篇：从硬件自检、容器部署的三个坑、模型下载与格式校验，到启动、冒烟验证、五类压测的完整脚本，以及调优清单与故障速查表。所有命令和输出都是这台机器上真跑过的。
如果本文对你有所帮助，欢迎点赞、收藏、转发，也欢迎关注 「全栈架构师笔记」。下一次，我们继续一起把这些新东西跑明白、测明白，也讲明白。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
The difference between 125B LLM: 18GB Z82Z for SMEs, how much is it worth?
Author: Wu Jia Ho Written : 2026-10-24 Test Environment:
- GPU: RTX 5090 (32GB VRAM, driver 580.142, CUDA 13.0, PCIE Gen3 x 16, measured 13.9 GB/s)
- CPU: Intel Xeon Gold 5218 x 2 (32 Physical core / 64 thread, 2 NUMA nodes, no AVX512-BF16)
- Memory：125.6 GiB RAM
- OS：Ubuntu 20.04.3 LTS（kernel 5.15.0-58）
- Engineering: Strata v. 0.2.2 (engine 0.1.47, CUDA 13.0, official container mirror)
- Model: Qwen 3.8-Flash-Next Q2 0 (125B Level MoE, 24,576 expert slot)
- Two rounds of configuration are identical: CONTEXT=32k / KV=int8 / PARALLEL=1 / VISION=cpu / MTP Spec4
- Single variable: 18,106 MiB of GPU1 in the first round; stop them in the second round
Introduction
Systematic analysis for engineers and researchers. The first set out the mechanisms. This one is just one thing: nailing every part of the mechanism with real data. I ran two rounds of the same machine, the same model, the same set of parameters, the only difference being if there was anyone else on GPU. And then I dig up 2,890 requests for telemetry from the engine logs (each with the "expert cache hit rate" and "PCIe spill ratio") and untangle the "VRAM Z Zipping" chain. You can use this as a "capacity planning reference sheet": give VRAM, I'll tell you how much tok/s you can expect; give VRAM, I'll tell you how much.
Note: All full-text performance figures are derived from two-wheel real pressure measurements (total 1,507 requests, 1,362 resource sampling sites, 317 service-end indicator sampling points) and uncertified extrapolations are clearly indicated. And because Allah was curious, he stopped the card, and he said, "Look at the 18 GBs.
Contents
I. Experimental design: one variable only
Why is this experiment worth doing?
The problem with most of the local LLM posts is that all variables change together. Changed the quantization, changed context, changed the context and sent it, and then reported a decode figure - which cannot be attributed.
This time, I deliberately compared it to a single variable:
1.2 Stopped things and how to roll back.
Let's get this straight.
- Stop by docker stop-t 30
- The container has not been deleted, and the docker start ms reranker7 ms reranker8 ms reranker9 can be fully restored
- Full container configuration snapshot archived (docker inspect full volume JSON)
- The vLM on the other card (RTX 4090) remained intact.
GPU1 status after stop:
index, name, memory.total, memory.used, memory.free
1, NVIDIA GeForce RTX 5090, 32607 MiB, 13743 MiB, 18367 MiB
pid, process_name, used_gpu_memory
2027940, .../strata, 13670 MiB
1.3 First thing after re-deployment: see how cache automatically expands
Same mirror, same-e environment variable, restarting the engine and eating cache all by itself:
Strata gender: original size: 8.31 GiB free, 700 MiB reserved (+184 MiB for the dry head) - > 5784 slots #
- > 19530 slots #
5,784 →19,530,3.38 times. This is the beginning of the second full story.
II. First jump: VRAM cache coverage (clean linear)
2.1 Actual values
The two rounds were fully aligned, with an error of 0.04 per cent (from byte alignment). It's the only part of the chain that can be calculated by a calculator, and it's the most engineering part of the paper. It means you don't need to run the benchmark to predict coverage.
2.2 Fixed spending is a prerequisite for this linear relationship
The difference is 118 MiB, 2%. The fixed costs do not increase with cache, so that almost all of VRAM has fallen into cache.
III. Second jump: coverage rate (long end victory)
3.1 Measurements
Draws from the engine log each request decode except carche hit rate, by rotation (causing 150-260 token comparable sample):
Coverage + 56 percentage points with a hit rate of only +13.2 percentage points. Looks like "regressive" is actually a long end distribution.
3.2 Why does 23.5% of coverage hit 86.5%?
The expert visit was extremely biased. Strata has an expert-profile.bin (24,576 rankings) that determines who first takes cache by route frequency:
strata generate: profile /opt/strata/terminal/data/expert-profile.bin: 24576 ranked pairs, built for 24576 slots
strata generate: pre-filled 5784 of 5784 slots from the profile in 0.6 s (12647 MB/s); slot 0 verified
So in the first round of 23.5% of the slots, they happen to be the hottest experts:
3.3 The point is not "high and low hit rate."
The hunch trap here is 86.5% sounds high (near 90%), so it's okay.
But the opposite is true:
First round: missed 13.5 per cent
Second round: 0.3 per cent missed
Well, it's 45 times worse.
Performance is driven by "uncution", not "hit." So all the data in the back point in the same direction: to the 0.3% to 13.5% range, instead of 86.5%.
iv. Third jump: Hit rate → PCIe spill
4.1 First split the 2,890 telemetry
From the engine logs, I paired the requests for "expert cache hit rate" and "PCIE spill ratio" with 2,890 sample (crossing two rounds, covering short questions and answers, long context, and delivering and overloading various loads). PCIe Spills:
This is the most important one in this paper.
Three characteristics:
- 0 → 0.5% is a cliff break: 231.8 fell to 141.4, -39%.
- 0.5% → 3% is almost platform: 141.4 → 137.7, only 2.6%. It says, "It's already slow. It doesn't matter if it's slow."
- 5% more than double-cut cliff: 141 po-72, half. And that's when the system's in the "disposal lead" mode.
4.2 Cross-validation of the hit rate subbarrel
The same group of sample, divided into barrels with a hit rate, reached the same conclusion:
Pearson correlation coefficient r = 0.779 (n = 2,890).
Note 95% jump: 90-95% still at 143,95-99% direct 280. It's not a linear climb, it's a shift over the threshold.
4.3 Direct comparisons in turn
In the second round, PCI was spilled to 0.00% -- this is a direct source of "+82% swallowing".
V. Why 1% of spills are expensive 39%
5.1 I'm going to press the bandwidth first. It doesn't work.
480 expert checks per token (48 floor x 10 experts)
x 1% Spill = 4.8 experts
x 1.38 MB / Specialist = 6.6 MB / token
÷ 13.9 GB/s (projected PCIe Gen3 x 16 bandwidth) = 0.47 ms / token
141 tok/s = 7.1 ms / token
→ 0.47 / 7.1 = 6.6%
Theoretically, it should be only 6.6 per cent slow and 39 per cent slow. Six times worse.
So the price is not "slow." It's "slow."
5.2 Real costs: Flow-line bubbles
GPU calculations are highly parallel, but relying on serials. A group of experts did not arrive, and the last number of calculations could only wait; SM was idle's, but the driver still reported "occupied," which is the source of "89% utilization rate but only half speed".
5.3 Sitting on "false busy" with a working curve.
The utilization rate is almost the same (89.1 per cent vs 89.8 per cent), while the cost of work is 47 per cent worse.
This is one of the most memorable things I've ever had in my life:
The GPU usage rate is "work or no work" and "work or no work." When the two depart, the bottlenecks must be on the data removal path.
VI. Three tiers of efficiency: absolute, marginal, energy
The same set of figures, viewed with three different denominators, leads to three different conclusions. This section calculates all three because they are often mixed.
6.1 Absolute efficiency: how much tok/s per GiB VRAM
Seeing here, a lot of people would say, "The marginal gains are diminishing." This conclusion is wrong, as it includes fixed expenses in the denominator.
6.2 Marginal efficiency: 1 GiB expert cache per additional
cache = 25,747 – 7,625 = 18,122 MiB ≈ 17.7 GiB
Δdecode = 256.2 − 140.6 = 115.6 tok/s
6.53 tok/s per GiB expert cache
The correct algorithm is a linear alignment, which calculates the distance:
First round: 7.45 GiB cache→ 1406 tok/s
Second round: 25.14 GiB cache → 256.2 tok/s
Slope = (256.2 - 140.6) / (25.14 - 7.45) = 6.53 tok/s per GiB
Intersection = 140.6 − 7.45 × 6.53 = 91.9 tok/s
This means that each plus 1 GiB cache is linear plus 6.53 tok/s. It means "the theoretical speed of cache at 0": This is an extrapolation value, which is not measured and is used only to verify the ease of linear alignment and cannot be cited as authentic performance.
This linear relationship is the most practical conclusion of this article: it makes the "predictation of VRAM" a multiplication rather than having to run a benchmark.
6.3 Energy efficiency: how much joules per token
This is the most often neglected but most important layer for production deployment:
Over 47% of electricity is spent, making 82% more effort, 20% less energy per unit. It's also the price of "fake-doing": when PCIe comes, the GPU consumes electricity, just not output.
6.4 3rd floor summary
VII. A contrario: VRAM. Why can ' t you open a parallel = 4
After VRAM got more, the most natural idea was to "go more together." I contrasted it with group four configurations and concluded that this instinct was denied.
7.1 Four group configurations measured
Key finding: Palallel = 4 is less than parallel = 1 (220.0) (179.3) and single-flow decode falls from 204 to 186,4 and ends with only 58 tok/s per client.
7.2 Mechanisms: Consistency is not expansion, it is division
“Short up the car, split the oil into N.” Each slot's coverage dropped to about 1/N, PCIe surged, and then the fourth section of the cliff was triggered at the same time.
7.3 When should we open a lot?
On this 32GB card, parallel=1 is the only correct solution.
VIII. Portrait of hardware resources and location of bottlenecks
8.1 Two-round comparison of resources
Sampling 2 seconds at 2 rounds totalling 1,362 effective sampling points:
8.2 Exclusion: None of the four "common suspicion"
8.3 Two hard restraints
Binding 1: The VRAM ceiling determines the maximum coverage.
Available on card 31.8 GiB
- Fixed costs 6.0 GiB
= 25.8 GiB to cache
÷ 1.318 MiB/ Slot = 19,600 Slot
÷ 24,576 = 79.8%
On the 32GB card, 80% of the cover is the ceiling. In order to be 100%, about 38.5 GiB needs to use VRAM (= 32.1 GiB cache + 6.0 GiB) or 48GB cards.
Limitation II: PCIe Gen3 x 16 = 13.9 GB/s.
strata generate: PCIe probe: 13.9 GB/s host->device (best of 13.8 13.8 13.9 13.9) -> pcie_frac 0.38 (default 0.55)
It's the engine itself, and it's the physical basis of section 4. RTX 5090 itself supports Gen5, but this card is inserted in Gen3 x 16's slot - only a quarter of the capacity of the link.
8.4 Bottlenecks have shifted
IX. Stability and simultaneous distribution: empirical evidence of the semantics of queues
9.1 Long-term stability: two rounds of zero failure
Both rounds are 100% successful, zero drift, zero VRAM leak.
9.2 Parallel scan: flat, delayed linear
This is direct evidence of the "queuing semantics":
c: 1 2 4 8 16
Swallow: 192 220 216 213 209 ← flat (or even slight decline)
TTFT: 0.16 0.80 1.94 3.54 7.28 ← approximate linear growth
The expert bandwidth of a single card is shared. Multiple clients will not make it faster, but will only keep each request waiting longer. And that explains why section 7 says "parallel" = 4 is worse -- it doesn't even change "real hair" and cache is cut apart.
9.3 Overload: 32 and simultaneous acts
Both rounds did not reject the request and all were lined up. This means that the deployment did not have access controls in place — the production environment should have a ceiling lined up, otherwise the delay would have been uncontrolledly inflated.
Conclusions and capacity planning tables
10.1 Factual findings of five sections of the chain
10.2 Capacity Planning Spacing Table
Based on measured two points (fixed cost 6.0 GiB, 1.318 MiB/tr.) and linear alignment (separation 92 tok/s, slope 6.53 tok/s per GiB):
convert Relationship: The cache ≈ whole card can be used - 6.0 GiB, slot = (cachebyte - 884 MiB) / 1.318 MiB, decode ≈ 92 + 6.53 x cacheGiB (92 for amplitude, extrapolation value).
10.3 End of three sentences
A key judgment: VRAM is not enough; it's the heat experts who can't put it down; the price of failure is not linear, it's a cliff break.
10.4 The measurement limits of this article
- Decode cross-cycle, not exactly cross-section: The MTP acceptance rate fluctuates with the creation of content (actually, 63.5 per cent-64.0 per cent, but with a wider range of individual requests) and the decode of the different requests in the same cycle has a natural fluctuations of 15 per cent.
- The hit rate and speed are relevant (r=0.779) and not strictly causal; both are driven by cache coverage, but the length of the request and the length of context are mixed factors.
- 12 / 18 / 24 / 38.5 GiB four behavioral extrapolation, non-existent; cut-off tok/s is also extrapolation.
- KV with "Activating/CUDA context" as a structural calculation, with an error of about 300 MiB.
- Only Q2 0 + 32K/64K + parallel 1/2/4, undetected IQ3 XXS, undetected Doc.
- PCIe fixed Gen3 x 16: The cliff position after Gen4/Gen5 will move and the "spray ratio" threshold will be relaxed, but the qualitative conclusion of "spray out and fall" will remain unchanged.
- Another 4090 co-pilot was occupied by v LLM (43,573 MiB, calculated utilization rate 0%), which was not measured but could cause minor interference with PCI/RAM bandwidth.
Appendix A: Two rounds of basic pressure measurement raw data
A.1 Official benchmark (strata bench run, agreement)
A.2 Long context comparison (cold prefix)
A.3 Capacity for new unlocking
Appendix B: 2,890 Telemetry
Each request is processed in a two-line log:
strata serve: prompt 26 tokens = 0 reused + 26 read in 43 ms, 192 generated in 1367 ms (140.5 tok/s), drafts accepted ...
strata serve: decode expert cache hit rate: 86.3% (23414 hits / 27131 lookups); 412 more read by the GPU over PCIe or from another GPU (2.31% of all 17850 routed)
I paired the two rows sequentially with scripts and obtained the three-track group of each request (cutoff rate, PCIe spill ratio, decode) for a total of 2,890 articles over two rounds.
- Sampling coverage: official benchmark (short/4K/16K), simultaneous scan (c=1/2/4/8/16), steady (c=1 and c=4), long context (4K/16K/18K/20K), photo input, overload (32 both)
- Matching: accept strict neighbour-to-neighborship only "to create a line before and after the hit rate" to avoid mismatches
- Data set: results-fullvram/vram-efficacy-dataset.json (581 KB)
- The relevant coefficient is Pearson, n=2,890, r=0.779
Friends, the second number is here.
Turning back, it's not the result of "+82%" that surprises me, but it's actually possible: a formula for a 1.318 MiB/s, a tilt of 6.53 tok/s per GiB, a threshold of 95% near. Before I ran, I wrote four predictions, four of them all -- that means more than any single number that the mechanism is "understandable" than "genetic."
The next one is the last in the series and is the most direct: from hardware self-inspection, three pits deployed in containers, model downloads and format validation, to startup, smoke validation, complete scripts for five types of pressure, and modulation lists and failure speed sheets. All the commands and outputs are really running on this machine.
If this is of help to you, welcome the praises, collections, forwardings, and the attention given to the "All-Inner Architects Notes." Next time, we continue to work together to figure out, measure and clarify these new things.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.