一块 24G 显存的显卡，想把 32B 的开源大模型跑成稳定服务，大多数人的第一反应是「显存不够」。其实缺的不只是显存，还有把显存用对的能力。这篇文章讲清楚两件事：vLLM 凭什么成为大模型厂商的默认推理引擎，以及你今天怎么在自己的机器上把它跑起来。
01 生成一个字，到底贵在哪
大模型回答问题时是一个 token 一个 token 往外蹦的（token 可以理解成「半个到一个词」）。每生成一个新 token，都要把前面所有内容完整过一遍神经网络。你问 1000 字、它答 2000 字，就是 2000 次前向计算，每次都带着前面越滚越长的上下文一起算。
工程师的解法叫 KV Cache：把已生成内容在每层网络里的中间计算结果（Key 和 Value 矩阵）存下来，下一个 token 只算增量。就像记账，核对过的账目不再重复核，只记新的一笔——计算省下来了，代价是显存。
KV Cache 的大小跟着「上下文长度 × 并发请求数」线性增长。以 7B 模型为例，一个 8K 上下文的请求大约要占几百 MB 到 1G 显存（具体看模型结构），32 个并发加起来就是几十 G——比模型权重本身还大。推理服务的并发能力，很大程度上取决于显存里能同时塞下多少个请求的 KV Cache。
02 浪费发生在分配方式上
早期推理框架的分配方式很「豪横」：请求一来，按模型支持的最大上下文长度预留一整块连续显存。模型支持 32K，哪怕你只问了三个字，也按 32K 的最大可能留位子。
结果三种浪费同时发生：
vLLM 的 PagedAttention 借了操作系统分页管理的思路。操作系统管内存不是按「每个程序可能要多少」预留，而是切成固定大小的小页，用到哪指到哪。PagedAttention 把 KV Cache 同样切成固定大小的块（block），每块存固定数量 token 的 KV，用一张块表做逻辑地址到物理位置的映射——请求要的块不连续也无所谓，按表查就行。
效果用数字感受：vLLM 论文的测试里，吞吐比当时主流推理方案高 2-4 倍，比 HuggingFace Transformers 原生实现最高能到 24 倍。同样的卡，能同时服务的人多了几倍。
顺带回答标题里的问题：厂商选 vLLM 不只因为省显存。它的模型支持面在开源引擎里数一数二，OpenAI 兼容 API 让业务切换几乎零成本，2026 年 10 月发布的 v0.32.0 还在继续压 CPU 开销（Model Runner V2）和优化大规模部署（多 API server 并行）。
03 分页顺手送的前缀缓存
分页还有个红利：KV 复用。两个请求如果开头一样——同一个 system prompt、同一段 RAG 检索结果、多轮对话的历史——开头那段 KV Cache 完全可以共用一份物理块。后到的请求直接「接着算」，不用把开头重新算一遍。
多轮对话是最大受益场景：聊到第 5 轮时，前 4 轮的 KV 已经在显存里，vLLM 只对新增部分做计算，响应速度能快一截。这个能力叫 Automatic Prefix Caching，vLLM 默认开启，不用配置。
04 亲手跑一个
先装（建议 Python 3.10+，Linux/CUDA 环境最顺；首次启动会自动从 HuggingFace 下载模型，国内可设 HF_ENDPOINT=https://hf-mirror.com）：
pip install vllm
vllm serve 后面跟的是 HuggingFace Hub 的模型 ID，首次启动会自动下载，缓存在 HF_HOME 指定的目录（默认 ~/.cache/huggingface/hub）。三种常见玩法：
# 1. 国内加速：设好镜像端点，照常启动
export HF_ENDPOINT=https://hf-mirror.com
# 2. 先手动下载好，再启动服务
huggingface-cli download Qwen/Qwen3-8B
# 3. 换 ModelScope（阿里源）：加一个环境变量
export VLLM_USE_MODELSCOPE=1
已经下好本地权重的话，--model 直接给路径，不走在线下载：
vllm serve /data/models/Qwen3-8B \
--served-model-name Qwen/Qwen3-8B
--served-model-name 让客户端调用时仍然用原来的模型 ID，业务代码不用跟着改。
起服务，一行命令：
vllm serve Qwen/Qwen3-8B \
--gpu-memory-utilization 0.85 \
--max-model-len 8192
三个最常调的参数：
启动后服务跑在 8000 端口，暴露的是 OpenAI 兼容 API。用官方 SDK 就能调：
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
model="Qwen/Qwen3-8B",
messages=[
{"role": "system", "content": "你是一个严谨的中文助手"},
{"role": "user", "content": "用一句话解释 KV Cache"},
],
stream=True,
)
for chunk in resp:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
到这里，你已经拥有了一个和 OpenAI 接口完全一致的本地大模型服务——业务代码里只要换掉 base_url，就能把云上模型切换成自托管。
05 什么时候不值得用
vLLM 不是装上就快，三种情况别硬上：
- 显存本来就放不下模型：vLLM 优化的是「怎么用显存」，不能变出显存。权重都装不下就先做量化（--quantization 支持 GPTQ/AWQ/FP8 等）或换小模型，再谈优化。
- 单人本地玩：Ollama 一行命令更省事。vLLM 的收益主要来自高并发，一个人对着模型提问，分页管理基本帮不上忙。
- 只追首 token 延迟：vLLM 强在「同时服务更多人」（吞吐），单请求的首 token 延迟不一定比小框架低，对延迟敏感的单路场景要先测再说。
一句话总结：vLLM 的核心是把显存从「按最大预留」改成「按需分页」，让同一块卡同时塞下更多请求。
如果这篇帮你搞懂了分页思想，关注大鱼AI研究室，后面我会继续拆推理工程里的其他门道。
你现在跑推理，卡住的是显存不够还是并发上不去？评论区聊聊你的场景和配置。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
GPU, a piece of 24G VRAM, tried to turn the 32B open source LLM into a stabilization service, and most people's first reaction was “VRAM is not enough”. It's not just VRAM, it's the ability to match VRAM. This article makes two things clear: how the v LLM became the default reasoning engine for LLM manufacturers and how you ran it on your machine today.
One word. Where is it?
LLM's answer is a token, a token jumps out (token can be understood as "half to one word". Every time a new token is generated, everything that's ahead of you goes through the neural network. You ask 1000 words, it answers 2,000 words, and it's 2000 forward calculations, each time with a long roller context.
The engineer's solution is KV Cache: Saves the intermediate calculations (Key and Value matrices) of the generated content in each layer of the network, and the next token is only incremental. As in the case of bookkeeping, the reconciled accounts are not repeated, but only a new one — the calculation saved at the cost of VRAM.
The size of KV Cache increases linearly with "context length x request number". In the case of the 7B model, a request for an 8K context is about hundreds of MB to 1G VRAM (concrete model structure), 32 of which add up to dozens of G-weights. The ability to co-operate with reasoning services depends largely on how many requests KV Cache can be plugged in VRAM.
02 Waste in distribution mode
The distribution of the early reasoning framework is “too luxurious”: a whole block of VRAM is set aside by the maximum context length supported by the model. Modeling supports 32K, even if you ask only three words, press 32K for the maximum possible place.
As a result, three types of waste occur simultaneously:
The vLM PagedAttention uses the idea of operating the system page break management. The operating system tube memory is not reserved by "how much each program may cost ", but by cutting into small pages of fixed size, to which point. PagedAttention cuts KV Cache into a block of a fixed size (block), a KV with a fixed number of tokens, maps the logical address to the physical position with a table of blocks - and it doesn't matter if the requested piece is not continuous, just check the table.
The effect is digitally felt: in the v LLM the test, swallowing is 2-4 times higher than the prevailing reasoning at the time, and up to 24 times higher than HuggingFace Transformers. The same card, many times more able to serve at the same time.
Answer the question in the title: The manufacturer chose v LLM not just because of the province of VRAM. Its model support surfaces count to two in open source engines, Open AI compatible API allows operations to switch to almost zero costs, and in October 2026, publish v. 0.32.0 continues to press CPU expenses (Model Runner V2) and optimize large deployments (many API servicers).
03 Page Break Prefix cache
There is also a bonus: KV reuse. If the two requests start with the same beginning - the same system program, the same RAG search results, the history of multiple rounds of dialogue - the opening KV Cache could well share a physical block. Then the request goes straight to the count, and it doesn't have to start over again.
Multi-wheel dialogues are the most beneficial scenario: at the time of round 5, the first four rounds of KV were already in VRAM, and v LLM only calculated the new parts and responded quickly. This capability is called Automatic Prefix Caching, v LLM default on, without configuration.
04, run with one hand.
First load (recommends Python 3.10+, Linux/CUDA environment best; first start will automatically download the model from HuggingFace, where HF ENDPOINT=https://hf-mirror.com):
pip install vllm
The vllm service is followed by the HuggingFace Hub model ID, which is automatically downloaded from the first start, and cache is in the directory specified by HF HOME (default ~/.cache/huggingface/hub). Three common games:
# 1. Domestic acceleration: set mirror endpoints and start as usual
export HF_ENDPOINT=https://hf-mirror.com
# 2. Download first manually and then start the service
huggingface-cli download Qwen/Qwen3-8B
# 3. Exchange ModelScope (Ari): Add an environment variable
export VLLM_USE_MODELSCOPE=1
When local weights are set, --Model directly to the path, without online downloads:
vllm serve /data/models/Qwen3-8B \
--served-model-name Qwen/Qwen3-8B
--served-model-name allows the client to call using the original model ID without changing the business code.
Service, one line of orders:
vllm serve Qwen/Qwen3-8B \
--gpu-memory-utilization 0.85 \
--max-model-len 8192
Three of the most common parameters:
Post-start service runs at the 8000 port and is exposed to Open AI compatible API. The official SDK can adjust:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
model="Qwen/Qwen3-8B",
messages=[
"Role": "system", "content": "You're a strict Chinese assistant,"
"Role": "user", "content": "KV Cache"
],
stream=True,
)
for chunk in resp:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
Here, you have a local LLM service that matches the Open AI interface -- a business code that replaces a base url with a cloud model.
When is it worth it?
vLM isn't just loaded, don't push in three situations:
- VRAM could not leave the model: v LLM optimized "how to use VRAM" and could not change VRAM. quantization (-quantisation supports GPTQ/AWQ/FP8, etc.) or change small models before optimising.
- Single-person local play: Ollama's line of commands is much less. The proceeds of vLM are mainly from high co-payment, with one person asking questions about the model, and page management is largely unhelpful.
- Following only token delays: v LLM is strong enough to "work for more people at the same time" (swallowing), the first token of a single request is not necessarily lower than a small frame, and one-way scenes that are sensitive to delays need to be measured first.
In summary, the core of v LLM is to change VRAM from "Maximum Reserved" to "As required page break" so that the same card can be plugged into more requests.
If this piece helps you to understand the idea of a page break, focus on the Big Fish Ai study, then I'll continue to tear down the other doors in the reasoning project.
Are you trying to figure out if VRAM is not enough or not? The comment section talks about your scene and configuration.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.