背景：不是跑 Demo，是要上产线
最近在做一个二维码缺陷质检的数据流水线，其中一层（L3 净化层）需要 VLM 判断两件事：
- 码制识别：QR / DPM 点阵码 / 其他条码 / 无码 / 无法判断
- 可见性判定：完整 / 部分可见 / 隐约可辨 / 完全不可见
判定结果直接决定下游标注数据的清洗和分级。硬件是一张 RTX 5070 Ti（16GB 显存），模型选了 Qwen2.5-VL-7B-Instruct 的 AWQ 量化版。
这不是跑 Demo，是每天要处理几万张裁剪图的批量任务，所以稳定性和显存控制比"能不能出结果"重要得多。
为什么是 7B + AWQ
7B 参数用 bf16 存储大约 14GB，加上推理时的 KV cache 和视觉编码器开销，16GB 卡基本塞不下，更别说上下文一长就 OOM。AWQ 量化后模型文件只有 6.5GB，留出足够空间给 KV cache 和批量推理。对"判定可见性"这种分类任务，量化带来的精度损失可以接受——毕竟 prompt 里已经把枚举定义写死了，模型只需要做规则性判断，不需要长文本生成。
最终跑通的环境配方
先说结论，这套组合在 5070 Ti 上是跑通的：
踩坑记录
坑一：flashinfer sampler 必须关掉
这是最实在的一个坑。代码里有一行看起来莫名其妙的环境变量，是踩过之后加的：
import os
os.environ.setdefault("VLLM_USE_FLASHINFER_SAMPLER", "0")
vLLM 默认会启用 flashinfer 的采样 kernel，但在当前这套版本组合（vllm 0.30 + flashinfer 0.6.18 + cu130）上会导致推理异常。关掉之后一切正常。
如果你也在类似环境遇到 vllm 生成结果异常、卡住或者崩溃，第一件事是把这个开关设为 0，能省掉半天排查。
坑二：CUDA Toolkit 版本和 torch 版本不需要"对齐"
这是很多教程误导人的点。机器上 nvcc --version 显示 CUDA 12.0，但 torch 是 2.13.0+cu130（CUDA 13.0 编译）——这是正常的，也是推荐的。
PyTorch 自带 CUDA 运行时，推理不需要系统 nvcc 和 torch 的 CUDA 版本一致。系统 nvcc 只有在编译自定义 CUDA 算子（比如手搓 flash-attn）时才需要匹配。如果你的场景只是"加载模型跑推理"，千万不要为了"版本对齐"去重装系统 CUDA，纯属给自己找事。
坑三：16GB 显存的核心参数
在 16GB 卡上跑 7B 多模态模型，这几个参数决定了能不能活：
llm = LLM(
model=MODEL,
max_model_len=4096, # 上下文别开太大，KV cache 吃显存
gpu_memory_utilization=0.85, # 留 15% 给视觉编码器和碎片
limit_mm_per_prompt={"image": 1}, # 关键！限制每条 prompt 最多 1 张图
)
limit_mm_per_prompt 是 vLLM 多模态的隐藏坑：不限制的话，多图输入会让显存峰值不可控。我们的场景每条样本就一张裁剪图，限制成 1 最安全。
另外图片输入还要控制分辨率上限：
{"type": "image", "image": "file://" + cpath, "max_pixels": 602112}
602112（约 784×768）对"判断码图是否清晰"这种任务足够了，再大只会浪费显存。
坑四：多模态输入不能直接拼字符串
这是 vLLM + Qwen-VL 的常见错误。正确姿势是先用 processor 套模板，再用 process_vision_info 处理图片：
from vllm import LLM, SamplingParams
from transformers import AutoProcessor
from qwen_vl_utils import process_vision_info
processor = AutoProcessor.from_pretrained(MODEL)
messages = [{"role": "user", "content": [
{"type": "image", "image": "file://" + cpath, "max_pixels": 602112},
{"type": "text", "text": PROMPT},
]}]
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
img_data, _ = process_vision_info(messages)
inputs = {"prompt": prompt, "multi_modal_data": {"image": img_data[0]}}
outs = llm.generate([inputs], SamplingParams(temperature=0.0, max_tokens=200))
直接把图片路径塞进 prompt 字符串，模型只会"想象"一张图然后胡说八道。
坑五：输出解析要靠 prompt 纪律，不是靠正则碰运气
判定任务最怕模型输出自由发挥。三个约束必须写进 prompt：
- 枚举值定义写死，包括边界情况。比如"很糊但能辨认出码结构，判 qr 而不是 no_code"、"DataMatrix 没有回字定位角"——这些说明比模型自己猜有用得多。
- 只输出一行 JSON，配合 temperature=0、max_tokens=200。
- 解析失败要有兜底：我们给失败样本单独记日志，而不是静默丢弃。
坑六：工程化细节比调参更影响能不能跑完
真跑几万条数据时，这几个处理是保命的：
- 断点续跑：输出 jsonl 按 (图片, 目标id) 去重，中断后重跑不重复计算
- 旋转框裁剪：二维码标注是旋转框，直接按外接矩形裁会带一堆背景，按角度旋转后再裁 + 10% padding 效果更好
- 失败不阻塞：单条样本裁剪失败或解析失败，记录后跳过，不中断整批
效果与结论
这套配置在 5070 Ti 上跑批量裁剪图推理是稳定可用的。对"码制分类 + 可见性分级"这种任务，7B AWQ 模型的判定结果可以直接进入人工抽检流程，替代了原来纯人工翻阅的工作量。
几个经验总结：
- 16GB 单卡跑 7B 多模态批量推理完全可行，关键是 AWQ 量化 + 显存参数收紧
- vLLM 的坑集中在版本组合，遇到怪问题先试 VLLM_USE_FLASHINFER_SAMPLER=0
- 多模态推理别省 processor 那一步
- prompt 里把枚举和边界写清楚，比换更大的模型有效
这套流水线后面还可以复用到文档识别、票据识别、缺陷分类等场景——本质上都是"给一张图 + 一个分类规则，让 VLM 输出结构化判定"。
标签建议：Qwen2.5-VL、vLLM、多模态、RTX 5070 Ti、模型部署
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Background: Not running, Demo, getting on the line.
Recently, a data flow line with a 2-dimensional calibration defect, one of which (L3 purified) requires VLM to judge two things:
- Code ID: QR / DPM point array / other bar codes / no code / undecided
- Visibility determination: Full / Partially Visible / Implicitly Visible / Unvisible
The findings directly determine the cleansing and ranking of the downstream marked data. Hardware is a RTX 5070 Ti (16GB VRAM) model of Qwen2.5-VL-7B-Instract version of AWQ quantization.
It's not a run, Demo, it's a batch of tens of thousands of clippings per day, so stability and VRAM control are much more important than "can't get results."
Why 7B + AWQ
7B parameters are stored in bf16 for approximately 14GB, plus the cost of the KV Cache and the visual encoder at the time of reasoning, and the 16GB card is largely impregnated, let alone context OOM. The post-AWQ quantization model file is only 6.5 GB, allowing sufficient space for KV Cache and volume reasoning. The precision loss caused by quantization for the classification task of "defining visibility" is acceptable - after all, the list is written down in the prompt, and the model needs only a rule-based judgement and no long text generation.
The environmental formula that eventually runs through.
First of all, this combination runs through 5070 Ti:
Peddle record.
Pipe 1: Flashinfer sampler has to be shut down.
This is the real pit. There's a line in the code that looks like an inexplicable environmental variable that's put on foot:
import os
os.environ.setdefault("VLLM_USE_FLASHINFER_SAMPLER", "0")
The vLM default will enable a sampling of flashinfer kernel, but this will cause an anomaly of reasoning in the current version combination (vllm 0.30 + flashinfer 0.6.18 + cu130). Turn it off and everything's fine.
The first thing you do is set this switch to zero, save half a day of checking.
Pit 2: CUDA Toolkit and Torch versions do not need alignment
That is the misleading point of many teachings. The nvcc-version shows CUDA 12.0, but the torch is 2.13.0+cu130 (CUDA 13.0 compiled) - this is normal and recommended.
The reasoning does not need to be consistent with the version of the system nvcc and the CUDA when PyTorch takes CUDA with itself. The system nvcc needs to be matched only when compiling custom CUDA algorithms (e.g. hand rubbing flash-attn). If your scene is just a "load-up model run," don't try to reload the CUDA just for yourself.
Pipe III: Core parameters of 16GB VRAM
Run a 7B multi-mode model on a 16GB card, and these parameters determine whether or not to live:
llm = LLM(
model=MODEL,
context don't go too far, KV Cache eat VRAM
gpu memory utilization=0.85, # 15% for visual encoders and debris
limit mm per prompt={"image":) # Critical! Limit to a maximum of 1 image per plan
)
The limit mm per prompt is a v LLM multi-modular hole: Multi-chart input makes VRAM peak uncontrollable. Our scenes are just a cropping sheet, restricted to one of the safest.
In addition, the resolution ceiling is controlled by the entry:
{"type": "image", "image": "file://" + cpath, "max_pixels": 602112}
602112 (approximately 784 x 768) is enough for the "Whether or not the judgement chart is clear" task to waste only VRAM.
Pipe IV: Multimodular input cannot spell string directly
This is a common error for v LLM+Qwen-VL. The correct position is first to use the Processor template and then to process pictures with the Procss vision info:
from vllm import LLM, SamplingParams
from transformers import AutoProcessor
from qwen_vl_utils import process_vision_info
processor = AutoProcessor.from_pretrained(MODEL)
messages = [{"role": "user", "content": [
{"type": "image", "image": "file://" + cpath, "max_pixels": 602112},
{"type": "text", "text": PROMPT},
]}]
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
img_data, _ = process_vision_info(messages)
inputs = {"prompt": prompt, "multi_modal_data": {"image": img_data[0]}}
outs = llm.generate([inputs], SamplingParams(temperature=0.0, max_tokens=200))
Put the picture path directly into the prompt string, and the model will only "imagine" a picture and then bullshit.
Pipe Five: Output resolution depends on prompt discipline, not regular luck. Gas
The task of determining is the fear that the model will be freely exported. Three constraints must be written:
- Death by definition, including boundaries. For example, "It's very lame, but it can recognize code structure, sentencing qr instead of no code," "DataMatrix has no backword locator angle," which is much more useful than models themselves.
- Just one line, JSON, with temperature = 0, max tokens = 200.
- There's a bottom line to solve the failure: we're writing a separate log for the failed sample, not throwing it away.
Pipe Six: Does engineering detail affect the end of a journey more than involvement?
When you run tens of thousands of pieces of data, these are the things you do:
- Breakpoint running: output jsonl press (photogram, targetid) to heavy, run again without double counting after interruption
- Rotation box cropping: 2-D code label is a rotation box, with a set of backgrounds to press the outside rectangle, rotated by angle, and then retracted + 10% padding better
- Failures do not block: single sample cropping or resolution failed, loging and skipping without interruption
Effects and conclusions
This set of configurations on 5070 Ti runs a batch tailoring reasoning that is stable. For tasks such as Code Classification + Visibility Rating, the findings of the 7B AWQ model can be accessed directly through the manual check-up process, replacing the original purely manual search load.
Several lessons learned:
- 16GB single-card run, 7B multi-mode mass reasoning is perfectly feasible, and the point is that AWQ quantization + VRAM parameters tighten
- v LLM pits are concentrated in version combinations and try first with a strange problem
- Multitemporal reasoning.
- Prompt, write the count and the boundary. It works better than a larger model.
This streaming line can also be used for document recognition, document recognition, defect classification etc. - It's basically, "Give me a map + a classification rule, let VLM output be structured."
Label recommendations: Qwen2.5-VL, v LLM, multimodular, RTX 5070 Ti, model deployment
Auto-fetched and translated by AI Atlas. All rights belong to the original author.