专门化可持续土地管理词汇如何降低实际应用的推论成本77%

专门化可持续土地管理词汇如何降低实际应用的推论成本77%

微调通用开源模型专门用于一个域或任务的模型，如Qwen和Gemma，继承了为许多域和一般应用而构建的词汇。

在一个我们制作的散文分级器中，这个词汇把模型所写的文字分成许多符号。 这篇文章预览了来自VocabX论文的一些结果，我们将很快发表：我们如何围绕每个信物的成本而不是其频率专门化词汇，并在不改变基准模型的情况下将测量推断成本削减77%.

标本器通常作为基底模型的固定财产处理，继承后留下。 对于一个单一领域专门化的模型，它可以是最大的成本杠杆之一，它设定在任何培训步骤之前。

我们生产中运行的一个系统是葡萄牙散文的自动化分级器，它建立在Qwen3-4B上。 它收到散文、即时文本和辅助文本，并返回JSON评估，评分为一等，对每项能力提出评论。

该模型已经很小，有40亿个参数，并且已经为此任务接受了培训。 然而，它的标致剂仍然是原版的Qwen3-4B，为多种语言和多种文字所建。 在分校中，它被用于单一语言，葡萄牙语，以及单一响应格式。 这有两个代价：分级员在每一个请求上写的文字花费比它需要的多，在每一个生成的符号上，模型评分了一个有151,669个条目的词汇，包括许多这个任务几乎从未使用过。

我们用我们称之为VocabX的方法重建了词汇，保持了同样的基础模型，相同的数据和相同的训练食谱。 本条中的编号来源于目前版本的论文，仍在审查中，发布前可能有所改变。

为每个人建造词汇

主要语言模型读写符号，标语器决定文本如何成为符号。 主要通用模型的标注器都接受了从许多领域和语言，从源代码到合同到中文的大量文本集的培训。 对于通用模型来说，这种选择是有道理的。

对于一个单一域中使用的模型，它同时会产生两种废物。

第一个是分裂。 标致器将域最重复的部分分割成许多块。 在分级生中，每一次回复都以相同的JSON信封，相同的能力名称，相同的评价短语开始。 在合同中，它是锅炉板条款。 在代码，标识符。 这些序列的每个都花费了数个代币，人们可以这样做，而模型在每次请求时都会支付每个代币。

第二是过量。 词汇中包含着数以万计的条目，域名从未使用过，模型在其生成的每个符号上计算出所有它们的分数。 一个根据美国学校生物水平考试的模型几乎不会用中文写作，但是它计算每个步骤的每个汉语符号的概率。 词汇将域最重复的部分分割成许多块，并带有域从未使用过的条目。

阅读是便宜的，写作是昂贵的

自然反应是计数符并试图减少计数。 但计数不是账单

为了了解为什么一个生成的代币成本更高，它有助于研究推论如何起作用。 它分两个阶段。 在预填中，模型在一次通过网络中处理整个输入：从GPU内存读取的每个重量被重复用于每个输入符，限制是计算。 在解码中，模型一次生成一个代号的响应，每个代号都需要一个完整的通过。 每读的重量只使用几次，GPU花费大部分时间等待数据从内存中到达：限制成为内存带宽。

这种效果正是我们运行的地方最强的：小模型服务于适量批量。 Batching请求将重心在它们之间分割开来，缓解了问题，但没有解决。 在每一个解码步骤中，模型还读取了KV缓存，即持有已处理的上下文并随其增长的注意力内存。 以我们测得的，分级器，一个生成的代币每读一个代币7.4倍，但比例随硬件，问题，批量大小，上下文长度和其他因素而不同，可以大于这个比例。 一般来说，方向是：产生比阅读更多的成本。

想想一个每箱收费的船运公司，但外出货箱的收费是进货货箱的七倍。 减少盒子总数有帮助。 减少外出箱数帮助七倍。 压缩输入和输出的方法同样错过了大部分可能的节省。

法案的第三个组成部分是词汇本身。 模型的最后一层 LM 头，将内部状态变为每个可能符号的分数。 为此，它每个词汇条目都持有一行参数。 在Qwen3-4B大小的模型中，该矩阵占参数的近10%. 这是一种大型模型在不注意的情况下吸收的股份，但它重在小型模型上。

而费用不只支付一次。 输入层只查看正在读取的符号的行，但LM头读取并乘以每个生成符号的整个矩阵。 由于解码器已经是内存绑定的，所以每增加一行的数据都会在每一步移动。 域名的条目从不使用与频繁域完全相同的成本。 这也是为什么添加符号可以降低计数，同时提高账单。 VocabX没有将这个术语纳入其选择标准，但它控制词汇大小分两个步骤：子词合并用一个较小的，特定域的词汇来取代基础词汇，宏在固定预算下输入。 在分级器中，输出层从大约15.2万行发展到大约6.5万行。

现有解决方案为何短

适应模型的词汇并不是一个新想法。 文献中的参考方法，如快速词汇传输(FVT)，多字符号化(MWT)和Liquid AI在位代碼器扩展(在位BPE)，这些方法从原始词汇中继续了BPE培训，已经减少了分裂。 当我们测试他们的问题时，我们发现了两个局限性。

他们根据频率而不是成本选择代币。 在投入中经常出现的表达比在反应中较少出现的表达值更值钱，即使第二种表达值是服务成本更高的表达值。

它们在单层工作。 他们要么调整字句如何切成块，要么将整个短语合并成一个符号。 这三个标准都不符合同一标准。 而分级生最大的浪费恰恰是长的，重复的跨度，比如JSON信封，这个子词方法只在每个部分合并后，一个接一个地到达。

我们是如何解决的

VocabX解决了这两种限制，再加上第三个问题出现在解决第一个问题的明显道路上。

首先，我们改变了问题。 我们不问候选人节省了多少物证，而是问它节省了多少费用和时间。 每个被消除的令牌乘以它出现在以下字段的价格：系统即时，用户消息或模型响应。 就我们测到的，分级器，一个在响应中保存的符号值7.4倍于输入中保存的符号，词汇也相应构建。

第二，我们同时进行两层工作。 回到船运公司：你可以把每件物品包装得更好，或者每周认清同样的十二件装具船，把装具装在一个盒子内，从而切开箱子。 VocabX都这样在子词层上，它重建了域词如何切成块。 在多字层中，它寻找整个跨度重复，从短表达到全行和JSON信封，每个都变成一个单一的符号，我们称之为宏符号。 这两层都是按照相同的费用标准选择的，每一层都有自己的预算。

为了找到这些跨度，我们使用五个矿工，自动搜索跨体重复的跨度，每个矿工都专门进行一种重复：长词，全线，表达式，完整句子，以及带有像 JSON 和标记之类的标点结构。 平均超过8科罗拉，最后一个贡献最大，仅根据代码就占了17.5分。 没有全部5分，降级员的象征性降分将下降55.6分。

第三，我们估计样本看不到的东西。 按成本选择代币要求知道每位候选人在已经选中的代币之后还能节省多少。 精确地测量需要后处理整个体每一个选择，数万次。 在尺度上这样做的一个常见方法是用一个文件样本来衡量，然后一个没有出现在样本中的候选人得到零收益。

一个在5万文档中出现37倍的表达式显示在400文档样本中上升，时间只有26%左右。 在一份合同书中，只有16.6%的候选人出现在400个文件样本中；其余的从未被评估。

一个候选人的收益有两个部分。 一个是它能节省多少整个体，这可以准确计算，没有样本。 另一种情况是，已经选择的代币尚未包括多少节余，仅需要估计这部分节余。 对于样本无法看见的候选人，我们使用基于其能看见者的行为的统计估计，而不是零。 一个在样本中是罕见的，但经常出现在体外的候选者又在运行中。

单是这一变化，用同样的样本和同样的预算，融资在合同书上的象征性削减从44.7%降至52.8%。

年级的变化

为了衡量真正的效果，我们用相同的基模，相同的数据，相同的食谱和同样的种子训练了七个版本的年级生。 他们之间的唯一区别是象征性的。 他们都在同一台测试机上运行了同样的500个请求，拥有一台RTX 5090 GPU和vLLM推论服务器，成本由测量运行时间计算。

该模型的反应平均从350个令牌下降到101个令牌。 费用下降了77.0%。 质量用板凳分数衡量，该分数结合了分级精度和筛选非专题或破规则散文的精度，从0.703上升到0.810.

赶到那里需要修正路线。 词汇的第一个版本将每个能力等级与周围的JSON融合成一个单一的宏符号。 它是最紧凑的选择，但质量下降：它的板凳分数为0.689，低于生产级数的0.703. 随着宏观内部的品级，品级和结构的每一个组合都成为了同物异物的变体，模型在单步中选择了两者。 赋予该等级自己的代币成本每回2个额外代币，融资板凳得分为0.810，在测量中甚至更便宜：降低77.0%，而引信版本为73.3%。

课课应用于分级者之外。 当压缩的跨度带有模型所要做出的决定时，压缩具有成本。 目标不是最便宜的词汇，而是保存重要符号的最便宜词汇。

超越年级

为了确定收益是否具体针对我们的情况，我们比较了VocabX与五个公共领域(合同、医疗文件、科学文件、新闻和Python代码)和三个模范家庭(Qwen3、Llama-3.1和Gemma-3)的三种参考方法。 所有方法都使用相同的采矿和测试文件，以及相同的词汇上限。

VocabX压缩了超过所有15种组合中最强的参考方法，比1.2到9.3分。 在估计推算成本中，普鲁兹版VocabX也领先于所有15个，平均比值为10.2分的Qwen3,7.5分的Llama-3.1和13.6分的Gemma-3. 在合同方面，套装版本以113 426个词汇条目达到62.6%的象征性减少，而最佳现场BPE配置为61.8%，使用207个

本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications

Models specialized for a domain or a task by fine-tuning general open-source models, such as Qwen and Gemma, inherit a vocabulary built for many domains and general applications.

In an essay grader we run in production, that vocabulary split the text the model writes most into many tokens. This article previews some results from the VocabX paper, which we will publish soon: how we specialized the vocabulary around the cost of each token rather than its frequency, and cut measured inference cost by 77% without changing the base model.

The tokenizer is usually treated as a fixed property of the base model, inherited and left alone. For a model specialized in a single domain, it can be one of the largest cost levers, and it is set before any training step.

One of the systems we run in production is an automated grader for Portuguese essays, built on Qwen3-4B. It receives the essay, the prompt and the supporting texts, and returns a JSON assessment with a grade and a comment for each competency.

The model was already small, at 4 billion parameters, and had already been trained for the task. Its tokenizer, however, was still the original Qwen3-4B one, built for many languages and many kinds of text. In the grader, it was used for a single language, Portuguese, and a single response format. That had two costs: the text the grader writes on every request took more tokens than it needed, and at every generated token the model scored a vocabulary of 151,669 entries, including many this task almost never uses.

We rebuilt that vocabulary with a method we call VocabX, keeping the same base model, the same data and the same training recipe. The numbers in this article come from the current version of the paper, still under review, and may change before publication.

A Vocabulary Built for Everyone

The leading language models read and write in tokens, and the tokenizer decides how text becomes tokens. The tokenizers of the leading general-purpose models are trained on large collections of text from many domains and languages, from source code to contracts to Chinese. For a general-purpose model, that choice makes sense.

For a model used in a single domain, it creates two kinds of waste at once.

The first is fragmentation. The tokenizer splits into many pieces exactly what the domain repeats most. In the grader, every response starts with the same JSON envelope, the same competency names, the same evaluation phrases. In contracts, it is the boilerplate clauses. In code, the identifiers. Each of these sequences costs several tokens where one would do, and the model pays for every one of them on every request.

The second is excess. The vocabulary carries tens of thousands of entries the domain never uses, and the model computes a score for all of them at every token it generates. A model tuned to grade American school biology exams will almost never write in Chinese, yet it computes the probability of every Chinese-language token at every step. The vocabulary splits into many pieces what the domain repeats most, and carries entries the domain never uses.

Reading Is Cheap, Writing Is Expensive

The natural reaction is to count tokens and try to reduce the count. But the count is not the bill.

To see why a generated token costs more, it helps to look at how inference works. It has two phases. In prefill, the model processes the whole input in a single pass through the network: each weight read from GPU memory is reused for every input token, and the limit is compute. In decode, the model generates the response one token at a time, and each token needs a full pass. Each weight read is used only a few times, and the GPU spends most of its time waiting for data to arrive from memory: the limit becomes memory bandwidth.

This effect is strongest exactly where we operate: small models served with moderate batches. Batching requests splits the weight reads across them and eases the problem, but does not solve it. At every decode step, the model also reads the KV cache, the attention memory that holds the context already processed and grows with it. In the case we measured, the grader, a generated token costs 7.4 times a token read, but the ratio varies with the hardware, the problem, the batch size, the context length and other factors, and it can be larger than that. In general, the direction holds: generating costs more than reading.

Think of a shipping company that charges per box, but charges seven times more for outgoing boxes than for incoming ones. Reducing the total number of boxes helps. Reducing the outgoing boxes helps seven times more. A method that compresses input and output equally misses most of the possible savings.

The third component of the bill is the vocabulary itself. The model's last layer, the LM head, turns the internal state into a score for every possible token. To do that, it holds one row of parameters per vocabulary entry. In a model the size of Qwen3-4B, that matrix accounts for almost 10% of the parameters. It is a share that large models absorb without noticing, but it weighs on a small model.

And the cost is not paid only once. The input layer looks up only the row of the token being read, but the LM head reads and multiplies the entire matrix for every generated token. Since decode is already memory-bound, every extra row is more data to move at every step. An entry the domain never uses costs exactly the same as a frequent one. That is why adding tokens can lower the count and raise the bill at the same time. VocabX does not include this term in its selection criterion, but it controls vocabulary size in two steps: the sub-word merge replaces the base vocabulary with a smaller, domain-specific one, and macros enter under a fixed budget. In the grader, the output layer went from about 152,000 to about 65,000 rows.

Why Existing Solutions Fell Short

Adapting a model's vocabulary is not a new idea. Reference methods in the literature, such as fast vocabulary transfer (FVT), multi-word tokenization (MWT) and Liquid AI's in-place tokenizer expansion (in-place BPE), which continues BPE training from the original vocabulary, already reduce fragmentation. When we tested them on our problem, we found two limitations.

They choose tokens by frequency, not by cost. An expression that appears often in the input is worth more to them than one that appears less often in the response, even if the second is the more expensive one to serve.

They work on a single layer. They either adjust how words are cut into pieces, or merge whole phrases into one token. None of these three does both under the same criterion. And the grader's biggest waste was precisely in long, repeated spans such as the JSON envelope, which a sub-word method only reaches after merging each part, one by one.

How We Solved It

VocabX addresses both limitations, plus a third problem that appears on the obvious path to solving the first.

First, we changed the question. Instead of asking how many tokens a candidate saves, we ask how much cost and time it saves. Each eliminated token is multiplied by the price of the field it appears in: system prompt, user message or model response. In the case we measured, the grader, a token saved in the response is worth 7.4 times one saved in the input, and the vocabulary is built accordingly.

Second, we work on both layers at once. Back to the shipping company: you can cut boxes by packing each item better, or by recognizing that the same twelve-item kit ships every week and sending the kit in a single box. VocabX does both. At the sub-word layer, it rebuilds how domain words are cut into pieces. At the multi-word layer, it looks for whole spans that repeat, from short expressions to full lines and JSON envelopes, and turns each into a single token, which we call a macro token. Both layers are chosen under the same cost criterion, each with its own budget.

To find these spans, we use five miners, automatic searches for spans that repeat across the corpus, each specialized in one kind of repetition: long words, whole lines, expressions, complete sentences, and structures with punctuation such as JSON and tags. Averaged over eight corpora, the last one contributed the most, and on code it alone accounted for 17.5 points. Without all five, token reduction on the grader would drop by 55.6 points.

Third, we estimate what the sample cannot see. Choosing tokens by cost requires knowing how much each candidate still saves after the ones already chosen. Measuring that exactly would require reprocessing the entire corpus after every choice, tens of thousands of times. A common way to do this at scale is to measure on a sample of documents, and then a candidate absent from the sample gets zero gain.

An expression that appears 37 times in 50,000 documents shows up in a 400-document sample only about 26% of the time. In a contracts corpus, only 16.6% of the candidates appear in a 400-document sample; the rest are never evaluated.

A candidate's gain has two parts. One is how much it saves over the entire corpus, and that can be counted exactly, without a sample. The other is how much of that saving is not yet covered by the tokens already chosen, and only that part needs to be estimated. For candidates the sample cannot see, we use a statistical estimate based on the behavior of the ones it can see, instead of zero. A candidate that is rare in the sample but frequent in the corpus is back in the running.

That change alone, with the same sample and the same budget, raised token reduction on a contracts corpus from 44.7% to 52.8%.

What Changed in the Grader

To measure the real effect, we trained seven versions of the grader with the same base model, the same data, the same recipe and the same seed. The only difference between them is the tokenizer. All of them ran the same 500 requests on the same test machine, with an RTX 5090 GPU and the vLLM inference server, and cost was calculated from measured runtime.

The model's response dropped from 350 to 101 tokens on average. Cost dropped 77.0%. Quality, measured by the bench score, which combines grading accuracy and the accuracy of screening off-topic or rule-breaking essays, rose from 0.703 to 0.810.

Getting there required a course correction. The first version of the vocabulary fused each competency's grade with the surrounding JSON into a single macro token. It was the most compact option, but quality dropped: its bench score was 0.689, below the production grader's 0.703. With the grade inside the macro, each combination of grade and structure became a different variant of the same token, and the model chose both in a single step. Giving the grade its own token cost 2 extra tokens per response, raised the bench score to 0.810 and, in the measurement, was even cheaper: a 77.0% reduction, against 73.3% for the fused version.

The lesson applies beyond the grader. Compression has a cost when the compressed span carries the very decision the model exists to make. The goal is not the cheapest possible vocabulary, but the cheapest one that preserves the tokens that matter.

Beyond the Grader

To find out whether the gain was specific to our case, we compared VocabX with the three reference methods on five public domains (contracts, medical papers, scientific papers, news and Python code) and three model families (Qwen3, Llama-3.1 and Gemma-3). All methods used the same mining and test documents, and the same vocabulary ceiling.

VocabX compressed more than the strongest reference method in all fifteen combinations, by 1.2 to 9.3 points. In estimated inference cost, the pruned version of VocabX also led in all fifteen, with average margins of 10.2 points on Qwen3, 7.5 on Llama-3.1 and 13.6 on Gemma-3. On contracts, the pruned version reached 62.6% token reduction with 113,426 vocabulary entries, against 61.8% for the best in-place BPE configuration, which uses 207

Auto-fetched and translated by AI Atlas. All rights belong to the original author.