2026年10月20日 — 8分钟读数
杰西·唐克夫利特和李俊豪
高级软件工程师
当我使用一个开放量级模型时，我看到的结果与质量基准测试的表示相差甚远。
软件开发者最近与AI模型合作
工具调用精度是服务开放量级模型的重要组成部分。 当您的工具调用不准确时，模型会建立在非感性上下文上。 这导致了低质量的结果，特别是在多回合会议常见的代理工作量方面。
可惜许多供应商和广泛使用的推论引擎并没有完全通过工具调用精度测试。 例如，当我们用vLLM版本0.28为Kimi K2.6服务时，我们只得到了92%的预期工具调用计划。 换句话说，每25个试图使用的工具电话中就有2个失败。
幸运的是，VLLM最近增加了对受约束的解码的支持，这是一种保证遵守计划的技术。 这个功能后来在vLLM0.31中作为服务器侧旗的发布。 通过启用它，可以得到100%的计划精度，使用较新的Kimi和DeepSeek模型。
在博客文章中，我们首先描述工具称为计划准确性，然后我们讨论受限制的解码以及如何解决计划准确性问题
您可以让 LLM 以两个步骤调用外部工具：(1) 告知 AI 模型存在外部可用的工具调用；(2) 向模型提供如何输出结构化工具调用的指示。 例如，OpenCode连接到Kimi时，OpenCode会发送Kimi一个描述可用工具的JSON阵列。 本列表中的工具之一是webfatch，它获取网页内容：
[{
"类型":"功能"
"函数":{
"名曰："网etch.
"描述":"- 从指定的 URL\n 获取内容 - 以 URL 和可选格式作为输入\n- 获取 URL 内容，转换为请求的格式( 默认情况下的标记下)\n- 返回指定格式的内容\n- 当您需要检索和分析网页内容时使用此工具\ n\ nUsage 指出：\ n [内容编辑为简洁]\ n]
"参数":{
"$schema":"https://json-schema.org/draft/2020-12/schema" (中文(简体) ).
"类型"：对象
"财产":{
"url": {
"类型":"弦"
"描述":"从中获取内容的URL"
},
"格式":{
"类型":"弦"
"enum": [
"文字" (中文(简体) ).
"降级"
"html" (中文(简体) ).
],
"描述":"返回内容在(文本，标记下，或html)中的格式。 默认减号".
"默认":"标记下"
},
"超时":{
"类型":"数"
"描述":"备选超时秒(最大120秒)"
}
},
"须"曰：[
内脏
]
}
}
}, ...]
推论引擎然后将这个JSON描述用LLM特异性符号包起来，然后传递到LLM，以匹配LLM如何被训练来识别工具声明。 在服务Kimi时，这看起来像：
\ im system tool declare im middle\ [{'类型]: "函数" 函数：{},...}\ im end {}
描述工具的JSON对象(或功能，在以上JSON中)包括工具的参数，其类型，需要哪些参数等。 这被称为工具的图谋。 为了成功执行由LLM生成的工具调用，调用必须遵循其计划中规定的规则。
然后，当一个用户要求通过他们的agent带取网页时，LLM可能会输出以下的令牌：
我去拿那页 QQToogle calls section begin tool call begin xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
这些令牌由推断服务器的解析器解析，提取LLM特定令牌，并将调用变为JSON对象，如：
{
"作用":"辅佐"
"内容":"我会拿那页。"
"工具调用":[
{
"id":"功能.webfetch:0" (中文(简体) ).
"类型":"功能"
"函数":{
"名曰："网etch.
"论语":{\"ur":\"https://example.com\","格式":\"markdown\"]
}
}
]
}
然后，它被送回到您的管理器( OpenCode，以我们为例)，它运行它的 Webfatch 工具，并将其输出返回模型。
若要成功，呼吁必须遵循工具的策略。
简单的工具调用方法是让模型生成工具自由调用，并相信模型的训练足以生成正确的调用。 这类似于我们如何信任模型来生成在战术上正确的传言。
简单的方法在大多数时候是有效的，因为现代模型擅长遵循结构化的模式。 但成功“大部分时间”意味着许多工具呼叫仍然可能失败，导致上下文窗口污染，原因是错误信息，模型性能较低，以及agent不得不重试一次或多次导致费用增加。
为了避免在使用Kimi时出现这样的经历，Moonshot AI发布了他们的K2V ToolCall 基准测试。 这个基准测试让提供者验证它们服务Kimi的质量与原版相同，包括它的默认行为，即只生成有效的工具呼叫。
K2VV ToolCall 的工作量包括选定2000项请求，以评价LLM何时生成工具调用，以及生成的工具调用是否在战术上正确(即遵守所提供的计划). 基准测试允许通过量化两个公尺来分析模型的质量：触发相似性和计划精度。 Trigger相似性考虑服务何时决定触发工具呼叫，以及它的决定与Moonshot AI自己的服务有多相似。 Schema精度将工具调用的部分量化，这些调用符合工具的策略，因此可以成功引用。 例如，当工具调用精度为50%时，模型生成的工具调用有一半是错的。
由于语言模型的概率性，Schema 精度通常会下降。 一个模型根据概率生成符号。 它没有运行编译器或类型检查器。 即使有明显的迅速，它仍然可以产生错误的工具调用。 例如，这可以是由于模型的训练数据与目前的推论利用不匹配。 对模型进行了许多工具调用数据集的培训，每个数据集都有自己的工具、计划及公约。 因此，一个模型可能会产生它在训练期间所见到的工具或场名，即使在当前生产环境中并不存在。 例如，当可用的工具是web search时，它可能会生成搜索 web，或者当活动计划预期位置时使用city name.
LLM计算每个符号的概率是序列中的下一个。 然后通过进行加权抽样来选择下一个符号。 约束解码通过防止模型对令牌进行采样从而导致综合无效的工具调用来避免工具调用不当。 在使用受约束的解码时，与其让模型取样符自由并验证之后的结果，推论引擎会一步一步地引导生成。 每一步，只允许有效的符号。
允许哪些符号由提供的工具方案决定。 vLLM 使用 XGrammar 库中的结构标记执行此功能。 显示下方的片段摘录了XGrammar如何在Kimi K2.6中执行受约束的解码。 它定义了工具调用的有效结构，包括调用开始地，索引出现地，参数开始地，以及必须生成schema-construction JSON的地方。
# Kimi K2.6 工具调用语法：
# # 工具 call begin 函数。 name}: { index}
# 工具调用参数参数
# 工具呼叫结束 #
标签格式( T)
开始=f"{TOOL CALL BEGIN PREFIX}{ name TOOL CALL SUFFIX}}"
内容=顺序格式( S)
元素=[
RegexFormat( pattern=r"\d+") 页面存档备份，存于互联网档案馆
ContString Format( 值=TOOL ALL ARGUMENT BEGIN) (中文(简体) ).
JSONSchemaFormat (英语：
json schema=参数
任意命令=任意命令
最大值白空 cnt=max 白空 cnt
),
]
),
结束=TOOL CALL 结束
)
你可以在这里找到完全执行。
使用这个定义，世代必须遵循一条有效的路径：
开始工具调用工具名工具调用索引开始参数 Schema- construction JSON 结束工具呼叫
无效路径在生成时被屏蔽。 例如，当生成达到工具调用索引阶段时，引擎将采样限制在整数符上，并遮盖了其他一切。 下图转载自XGrammar的文章
资料来源：董，叶欣等"XGrammar：大型语言模型的灵活高效结构化生成引擎". "机器学习与系统7集"(2025). [link].
这改变了计划准确性，从最佳方法转变为运行时间保证。 模型仍然决定要生成哪一个符号，但现在该符号是从一个限制性的，在战术上正确的集合中选择的。
值得注意的是，这种运行时间执行新增可忽略不计的性能间接费用。 XGrammar通过预先计算每个阶段的代号口罩，并在生成时高效地将其应用到GPU上，保证正确的工具调用而不交换显著性能。
Moonshot AI建议打开受限制的解码，并且它自己的API也这样做。 因此，我们最初在我们的fork of vLLM中加入了受约束的解码。 VLLM在v0.25中添加了这个功能后，我们从上游切换到使用这个功能用于Kimi和DeepSeek. 不幸的是，不久后，我们发现我们的工具召唤精度分数回落到91%!
在分析这个问题后，我们确认VLLM实施受限制的解码仍然是正确的。 相反，我们发现vLLM改变了它对于没有明确打开受约束解码的请求的默认行为。 这主要是因为vLLM的设计是为了支持广泛的模型。 支持受限制的解码是一项按模式进行的任务，问题属于三个营地：
- 一般模型问题：对于一些模型来说，其不受限制的输出与工具的图案或模型的调用工具格式有很大不同。 这迫使模型选择它本来不会拥有的标志，导致它偏离航线[1]. 例如，gpt-oss-20b,Qwen 3.6,GLM 5.1用户在默认启用此功能后报告了问题。
- 语法翻译bugs:vLLM可能在将模型的工具计划或工具调用格式翻译为模型的具体语法时有bugs. 这会造成一些问题，包括无效的工具调用，请求失败，默不作声地放弃受约束的解码，以及降低工具调用精度[1,2].
- 特性交互bugs：在受约束的解码功能和其他服务引擎功能之间的相互作用中存在bugs. 例如，在受约束的解码和投机解码之间的相互作用中出现一个bug，可能导致投机符被语法错误拒绝，导致请求失败[1,2].
由于这些问题，vLLM维护者报告说，在启用了受限的解码后，发现几个模型存在问题。 这使得在默认情况下受限制的解码功能失效，成为VLLM等模型不可识别服务引擎更安全的选项。 后来，vLLM增加了-tool-限制性级别旗，这样用户就可以允许工具调用严格的部署。
Ubicloud的焦点与VLLM不同。 我们注重服务Kimi和DeepSeek模型，其质量是用户对第一当事方供应商的期望。 为此，我们启用了受限的解码并重新运行了K2V ToolCall 基准测试。 一旦我们验证了100%的工具呼叫计划精度在所有服务模型版本，我们部署的改变生产。 然后我们输入博客文章
当您使用 LLMs 时，所观察到的质量并不仅仅基于模型。 推论引擎也发挥着重要作用。 改变推论引擎及其设置，您会改变模型的输出质量。
在服务Kimi K2.6和DeepSeek V4 Flash时，我们观察到由于工具调用计划不准确导致的可见质量退化。 我们解决了这个问题首先携带我们自己的vLLM叉然后启用新的合作
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
October 20, 2026 · 8 min read
Jesse Donkervliet and Junhao Li
Senior Software Engineers
When I use an open-weight model, I see results that are nowhere close to what the quality benchmarks indicate.
Software developer working with AI models these days
Tool call accuracy is an important part of serving open-weight models. When your tool calls aren’t accurate, the model builds up on nonsensical context. This leads to low quality results, particularly for agentic workloads where multi-turn sessions are common.
Sadly, many providers and widely-used inference engines don’t fully pass tool-call accuracy tests. For example, when we served Kimi K2.6 with vLLM version 0.28, we got the expected tool call schema only 92% of the time. In other words, 2 out of every 25 attempted tool calls failed.
Fortunately, vLLM recently added support for constrained decoding, a technique to guarantee schema-adherence. This feature has then been released as a server-side flag in vLLM 0.31. By enabling it, you can get to 100% schema accuracy with the newer Kimi and DeepSeek models.
In this blog post, we first describe tool call schema accuracy and why it’s important in serving agentic workloads. We then talk about constrained decoding and how it fixes the schema accuracy problem.
You can enable an LLM to call an external tool in two steps: (1) inform the AI model about the existence of externally available tool calls and (2) provide the model with instructions on how to output structured tool-calls. For example, when connecting OpenCode to Kimi, OpenCode sends Kimi a JSON array describing available tools. One of the tools in this list is webfetch, which fetches Web page contents:
[{
"type": "function",
"function": {
"name": "webfetch",
"description": "- Fetches content from a specified URL\n- Takes a URL and optional format as input\n- Fetches the URL content, converts to requested format (markdown by default)\n- Returns the content in the specified format\n- Use this tool when you need to retrieve and analyze web content\n\nUsage notes:\n [content redacted for brevity]\n",
"parameters": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "The URL to fetch content from"
},
"format": {
"type": "string",
"enum": [
"text",
"markdown",
"html"
],
"description": "The format to return the content in (text, markdown, or html). Defaults to markdown.",
"default": "markdown"
},
"timeout": {
"type": "number",
"description": "Optional timeout in seconds (max 120)"
}
},
"required": [
"url"
]
}
}
}, ...]
The inference engine then wraps this JSON description in LLM-specific tokens before passing it into the LLM, to match how the LLM was trained to recognize tool declarations. When serving Kimi, this looks something like:
<|im_system|>tool_declare<|im_middle|>[{"type":"function","function":{...}, ...]<|im_end|>
The JSON object describing the tool (or function, in the JSON above) includes the tool’s parameters, their types, which of the parameters are required, etc. This is known as the tool’s schema. For an agent harness (e.g., OpenCode) to successfully execute a tool call generated by the LLM, the call must follow the rules specified in its schema.
Then, when a user asks to fetch a Web page via their agent harness, the LLM may output the following tokens:
I'll fetch that page. <|tool_calls_section_begin|><|tool_call_begin|>functions.webfetch:0<|tool_call_argument_begin|>{"url": "https://ubicloud.com", "format": "markdown"}<|tool_call_end|><|tool_calls_section_end|>
These tokens are parsed by the inference server’s parser, extracting the LLM-specific tokens and turning the call into a JSON object such as:
{
"role": "assistant",
"content": "I'll fetch that page.",
"tool_calls": [
{
"id": "functions.webfetch:0",
"type": "function",
"function": {
"name": "webfetch",
"arguments": "{\"url\": \"https://example.com\", \"format\": \"markdown\"}"
}
}
]
}
This is then sent back to your harness (OpenCode, in our example), which runs its webfetch tool and sends its output back to the model.
For a tool-call to succeed, the call must adhere to the tool’s schema.
The simple approach to tool calling is to let the model generate tool calls freely and trust that the model’s training is sufficient to generate correct calls. This is similar to how we trust the model to generate syntactically correct prose.
The simple approach works most of the time because modern models are good at following structured patterns. But success “most of the time” implies that many tool calls can still fail, resulting in context window pollution due to error messages, lower model performance, and increased cost due to the agent having to retry the call one or more times.
To avoid such experiences when using Kimi, Moonshot AI published their K2VV ToolCall benchmark. This benchmark lets providers verify that they serve Kimi with the same quality as the original, including its default behavior of only generating valid tool calls.
K2VV ToolCall’s workload consists of 2000 requests selected to evaluate when the LLM generates tool calls and whether the generated tool calls are syntactically correct (i.e., adhere to the provided schema). The benchmark allows analyzing the model’s quality by quantifying two metrics: trigger similarity and schema accuracy. Trigger similarity considers when the service decides to trigger a tool call, and how similar its decisions are to Moonshot AI’s own service. Schema accuracy quantifies the fraction of tool-calls that adhere to the tool’s schema and can therefore be invoked successfully. For example, when tool-call accuracy is 50%, half of the tool calls generated by the model are malformed.
Schema accuracy usually falls short due to the probabilistic nature of language models. A model generates tokens based on likelihood. It is not running a compiler or type checker. Even with a clear prompt, it can still produce malformed tool calls. This can be caused by, for example, a mismatch between the model’s training data and the current inference harness. Models are trained on many tool-calling datasets, each with its own tools, schemas, and conventions. As a result, a model may generate a tool or field name that it has seen during the training, even if it does not exist in the current production environment. For example, it may generate search_web when the available tool is web_search, or use city_name when the active schema expects location.
An LLM computes for each token the probability that it is the next in the sequence. It then selects the next token by performing a weighted sample. Constrained decoding avoids malformed tool calls by preventing models from sampling tokens that would result in a syntactically invalid tool call. When using constrained decoding, instead of letting the model sampling tokens freely and validating the result afterward, the inference engine guides generation step by step. At each step, only valid tokens are allowed.
Which tokens are allowed is determined by the provided tool schema. vLLM implements this using structural tags from the XGrammar library. The snippet below shows an excerpt of how XGrammar implements constrained decoding in Kimi K2.6. It defines the valid structure of a tool call, including where the call begins, where the index appears, where the arguments begin, and where schema-constrained JSON must be generated.
# Kimi K2.6 tool call syntax:
# <|tool_call_begin|>{function.name}:{index}
# <|tool_call_argument_begin|>{arguments}
# <|tool_call_end|>
TagFormat(
begin=f"{TOOL_CALL_BEGIN_PREFIX}{name}{TOOL_CALL_SUFFIX}",
content=SequenceFormat(
elements=[
RegexFormat(pattern=r"\d+"),
ConstStringFormat(value=TOOL_CALL_ARGUMENT_BEGIN),
JSONSchemaFormat(
json_schema=parameters,
any_order=any_order,
max_whitespace_cnt=max_whitespace_cnt,
),
]
),
end=TOOL_CALL_END,
)
You can find the full implementation here.
With this definition, generation must follow a valid path:
Begin tool call → Tool name → Tool call index → Begin arguments → Schema-constrained JSON → End tool call
Invalid paths are blocked during generation. For example, when generation reaches the tool-call index phase, the engine restricts sampling to integer tokens and masks out everything else. The figure below, reproduced from XGrammar’s article, provides a good visualization of how invalid tokens are masked.
Source: Dong, Yixin, et al. "XGrammar: Flexible and efficient structured generation engine for large language models." Proceedings of Machine Learning and Systems 7 (2025). [link]
This changes schema accuracy from a best-effort approach to a runtime guarantee. The model still decides which token to generate, but now the token is selected from a restricted and syntactically-correct set.
Notably, this runtime enforcement adds negligible performance overhead. By precomputing token masks for each phase, and applying them efficiently on the GPU during generation, XGrammar guarantees correct tool calls without trading off significant performance.
Moonshot AI recommends constrained decoding to be turned on and its own API behaves as such. So we initially added constrained decoding in our fork of vLLM. After vLLM added this feature in v0.25, we switched to using this feature from upstream for Kimi and DeepSeek. Unfortunately not long after, we found out that our tool-call accuracy score dropped back to 91%!
After analyzing the problem, we confirmed that vLLM’s implementation of constrained decoding was still correct. Instead, we found that vLLM changed its default behavior for requests that do not explicitly turn constrained decoding on. This is primarily because vLLM is designed to support a wide range of models. Supporting constrained decoding is a per-model task and problems fall into three camps:
- General model issues: For some models, their unconstrained output diverges significantly from the tool's schema or model's tool-calling format. This forces the model to select tokens it otherwise wouldn't have, leading it off course [1]. For example, gpt-oss-20b, Qwen 3.6, and GLM 5.1 users reported issues after having this feature enabled by default.
- Grammar translation bugs: vLLM might have bugs in translating a model's tool schema or tool-calling format into the model's specific grammar. This can result in problems including invalid tool calls, request failures, silently dropping the constrained decoding, and lower tool-calling accuracy [1, 2].
- Feature interaction bugs: There are bugs in the interaction between constrained decoding and other serving-engine features. For example, a bug in the interplay between constrained decoding and speculative decoding can result in speculative tokens being wrongly rejected by the grammar, leading to failing requests [1, 2].
Due to these issues, vLLM maintainers reported seeing problems with several models after constrained decoding was enabled. This made disabling constrained decoding by default the safer option for a model-agnostic serving engine such as vLLM. Later, vLLM added the --tool-strict-level flag so that users can enable tool-call strictness for their deployments.
Ubicloud’s focus is different from that of vLLM however. We focus on serving Kimi and DeepSeek models with the quality that users expect from first-party providers. For this reason, we enabled constrained decoding and re-ran the K2VV ToolCall benchmark. Once we verified 100% tool-call schema accuracy across all served model versions, we deployed the change to production. We then typed up this blog post to share our learnings.
When using LLMs, the quality you observe isn’t based on the model alone. The inference engine also plays an important role. Change the inference engine and its settings, and you change the model’s output quality.
When serving Kimi K2.6 and DeepSeek V4 Flash, we observed visible quality degradations due to tool-call schema inaccuracies. We solved this problem by first carrying our own vLLM fork and then enabling the new co
Auto-fetched and translated by AI Atlas. All rights belong to the original author.