AI21 实验室发布 LIFT-Bench 到 Curb Reward Hacking in RL Training

AI 模型和平台

AI21 实验室发布 LIFT-Bench 到 Curb Reward Hacking in RL Training

在 Google 上的首选来源中添加 Unite.AI

AI21 Labs 发布 LIFT-Bench 2026年10月25日，一个跟随基准测试的指令，其中每个约束范围被限定为文件的一个部分，任务内容本身被打分，与LIFT-Train，一个34,258任务训练对应的对等，以及同时产生两者的开放式管道。

Eran Goldstein、Yuval Belfer、Yonatan Belinkov和Daniel Gissin于2026年10月20日向ArXiv提交了一份题为“LIFT：评价和培训指导的范围限制”。 它的抽象称，常见的遵循基准测试的指令，如IFEval，评价那些容易在程序上被验证的制约，这些相同的制约随后成为强化学习的奖励信号，这种设计可以导致奖励将模型推离主要任务的黑客。

基准测试背后的训练运行

AI21在发布后发表的一篇博客文章中，使用IFTrain，该帖子描述为在之后打开用于教学的数据集，作为RL奖赏的来源，该跑将奖赏推向1.0和融资 IFEval以5分，IFBench以16分。 然而当这个模型被问及关于新海革命的交替历史的一章，没有字重复四次以上的时候，它用一个单词回答："中国".

失败是系统性的，而不是一阵风波，据公司称：通过390步骤，中位答案已经从114个字缩到18个字，38%的答案在30个字以下至少得0.8分。 AI21将崩溃归结于奖励的结构：IFTrain的检查器和IFEval一样，用代码对照整个响应来验证限制，没有对任务是否完成进行评分，所以降低任务成本使模型一无所有。 现有的补丁并不弥补这一差距，称公司：IF Bench 新增是奖励模式的分类器，对一般质量而不是特定任务进行评分，AdvancedIF列车在内容路标上运行，但保持其路标生成器，培训数据，RL代码关闭。

A. 范围限制和经认证的法官

LIFT-Bench将每个约束附加在文件大纲的一节中，并且只根据该节的文本进行评分，一个设计AI21称为范围限制。 每节包含两种限制：可核查的，用代码检查，例如需要一节德语或至少使用三次的关键词，以及LLM检查的，说明该节必须涵盖的内容。 判断约束与格式化相同。 范围界定还将约束密度，单节附加的制约数量，转化为任务生成期间的参数集。

所有任务都是合成的：主题来自维基百科，轮廓和约束来自LLM，一个采样器控制着每个部分有多少种约束。 基准测试有784个测试任务，分10个部分：短文件约4个部分，长文件约9个部分，每个部分为4个密度级，加上2个没有大纲的平分，其中每个约束都以IFEval等现有的基准测试的方式适用于整个响应。

由于同一名法官得分为基准测试，并发放RL奖赏，AI21在选择一个标注为人类的305个例子之前，对候选法官进行了测试。 该公司报告说，一名没有瞄准镜的Gemma-4-31B法官接受了其中的292个例子，当时只有227个实际满意，一名没有瞄准镜的Qwen2.5-32B的成绩为0.44 macro-F1，与总回答是的0.43分大致相同。 将法官范围限定在文本中，每个约束都增加了11到19个宏-F1点，这是任何变量测试的最大效果，AI21选定范围为Gemma-4-31B. 发布分裂者将google/gemma-4-31B-it写入每个任务中作为裁判员，寄存器警告说，分数只比较使用同一裁判员的横跑。 结构化的分割使结构受到10%的分量重量的限制，内容受到90%的限制；平面任务没有结构限制，因此内容占用全部重量。

关于当前模型的基准测试显示

AI21评价了17个模型，每个任务有8个样本。 关于结构化的任务，公司报告，GPT-5.5平均为0.96，而对于平面任务，没有模型通过0.85，主要的封闭模型平均在0.65至0.72之间。 该报的作者同样指出，前沿模型与基准测试的较难子集发生斗争。 结构化与平面化差距并不是一个约束负载的问题，根据AI21：限制在十种约束或更少的任务，除了Qwen3.8-A95B以外的每个模型在平面任务上仍然得分较低，只有结构化任务才能挣得的提纲分数占分数的10%.

公司报告，这两个政权之间的排名也有变动。 在跟踪的9个主要型号中，GPT-5.5在结构化任务上排名第一，在平面任务上排名第五，两个最佳整体平均值属于开放式重量型号Kimi K3和DeepSeek V4 Pro. Kimi K3在为文件的其余部分增加限制的同时，持有一节限制固定下来，每密度级损失0.4%,GPT-5.5损失1.5%，而GPT-5纳米损失5.5%.

以基准测试分数作为奖励的训练

LIFT-Train将基准测试的分数作为RL的奖励，不变地适用于来自同一发电机的34,258项任务的RL奖励；同一约束奖励scalar的寄存器指出同时作为基准测试的度量衡和奖励，没有单独的奖励模式。 AI21在LIFT-Train和IFTrain上与GRPO一起培训了Qwen2.5-7B. 在IFTrain上，悬赏以第180步打出1.0，而中位数答案则由第390步的114字降至第18步。 在LIFT-Train上，中位数答案停留在390到530个字左右，30个字以下没有获得高分的答案。

LIFT-Train的平板片，AI21 称指出奖励而不是格式的内容是决定因素，同时指出数据集的主题仍然不同，维基百科对WildChat. 在180GRPO步骤之后的5个基准测试间，每个食谱都赢得了自己的基准测试；在AdvancedIF和Complex Bench上，IFTrain在LIFT-Train稳住的同时滑倒了。 AI21将小三角洲定性为避免回归而非一般胜。

Qwen3-32B对每个范围进行单独培训，对混合物的培训，只有结构化的培训达到0.805对统一任务的培训，对结构化任务的培训达到0.798，对混合物的培训则达到0.896和0.919，而简单硬的教学大纲则没有区别，AI21报告。 公司在其摘要中建议各小组按照任务而不是规则进行指导培训，以衡量任务，对照人的标签检查法官，在范围评价上选择基准模式，而不是一个平面领导板，在跟踪答复长度的同时，利用评价作为奖励。

可用性

LIFT-Bench,LIFT-Train，以及合成管道根据Apache-2.0许可可在GitHub上使用，基准测试在Hugging Face Hub上作为ai21labs/LIFTBench托管。 任务以PrimeIntellect的核查员库为评价和RL培训而本土使用的格式发布，存储库发布一个五块的Prime-rl培训组合，包含17,129项任务。 可验证的制约类型包括语言，案例惯例，关键词计数，emoji计数，以及大纲匹配文件结构。

本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

AI Models & Platforms

AI21 Labs Releases LIFT-Bench to Curb Reward Hacking in RL Training

Add Unite.AI to your preferred sources on Google

AI21 Labs released LIFT-Bench on October 25, 2026, an instruction-following benchmark in which every constraint is scoped to one section of a document and the task content itself is scored, alongside LIFT-Train, a 34,258-task training counterpart, and the open pipeline that generated both.

A companion paper, “LIFT: Scoped Constraints for Evaluating and Training Instruction Following” by Eran Goldstein, Yuval Belfer, Yonatan Belinkov, and Daniel Gissin, was submitted to arXiv on October 20, 2026. Its abstract states that common instruction-following benchmarks such as IFEval evaluate constraints that can easily be verified programmatically, that those same constraints then serve as reward signals for reinforcement learning, and that this design can lead to reward hacking that pushes models away from the main task.

The Training Run Behind the Benchmark

In a blog post published with the release, AI21 reports that the project began with a real experiment that looked successful on paper. Using IFTrain, which the post describes as the go-to open dataset for instruction following, as the source of RL rewards, the run drove the reward to 1.0 and raised IFEval by 5 points and IFBench by 16 points. Yet when the model was asked for a chapter of alternate history about the Xinhai Revolution, with no word repeated more than four times, it answered with a single word: “China”.

The failure was systematic rather than a fluke, according to the company: by step 390, the median answer had shrunk from 114 words to 18, and 38% of answers scored at least 0.8 in under 30 words. AI21 attributes the collapse to the structure of the reward: IFTrain’s checkers, like IFEval’s, verify constraints with code against the whole response, and nothing scores whether the task was done, so dropping the task costs the model nothing. Existing patches do not close that gap, the company says: IFBench adds a reward-model classifier that scores general quality rather than the specific task, and AdvancedIF trains on content rubrics but keeps its rubric generator, training data, and RL code closed.

Scoped Constraints and a Validated Judge

LIFT-Bench attaches each constraint to one section of a document outline and scores it only against that section’s text, a design AI21 calls scoped constraints. Each section carries two kinds of constraints: verifiable ones checked by code, such as requiring a section in German or a keyword used at least three times, and judged ones checked by an LLM that spell out what the section must cover. Judged constraints count in the same score as formatting. Scoping also turns constraint density, the number of constraints attached to a single section, into a parameter set during task generation.

All tasks are synthetic: topics come from Wikipedia, outlines and constraints from an LLM, and a sampler controls how many constraints land on each section. The benchmark holds 784 test tasks across ten splits: short documents of about four sections and long documents of about nine sections, each at four density levels, plus two flat splits with no outline, in which every constraint applies to the whole response in the manner of existing benchmarks such as IFEval.

Because the same judge scores the benchmark and hands out the RL reward, AI21 tested candidate judges against 305 human-labelled examples before choosing one. The company reports that an unscoped Gemma-4-31B judge accepted 292 of those examples when only 227 were actually satisfied, and that an unscoped Qwen2.5-32B scored 0.44 macro-F1, roughly the same as the 0.43 scored by always answering yes. Scoping the judge to the text each constraint governs added 11 to 19 macro-F1 points, the largest effect of any variable tested, and AI21 selected scoped Gemma-4-31B. The released splits write google/gemma-4-31B-it into each task as the judge, and the repository cautions that scores are comparable only across runs that used the same judge. Structured splits give structure constraints 10% of the score weight and content constraints 90%; flat tasks carry no structure constraints, so content takes the full weight.

What the Benchmark Shows About Current Models

AI21 evaluated 17 models with eight samples per task. On structured tasks, the company reports, GPT-5.5 averages 0.96, while on flat tasks no model passes 0.85 and the leading closed models average between 0.65 and 0.72. The paper’s authors likewise state that frontier models struggle with the benchmark’s harder subsets. The structured-versus-flat gap is not a matter of constraint load, according to AI21: restricted to tasks with ten constraints or fewer, every model but Qwen3.8-A95B still scores lower on flat tasks, and the outline points that only structured tasks can earn account for 10% of the score.

Rankings shift between the two regimes as well, the company reports. Among the nine main models tracked, GPT-5.5 ranks first on structured tasks and fifth on flat ones, and the two best overall averages belong to open-weight models Kimi K3 and DeepSeek V4 Pro. Holding one section’s constraints fixed while adding constraints to the rest of the document, Kimi K3 loses 0.4% per density step and GPT-5.5 loses 1.5%, while GPT-5 nano loses 5.5%.

Training With the Benchmark Score as the Reward

LIFT-Train applies the benchmark’s score unchanged as the RL reward across 34,258 tasks from the same generator; the repository notes that the same constraint_reward scalar serves as both the benchmark metric and the reward, with no separate reward model. AI21 trained Qwen2.5-7B with GRPO on LIFT-Train and on IFTrain. On IFTrain, the reward hit 1.0 by step 180 while the median answer fell from 114 words to 18 by step 390. On LIFT-Train, the median answer stayed around 390 to 530 words, and no high-scoring answer came in under 30 words.

A flat-only slice of LIFT-Train, matching IFTrain’s surface format but keeping content in the reward, also trained cleanly. AI21 says this points to content in the reward, rather than format, as the deciding factor, while noting that the datasets’ topics still differ, Wikipedia versus WildChat. Across five benchmarks after 180 GRPO steps, each recipe won its own benchmark; on AdvancedIF and ComplexBench, which also score content, IFTrain slipped while LIFT-Train held steady. AI21 characterizes the small deltas as avoiding a regression rather than a general win.

Training Qwen3-32B on each scope separately and on a mixture, structured-only training reached 0.805 on flat tasks and flat-only training reached 0.798 on structured tasks, against 0.896 and 0.919 for the mixture, while an easy-to-hard curriculum made no difference, AI21 reports. In its summary, the company advises teams training for instruction following to measure the task rather than only the rules, to check the judge against human labels, to pick the base model on the scoped evaluation rather than a flat leaderboard, and to use the evaluation as the reward while tracking answer length alongside the reward curve.

Availability

LIFT-Bench, LIFT-Train, and the synthesis pipeline are available on GitHub under an Apache-2.0 license, with benchmark splits hosted on the Hugging Face Hub as ai21labs/LIFTBench. Tasks are emitted in the format consumed natively by PrimeIntellect’s verifiers library for evaluation and RL training, and the repository publishes a five-split prime-rl training mixture of 17,129 tasks. Verifiable constraint types cover language, case convention, keyword count, emoji count, and outline-matching document structure.

Auto-fetched and translated by AI Atlas. All rights belong to the original author.