TL;DR
做 LLM 后训练的都知道 OPSD（在线自蒸馏）香：不用奖励模型、信号稠密。但它有个老毛病——老师看了完整参考答案后分布过尖，监督信号跟学生的学习节奏对不上。这篇论文的解法很直白：别让整条回答共用一位老师，搭个 7 人候选池，让每个 token 自己挑。Qwen3-1.7B 数学推理平均分直接从 51.1 涨到 57.0（+5.9），代码已开源。
导语：调过蒸馏的都见过这个怪现象
先问一个可能戳到你的问题：你有没有遇到过，蒸馏训练前期涨分如飞，中后期分数不升反降，删数据、换教师、调系数都试遍了，就是没想过「监督信号本身可能就不对劲」？
这就是 OPSD（On-Policy Self-Distillation，在线自蒸馏——同一个模型既当学生又当老师，老师唯一特权是能「偷看」参考答案）这两年被反复报告的坑：老师拿到完整参考答案后，输出分布会变得过度尖锐，给学生的监督信号又硬又急，跟学生当前的学习动态完全错配。轻则涨分停滞，重则推理能力倒退。
那怎么办？2026 年已经有一批「自适应曝光」的工作（比如 Paint），思路是让老师少看一部分参考答案，看看多少「剧透」最合适。但这类方法有个共同的隐含假设：调整是回答级的——一条回答里几百个 token，全体共用同一位老师。
来自慕尼黑工业大学、京都大学、ELLIS Institute Finland 的团队在 arXiv 新发的这篇《Every Token Needs Its Own Teacher》直接对这个假设开炮：回答级的粒度太粗了，每个 token 都该有自己的老师。标题说得很直白：Every Token Needs Its Own Teacher。
想自己动手试？
- 📄 论文：arXiv 2610.40574
- 💻 代码：GitHub · xingzhejun/MTA-OPSD
有意思的是，本站之前解读过的 one-shot-opd（1 条训练数据追平 17K 数据集那篇）正好是这条 OPSD 线上的前作——那篇回答「数据要多少」，这篇回答「老师怎么给」，凑一起正好是 OPSD 的两大工程痛点。
这篇论文到底想解决什么问题？
先补 30 秒背景。OPSD 的设定是：学生自己采样一条回答，然后老师在回答的每个 token 位置上给出「下一步该输出什么」的完整分布，学生逐位模仿。老师和学生是同一个模型，唯一区别是老师的输入里多拼了一份参考答案——相当于开卷考试的学生。
听起来很美，但问题就出在这份参考答案上。
为什么会这样？因为参考答案信息量太大。老师一看到完整解题过程，下一个 token 的分布立刻变得非常确定（想象你做完型填空时手里已经有标准答案——每个空的答案都「显而易见」）。可学生这时候还没学到那个程度，被这种过尖的分布拽着走，反而会压抑模型自身的不确定性，把推理能力练歪。
已有的自适应方法（Paint 等）发现：让老师「少看点」参考答案（比如只看前一半），分布就不会那么尖，监督更温和。这确实有效，但它们只在整个回答的粒度上做调整——同一回答内的所有 token，拿到的还是同一份「半剧透」。
这篇论文的作者们提出了一个更细的疑问：在同一个回答里，有的 token 是关键推理步骤，有的只是「的」「了」这类连接词，它们需要的老师强度能一样吗？
一句话概括研究问题：教师的偏好是否随 token 而变？如果是，怎么在训练时高效识别每个 token 的偏好老师？
它的思路是什么？
整个方法分两步走：先用一个很聪明的实验证明「token 偏好确实存在」，再基于实验发现的两条规律设计选择算法。这个「现象 → 规律 → 方法」的推进方式，读起来相当顺。
第一步：怎么知道一个 token 喜欢哪位老师？
直接「逐组合训练」显然不可行（token 数 × 老师数，组合爆炸）。作者们想了个办法——教师干预实验，流程像接力赛：
- 保留学生已生成的前缀；
- 让候选老师从某个 token 位置接管，续写 64~256 个 token；
- 控制权交还学生，让它把回答写完；
- 对比「接力回答」和纯学生回答的正确率——提升大，说明这位老师在这个位置帮得上忙。
这个「学生-老师-学生」接力设计妙在：它用最终任务成绩说话，不依赖任何代理指标。作者在 Qwen3-1.7B/4B 上跑了 75264 次生成，横跨 7 个干预位置 × 7 位候选老师。
结果干净利落：同一回答里，不同位置的首选老师重合率只有 48%~52%——跟抛硬币差不多。如果 token 们没有各自的偏好，这个数字应该接近 100%。结论成立：每个 token 确实需要自己的老师。
第二步：规律拿来就用。
实验还顺带挖出两条可操作的规律：其一，让 7 位老师对某个 token「投票」（支持还是反对学生已采样的 token），多数派子集能覆盖 78%~91% 的首选老师——投票可以当可靠的第一道筛子；其二，首选老师几乎总是与学生 KL 距离最大的那个（KL 距离衡量两个分布差多远，越大说明老师给的信息跟学生当前认知差异越大、信息量越足）——KL 可以当第二道筛子。
于是 MTA-OPSD（Multi-Teacher Adaptive OPSD）水到渠成：
图说：左边搭 7 人候选老师池，中间对每个 token 投票筛方向，右边在多数派里挑 KL 最大的当专属老师。
- 搭老师池：还是那个模型，只是给参考答案做 3 种截断（前缀 / 后缀 / 随机段）× 2 种曝光比例（0.5 / 0.75），加上完整参考，共 7 位候选老师。截断时最终答案永远保留——保证每位老师都知道正确方向，只是「剧透程度」不同。
- 多数投票定方向：每个 token 位置，7 位老师各自表态「支不支持学生已采的这个 token」（看优势信号的正负号），跟多数派站一边的老师进入子集。这一步解决「方向可不可信」。
- KL 选择定人选：在多数派子集里，挑与学生 KL 距离最大的老师。这一步解决「信息量够不够」。
图说：三位老师对同一条回答的逐 token 信号，有的位置英雄所见略同，有的位置直接对着干——这就是 token 级选师的依据。
顺序为什么不能反？论文说得很透：KL 大不等于老师好。如果多数派方向是「反对」，而这位老师恰好跟学生很接近（KL 小），它最多轻微地往下压一压这个 token；只有 KL 大的老师才能提供足够强的纠正信号，把概率质量真正推向别的候选。反过来，方向一致时，KL 大的老师能用更丰富的分布「强化」学生的正确选择，而不是让学生死记硬背。消融实验也验证了这一点：跳过投票直接选最远 KL，平均分从 57.0 掉到 52.9；保留投票但选最近 KL，只有 53.3——方向和信息量，缺一不可。
整个机制没有任何新增网络或可学习参数，纯粹是训练时的选择逻辑，实现上基于 veRL 训练框架 + vLLM 推理，4 张 H200 就能跑全参数训练。
效果到底怎么样？
主结果一句话：三个规模全部第一，规模越小涨得越多。
平均分是 AMC23 / AIME24 / AIME25 三个数学推理基准的均值（Avg@12 采样，取训练中最优 checkpoint，越高越好）。几个值得咂摸的细节：
- AIME25 上提升最猛：1.7B 从基线 33.6 拉到 41.1，而原版 OPSD 只到 37.5——越难的题，老师给错信号的危害越大，token 级调整的收益也越大。
- SDPO 在 1.7B 上居然倒退（49.4 < 51.1）：说明光换散度函数解决不了粒度问题，这从侧面支持了「监督粒度才是关键矛盾」。
- 消融最扎心的一行：同样的 7 人老师池，每个 token 随机挑老师只有 53.6，把 7 位老师的损失取平均也只有 53.7——都接近原版 OPSD 的 54.0。老师池本身不涨分，涨分全靠「选对」。
还有个容易被忽略的工程彩蛋：单教师里最强的 Prefix-0.75（老师只看参考答案前 75%）单独训练就能拿 56.2，已达到完整方法收益的 94%。预算紧张的团队，从这个数字开始抄作业性价比最高。
为什么你要关心？
如果你在做（或打算做）模型后训练，这篇论文至少有三处直接可用：
- 成本敏感路线：先试「部分参考答案教师」——不用 7 人池，先让老师看 50%~75% 的参考答案，就可能拿到大部分收益。这是一个下午就能验证的改动。
- 免奖励模型的监督新思路：多数投票 + KL 最大的两步选择是通用模板，本质是「用集体共识保证方向、用距离保证信息量」。任何「监督该给多少、给谁」的场景（RL 负样本蒸馏、agent 轨迹训练）都能套。
- 一个现成的批判武器：论文附录里有个隐藏炸弹——曝光比例和学生-老师 KL 距离几乎不相关（0.75 曝光只有 49.1% 的概率 KL 更大）。这意味着「调节参考答案曝光 = 调节老师强度」的直觉根基并不牢，下次评审或选型讨论曝光调节类方案时，这是个有分量的反例。
再往大了说，OPSD 这条线（免 RL 奖励模型、稠密 token 级信号）正是当下开源模型后训练的热门选择，「教师错配」是这个方向公认的主要瓶颈。这篇论文没换范式、没改架构，纯靠调整监督粒度就白捡了几分——这种「把已有方法调细一个数量级」的思路，比动辄换个新算法的工作对工程团队友好得多。
理性看待
需要泼两盆冷水。一是开销：每个 token 要做 7 个条件上下文的前向，训练时教师侧算力大约是单教师的 7 倍，论文通篇没报训练时长对比，落地前务必自己测。二是泛化面：目前只有数学推理 + Qwen3 家族的证据，AIME 每榜仅 30 题，4B/8B 的 +1.5/+2.8 增益处在采样噪声边缘，代码、写作、工具调用等其他域能否复现还是问号。作者也把「自进化教师池」留作了未来工作——值得蹲一个续作。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
TL;DR
Those who do LLM training know the OSD scent: no reward models, dense signals. But there's an old problem -- teachers overstretch the full reference and monitor the signals against the students' learning rhythm. The paper is very straightforward: don't let the whole answer be a teacher and a pool of seven candidates for each token. The average of Qwen3-1.7B mathematical reasoning has increased directly from 51.1 to 57.0 (+5.9), the code is open.
Introduction: I've seen this weird thing before.
A question that may have been pointed at you: Have you ever encountered an increase in the pre-retort training rate, that the post-production score does not go up and down, that all the data are deleted, that all the teachers are replaced, that all the triangulations are tried, that is, that the "supervising signals themselves may not be right"?
This is the pit that has been repeatedly reported over the last two years: once the teacher has a full reference, the output distribution becomes too sharp, the monitoring signals for the students are hard and fast, and they are completely out of step with the student's current learning dynamics. The increase in light points has stalled and the ability to reason has been reversed.
So what? In 2026, there was a series of "self-adaptation" jobs (e.g. Pant), the idea being to give teachers less reference to the answers and to see how many of them were the best. However, there is a common implicit assumption that adjustments are of an answer level — hundreds of tokens in one response, all shared by the same teacher.
The team from Munich University of Industry, Kyoto University, Ellis Institute Finland, in arXiv's new edition of "Every Token Needs Its Own Taacher " directly opened fire on this hypothesis: the size of the response level is too thick, each token should have its own teacher. The title is plain: Every Token Needs Its Own Teacher.
You want to try it yourself?
- Paper: arXiv 2610.40574
- Code: GitHub. xingzhejun/MTA-OPSD
Interestingly, one-shot-opd (the one that trained the 17K data set) read before this station is the front of this OPSD line -- the answer to the question "How much is the data?" The answer to the question "how does the teacher give" is exactly what happened to the two major projects of OPSD.
What is this paper trying to solve?
30 seconds background. OPSD is designed to sample one answer for the students themselves, and then the teacher gives the complete distribution of what "next step should output" at each token point of the answer, and the students copy it. Teachers and students are the same model, and the only difference is that the teacher's input spell out an additional reference answer - the equivalent of the student who opens the examination.
Sounds beautiful, but that's the point of reference.
Why? Because the references are too big. As soon as the teacher sees the complete solution, the distribution of the next token becomes very certain (imagining that you have a standard answer when you're done empty - every empty answer is "obvious"). But instead of learning that level at this point, being dragged away by such an overstretched distribution, students tend to suppress the uncertainty of the model itself and to distort its reasoning.
The existing self-adaptation methods (Paint, etc.) have found that, by giving teachers the "less to read" answer (e.g., just the first half), distribution is less sophisticated and monitoring is more moderate. This does work, but they only adjust the size of the whole response — all the tokens in the same answer, or the same "half-perfect".
The authors of the paper raised a more detailed question: in the same answer, some token is a key line of reasoning, others are just a "go" connection, and do they need the same teacher strength?
One sentence summed up the question of whether teachers' preferences changed with token. If so, how do you efficiently identify every token teacher in training?
What's the idea?
The whole approach goes in two steps: first, using a very smart experiment to prove that "token preferences do exist" and then designing algorithms based on two patterns that were discovered. The way this "situational pattern method" moves forward, it's pretty good.
Step one: How do you know which teacher token likes?
Direct "package-by-train" is obviously not feasible (token numbers x number of teachers, group explosion). The authors think of a way - the teacher intervention experiment, the process like relay:
- The retention of prefixes created by students;
- To take over from one of the token positions and continue 64 to 256 tokens;
- Return control to the student so that it can complete the answer;
- A comparison of the "relay answer" and the correctness of a pure student's answer - an improvement that suggests that the teacher can help in this position.
This "student-teacher-student" relay is well designed: it speaks with final performance, without relying on proxy indicators. The author ran 75264 times on Qwen3-1.7B/4B, spanning 7 intervention positions x 7 candidate teachers.
The result was clean: in the same answer, there was only 48% ~52% of the preferred teachers at different locations -- almost the same as throwing coins. If the tokens don't have their own preferences, it should be close to 100%. The conclusion is: each token does need his teacher.
Step two: The law will work.
The experiment also led to two operational patterns: one, allowing seven teachers to "vote" for a token (support or oppose student sampled token), the majority group can cover 78% ~91% of the preferred teacher-- – Votes can be a reliable first screen; second, the preferred teacher is almost always the one who is the largest with KL. (The greater the distance between the two distributions, the greater the difference between the information given by the teacher and the student’s current perception, the greater the amount of information – KL can be the second screen.
So MTA-OPSD (Multi-Teacher Adaptive OPSD) reached the channel:
Figure: Seven candidates for the teacher pool on the left, with each token cast in the middle, and KL the biggest exclusive teacher in the majority.
- The teacher pool: the model, which is also the reference answer, is three cut-off (prefix/suffix/ random segment) x two exposure ratios (0.5 / 0.75), plus a full reference to seven candidates. The final answer remains — to make sure that every teacher knows the right direction, but that it's a different "draught."
- The majority vote was directed: each token position, seven teachers each said that they did not support this token that the students had taken (see the positive and negative sign of the dominant signal) and entered the subset with the majority teacher. This is a step to address the "untrustable direction".
- KL chooses the chosen teacher: in the majority collection, the most distant teacher from KL. This is a step towards solving the problem of "sufficient information".
The figure shows that the three teachers responded to the same article with a token signal, which is somewhat similar to that of the hero of the position, and that some of the positions were directly directed towards the candidate at the token level.
Why can't the order be reversed? Thesis is very clear: KL is not the best teacher. If the majority direction is "opposed," the teacher happens to be close to the student (KL small), and it presses this token down at the most slightly; only a KL large teacher can provide a sufficiently strong correction signal to actually push the probability quality to another candidate. On the other hand, when the direction is the same, KL’s big teacher is able to make the right choices with a larger distribution of “enhanced” students, rather than keeping them alive. The digestion experiment confirmed this: Directly selected KL, skipping the ballot, falls from 57.0 to 52.9 on average; the vote is retained but the most recent KL is only 53.3 — in the direction and in the amount of information that one cannot afford.
The entire mechanism does not have any additional network or learning parameters, it is purely a selection logic for training, and achieves an all-parameter training based on veRL training framework + vLM reasoning.
How did it work?
The main result, in one sentence, is that the three sizes are the first, and the smaller the size, the larger the size.
The average score is the average of AMC23 / AIME24 / AIME25 three mathematical reasoning benchmarks (Avg@12 sampling, taking the best checkpoint in training, as high as possible). A few details worth touching:
The biggest increase on AIME25 is 1.7B from the baseline 33.6 to 41.1, while the original version of OPSD reaches only 37.5 - the harder the problem, the more dangerous the teacher gives the wrong signal, the greater the benefit of the token level adjustment.
- SDPO reverses on 1.7B (49.4 < 51.1) by explaining that the light-replaced dispersing function does not solve the problem of particle size, which supports "the key contradiction is the monitoring of particle size".
- The best line: the same 7-person teacher pool, each token randomly selects a teacher only 53.6, and the average loss of seven teachers is 53.7 - close to the original version of OPSD 54.0. The teacher pool itself does not increase the points. The increase is based on "check-in."
There is also an easy-to-neglect project egg: the strongest of the teachers alone, Prefix-0.75 (75% before the teachers read the answers), can get 56.2, which is 94% of the total methodological gain. The team with a tight budget starts with the highest operational value.
Why do you care?
If you're doing (or planning) post-model training, there are at least three direct uses for this paper:
- Cost-sensitive route: a partial reference teacher - without a pool of 7 people, a 50% or 75% reference answer is likely to yield most of the benefits. This is a change that can be validated in the afternoon.
- New approaches to monitoring the incentive-free model: the largest two-step option for majority votes + KL is a common template, which is essentially a "collective consensus to ensure direction, distance and information". Any "supervising what to give, to whom" scenario (RL negative Z111Z distillation, angent track training) can be arranged.
- An off-the-shelf critical weapon: There's a hidden bomb in an appendix to the paper - the exposure ratio and the student-teacher KL distance is almost irrelevant (0.75 with only 49.1% probability of exposure KL). This means that the intuitive foundation of "regulating the exposure of reference answers = regulating teacher strength" is not sound, and the next time an evaluation or selection of options is discussed, it is a counterproductive example.
And to put it even higher, the OPSD line (free of RL reward models, dense token level signals) is the popular choice for training after the current open source model, and "teacher mismatch" is the main bottleneck in this direction. The paper, which has not changed the paradigm, has not changed the structure, and has taken a few points simply by adjusting the size of the monitoring particles — the idea of “delineating the existing method by one order of magnitude” is much more friendly to the engineering team than moving to a new algorithm.
A rational view.
We need two bowls of cold water. One is the cost: each token has to do seven conditions for the forward direction of context, and the teacher's side count is about seven times more than a single teacher at the time of the training, and the whole essay is not compared, so it must be measured by itself before landing. The second is panorama: there is now only evidence of mathematical reasoning + Qwen3 family, AIME has only 30 questions per listing, 4B/8B +1.5/+2.8 benefits on the edge of sample noise, replicability of other domains such as codes, writing, tool call or question mark. The author also left the "self-evolutionary teacher pool" for future work — worth going over.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.