同一段代码，前面只增加无关Token，DeepSeek会改变答案吗？
作者丨李娜
编辑丨岑峰
这是字节Seed在9月底一项研究中发现的现象。研究人员观察到，DeepSeek-V4-Flash基础模型会随着代码前方无关注释的长度变化，在正确答案“8”和错误答案“32”之间周期性切换。 (雷峰网(公众号：雷峰网))但这个存在于旧版基础模型中的异常，今天还能测出来吗？我们找来Seed论文使用的代码，通过DeepSeek官方API，对现役V4.1-Flash进行了两轮测试。结果与论文中的典型案例并不相同：固定采样的64组全部答对，另一轮测试中正确答案也始终占据预测概率首位，但它得到的概率却随着输入发生变化。为了降低长上下文的推理成本，DeepSeek V4采用了两种交替使用的注意力机制。
- CSA（压缩稀疏注意力），将8个Token组成一个压缩窗口，每隔4个Token生成一条压缩记录，再从中筛选与当前问题最相关的信息。
- HCA（重度压缩注意力），压缩得更加激进，每128个Token只保留一条压缩记录，但会读取全部历史压缩记录。
两种机制承担不同的工作：前者尽可能保留细粒度的检索能力，后者以更高的压缩率提供全局信息。论文发现，DeepSeek V4系列的性能漂移与CSA和HCA的分块KV压缩有关。同一条信息只要前后挪动几个token，就可能落入不同的压缩位置，改变保留和检索的效果。研究人员将这种位置称为“相位”（Phase）。如果模型更擅长检索某些相位的信息，却容易忽略另一些相位，即使关键信息本身没有变化，模型也可能从答对变成答错。这就是所谓的“相位敏感性”。图：8 Token压缩窗口、4 Token步幅示意图为了验证这一现象，团队将实验扩展到128K上下文，让模型从1.6万条键值记录中寻找指定信息。结果，DeepSeek-V4-Flash-Base表现最好的位置分组准确率达69.9%，最差的只有29.7%，相差40.2个百分点，且波动周期与压缩步幅一致。Seed进一步基于Qwen3-0.6B架构训练了多组对照模型，尝试排除其他因素。研究人员去掉位置编码RoPE，周期性弱点依然存在；取消可学习的压缩门控，改用均匀平均，弱点也没有消失。当压缩步幅分别设为4、6、8、12时，准确率波动的周期也随之改变，而全注意力对照模型没有出现同样明显的现象。这些实验将问题指向了分块缓存压缩机制，而不只是某次代码补全的偶然错误。当然，40.2个百分点是特定基础模型在受控检索实验中的最好与最差位置差距，不能直接理解为DeepSeek在日常使用中的准确率下降。我们参考论文中的FP8代码补全实验，调用DeepSeek官方API，对现役V4.1-Flash进行了一轮测试。（Seed最初观察到答案翻转的模型是DeepSeek-V4-Flash-Base，目前已无法通过我们使用的官方API调用。）测试方法并不复杂：保持代码不变，只调整前面无关注释的长度，让模型反复补全同一道题。正确答案是“8”，论文中曾观察到模型有时更倾向于错误的“32”。为了尽量减少其他因素的影响，我们取用了论文指定版本的官方推理代码，截取到相同的补全位置。经过Tokenizer核对，代码前缀与论文一致，均为983个Token。接下来，我们在代码前方添加不同长度的无关内容，保持真正需要补全的代码不变。测试共采用四类填充方式，每类设置16种长度，从24个Token逐步增加到39个Token。这样一共得到64组输入，相邻两组的输入Token数只相差1个。 图：实测输入对照，根据保存的API请求整理，两组输入仅增加一个Token。在64次固定采样设置的测试中，模型全部补全为正确答案“8”，没有出现Seed论文中正确答案与错误答案之间的翻转。不过，模型答对了，并不意味着它每次对答案都有同样的把握。我们又进行了64次概率测量，记录模型对正确答案“8”和错误答案“32”的预测偏好。结果，“8”始终是概率最高的候选，但它得到的概率并不固定。在文档字符串这组测试中，仅把前方填充从24个Token增加到25个Token，P(8)就从83.33%升至92.29%。我们把四种填充方式分别统计，发现在这一轮测试里，奇数长度下，“8”的平均预测概率都高于偶数长度，差距在3.6至5.8个百分点之间。不过，这些变化还不能说明我们复现了Seed发现的周期性弱点。虽然四类填充都呈现出奇数长度平均概率更高的现象，但并不是每组相邻输入都遵循这一规律。我们还对8组输入进行了重复测量，发现即使输入完全相同，模型给“8”的概率也会波动，最大相差约5.35个百分点。此外，概率测试中有6次随机生成了错误答案“32”，但这几次模型仍然认为“8”更可能出现，只是最终采样选中了“32”，并不意味着预测偏好发生了逆转。因此，这组实测没有观察到Seed论文中的答案偏好翻转，但记录到了不同填充长度下的概率变化。考虑到我们测试的是更新的V4.1-Flash，而且尚未完全排除填充内容和接口波动的影响，还不能把这种变化归因于缓存压缩架构。我们没有在此次测试中观察到答案翻转，但这不意味着V4.1已经完全消除了这一问题。Seed论文也曾在V4.1的部分测试条件下发现周期性的预测偏好变化。不同的输入设置和模型调用方式，都可能影响最终结果。不过，从长上下文检索实验来看，DeepSeek的新版本确实出现了明显改善。Seed论文显示，V4-Flash基础模型最好与最差相位的准确率相差40.2个百分点，经过后训练后缩小至19.1个百分点；到了9月发布的V4.1-Flash，这一差距进一步降至6.1个百分点。值得注意的是，V4.1并没有放弃缓存压缩，反而进一步提高了压缩效率。DeepSeek将部分压缩路径的步幅从4个Token缩小至2个Token，并采用新的Causal Encoder–Decoder（CED）架构，结合跨层缓存复用、低精度KV存储等技术继续降低成本。根据DeepSeek官方数据，相比上一代，V4.1的KV Cache显存需求降至四分之一，持久化缓存存储需求降至八分之一。图2｜不同版本在最好和最差相位上的准确率差异（据Seed论文数据制图）2025年2月，Seed参与的FlexPrefill研究尝试根据输入内容动态选择需要计算的注意力位置；今年9月，另一项PISA研究又提出通过分层检索降低稀疏注意力的计算开销。值得注意的是，FlexPrefill作者之一Yiyuan Ma，也是此次发现DeepSeek周期性弱点论文的通讯作者。在我们的64组概率测试中，DeepSeek始终更倾向于正确答案。如果只统计最终答案是否正确，这些输入之间的差异就很难体现出来。Seed在论文中也发现了类似的评测盲区。在一组自行训练的0.6B模型对照实验中，使用全注意力和分块缓存压缩的两个模型，平均检索准确率分别为61.1%和59.4%，仅相差1.7个百分点。但将结果按信息所处的位置拆开后，情况完全不同。全注意力模型表现最差的位置分组，准确率仍有58.4%；分块压缩模型却只有9.9%。两个平均分接近的模型，在某些位置上，可靠性可能完全不同。 图3｜平均分相近，最差位置表现却悬殊（据Seed论文0.6B实验数据制图）2024年，NVIDIA推出RULER，发现模型能通过简单的“大海捞针”测试，不代表它能完成复杂的信息检索与汇总。2025年，Adobe等机构提出NoLiMa，发现减少问题与答案之间的关键词重合后，模型表现也可能明显下降。从“能装下多少Token”到“能不能准确找回和理解信息”，长上下文评测不断变得更细。而Seed此次研究又揭示了新的维度：模型所采用的压缩架构，也可能让某些固定位置成为检索弱点。这种差异在Agent场景下尤其值得关注。Coding Agent需要持续读取代码、日志和工具反馈，关键信息的位置会随着任务推进不断变化。一次检索错误可能影响后续操作。不过，Seed目前的实验还不能直接证明相位敏感性导致了多少真实Agent任务失败，这仍需要更接近实际工作流的测试。因此，对采用分块压缩的模型来说，除了平均成绩，还可以进一步比较不同相位上的表现、最差位置的准确率，以及这些差异是否会影响连续任务的完成。对于其他架构，可能又需要设计不同的测试方法。回头看这次研究，DeepSeek将新的压缩架构应用于实际模型，字节Seed则借助公开权重，找到了此前不容易察觉的性能弱点。而在论文发表之前，DeepSeek已经通过新一轮迭代，让相关位置差异明显减轻。这也是当下大模型技术竞争值得关注的一面：一家公司的架构探索，可能成为另一家公司的研究对象；一次技术突破尚未被完全理解，新的问题和改进方向就已经出现。
可独家畅览：
专家演讲PPT
大会报告全文
热门论文解读
学术新星访谈
扫描上方二维码
或点击「阅读原文」关注专区。
雷峰网原创文章，未经授权禁止转载。详情见转载须知。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
The same code, adding nothing but Token. Does DeepSeek change the answer?
The author, Leena.
Edit Peak
This was the phenomenon found by byteSeed in a study at the end of September. Researchers observed that the DeepSeek-V4-Flash base model rotates cyclically between the correct answer “8” and the wrong answer “32” as the length of the unrelated note changes in front of the code. But can this anomaly, which exists in the old version of the basic model, be detected today? We found the code used in the Seed paper and conducted two rounds of tests on active V4.1-Flash through the official DeepSeek API. The results are not the same as the typical cases in the paper: all 64 sets of fixed samples corresponded, and the correct answers in the other round have always been at the top of the predictive probabilities, but the probabilities obtained change with the input. In order to reduce the reasoning costs of long context, DeepSeek V4 introduced two alternate attention mechanisms.
- CSA (compressed thin attention), which forms eight Tokens into a compressed window, produces a compressed record every four Tokens, from which the most relevant information on the current problem is screened.
- HCA (heavy compression of attention), which is more aggressive, keeps only one compressed record per 128 Token, but reads the entire historical compression record.
The two mechanisms undertake different tasks: the former retain as much as possible the ability to retrieve fine particles and the latter provide global information at a higher compression rate. The paper found that the performance of the DeepSeek V4 series was associated with the KV compression of sections of CSA and HCA. Any movement of several tokens of the same information may lead to different compression positions and change the effects of reservations and retrieval. Researchers refer to this position as “phase”. If the model is better at retrieving information on certain phases, it is easier to ignore other phases, even if the key information itself remains unchanged, the model may change from a yes to a yes. This is called “phase sensitivity”. Figure 8 Token Compressed Window, 4 Token Step Maps to validate this phenomenon, the team expanded the experiment to 128 Kcontext to allow the model to search for specified information from the 16,000 key logs. As a result, DeepSeek-V4-Flash-Base performed the best position grouping with an accuracy of 69.9 per cent, the worst being only 29.7 per cent, with a 40.2 percentage point difference, and the swing cycle was consistent with the compression step. Seed further trained multiple control models based on the Qwen3-0.6B structure, trying to exclude other factors. The cyclical weakness of researchers who remove the location code RoPE remains; the learning-able compression of the door is removed and evenly averaged, and the weaknesses remain intact. When compression steps were set at 4, 6, 8 and 12 respectively, the cycle of accuracy rate fluctuations changed, and the full attention control model did not show the same marked pattern. These experiments point the problem to a segment cache compression mechanism, not just an occasional error in the code completion. Of course, 40.2 percentage points represent the difference between the best and worst position of a given base model in a controlled search experiment and cannot be directly understood as a decline in the accuracy of DeepSeek in daily use. Referring to the FP8 code completion experiment in the paper, we called DeepSeek Official API to conduct a round of tests on active V4.1-Flash. (Seed initially observed that the model of the reversal of the answer was DeepSeek-V4-Flash-Base, which is currently unavailable through the official API we use. ) The test method is not complex: the code is kept unchanged and only the length of the non-note preceding is adjusted to allow the model to repeat the same subject. The correct answer is “8”, which was observed in the paper as sometimes more inclined to be the wrong “32”. In order to minimize the impact of other factors, we have taken the official reasoning code of the paper-designated version to the same complement position. Following a check by Tokenizer, the prefixes were consistent with the paper and were 983 Tokens. And then we're going to add to the code a different length of irrelevant content and keep the code that really needs to be completed. Four types of filling were used, each setting up 16 lengths, increasing gradually from 24 Token to 39 Tokens. This resulted in a total of 64 input groups, with only one difference between the number of Tokens entered by the two adjacent groups. Figure: The comparison of measured inputs is organized according to the saved API request, and only one Token is added to the two sets of inputs. In 64 tests of fixed sampling settings, the model was fully completed as the correct answer “8,” and there was no reversal between the correct answer and the wrong answer in the Feed paper. However, the fact that the model is correct does not mean that it has the same certainty about the answer each time. We did another 64 probabilistic measurements to document the model's prognosis for the correct answer “8” and the wrong answer “32”. As a result, “8” remains the highest-probability candidate, but the probability it receives is not fixed. Test in the document string group P (8) rose from 83.33 per cent to 92.29 per cent, with an increase in the front fill from 24 to 25 to 25. When we counted the four types of filling, we found that the average projection probability for “8” was higher than even length in this round of tests, with a gap between 3.6 and 5.8 percentage points. These changes, however, do not explain the cyclical weaknesses we found in Seed. While each of the four types of fill shows a higher average probability of odd length, not every group of adjacent inputs follows this pattern. We also repeated measurements of eight sets of inputs and found that even if they were identical, the probability that the model gave to 8 would fluctuate, with a maximum difference of about 5.35 percentage points. In addition, six of the probability tests randomly generated the wrong answer “32”, but the models still believe that “8” is more likely to appear, except that the final sample selected “32”, which does not imply a reversal of the prognosis. As a result, this set of measurements did not observe a preference for a reversal of the answers in the Seed paper, but the probability changes under different filling lengths were recorded. Considering that we have tested the updated V4.1-Flash and that the effects of fluctuations in filling and interfaces have not been completely excluded, this change cannot be attributed to the cache compression structure. We did not observe a reversal of the answer in this test, but this does not mean that V4.1 has completely eliminated the problem. The Seed paper also found cyclical prognosis changes under some V4.1 test conditions. Different input settings and models may affect the end result. However, from the long context search experiment, there has indeed been a marked improvement in the new version of DeepSeek. According to the Seed paper, the V4-Flash base model is ideally 40.2 percentage points different from the accuracy of the worst phase and has been reduced to 19.1 percentage points after training; by September, the gap had fallen further to 6.1 percentage points in V41-Flash of publish. It is noteworthy that, instead of abandoning the cache compression, V4.1 further increased the compression efficiency. DeepSeek reduced the range of partial compression paths from four Tokens to two Tokens and continued to reduce costs using the new Causal Encoder-Decoder (CED) architecture, combining cross-layer cache reuse and low precision KV storage. According to DeepSeek official data, the demand for KV CacheZ82Z in V4.1 fell to one quarter and the storage demand for sustainable cache to one eighth of the previous generation. Figure 2 | Differences in accuracy rates for different versions of the best and most differential phase (based on Seed thesis data) In February 2025, the FlexPrefil study, in which Seed was involved, attempted to select the desired focus based on input content dynamics; in September this year, another PISA study offered to reduce the cost of calculating attention by stratification. It is noteworthy that Yiyuan Ma, one of FlexPrefil ' s authors, was also the correspondent for the discovery of DeepSeek ' s periodic vulnerability paper. In our 64-group probability test, DeepSeek always preferred the right answer. If only the final answer is correct, the differences between these inputs are difficult to reflect. In his paper, Seed found similar assessment blind areas. In a group of self-trained 0.6B models, two models were compressed using full focus and segment cache, with average search accuracy of 61.1 per cent and 59.4 per cent, respectively, only 1.7 percentage points. But when the results are removed from the information, It's completely different. The worst-performing position grouping of the full attention model is still 58.4 per cent, while the segment compression model is only 9.9 per cent. Two models of average convergence may be completely different in some locations. In 2024, NVIDIA introduced RULEER and found that the model could be tested by simple “sea needles” and did not mean that it could complete complex information retrieval and aggregation. In 2025, agencies such as Adobe proposed NoLima, which may also see a significant decline in model performance as a result of the overlap of keywords between reduction and answer. From “how much token can be loaded” to “can information be accurately recovered and understood”, the long context assessment is becoming more sophisticated. The Seed study revealed new dimensions: the compressed structure used in the model may also make certain fixed positions vulnerable to retrieval. This difference is of particular concern in the Agent scene. Coding Agent requires continuous reading of codes, logs and tool feedback, and the location of key messages will change as the mission moves forward. A search error may affect subsequent operations. However, the current experiment of Seed cannot prove directly how much of the real Agent mission has failed as a result of phase sensitivity, which still requires closer testing to the actual workflow. Thus, in addition to average accomplishments, a further comparison of the performance of different phases, the accuracy of the worst position and whether these differences will affect the completion of successive missions can be made for the application of the block compression model. For other structures, different testing methods may need to be designed. Looking back at the study, DeepSeek applied the new compression architecture to practical models, while byteSeed, with its open weight, identified performance weaknesses that were previously difficult to detect. Before the paper was published, DeepSeek had adopted a new series of iteratives, which had led to a marked reduction in location differences. This is also a notable aspect of the current LLM technological competition: the structural exploration of one company may be the subject of research by another, and a technological breakthrough has not yet been fully understood and new problems and directions for improvement have emerged.
It's an exclusive view:
Expert presentation PPT
General Assembly
The reading of popular papers
Academic New Star Interview
Scan 2D above
This post is part of our special coverage Syria Protests 2011.
It's an original Ray Peak article, which is forbidden to reproduce without authorization. For further information, see the information below.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.