关键是企业 Agents 不理解突出显示

安德森角

关键是企业 Agents 不理解突出显示

在 Google 上的首选来源中添加 Unite.AI

世界上有许多重要、没有被扫描的报纸，据估计，每年通过全球贸易流通的40亿份纸质文件中只有1%是数字化的，剩下的则超出了机器学习的分析能力。

这仅仅是对商业部门问题的粗略评估；政府也生产了大量的印刷品或纸质产出，美国国家档案馆今年报告说，它所保存的模拟文件只有4.04%是数字化的，还有119.7亿美元有待实现。

我承认自己也是问题的一部分：

在审查论文时，我打印它们，因为它们需要大量的人文分析以及机器摘要。 因为时间很紧，我把大块的重心放在关键部分和可能的引文上，这样数据形状在写作的时候就会飞跃出来。 由于我没有足够的OCD来烦恼亮点

但是，很多人，包括许多写重要东西的重要人物，确实使用色彩亮点，以及替代文字颜色，来进行强调。

根据早期照片中描绘的论文(黑话上的那张)，可能存在数以亿计的文件，其中包含一层这样的强调，像Docling这样的文件摄入系统目前无法看到。

这些手动强调相当重要，因此，通过OCR(或更先进的方法，例如VLMs)将这样突出的一页内容简化为仅仅文字并不能传达作者的全部意图，甚至可能歪曲文件的宗旨和意图。

只是重点

新纸* 发现，广泛使用的文档转换器经常丢弃突出显示和文字颜色，因此AI agents仍然无法辨别重要指令。 在研究中审查的6,462份政策文件中，将近三分之一载有红色文字或突出内容，大约十分之一使用这些标记来提请注意重要的要求。

关于企业文件摄入agents，提交人指出：

`成熟转换器忠实地回收文字、桌子和结构，因此管道看起来很健康。

`我们显示，所有这些人都丢失了一层——意思是作者在文本的外观中编码——而且丢失使agents充满信心，而不是明显不完整，这是部署中更难发现的。

在作者的测试中，当AI模型被要求确定真实政策文件中的重要要求时，当原始的突出显示和文本颜色得到保存时，它们的表现要好得多 -- -- 这一发现在包括GPT-5.6 Terra,Claude Opus 5和Claude Sonnet 5在内的6个模型中得到了证实。

实际上，从强调以下两点中可以得出两个信号：(a) 任何事物都被强调，以及(b) 其颜色被强调。 在使用多种颜色进行手工突出显示的事例中，逻辑标题会以某种方式将红色编码为负色(如我们早些时候在温斯顿·丘吉尔对他自己发言的更正中看到的):

作者提出，现有的管道可以相当简单的解决问题，在文件转换过程中保留原有的突出显示和文字颜色，并添加说明其对于AI模型的意义的标签。 虽然这不需要额外的AI处理，但它几乎是代表每个文件所需的符号的两倍：

LLM的通话费用；这里报告的全部效果仅来自解析。 它确实花费了象征，大概是简单的两倍，因此它是一种蓄意的上下文预算交易，而不是自由赢。

大胆的漠不关心

研究的一个有趣的方面是，通过过度使用，打入数字文件的现有强调手段显然失效的程度：

`[即新管道]的全部效果通过突出[和][色来承载。 粗体、斜体、单空间和击球都无济于事[可衡量]，信任的间隔横跨着它。

这并非因为没有这些提示，这是因为它们太常见，无法提供信息。

`作者大胆地提出每个标题和定义的术语，并没有因此挑出任何内容，而彩色或强调要求则是故意的、省略的行为。 Cue 打字只支付足够罕见的提示

任务和测试

为了建立这一系统，提交人从一个未具体说明的公文库中以.docx格式扫描了6,462份真正的政策文件，检查这些文件是否为红色文本，并突出其内容，而没有根据措辞作出任何猜测。 然后要求这些模型挑出被标记为特别重要的要求，并留下没有标记的要求，核心迅速：

`在本文件中列出作者特别要注意或优先注意的每一项要求。 '

在主要测试中，选择了84份文件，其中载有332个标记要求和4,192个无标记要求(即没有红色文字或彩色突出显示的要求)。 每个文件必须包含至少一个标记的要求和三个没有标记的要求，因此AI不能简单地列出所有的东西。

基本真相直接来自文件的格式，而不是通过人类判断或另一种AI模式。 业绩是通过精确、召回和F1分数衡量的，既考虑到正确确定的要求，也考虑到错误地列入无标记的要求的假阳性。

评分是程序性的，没有涉及LLM法官。 要算作匹配，一个答案必须包含从原始要求中至少连续四个内容单词，有助于防止类似但不同的要求被算作正确。

突出事项

还进行了进一步测试，以确定是否仅从其措辞中即可确定重要要求，而不突出突出其内容或颜色。 接受过实例培训的仅文本分类器实现了ROC AUC的0.664分(其中0.5表示随机猜测).

更简单的方法表现的并不比机会好：当表示紧急的词被作为线索使用时记录了0.508的分数，而句长则产生0.492. 这些结果表明，仅从措辞上就很难了解突出作者的意图。

这有道理：大概，如果措辞足够有力，作者就不会感到有必要事后强调。

参赛者

比较的方法包括作者自己的解析器(OURS)；具有突出作用的OURS；无型号标记的OURS;docx2python;Docling；以及纯文本转换。

每种方法都使用了同样的文件和问题，只有文件的转换方式才有改变。 原则比较涵盖76份文件，每次测试重复三次：

如上所示，作者系统取得了0.690的F1总得分最佳，Docx2python的得分为0.586;Docling的得分为0.150；纯文本的得分仅为0.075(F1结合精确度和召回度成为单一的精确度，其中1.0是完美的得分)。

保留格式的重要性也表现在虚假的正面结果上：在Docling的情况下，未加标记的要求被错误地选取，其比例是作者系统记录的近四倍(4.3%,1.1%)。

在审查AI模型是否从被告知格式意味着什么，而不是简单地显示原始颜色和突出显示中受益时，使用了单独的测试，在提供这些额外标签时，发现一个很小但统计学上显著的改进。 然而，大多数好处是通过保留格式本身获得的。

外部压力

研究人员利用Microsoft办公室的 " 基准测试 " (OCB)进行了额外评估，该评估衡量了从办公室文件中检索资料的准确性。

从59个Word文件中提取了326个问题，涉及格式、表格、超链接、评论和脚注。 问题和评分标准均由Microsoft提供，独立于研究：

如上所示，作者系统取得了0.959的优异总分，而Docx2python的好分是0.810,Docling的好分是0.720，纯文本的好分是0.634。

新的元数据、评论和脚注系统取得了完美的分数，而所有相互竞争的方法都得分为零 — — 尽管这些改进不仅仅是因为保留了突出内容和文字颜色；而且作者系统在从表格、文件布局、字体和超链接中恢复信息方面也证明得更好。

结论

我最近试图看看我个人在PDF科学论文中对表格和数字的手动强调是否能通过LLMs和/或Python自动化，科学论文的编集、量化和解析如今已成为文献中最受欢迎的部分，对于业余爱好者来说，没有太多的余地 — — 或希望。

关于这份文件，作者的胜利是确定了翻译中丢失的重要重点指标。 然而，从OCR和文档摄入层需要操作的规模来看，其解决方案所要求的象征性开支翻一番并不是笑话。 因此，我认为，这里的奖项是确定这一问题。

* 标题为 " 作者写什么，Agent读什么：企业文件摄入中寂静的Cue损失 "。

- 都来自Amazon。

虽然我经常将作者的内在引用转换为超链接

2026年10月30日星期五首次出版。

本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

Anderson's Angle

Critically, Enterprise Agents Don’t Understand Highlighting

Add Unite.AI to your preferred sources on Google

There’s a lot of important, unscanned paper in the world, and particularly in the business world. It’s estimated that only 1% of the four billion paper documents circulating through global trade each year are digitized, leaving the rest beyond the analytical powers of machine learning.

That’s just a rough appraisal of the problem in the business sector; governments also produce a huge amount of printed or paper-bound output, and the US National Archive reported this year that only 4.04% of the analog documents it maintains are digitized, with 11.97 billion yet to go.

Therefore it’s an ongoing rather than a legacy problem, and I confess that I am part of the problem too:

In reviewing papers, I print them, because they need a lot of human parsing as well as machine summary. Because time is tight, I scrawl large emphases over key sections and potential quotes, so that the shape of the data leaps out at me when it’s time to write. Since I’m not quite OCD enough to bother with highlighters, added emphasis is indicated by diverse symbols, larger text, and other very personalized signals of importance.

But a lot of people, including a lot of important people writing important things, do use color highlighters, and alternate text colors, for emphases.

According to the very paper depicted in the earlier photo (the one on the black lectern), there may be documents numbering in the hundreds of millions that contain a layer of such emphasis that document-ingestion systems such as Docling are not currently able to see.

These manual emphases are significant, giving weight to text in a way that isn’t necessarily reflected in the construction of the language. Therefore reducing the content of such a highlighted page to mere text through OCR (or more advanced methods, such as VLMs) does not convey all of the author’s intent, and may even misrepresent the purposes and intentions of the document.

Just the Highlights

The new paper* finds that widely-used document converters routinely discard highlighting and text colors, so that AI agents remain unable to discern important instructions. Of the 6,462 policy documents examined in the study’s tests, nearly a third contained red text or highlighting, and around one in ten used these markings to draw attention to important requirements.

In regard to enterprise document ingestion agents, the authors† state††:

‘Mature converters recover text, tables and structure faithfully, so the pipeline looks healthy.

‘We show that one layer is lost by all of them—the meaning an author encodes in how text looks—and that the loss makes agents confidently wrong rather than visibly incomplete, which is the harder failure to detect in deployment.’

In the authors’ test, when AI models were asked to identify important requirements in real policy documents, they performed far better when the original highlighting and text colors were preserved – a finding confirmed across six models, including GPT-5.6 Terra, Claude Opus 5, and Claude Sonnet 5.

Effectively, it’s possible to extract two signals from highlighting: a) the fact that anything was highlighted at all and b) the color it was highlighted in. In an instance where multiple colors were used for manual highlighting, a logical rubric would code red as in some way negative (as we saw earlier, in Winston Churchill’s corrections to his own speeches):

The authors suggest that existing pipelines can solve the problem quite simply, by preserving the original highlighting and text colors during document conversion, and adding labels that explain their significance to the AI model. While this requires no additional AI processing, it does roughly double the number of tokens needed to represent each document:

‘Recovering these cues is deterministic, code-only, and costs no LLM calls; the entire effect reported here comes from parsing alone. It does cost tokens—roughly twice plain text—so it is a deliberate trade of context budget for factuality rather than a free win.’

Bold Indifference

One interesting facet of the study is the extent to which the existing means of emphasis in typed digital documents is, apparently, ineffective through overuse††:

‘The entire effect [i.e., of the new pipeline] is carried by highlight [and] [colour]. Bold, italic, monospace and strikethrough contribute nothing [measurable], with confidence intervals straddling it.

‘This is not because those cues are absent: a single document carries over a thousand bold markers. It is because they are too common to be informative.

‘An author who bolds every heading and defined term has not thereby singled anything out, whereas colouring or highlighting a requirement is a deliberate, sparing act. Cue typing pays only for cues rare enough to signal intent.’

Tasks and Tests

To establish the system, the authors scanned 6,462 real policy documents from an unspecified public corpus in .docx format, checking the documents for red text and highlighting, without making any guesses based on the wording. The models were then asked to pick out the requirements that had been marked as especially important, and leave the unmarked ones alone, with the core prompt:

‘List every requirement in this document that the author has singled out for special attention or elevated priority.’

For the main test, 84 documents were selected, containing 332 marked requirements and 4,192 unmarked ones (i.e., requirements without red text or colored highlighting). Each document had to contain at least one marked requirement and three unmarked ones, so the AI couldn’t simply list everything.

The ground truth was established directly from the documents’ formatting, rather than through human judgment or another AI model. Performance was measured using precision, recall and F1 scores, taking account of both correctly identified requirements and false positives, where unmarked requirements were mistakenly included.

Scoring was programmatic, with no LLM judge involved. To count as a match, an answer had to contain at least four consecutive content words from the original requirement, helping to prevent similar but different requirements from being counted as correct.

Ablating Highlights

A further test was carried out to see whether important requirements could be identified from their wording alone, without highlighting or colors. A text-only classifier trained on examples achieved an ROC AUC score of 0.664 (where 0.5 represents random guessing).

Simpler methods performed no better than chance: a score of 0.508 was recorded when words suggesting urgency were used as clues, while sentence length produced a score of 0.492. These results suggest that very little can be learned about what the highlighter-wielding author intended, just from the wording alone.

This makes sense: presumably, if the language had been adequately forceful, the author would not have felt the need for any post facto emphasis.

Contenders

The approaches compared comprised the authors’ own parser, OURS; OURS with salience roles; OURS with untyped markup; docx2python; Docling; and plain-text conversion.

The same documents and questions were used for each approach, with only the way the documents were converted being changed. The principle comparison covered 76 documents, with each test repeated three times:

As shown above, the best overall F1 score of 0.690 was achieved by the authors’ system, compared with 0.586 for docx2python; 0.150 for Docling; and just 0.075 for plain text (F1 combines precision and recall into a single measure of accuracy, where 1.0 represents a perfect score).

The importance of preserving formatting was also demonstrated by the false-positive results: with Docling, unmarked requirements were incorrectly selected at nearly four times the rate recorded for the authors’ system (4.3% versus 1.1%).

A separate test was used to examine whether AI models benefited from being told what the formatting meant, rather than simply being shown the original colors and highlighting, and a small but statistically significant improvement was found when these extra labels were provided. However, most of the benefit had already been obtained by preserving the formatting itself.

External Pressure

The researchers conducted an additional assessment using Microsoft’s Office Comprehension Benchmark (OCB), which measures how accurately information can be recovered from Office documents.

326 questions were extracted from 59 Word documents, covering formatting, tables, hyperlinks, comments and footnotes. Both the questions and the scoring criteria were supplied by Microsoft, independently of the study:

As shown above, an excellent overall score of 0.959 was achieved by the authors’ system, compared with a good score of 0.810 for docx2python, a moderate 0.720 for Docling, and a relatively poor 0.634 for plain text.

Perfect scores were achieved by the new system for metadata, comments and footnotes, while all competing methods scored zero – though these improvements weren’t solely due to preserving highlighting and text colors; and the authors’ system also proved better at recovering information from tables, document layouts, fonts and hyperlinks.

Conclusion

I tried with little success recently to see if my own manual emphasis of the tables and figures in a PDF science paper could be automated via LLMs and/or Python, and the results were…well, hilarious. Stratifying, quantifying and parsing science papers is now a well-subscribed strand in the literature, and there’s not much room – or hope – for the casual amateur.

As regards to this paper, the authors’ victory is to have identified a significant emphasis indicator that was being lost in translation. However, the doubling of token spend required by their solution is no joke, at the kind of scale that OCR and document ingestion layers need to operate at. Therefore, the prize here is to have identified the issue, I think.

* Titled ‘What the Author Wrote, What the Agent Reads: Silent Cue Loss in Enterprise Document Ingestion’.

† All from Amazon.

†† Authors’ emphases, not mine, though I do routinely convert authors’ inline citations to hyperlinks.

First published Friday, October 30, 2026

Auto-fetched and translated by AI Atlas. All rights belong to the original author.