2026年11月4日 (英语).
研究科学家Otilia Stretcu和学生研究员Leijie Wang,Google研究
Agile De Leament通过从范围界定子类到解决边缘案例的交互审议工作流程，通过使用LLM驱动优化循环，自动构建一个准确，对齐的分类器，来解决构建AI图像分类器对主观类的挑战。
机器学习模型从人类所标注的事例中学习认识我们周围的世界。 这一过程的基础是一种假设，即进行标签的人在开始前对类别有一个明确而稳定的定义。 这种假设在培训一个客观类别的模型时是很好的，例如停止标志或金色的检索器，其边界是明确和普遍承认的。 然而在实践中，现实世界的应用往往需要视觉系统来导航人类经验的主观"灰色区域". 例如，考虑培训一个视觉分类员，以识别“健康食品”，管理“启发艺术”或协助内容节制。 在此类领域，类别界限可能不是普遍性或静止性的，因为它们可能随着文化背景、个人哲学或不断变化的业务政策而改变。 此外，人们很少从对主观概念的完整阐述开始，甚至在他们自己的思想中也是如此。 相反，它们从概念的直观概念开始，仅仅发现概念的真正界限、例外和细微差别，因为它们遇到现实世界的例子。 传统培训工作流程不包含这个 c oncept 演化，迫使用户向AI系统提供刚性标签，即使他们自己的精神模型仍在转移。
为了弥补这一差距，我们引入了Agile De control，这个框架将主观AI分类器的创建视为迭代对话。 Agile De Releasing 与其要求人类充当静态数据标签，不如将AI系统转变为协作伙伴，帮助用户"精心设计"，并磨练自己的定义。 该框架首先解决了人类的模糊性问题，从而系统地建立了更有效、更有力和更一致的人工智能系统。 为了支持更广泛的研究界，我们还拥有开源在GitHub上的基础代码和算法。
这部作品的一个关键不同之处在于它专注于现场用户体验。 因为标准、静态的AI 基准测试无法捕捉主观概念，我们绕过模拟测试，这使得我们能直接观察和测量一个人的意图在与智能系统配对时是如何动态演变的。 我们发现，Agile De Relief通过用直接反馈边界线图像取代手动即时工程，大大降低了用户的疲劳和焦虑，从而导致更一致，性能更高的AI模型超过标准基线。
积极审议将定义主观概念的复杂任务分为两个结构化阶段：概念范围界定和概念迭代。 这两个阶段共同将模糊的想法转化为AI模型的高度精确的教学蓝图。
当面对"健康食物"这样的广泛话题时，人类的直觉自然会挣扎着列出每一个规则的前面。 在概念范围界定阶段，系统指导用户将其总体概念细分为分概念和明确排除的层次结构。
例如，用户可能将“健康食品”分解成正亚类，如“萨拉兹”、“格兰碗”和“莲蛋白”。 同时，它们可能定义明确的排除，如"加工食品"或"劳，未备料". 这创造了一个初步的结构化基线，以证明模型的理解。
随着初始结构的建立，框架从静态规划转向主动测试。 传统的机器学习假设人类的注释者是"oracle"，已经想到一套完整的定义规则。 积极审议改进了这一前提，而侧重于协作完善。
这个阶段从积极学习中汲取灵感，这种方法传统上用来检索信息数据点，以最大限度地提高机器的训练效率。 然而，Agile De Releasing并没有像在主动学习中那样，纯粹为了尽量减少机器的不确定性而选择例子，而是使用视觉语言模型(VLMs)来积极挖掘和表面的边缘图像，以挑战目前基于文本的定义。
例如，这个系统可能会呈现水果滑冰的画面来挑战"健康饮食"的概念，迫使用户决定：这是一种健康饮料，因为它含有原始水果，还是由于糖含量高而被排除在外的加工食品？
面对这些模棱两可的边缘案例，用户必须反思自己的隐性假设，澄清其真实意图，并相应更新政策规则。 这种人类的反馈随后直接被引导到自动的即时优化循环，确保VLM的分类边界随用户精细的定义同步转移。
评估主观视觉分类者是一个根本性的挑战：没有静态、客观的地面事实来衡量业绩，因此，对预先贴有标签的数据集的离线评价无法衡量框架作为人的思想模型在考虑过程中如何适应概念转变。
为了抓住这一动态，我们构建了一个功能原型，并通过现场用户研究来评价。 我们举办了18次现场用户会议，每次持续1.5小时，参加者与我们的系统和替代基线进行互动，随机排列，以防止学习偏差。
我们选择了两个复杂，高度主观的域来测试这些系统的极限：
- 健康食品：个人健康哲学、饮食限制和文化背景能够产生高度多样化、主观观点的领域。
- 对付：现实世界的内容适度挑战，需要抓住专家层面的细微差别和具体背景。 这涉及在网络娱乐中识别闪烁图像，或游戏中提供误导性的财政或数字奖励。
我们对照三个共同的独特模式评估了Agile Dele control:
——零射分级：标准"一弹一弹"办法。 用户写一个初始的提示，但无法看到边缘大小写或者在模型误解边界时进行调整。
建模合作者：一种自动化的自上而下的方法，由AI agent试图自行分解概念。 虽然它会自动生成标准，但它会将用户从循环中移除，从而无法引导或纠正其逻辑。
- 人工审议：乏味的、野蛮的、强制的现状过程，用户在电子表格中检查数据，猜测搜索查询以发现异常，并手写原始提示。
现场用户会话显示，Agile De Relections产生性能较高的分类器模型，同时大大改善用户体验。
我们发现，经过有条理的审议过程的研究参与者表明，随着时间推移，标签的一致性明显提高。 通过早期解决边界模糊问题，它们建立了大大超过所有标准基线的分类器。 这些性能收益在最具挑战性的主观图像上最为戏剧性——即传统分类器通常因为依赖于默认，通用AI假设而不是用户明确，细微的意图而失败的确切领域。
与会专家报告说，该系统是有效的诊断工具。 AI系统通过故意服务于混淆的，边缘的图像，帮助他们在最初的定义中发现"盲点"——发现他们没有预料到的情景，直到呈现出模棱两可的现实世界的例子。
我们现场研究的一个重要发现是人工即时工程的认知成本。 我们的研究表明，用户在试图预测在人工概念审议过程中可能发现边缘案例的搜索术语时，会经历明显的认知疲劳。 此外，他们报告说，他们一直感到焦虑，因为手动调整一个文本，以纠正一个错误，可能会无意中降低他们已经修好的分类器的其他部分的性能。
积极审议通过处理繁重的计算升力来解决这种摩擦。 用户不需要成为即时工程师；他们只是看一看被整理的边线图像画廊，提供自然反馈。 系统无缝地吸收了它们的意向，并自动更新了基础模型。
因此，使用Agile Delectric法的与会者报告说，他们感到压力小得多、沮丧少、精神消耗少，每个与会者都表示倾向于Agile Delectric法而不是人工方法。
归根结底，Agile De Lealy将人与人间机器学习的范式从单行道——人工智能系统只是从人类的标签中学习——转变为人类与人工智能共同完善概念的双行道合作。
随着机器学习系统越来越多地用于处理复杂的现实世界社会任务，确保其界限反映细微的主观概念至关重要。 无论是增强信任和安全团队的能力，以建立更安全的内容节制过滤器，帮助馆长塑造数字收藏，还是帮助科学家对复杂的视觉数据进行分类，Agile De Release为人与AI对齐提供了基础。
要深入了解我们范围界定和集群算法的基本积极学习技术和技术实施情况，阅读我们的论文，检查GitHub上的开源实施情况。 要开始，请参看我们互动的Colab/Jupyter演示，走过这篇文章中"健康食品"的例子。 从那里，你可以将它适应任何你作为主题专家的领域(例如，标注误导性点击)，或者用它来围绕你自己的主观品味和标准(例如，判定"启发艺术")来构建个性化的分类器。
我们感谢我们的合作者——魏巧、托马斯·登比、克里希纳穆尔蒂·维斯瓦纳坦、恩明·罗、卢春塔、图沙尔·多格拉、兰杰·克里希纳和阿里埃勒·福克斯曼——对这项工作和所附文件的宝贵见解和贡献。 通过Google Research和Google Ads之间的密切合作关系，这一研究得以实现。
×
❮
❯
×
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
November 4, 2026
Otilia Stretcu, Research Scientist, and Leijie Wang, Student Researcher, Google Research
Agile Deliberation addresses the challenges of building AI image classifiers for subjective categories by guiding users through an interactive deliberation workflow, from scoping sub-categories to resolving edge cases, using an LLM-driven optimization loop to automatically build an accurate, aligned classifier.
Machine learning models learn to recognize the world around us from examples labeled by humans. Underlying this process is an assumption that the person doing the labeling has a clear and stable definition of the category in mind before they begin. This assumption holds well when training a model on objective categories, like a stop sign or a golden retriever, where boundaries are clear and universally recognized. In practice, however, real-world applications often require vision systems to navigate the subjective "gray areas" of human experience. For example, consider training a vision classifier to identify "healthy food", curate "inspiring artwork”, or assist in content moderation. In such domains, category boundaries may not be universal or static, since they can shift with cultural context, personal philosophies, or evolving operational policies. Furthermore, people rarely start with a complete specification for subjective concepts, even in their own minds. Instead, they begin with an intuitive idea of a concept and only discover the concept’s true boundaries, exceptions, and nuances as they encounter real-world examples. Traditional training workflows don’t account for this concept evolution, forcing users to provide rigid labels to an AI system even when their own mental model is still shifting.
To bridge this gap, we introduce Agile Deliberation, a framework that treats the creation of subjective AI classifiers as an iterative conversation. Rather than requiring humans to act as static data labelers, Agile Deliberation turns the AI system into a collaborative partner that helps users "deliberate" and sharpen their own definitions. By resolving human ambiguity first, the framework systematically builds more effective, robust, and aligned AI systems. To support the wider research community, we have also open-sourced the underlying code and algorithms on GitHub.
A key differentiator of this work is its focus on the live user experience. Because standard, static AI benchmarks can’t capture subjective concepts, we bypassed simulated testing and evaluated Agile Deliberation across 18 live, 1.5-hour sessions with human participants. This allowed us to directly observe and measure how a person's intent dynamically evolves when paired with an intelligent system. We found that Agile Deliberation drastically reduced user fatigue and anxiety by replacing manual prompt engineering with direct feedback on borderline images, which resulted in more consistent, higher-performing AI models that outperform standard baselines.
Agile Deliberation splits the complex task of defining a subjective concept into two structured stages: concept scoping and concept iteration. Together, these two phases transform a fuzzy idea into a highly precise instructional blueprint for an AI model.
When confronted with a broad topic like "Healthy Food," human intuition naturally struggles to list every rule upfront. In the concept scoping stage, the system guides the user to break down their overarching concept into a structured hierarchy of sub-concepts and explicit exclusions.
For instance, instead of just using a single broad label, a user might decompose "Healthy Food" into positive sub-categories like "Salads," "Grain Bowls," and "Lean Proteins." Simultaneously, they might define explicit exclusions, such as "Processed Foods" or "Raw, Unprepared Ingredients." This creates an initial structured baseline that grounds the model's understanding.
With the initial structure established, the framework moves from static planning to active testing. Traditional machine learning assumes the human annotator is an "oracle" with a complete set of fully defined rules already in mind. Agile Deliberation upends this premise, focusing instead on collaborative refinement.
This stage draws inspiration from active learning, a method traditionally used to retrieve informative data points to maximize a machine's training efficiency. However, instead of selecting examples purely to minimize machine uncertainty as in active learning, Agile Deliberation uses vision-language models (VLMs) to actively mine and surface semantically borderline images that challenge the current text-based definitions.
For instance, the system might present an image of a fruit smoothie to challenge a "healthy diet" concept, forcing the user to decide: Is this a healthy beverage because it contains raw fruit, or is it an excluded processed food due to high sugar content?
Confronting these ambiguous edge cases drives users to reflect on their own hidden assumptions, clarify their true intent, and update the policy rules accordingly. This human feedback is then routed directly into an automatic prompt optimization loop, ensuring the VLM's classification boundaries shift in tandem with the user’s refined definitions.
Evaluating subjective visual classifiers presents a fundamental challenge: there is no static, objective ground truth to measure performance against and therefore, offline evaluation on pre-labeled datasets can’t measure how well the framework adapts as a person's mental model of a concept shifts during the deliberation process.
To capture this dynamic, we built a functional prototype and evaluated it through a live user study. We conducted 18 live user sessions, each lasting 1.5 hours, where participants interacted with our system and alternative baselines in a randomized order to prevent learning bias.
We chose two complex, highly subjective domains to test the limits of these systems:
- Healthy Food: A domain where personal health philosophies, dietary restrictions, and cultural backgrounds can produce highly diverse, subjective viewpoints.
- Pay-to-Play: A real-world content moderation challenge that requires capturing expert-level nuance and specific context. This involves identifying clickbait images in online entertainment or gaming that promise misleading financial or digital rewards.
We evaluated Agile Deliberation against three common distinct paradigms:
- Zero-shot classifiers: The standard "one-and-done" approach. Users write an initial prompt, but have no way to see edge cases or adapt when the model misinterprets the boundary.
- Modeling collaborator: An automated, top-down approach where an AI agent tries to decompose the concept on its own. While it generates criteria automatically, it removes the user from the loop, leaving no way to guide or correct its logic.
- Manual deliberation: The tedious, brute-force status quo process, where users examine data in spreadsheets, guess search queries to find anomalies, and rewrite raw prompts by hand.
The live user sessions revealed that Agile Deliberation yields higher-performing classifier models, while substantially improving the user experience.
We found that study participants who went through the structured deliberation process demonstrated significantly higher labeling consistency over time. By resolving boundary ambiguities early on, they built classifiers that substantially outperformed all standard baselines. These performance gains were most dramatic on the most challenging subjective images — the exact areas where traditional classifiers typically fail because they rely on default, generic AI assumptions rather than the user's explicit, nuanced intent.
Expert participants reported that the system acted as an effective diagnostic tool. By intentionally serving up confusing, borderline images, the AI system helped them uncover "blind spots" in their initial definitions — uncovering scenarios they hadn't anticipated until presented with ambiguous real-world examples.
A key finding of our live study was the cognitive toll of manual prompt engineering. Our study showed that users experienced pronounced cognitive fatigue when trying to predict search terms that might uncover edge cases during manual concept deliberation. Furthermore, they reported experiencing persistent anxiety that manually tweaking a text prompt to fix one mistake might unintentionally degrade performance for other parts of the classifier they had already fixed.
Agile Deliberation addresses this friction by handling the heavy computational lifting. Users don't need to be prompt engineers; they simply look at a curated gallery of borderline images and provide natural feedback. The system seamlessly absorbs their intent and updates the underlying model automatically.
As a result, participants using Agile Deliberation reported feeling significantly less stressed, less frustrated, and less mentally drained, with every participant expressing a preference for Agile Deliberation over the manual methods.
Ultimately, Agile Deliberation shifts the paradigm of human-in-the-loop machine learning from a one-way street — where the AI system simply learns from human labels — to a two-way collaboration where humans and AI refine concepts together.
As machine learning systems are increasingly deployed to handle complex, real-world societal tasks, ensuring that their boundaries reflect nuanced subjective concepts is essential. Whether empowering trust and safety teams to build safer content moderation filters, helping curators shape digital collections, or assisting scientists in categorizing complex visual data, Agile Deliberation provides a foundation for human-AI alignment.
To learn more about the underlying active learning techniques and the technical implementation of our scoping and clustering algorithms, read our paper and check out the open-source implementation on GitHub. To get started, see our interactive Colab/Jupyter demo to walk through the "Healthy Food" example featured in this post. From there, you can adapt it to any domain where you are a subject-matter expert (e.g., flagging misleading clickbait) or use it to build a personalized classifier around your own subjective taste and standards (e.g., curating "inspiring artwork”).
We would like to thank our collaborators — Wei Qiao, Thomas Denby, Krishnamurthy Viswanathan, Enming Luo, Chun-Ta Lu, Tushar Dogra, Ranjay Krishna, and Ariel Fuxman — for their valuable insights and contributions to this work and the accompanying paper. This research was made possible through a close partnership between Google Research and Google Ads.
×
❮
❯
×
Auto-fetched and translated by AI Atlas. All rights belong to the original author.