Google希望视频AI理解人们在工作时需要什么
一个由Google领导的研究团队拥有推出 StreamVibe，一个视频agent，旨在自行决定何时说话。 测试表明，对于今天的AI模型来说，这尤其困难。
Google，斯坦福，牛津，和几所美国大学的研究人员正在寻找一个助手，看着你做饭或组装家具，并且只在需要的时候说话。 与StreamVibe合作，他们制作了一个agent的视频，旨在理解甚至模糊的请求，比如"如果遇到任何困难，请帮助我".
据报纸报道，现有的视频助手是为"水开始沸腾时告诉我"等清晰的指令而建造的，因此他们很少发现不明的意图。 他们也很难追踪已经做过的事情，这导致他们重复建议或回复太晚，当用户在中途更改请求时，助手们经常坚持原指令。
一个更关心的助手的三个组成部分
为了衡量这个问题，团队从超过183小时的日常生活和软件录音中汇编了大约4100个基准测试的例子。 Gemini 2.5 数据 Flash建议何时提供帮助和说什么，而4名人类评估人员必须一致赞同每个例子。 测试包括三个阶段，包括明确的指示，模糊的目标，并要求随着时间推移而改变。 一个例子是，一个用户拆卸一个玩具挖掘器，要求更新进度，然后在3分钟后切换到要求质量检查。
从那里，研究人员训练了StreamVibe-VL，这个模型显然基于Alibaba的Qwen3.5-9B，它决定每秒是安静还是说话。 第二个训练阶段使用强化学习使模型更加保留，以免不断中断。 一个单独的内存系统追踪那些等待处理的请求和哪些步骤是完整的，尽管只有模型才能决定是否说话和说什么。
所有测试过的模型在连续操作中失去对话的跟踪
研究者认为，StreamVibe-VL比其他模型提高10.7%的反应时间和质量，而内存系统的设计则是为了促进12.3%的主动互动，减少不必要的中断，等等。
StreamVibe-VL一般会以宽边优势比其基础模型和其他开源模型强，尽管Google的专有Gemini 3.8 Flash在几次测试中得分更高。 根据论文，专业训练不会损害一般的视频理解。
模型往往在只需要在特定时刻决定一个即时是否合适时才发出正确的呼声。 但是，当他们处理整个视频，并且必须依靠自己先前的反应时，所有被测试系统的性能都会因为错误而急剧下降，而根据系统本身的历史，答复的关键内容衡量标准下降了61.5%。 在超过5分钟的视频中，模特们很少选择正确的时机，发现在很长的一段时间里跟踪一个改变的指令特别困难。
在一张关于将一张图片转换成黑白的GIMP教程中，在纸上和项目页面上记录的对比模型JoyAI 称中，完成的图像虽然用户尚未打开，但已经保存下来，同时StreamVibe-VL给出了正确的时机的下一步，如"Colors"，然后是"Desaturate". 在用果酱制作卷面包的一步步指南中，其他模型只是描述了屏幕上发生的事情，如面包被扁，而StreamVibe-VL则建议先切除地壳。
研究者承认，现实世界的情况可能比他们的测试设置更为复杂，今后的工作将研究用户偏好和环境如何随着时间的推移相互影响。
Google一直致力于AI系统，目前积极分析视频。 9月，Google 推出 agent基于Gemini的视频分析，模型本身决定更仔细地检查哪些场景。 通过Gemini 4 Argon,Google最近还展示了AI如何在机器人视频中跟踪较长的动作序列和斑点错误。 StreamVibe的助手旨在了解随时发生的事情，并识别何时需要它的帮助。
在"主动奔驰基准测试"上，AI模型经常在视觉信息缺失时猜测或拒绝回答而不是提出具体问题。 一个单独的视频理解基准测试揭示了空间和时间推理的重大弱点。 而对于自主的AI agents来说，对话本身会损害性能，正如Microsoft研究者所显示的那样。
AI News Without the Hype – 由人类治愈
订阅THE DECODER免费阅读，每周AI通讯，我们独家的"AI雷达"前沿报告每年6次，全面存档访问，访问我们的评论部分。
现在订阅
读取全照。
订阅无韵报道。
- 全面查阅THE DECODER的每篇文章
- 没有广告
- 加入评论和社区讨论
- 每周通过邮件转载AI新闻
- 6x/年："AI雷达"——对AI最重要的话题进行深度潜水。
- 每日AI新闻，总是最新的
- 我们完整的十年档案
- 由一个在AI工作10年以上的团队负责
订阅到 The Decoder
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Google wants video AI to understand what people need as they work
A research team led by Google has introduced StreamVibe, a video agent designed to decide for itself when to speak up. The tests show above all how hard that remains for today's AI models.
Researchers from Google, Stanford, Oxford, and several U.S. universities are working on an assistant that watches you cook or assemble furniture and speaks up only when needed. With StreamVibe, they've built a video agent designed to understand even vague requests such as "Please help me if I run into any difficulties."
According to the paper, existing video assistants are built for clear-cut commands like "Let me know when the water starts boiling," so they rarely pick up on unspoken intentions. They also have a hard time tracking what's already been done, which causes them to repeat advice or respond too late, and when users change their requests midway through, the assistants often stick to the original instructions.
Three building blocks for a more attentive assistant
To measure the problem, the team compiled a benchmark of about 4,100 examples from more than 183 hours of everyday life and software recordings. Gemini 2.5 Flash suggested when to offer help and what to say, while four human evaluators had to unanimously approve each example. The test covers three stages including clear instructions, vague goals, and requests that change over time. In one example, a user taking apart a toy excavator asks for progress updates and then switches to requesting a quality check after three minutes.
From there, the researchers trained StreamVibe-VL, a model apparently based on Alibaba's Qwen3.5-9B that decides every second whether to stay quiet or speak. A second training stage uses reinforcement learning to make the model more reserved so it doesn't constantly interrupt. A separate memory system tracks which requests are pending and which steps are complete, though the model alone decides whether to speak and what to say.
All tested models lose track of the conversation in continuous operation
According to the researchers, StreamVibe-VL improves response timing and quality by 10.7 percent over other models, while the memory system is designed to boost proactive interaction by another 12.3 percent and cut down on unnecessary interruptions, among other things.
StreamVibe-VL generally beats its base model and other open-source models by a wide margin, though Google's proprietary Gemini 3.8 Flash scores higher in several tests. According to the paper, the specialized training doesn't hurt general video comprehension.
Models often make the right call when they only have to decide at specific moments whether a prompt is appropriate. But when they process an entire video and have to build on their own earlier responses, performance across all tested systems plummets as errors compound, and a key content metric for the responses drops by up to 61.5 percent based on the system's own history. In videos longer than five minutes, the models rarely pick the right moment and find it especially hard to track a changed instruction over a long stretch.
In a GIMP tutorial on converting a photo to black-and-white documented in the paper and on the project page, the comparison model JoyAI says the finished image has already been saved even though the user hasn't opened it yet, while StreamVibe-VL gives the next step at the right moment, such as "Colors" and then "Desaturate." In a step-by-step guide for making rolled toast with jam, other models just describe what's happening on screen, such as the bread being flattened, while StreamVibe-VL advises cutting off the crust first.
The researchers acknowledge that real-world situations are likely more complex than their test setup, and future work will examine how user preferences and the environment influence each other over time.
Google has been working on AI systems that actively analyze videos for a while now. In September, Google introduced agent-based video analysis for Gemini, where the model itself decides which scenes to examine more closely. With Gemini 4 Argon, Google also recently showed how AI can track longer action sequences in robot videos and spot errors. StreamVibe's assistant is designed to understand what's happening at any moment and recognize when its help is actually needed.
On the ProactiveBench benchmark, AI models often guess or refuse to answer rather than ask specific questions when visual information is missing. A separate video comprehension benchmark revealed major weaknesses in spatial and temporal reasoning. And for autonomous AI agents, the conversation itself can hurt performance, as Microsoft researchers have shown.
AI News Without the Hype – Curated by Humans
Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.
Subscribe now
Read on for the full picture.
Subscribe for hype-free coverage.
- Full access to every article on THE DECODER
- No ads
- Join the comments and community discussions
- A weekly AI news recap via mail
- 6x/year: "AI Radar" — deep dives on the AI topics that matter most
- Daily AI news, always up to date
- Our full ten-year archive
- Covered by a team with 10+ years in AI
Subscribe to The Decoder
Auto-fetched and translated by AI Atlas. All rights belong to the original author.