新智元报道
AI已经能听、能看、能回答。
但面对持续变化的对话，它如何一边回应，一边保持理解与推理？
当你让AI分析屏幕上的图表，又突然补充「等一下，看右边那条曲线」，系统需要停止旧回答、更新目标，并检查后台计算是否仍然有效。
实时交互的难点，就在这些瞬间。
伊利诺伊大学厄巴纳—香槟分校、卡内基梅隆大学、哥伦比亚大学等机构的研究人员最新提出的综述提出了连接全模态智能与交互能力的统一概念框架，系统梳理从流式表征、模型架构、训练方法到实时部署与评测的技术链路，并进一步总结智能与交互之间的关键权衡及未来研究方向。
论文地址：https://doi.org/10.13140/RG.2.2.20245.69602
项目地址：https://github.com/ShenYifanS/Awesome-Interactive-Omni
图 1｜综述共总结了349篇工作，提出统一概念框架，将全模态智能与交互能力作为两个相互关联的组成部分，串联表示、架构、训练、部署与评测。来源：论文 Figure 1。
Omni AI（全模态智能）追求对文本、图像、音频和视频等多种模态的统一理解、推理与生成。但具备多模态能力，并不意味着具备自然交互能力。
综述将实时交互式全模态智能划分为两个相互关联的组成部分：
- Omni Intelligence（全模态智能）： 关注跨模态理解、推理与生成。
- Omni Interactivity（全模态交互）： 关注回应时机、轮次管理、打断处理和动态调整。
论文进一步总结了三个关键特征：流式输入、持续演进的内部状态，以及在线响应控制。这意味着模型需要持续接收信息、更新判断，并动态决定何时回应、等待或修订输出。
图2｜全双工交互允许同时听说，并需要处理附和、语音重叠与打断。「原生」进一步强调，时间对齐、增量处理和交互控制被纳入模型架构与训练本身。单纯降低回答延迟，还不足以覆盖这些要求。来源：论文 Figure 2。
在统一概念框架下，综述归纳了两条主要技术路径。
一是基于现有模型进行适配。通过流式处理、模态模块、后训练和外部编排，为预训练语言模型或全模态模型增加实时交互能力。优势是复用已有能力，挑战在于延迟、状态同步与打断控制。
二是构建原生交互模型。将连续多模态处理、时间结构和交互控制融入模型架构与训练，实现更紧密的感知与交互协同，但对训练数据和能力保留提出了更高要求。
实时交互要求模型准确判断何时回应、等待，以及如何处理中途打断。
围绕这些问题，综述系统梳理了响应时机控制、轮次管理与打断恢复机制，分析了现有方法如何结合语音、视觉、语义和任务状态，实现动态交互决策。
在训练层面，文章总结了如何利用停顿、附和、语音重叠、纠正与沉默等交互信号，让模型不仅学习回答什么，也学习什么时候回答。
针对打断后的交互恢复，综述进一步归纳了六个关键环节：
- 检测（Detect）： 识别用户是否真正发起打断，区分有效指令与背景声音。
- 让出话语权（Yield）： 及时停止或降低当前语音输出。
- 回退（Rollback）： 对齐用户实际听到的内容。
- 取消（Cancel）： 中止过时的任务。
- 修订（Revise）： 根据新指令更新意图和任务。
- 恢复（Resume）： 基于更新后的状态继续交互。
面对复杂推理和工具调用，模型过早回答可能导致信息不足，而等待计算完成又会影响交互体验。
综述将这一问题概括为智能与交互之间的权衡（Intelligence–Interactivity Trade-off）：训练阶段关注交互适配对原有能力的影响，运行阶段则需要平衡响应速度与答案质量。
图 3｜智能与交互之间的权衡：训练阶段关注能力保留，运行阶段关注响应时机与任务完成质量。来源：论文 Figure 14。
为应对这一权衡，综述进一步讨论了多时间尺度交互智能（Multi-Rate Interactive Intelligence），通过协调快速响应与较慢的推理、检索和工具调用，兼顾交互实时性与复杂任务处理能力。
一种实现思路是前后台协同，前台负责持续感知和快速响应，后台异步执行复杂推理与工具任务，并通过状态同步避免使用过时的计算结果。
图 4｜前台维持低延迟交互，后台异步执行检索、工具调用、推理和记忆操作。来源：论文 Figure 6。
针对现有评测难以同时衡量回答质量与交互行为的问题，综述从延迟、时机质量、内容质量、鲁棒性、长会话质量和修复能力六个维度系统整理了现有评测方法，强调需要在可比较的交互场景与资源预算下，同时衡量智能水平与交互表现。
综述进一步总结了四个值得关注的未来方向：
- 多时间尺度协同： 协调不同时间尺度下的感知、推理与交互。
- 持续状态与长期记忆： 维护长时间交互中的任务状态与记忆。
- 主动与社会性交互： 判断主动回应时机，理解多人交互中的角色与权限。
- 开放环境中的安全与效率： 兼顾安全、计算成本与系统可靠性。
从「能听、能看、能说」，到持续理解、适时回应并动态调整行动，是实时交互式全模态智能下一阶段的重要研究方向。
参考资料：
https://doi.org/10.13140/RG.2.2.20245.69602
编辑：LRST
不喜欢
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
New Ji-won.
AI can hear, see, answer.
But in the face of a constantly changing dialogue, how can it respond while maintaining understanding and reasoning?
When you get AI to analyze the charts on the screen and suddenly add "Wait a minute, look at the curve on the right," the system needs to stop old answers, update targets and check if the back-office calculations are still valid.
The hard points of real-time interaction are at these moments.
The most recent overview presented by researchers at the University of Illinois, Erbana, the Champagne Branch, Carnegie Mellon University, Columbia University, etc., provides a unified conceptual framework for connecting all-modular intelligence and interactive capabilities, systematically combing the technological links from current manifestations, modelling structures, training methods to real-time deployment and evaluation, and further summarizing the key trade-offs between intelligence and interaction and the direction of future research.
Address: https://doi.org/10.13140/RG.2.2.2024456902
Project address: https://github.com/ShenYifans/Awesome-Interactive-Omni
Figure 1 summarizes 349 work and proposes a unified conceptual framework, with full-modular intelligence and interactive capabilities as two interrelated components, with symmetry, architecture, training, deployment and evaluation. Source: Figure 1.
Omni AI pursues a uniform understanding, reasoning and generation of a variety of modes, such as text, images, audio and video. However, having a multi-modular capability does not mean having a natural interactive capability.
The synthesis divides real-time interactive whole-state intelligence into two interrelated components:
- Omni Intelligence: Focus on cross-modular understanding, reasoning and generation.
- Omni Interactivity (total modular interaction): Focus on response timing, rotational management, interruption and dynamic adjustments.
The paper further summarized three key features: flow input, the internal state of continuous evolution and online response controls. This means that the model needs to receive information on an ongoing basis, update judgement and dynamically decide when to respond, wait or revise the output.
Figure 2 shows that dual-work interaction allows simultaneous hearing and needs to be addressed with adhesion, voice overlap and interruption. It is further emphasized that time alignment, incremental processing and interactive control are integrated into model structures and training itself. The mere reduction of the delay in response is not sufficient to cover these requirements. Source: Figure 2.
Within the framework of a unified conceptual framework, two main technical pathways were summarized.
One is adaptation based on existing models. Increase real-time interactive capacity for pre-trained language models or full-modular models through fluid processing, modular modules, post-training and external layout. The advantage is that reuse is already capable, and the challenge is to delay, synchronize and break controls.
And the second is to build a primary interactive model. The integration of continuous multi-module processing, time structure and interactive controls into model structures and training to achieve closer perception and cross-fertilization, has increased the demand for training data and capacity retention.
The real-time interactive requirement model accurately determines when to respond, when to wait and how to deal with interruptions.
With these issues in mind, the synthesis system has streamlined response timing control, rotational management and disruption of recovery mechanisms and analysed how existing methods combine voice, vision, semantics and mission status to achieve dynamic interactive decision-making.
At the training level, the articles summarize how to use the interactive signals of standstills, adsorption, speech overlaps, correction and silence so that models learn not only what to answer, but also when.
In response to the interactive recovery following the interruption, the synthesis further summarizes six key elements:
- Test: Identify if the user actually initiates the interruption and differentiates between valid commands and background sounds.
- Release voice: Stop or lower current voice output in time.
- rollback (Rollback): Match what the user actually heard.
- Cancel: Abort outdated missions.
- Revision (Revise): Update intent and tasks in accordance with the new directive.
- Resume: An updated state continues to interact.
In the face of complex reasoning and tool call, premature responses to the model may result in insufficient information, while waiting for the calculations to be completed will affect the interactive experience.
The synthesis summarizes this issue as a trade-off between intelligence-interactivity Trade-off: while the training phase focuses on the impact of cross-adaptation on original capabilities, the operational phase requires balancing response speed with the quality of answers.
Figure 3 Trade-offs between intelligence and interaction: training phases focus on retention of competencies and operational phases on timing of response and quality of mission completion. Source: Figure 14.
To address this trade-off, the synthesis further discussed multi-temporal interactive intelligence (Multi-Rate Interactive Inteligence), which combines interactive real-time and complex mission processing capabilities by coordinating rapid response with slower reasoning, retrieval and tool call.
One way to achieve this is to work together back-to-back, with the front desk responsible for continuous perception and rapid response, moving backstage to perform complex reasoning and tool tasks and avoiding the use of outdated calculations through state synchronization.
Figure 4 Maintenance of low-delayed interaction at the front desk, backstage step execution of retrieval, tool call, reasoning and memory operations. Source: Figure 6.
In response to questions about the difficulty of measuring the quality of responses and the behaviour of interactions at the same time, the overview systematized the existing assessment methodology from the six dimensions of delay, timing quality, content quality, robustness, long-talk quality and resilience, emphasizing the need to measure the level of intelligence and the performance of interactions under comparable interactive scenarios and resource budgets.
The overview further summarizes four future directions of concern:
- Multi-timescale synergy: coordination of perception, reasoning and interaction at different time scales.
- Sustained status and long-term memory: Maintenance of mission status and memory during long interaction.
- Active and social interaction: To judge the timing of a proactive response and to understand the role of multiple interactions with permission.
- Safety and efficiency in an open environment: balancing security, costing and system reliability.
From "hearing, seeing, speaking" to sustained understanding, responsiveness and dynamic adaptation actions, the next phase of real-time interactive all-modular intelligence is an important direction for research.
References:
https://doi.org/10.13140/RG.2.2.20245.69602
Edit: LRST
I don't like it.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.