新智元报道
人类说一句「我不是狼」，多模态大模型既能看到表情，也能听见声音。但它真的知道这句话是真是假吗？
最近，来自东京大学的研究团队提出了多模态交互式欺骗评估基准 MIDA（Multimodal Interactive Deception Assessment），系统测试 GPT、Gemini、Claude、Qwen、DeepSeek 等开源和闭源模型在多人社交互动中的欺骗判断能力。
结果显示，即使是当前能力较强的多模态大模型，也很难可靠地区分真话与谎言。它们能够理解图像、声音和文本，却未必能弄清楚一个人在当时「知道什么、相信什么、又试图让别人相信什么」。
相关研究已被 EMNLP 2026 主会（Main Conference）接收。
论文链接： https://arxiv.org/abs/2511.16221
项目主页： https://ut-vision-social.github.io/
代码地址： https://github.com/kkkcx/Can-MLLMs-Read-the-Room
数据集地址： https://huggingface.co/datasets/Ukpkmkkk/MIDA
论文作者康彩新、黄逸飞、欧阳良扬、张铭方、刘锐聪、佐藤洋一均来自东京大学。团队关注多模态理解、视频分析与社会智能，本研究主要聚焦复杂多人互动中的欺骗判断与心智推理。
图1：MIDA整体框架。左侧为基于游戏记录、规则与人工标注的标签构建流程；中间是策略分类和欺骗判断任务；右侧为评测模型及SoCoT、DSEM两个推理模块。
过去的欺骗检测研究，大多围绕单人陈述或两人对话展开。但现实中的谎言通常不会孤立出现，它往往夹在追问、站队、辩护和群体压力中，并随着新信息不断变化。
更棘手的问题是，现实世界中的一句话究竟是不是谎言，通常不存在绝对可靠的标准答案。
研究团队选择「一夜终极狼人」作为实验环境，正是因为这款游戏同时解决了上述两个问题。
游戏中，每名玩家都有隐藏身份，夜间行动决定了玩家知道哪些信息，白天发言则体现其判断与策略。由于角色、行动和游戏规则均可追溯，研究者可以判断某句话是否违背了说话者当时掌握的信息。
因此，MIDA判断的并不是一个人「看起来像不像在撒谎」，而是他明明知道什么，却说了什么。
MIDA共包含2360条发言。其中，819条来自Ego4D中的40局线下游戏，1541条来自151段YouTube游戏视频。
两个子集形成了一个自然的难度梯度：Ego4D中的参与者以新手为主，YouTube视频中的玩家则更有经验，也更擅长使用复杂的欺骗策略。
每条样本都包含视频、音频、对话历史和游戏规则，并被标注为真、假或中性。这里的「中性」并不是指一句话态度中立，而是它不包含能够根据说话者当前知识状态进行核验的事实，例如提问、猜测、意见或假设。
为了获得可验证的标签，研究团队首先人工记录游戏中的夜间行动，再结合角色、规则和对话内容生成并校验每条发言的标签。
需要说明的是，角色、夜间行动和投票结果只用于构建可验证的真值标签，并不会在测试阶段提供给模型。模型仍然需要根据视频、音频和对话历史，自行推断说话者掌握的信息及其真实意图。因此，游戏规则提供的是一个可控、可验证的评测环境，而不是帮助模型直接获得答案的捷径。
除了判断真假，MIDA还要求模型识别发言使用的说服策略，包括身份声明、指控、辩护、证据、询问和行动号召。
图2：两个数据子集中不同说服策略的占比及TRUE、FALSE、NEUTRAL标签分布。身份声明和证据类发言承载了更多可核验信息，而询问和无明确策略的发言大多属于NEUTRAL。
研究覆盖GPT-4o、GPT-5、Gemini、Claude，以及Llama、InternVL、Qwen、DeepSeek等模型。
从总体准确率看，一些模型的表现似乎并不差。在Ego4D子集上，GPT-4o的总体准确率达到74.0%，Qwen2.5-VL-7B达到76.4%。
但MIDA中存在大量中性发言。模型只要频繁选择「中性」，就能得到一个看起来不错的总分。
当研究者只保留必须判断真假的样本时，结果发生了明显变化：GPT-4o的二分类准确率只有27.6%，GPT-4o-mini也只有39.4%。Qwen2.5-VL-7B虽然总体准确率最高，但其对「假话」的F1值为0。
换句话说，一个模型可能在大多数时候回答「无法判断」，由此获得较高的总体准确率，却依然不会真正识别谎言。
表1：不同模型在MIDA-Ego4D和MIDA-YouTube上的主要评测结果。总体准确率与仅考察TRUE、FALSE样本的二分类准确率之间存在明显落差。
最需要社会推理的发言，恰好也是模型表现最差的部分。
在「身份声明」类发言上，各款模型的平均准确率在两个子集上分别只有15.7%和25.3%；对于包含证据的发言，平均准确率也都不到40%。模型或许能够识别「这句话是在指控别人」，却不一定知道这项指控是否可能成立。
经验玩家还会进一步放大难度。YouTube玩家给出虚假证据的比例达到36.8%，接近新手玩家的3倍；身份谎言占比为51.6%，也远高于新手玩家的18.9%。
研究团队通过消融实验分析了不同信息的作用。
去掉此前的对话历史后，GPT-4o-mini在真假样本上的准确率从39.4%骤降至13.4%。
这说明，识别一句话属于提问、辩护还是指控，可能只需要观察局部文本；判断这句话是真是假，则必须记住谁说过什么、谁知道什么，以及不同玩家的说法能否相互印证。
视觉信息也不是越多越好。加入一帧画面后，模型的宏平均F1提高了2.5个百分点；增加到三帧后，增益反而缩小至2.1个百分点。
更多画面带来了更多表情和动作，也带来了更多噪声。模型虽然「看见」了这些线索，却未必知道哪些线索真正重要。
表2：对话历史与视觉帧数的消融实验。移除对话历史后，MIDA二分类准确率从39.4%降至13.4%；增加视觉帧数也没有带来持续增益。
针对上述问题，研究团队设计了社会思维链SoCoT和动态社会认知记忆DSEM两个模块。
图3：SoCoT将面部、身体和声音转化为结构化线索；DSEM则维护每位参与者随时间变化的知识、发言和关系状态。
SoCoT要求模型先描述面部表情、身体姿态和声音特征，再推测说话者的目标与心理状态，最后给出真假判断。
这一过程相当于把「他说话时声音发抖，所以可能有问题」这样的直觉，拆解成可以检查的推理步骤。
不过，非语言线索并不总能提高判断能力。以GPT-4o-mini为例，只分析面部线索时，模型总体准确率提高了8个百分点，真假二分类准确率却下降了7.1个百分点。
紧张、回避视线等看起来「可疑」的表现，并不等同于撒谎。模型即使成功识别出这些表现，也可能对它们作出错误解释。
DSEM采用了另一种思路：为每名玩家维护一块持续更新的「记忆板」，记录其角色声明、行动、见闻、已知事实、人物关系和事件时间线。
面对一条新发言，模型不再只分析当前画面，而是持续核对：这个人此刻应该知道什么？他的说法是否与此前信息冲突？其他玩家的发言能否为其提供支持？
实验中，DSEM带来的提升更加稳定。它使GPT-4o-mini的二分类准确率提高2.3个百分点、宏平均F1提高3.3个百分点；在Qwen2.5-VL-7B上，宏平均精确率提高7.6个百分点。
相比从某个瞬间的表情中寻找「破绽」，持续追踪每个人的知识与立场，似乎更接近人类实际使用的社会推理方式。
表3：SoCoT和DSEM对闭源、开源模型基线的影响。SoCoT对不同指标的影响并不一致，而DSEM在两类模型上均稳定提高了真假二分类准确率。
MIDA的意义并不局限于狼人杀。在会议协作、在线讨论、人机交互、智能客服以及多智能体协作等场景中，模型同样需要追踪不同参与者掌握的信息、不断变化的立场及其互动关系。
狼人杀为这类能力提供了一个可控且足够复杂的实验环境：参与者拥有不同的私有信息，会结盟、质疑、辩护，也会有意识地影响他人的判断。通过在这一环境中暴露模型的能力缺口，MIDA为更广泛的多人社会推理研究提供了可量化的起点。
作者将当前的多模态大模型概括为强大的「知识引擎」，而不是成熟的「社会智能体」。它们缺少的可能不只是更强的视觉编码器，而是对他人知识、信念和意图进行持续建模的能力，也就是通常所说的「心智理论」。
当模型离开答案明确的视觉问答，进入一个人人掌握不同信息、观点持续变化、甚至会主动影响他人判断的环境时，今天最强的多模态大模型，依然很难真正「读懂空气」。
参考资料：
https://arxiv.org/abs/2511.16221
编辑：LRST
不喜欢
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
New Ji-won.
Humans say, "I'm not a wolf," and multi-model LLM can see both the expression and the sound. But does it really know if that's true?
Recently, the research team from the University of Tokyo proposed a multimodular interactive deceptive assessment benchmark MIDA (Multimodal Interactive Deception Assessment), a system test of the ability of open and closed-source models such as GPT, Gemini, Claude, Qwen, DeepSeek to defraud people in social interactions.
The results show that even the current multi-modular LLM, which is highly capable, is difficult to establish reliable areas where truth and lies are divided. They can understand images, voices and texts, but they may not be able to figure out what a person "knows, believes, and tries to make others believe."
The study has been received by the Main Conference of EMNLP 2026.
Paper link: https://arxiv.org/abs/2511/16221
Project homepage: https://ut-vision-social.github.io/
Code address: https://github.com/kkkxx/Can-MLMs-Read-the-Room
Data set address: https://huggingface.co/datasets/Ukpkmkkk/MDA
The authors of these papers are Kosuke Kosaka, Yosemite, Ouyang Liang, Zhang Zhang, Liu Zhong and Yoichi Sato from Tokyo University. The team focused on multimodular understanding, video analysis and social intelligence, and the study focused mainly on deceptive judgement and mental reasoning in the interaction of complex people.
Figure 1: MIDA overall framework. The left side is the build process based on game records, rules and manually marked labels; the middle is the policy classification and deceptive judgement task; and the right side is the evaluation model and the SoCoT, DSEM, the reasoning module.
In the past, deceptive testing was mostly conducted around individual statements or two-person dialogue. In reality, however, lies often do not appear in isolation, often caught up in questioning, standing, defending and group pressure, and changing with new information.
The more difficult question is whether the word in the real world is a lie or not and there is usually no standard answer that is absolutely reliable.
The research team chose "The Ultimate werewolf One Night" as an experimental environment, precisely because the game solved both problems.
Every player has a hidden identity in the game, and night action determines what the player knows, while speaking during the day reflects his judgement and strategy. Since roles, actions and rules of the game can be traced, researchers can judge whether a sentence is contrary to the information available to the speaker.
Therefore, MDA does not judge a person as "not like he's lying" but what he says when he knows what.
MIDA contains 2360 statements. Of these, 819 are offline games from Ego4D and 1541 from 151 YouTube video games.
Two subsets create a natural difficulty gradient: the participants in Ego4D are dominated by beginner, while the players in YouTube video are more experienced and better equipped to use sophisticated deception tactics.
Each sample contains video, audio, conversation history and rules of the game and is marked as true, false or neutral. The term “neutral” here does not mean that a sentence is neutral, but rather that it does not contain facts that can be verified on the basis of the current state of knowledge of the speaker, such as questions, speculation, opinions or assumptions.
In order to obtain verifiable labels, the team first manually records night action in the game and then generates and validates the labels for each statement in combination with roles, rules and dialogue content.
It should be noted that role, night action and voting results are only used to construct verifiable true-value labels and are not available to models during the testing phase. The model still requires a self-inducement of the information held by the speaker and his or her true intentions based on the history of video, audio and conversation. Thus, the rules of the game provide a manageable, verifiable assessment environment rather than a shortcut to help the model obtain answers directly.
In addition to the determination of authenticity, MDA requires model identification of the persuasion strategies used in statements, including identification statements, charges, defence, evidence, questioning and calls for action.
Figure 2: The share of the two data sets with different persuasion strategies and the distribution of TRUE, FALSE, NEUTRAL labels. Statements of identity and evidence contain more verifiable information, while inquiries and statements with no clear strategy are mostly NEUTRAL.
The study covered GPT-4o, GPT-5, Gemini, Claude, and models like Llama, InternVL, Qwen, DeepSeek.
In terms of overall accuracy, some models appear to perform well. The overall accuracy rate of GPT-4o was 74.0 per cent and that of Qwen2.5-VL-7B was 76.4 per cent in the Ego4D subset.
However, there are numerous neutral statements in MDA. Models often choose "neutral" and get a total score that looks good.
When researchers retained only sample, which had to be judged as true or false, the results changed markedly: the GPT-4o II classification accuracy rate was only 27.6 per cent and the GPT-4o-mini was 39.4 per cent. Qwen2.5-VL-7B, although the highest overall accuracy rate, has an F1 value of 0 for "fake."
In other words, a model may answer “unjudgeable” at most times, resulting in a high overall accuracy rate, but still does not really recognize the lie.
Table 1: The main findings of the different models on MIDA-Ego4D and MIDA-Youtube. There is a clear discrepancy between the overall accuracy rate and the accuracy rate of the two categories of TRUE, FALSEZ sample only.
The statements most in need of social reasoning coincided with the worst performance of the model.
The average accuracy rate of each section's model is only 15.7 per cent and 25.3 per cent, respectively, in the “declaration of identity” category, and less than 40 per cent for statements containing evidence. The model may be able to identify "this word is accusing others" but may not be aware of the possibility of such an accusation.
Empirical players also make it more difficult. The proportion of false evidence given by YouTube players reached 36.8 per cent, nearly three times that of beginner players; identity lies accounted for 51.6 per cent, well above 18.9 per cent of beginner players.
The team analysed the role of different information through digestion experiments.
After removing the previous history of dialogue, GPT-4o-mini's accuracy rate on genuine sample dropped sharply from 39.4 per cent to 13.4 per cent.
This suggests that the identification of a sentence as a question, a defence or an accusation may require only observation of a local text; the judgement is true, and it must be borne in mind who said what, who knows what and whether the statements of different players are mutually corroborating.
The more visual information the better. With the inclusion of a frame, the macro average F1 of the model increased by 2.5 percentage points; when it increased to three frames, the gain narrowed to 2.1 percentage points.
More images bring more expression and action and more noise. The model "sees" these clues, but does not necessarily know which ones really matter.
Table 2: Excerpts of digestion of dialogue history and visual frames. After removing the history of the dialogue, the MIDA II classification was reduced from 39.4 per cent to 13.4 per cent; the increase in the number of visual frames did not bring about sustained gains.
In response to the above, the research team designed two modules, SoCoT, and DSEM, the social think chain.
Figure 3: SOCoT transforms facial, body and sound into a structured thread; DSEM maintains each participant ' s knowledge, speech and relationship over time.
SoCoT requires that the model first describe facial expressions, body postures and sound features, then speculate on the target and psychological state of the speaker, and finally give a false judgement.
This process amounts to an intuitive perception of "his voice shivering while he is talking, so there may be problems" and is broken down into a step of reasoning that can be examined.
However, non-linguistic leads do not always improve judgement. In the case of GPT-4o-mini, the overall accuracy of the model increased by 8 percentage points, while the accuracy of the true and false classification decreased by 7.1 percentage points.
The appearance of "suspicious" behaviour, such as tension and circumvention, does not amount to lying. Even if the models are successful in identifying these manifestations, they can be misinterpreted.
DSEM uses another approach: to maintain a continuously updated "remember board" for each player, documenting his/her role statements, actions, perceptions, known facts, personal relationships and incident time lines.
In the face of a new statement, the model is no longer merely an analysis of the current picture, but an ongoing check: what should this person know at this time? Does his statement conflict with previous information? Could other players be supported in their statements?
In the experiment, the upgrades brought by DSEM were more stable. It increases the accuracy of the second GPT-4o-mini classification by 2.3 percentage points, the macro average F1 by 3.3 percentage points, and the macro average accuracy by 7.6 percentage points in Qwen2.5-VL-7B.
The constant tracking of everyone's knowledge and position seems to be closer to the social reasoning that humans actually use than looking for a "breakdown" from an instant look.
Table 3: Impact of SOCOT and DSEM on closed source, open source model baselines. The impact of SOCOT on different indicators is not uniform, and DSEM has stabilized on both models to improve the accuracy of the true and false classification.
MIDA is not limited to werewolves. In settings such as conference collaboration, online discussions, human interaction, intelligent passenger service and multi-AI agent collaboration, models also need to track information, changing positions and interactions among different participants.
The werewolf killings provide a manageable and sufficiently complex experimental environment for such capabilities: participants possess different private information, form alliances, challenge, defend and consciously influence the judgement of others. By exposing the capability gap in models in this environment, MIDA provides a starting point for a broader multi-person social reasoning study that can be quantization.
The author summed up the current multimodern LLM as a powerful "knowledge engine" rather than a mature "social AI agent". The possibility that they lack is not just a stronger visual encoder, but rather the ability to continuously model the knowledge, beliefs and intentions of others, commonly referred to as "theory of mind".
When models leave clear-cut visual questions and answers and enter an environment where one person has different information, where views change and can even influence others' judgement, today's strongest multi-modular LLM is still difficult to truly “read the air”.
References:
https://arxiv.org/abs/2511.16221
Edit: LRST
I don't like it.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.