2024年8月，王鹏伟正式加入智源研究院，开始着手搭建具身模型研究中心。而在他之前，智源这一板块的负责人叫王鹤。
与王鹤纯正的机器视觉和机器人学的背景不同，王鹏伟履历里的“模型味”要重得多。
在微软亚洲研究院（MSRA）攻读博士期间，他的研究方向就是知识图谱与自然语言处理。2018年毕业后，王鹏伟加入阿里达摩院，主要负责预训练工作，一干就是五年。其后，他又加入快手大模型团队，负责从零到一搭建快手的多模态大模型。直到2024年2月，王鹏伟才正式接触到具身智能。
“第一次听到具身智能这个词，感觉很陌生，当时整个行业里听过的人也很少，我还特地查了查是哪几个字。”王鹏伟回忆说。
很快，他就意识到多模态与大模型在数字世界已经非常成熟，要进入物理环境，具身智能就是首要问题。而具身大脑这个概念，可能会成为继大语言模型和多模态大模型之后，下一波技术浪潮的高点。
当时的行业里并没有具身大脑的概念，Embodied brain 、Robo brain这些名词也没有被业界广泛认同。但王鹏伟坚定地认为：具身大脑就是具身智能行业的核心。“直到现在，制约具身智能行业的关键依然还是那个问题——大脑不够泛化。而我们自始至终一直在解决的问题就是模型的泛化问题。”
2026年智源大会，在加入智源即将满2年的时间点上，王鹏伟终于带着他的悟界·RoboBrain Orca世界模型正式走上了牌桌。而我们也借此机会，与王鹏伟就具身大脑与世界模型等话题，进行了一番深入的对话。
▎以下为与王鹏伟的对话全文，略有删减：
创投家：咱们先聊聊“具身大脑不够泛化”这个问题？
王鹏伟：我们一直认为，到目前为止，制约具身智能行业发展的核心问题就是大脑还不够智能，也就是不够泛化。
本质上行业也还没有找到一种高效消费数据的范式。哪种模型架构消费数据的效率最高？消费哪种数据的效率最高？这些问题的答案都还没有找到。
所以，我们所做的所有事情其实都是为了解决泛化的问题。而且我们一直坚持在通用场景去解决泛化，而不是去某一个场景里去做过拟合，去刷榜。
创
创投家：您在演讲里还提到两个概念“有意识学习”和“无意识学习”，这两种学习模式是否就是定义Orca“像人类一样学习”的关键？
王鹏伟：对。我们回头看人到底是怎么学习这个世界的？
我们认为婴儿在学会说话之前的学习过程，是通过连续的观察感知整个世界的变化，这个学习的过程里没有人明确告诉他这就是物理的规律。这就是无意识学习。而随着人类长大，掌握了语言能力，就可以从大量被浓缩成语言的物理规律中学习，这就是有意识地学习。而Orca也是follow这两种范式去逐步建模的。
无意识的学习就是learning from video，和目前大多数具身大脑一样，给定纯视频的当前观测，让模型在Latent层面学会状态转移。这里没有显性的语言指导、没有显性的物理规律，Orca完全纯模拟无意识的学习，去学习最自然的自然规律。
而有意识的学习，其实是在干预条件下学习状态的转移，同时“由因推果”和“由果溯因”，由当下状态去预测之前和之后的状态。相比无意识学习中的相邻帧预测，有意识学习更关注事件级、语义化、稀疏的状态转移。
而目前有意识学习范式是极度缺乏训练数据集的。我们需要去构建大量包含action和action引发的状态转移的这种关键event数据集。
创投家：VLA目前的泛化效果到底怎样呢？
王鹏伟：可以这么说，目前VLA路线在泛化能力上，暂时还没有看到特别明确的突破路径。
VLA的本质是用多模态大模型的思路去生成action，这种数据映射并不是非常高效。
其实从一些限定场景的任务成功率来看，VLA的数据利用率已然是最高了。只不过VLA对于泛化来说，基本没有起到任何的作用。
如果拿世界模型去和VLA去对比，在某些任务的数据利用效率和执行成功率上，世界模型是有可能比不过VLA的。但世界模型其实是在探索一种新的解决泛化问题的思路，而不是去追求某一个任务的成功率。
创投家：所以VLA只能用来解决特定场景下的任务，而世界模型才能解决通用泛化问题？
王鹏伟：其实VLA和世界模型的关系会让我常常想起两个案例。
一个是Bert（全称Bidirectional Encoder Representations from Transformers，是Google在2018年提出的预训练语言模型）。当时的Bert，基本上在所有领域都可以和专家模型去较量一下。可能有个别领域会弱于专家模型，但Bert的价值在于提供了一个统一的标准，只需要在垂直领域再优化优化，它就可以打败专家模型；
另一个就是ChatGPT没有大火之前，它一直在尝试用同样的一套建模范式去解决所有问题，而不是去追求在具体某一个任务上击败专家模型。所以ChatGPT刚刚问世的时候，最受益的任务，是那些长尾的，那些专家模型没有覆盖到的场景。
现在如果把VLA看作专家模型，世界模型就是当时的Bert和ChatGPT。
创投家：那您是如何看待VLA和世界模型的终局的呢？
王鹏伟：我觉得他们俩一定会并存一段时间，但最后哪种范式会留下来，其实不太好说。但我会更看好世界模型，因为它其实是在解决更本质的问题。
而未来，世界模型也不是说要把VLA完全替代掉。
有一些场景和任务，比如工业场景里的非柔性分拣这类任务，其实不需要泛化，使用低成本的数据映射就可以搞定，VLA就非常适用。而世界模型更适合用来解决对泛化要求更高的场景任务。
创投家：您在智源大会的演讲中曾提到当下各种世界模型路线的一些问题，您方便再展开说说吗？
王鹏伟：对，我们把目前世界模型分为四类，分别是以语言、像素、三维结构和视觉表征为中心的世界模型路线。
以语言为中心的世界模型，比如大语言模型，本质上也是一种世界模型。这种模型理解的世界是用语言描述的世界，输入的信息只有语言，而缺失了视觉、听觉、触觉等必要的信息，这种对世界的理解是片面的。
以像素为中心的世界模型，比如各类视频生成类模型。它的问题在于生成视频的时候需要还原出每一个像素。但其实人类在执行动作或者预测状态时，并不需要把每一个像素点都考虑得那么清晰，因为大部分都是无用的。
以三维结构为中心的世界模型，包括3D重建以及李飞飞团队的World Labs Marble模型，不过模型重建3D空间不等于理解世界，几何结构也不代表物理状态。
以视觉表征为中心的世界模型，比如JEPA，其实还是learning from video，其本质还是在学习数据分布。没有人能够告诉你，这个表征背后代表的物理规律到底是什么。
以上四种世界模型，其实问题都在于只是focus在单点、单个模态上去做建模。这也是我提到的世界模型的第一个关键问题——只学到了片面的世界。
创投家：您在演讲里提到的第二个问题是目前的世界模型都是“被动任务驱动”？
王鹏伟：对，目前具身智能的所有数据采集和任务学习，都需要人工去布置，模型被动去执行、被动去收集任务、被动生成数据，这非常不利于物理AGI的实现。
如何让具身智能的学习过程变成主动式？让它自己可以去环境里去做探索，目前行业里最多的声音就是通过规模化部署，把成百上千台机器人部署到各种不同的场景中去数采。
但整个行业目前还没有大规模验证这一思路，原因并不是没有那么多机器人，而是目前的具身大脑能力太差了，探索的结果往往是失败的数据占据了大多数。而如果用人工干预来提升准确性，又会极大降低探索效率，难以scaling。
创投家：那Orca的解决方案是什么呢？
王鹏伟：这个问题的本质还是具身大脑不够聪明，它的行为物理很差。
Orca解决这个问题的核心就在于我们采用了一种独特地建模范式：Next State Prediction：从多模态世界信号中学习到当前状态，并“由因推果”和“由果溯因”地去预测前后状态演化。核心理念是从大语言模型的语义理解，到视频生成的视觉预测，再到动作生成的动作预测，最终推进到真实世界建模。
也就是说，Orca会预测多种未来的情况，并决定要选哪一个，而并非被动式地去执行一个片段。
创投家：您在演讲里还提到两个概念“有意识学习”和“无意识学习”，这两种学习模式是否就是定义Orca“像人类一样学习”的关键？
王鹏伟：对。我们回头看人到底是怎么学习这个世界的？
我们认为婴儿在学会说话之前的学习过程，是通过连续的观察感知整个世界的变化，这个学习的过程里没有人明确告诉他这就是物理的规律。这就是无意识学习。而随着人类长大，掌握了语言能力，就可以从大量被浓缩成语言的物理规律中学习，这就是有意识地学习。而Orca也是follow这两种范式去逐步建模的。
无意识的学习就是learning from video，和目前大多数视觉表征学习一样，给定纯视频的当前观测，让模型在Latent层面学会状态转移。这里没有显性的语言指导、没有显性的物理规律，Orca完全纯模拟无意识的学习，去学习最自然的自然规律。
而有意识的学习，其实是在干预条件下学习状态的转移，同时“由因推果”和“由果溯因”，由当下状态去预测之前和之后的状态。相比无意识学习中的相邻帧预测，有意识学习更关注事件级、语义化、稀疏的状态转移。
而目前有意识学习范式是极度缺乏训练数据集的。我们需要去构建大量包含action和action引发的状态转移的这种关键event数据集。
创投家：针对这两种学习模式，Orca需要怎样的数据集呢？
王鹏伟：我们需要的数据分为三部分：
第一部分，用了大量互联网Web data，大概有12.5万个小时。数据构成由第一视角、第三视角、机器人视角以及其他视频数据构成。
第二部分，是为了支撑有意识的学习，我们用了大量专家模型做标注，得到一些事件级状态转移的数据，这些数据大概是130m，去支撑有意识学习。
第三部分，我们想让整个世界模型在语言层面有一些输出，所以构建了VQA数据，包含因果VQA和通用VQA数据。
创投家：您特别强调了Orca的预训练没有用到任何一条action数据，这件事的意义是什么？
王鹏伟：这个是让我特别surprise的地方，但也符合我们当时预期的。
action数据以现在的增长的速率，想要达到GPT3的数据体量，要上百年。而如果说我们的模型架构，数据的消费方式都已经固定，那泛化问题的本质就变成了：到底什么样的数据才能够真正Scaling起来？
而现在整个web data中其实存在着大量的交互数据，模型理应从这些更广泛的交互数据中去学习。另一方面，训练一个world foundation model必然要消费大量的数据，那点action数据一定会被海量的其他数据给淹没掉。
所以我们一开始就定义：世界模型的泛化一定是用action free的data，也就是不用任何一条action数据。
创投家：目前行业里关于世界模型能力的评测是否已经有共识？
王鹏伟：其实，现在行业里其实针对状态转移的世界模型的benchmark还没有，对world model的评测，尤其是状态转移的world model的评测还是一片蓝海。
因为VGM的大部分评测都是去衡量模型预测生成的像素与目标图片或者视频的一致性。用这种方式去评测状态转移的世界模型没有任何意义。包括对泛化性的测试，目前行业里的方法也还很不完善。
所以我们现在已经构建出来一个专门针对状态转移世界模型的评测集。会根据现有的 video 切出一段关键的event，并考察模型预测状态与真实未来状态的一致性。
创投家：您觉得具身大脑要实现在泛化性上0的突破，大概还需要多久？
王鹏伟：我觉得一年之内，一定会有一些方案实现0%的突破。但是要让泛化能力真的可用，必须得达到90%以上，这就是一条漫长的路径了。
而现在最大的问题就是，还没有一条路线能够看到突破0的可能性。这个是整个行业最焦虑的地方。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
In August 2024, Wang Peng Wei formally joined the Institute and began work on the development of a body model research centre. And before him, the head of the board is Wang Tsuruo.
Unlike Wang's pure machine vision and robotics, Wang Peng Wei's history has a much heavier "model" taste.
During his Ph.D. at Microsoft Asia Institute (MSRA), his research was directed towards knowledge mapping and the processing of natural languages. After graduating in 2018, Wang Peng Wei joined the Arida Mo compound and was mainly responsible for pre-training, which lasted five years. He then joined the fast-hand team LLM, which was responsible for the multi-modular LLM from zero to one. It was only in February 2024 that Wang Peng Wei was formally exposed to intelligence.
"It was strange to hear the word smart for the first time, and very few people had heard it in the whole industry, and I looked specifically into the words."
Soon, he realized that polymodulates and LLM were very mature in the digital world, and that being smart was the first thing to get into the physical environment. The concept of having a brain, on the other hand, is likely to be the high point of the next technological wave, following the mega-linguistic model and multimodern LLM.
There was no brain in the industry at the time, and the terms Embodied brain and Robo brain were not widely accepted by industry. But Wang Peng Wei firmly believes that having a brain is at the heart of a smart industry. “To now, the key to regulating the smart industry is still that problem — the brain is not sufficiently broad. And the problem that we've been dealing with all along is the generalization of models."
In 2026, Wang Peng Wei finally officially took his world model, Robobrain Orca, to the card table, at the point when he was two years old. And we took this opportunity to have an in-depth dialogue with Wang Peng Wei on topics like brain and world models.
The following is the full text of the dialogue with Wang Peng Wei, with a slight reduction:
So let's talk about "not having a brain full" first.
Wang Peng Wei: We have always believed that the central problem that has constrained the development of smart industries so far is that the brain is not smart enough, that is, not broad enough.
In essence, industry has yet to find a paradigm for efficient consumption data. What model structure consumption data are most efficient? Which data are the most efficient to consume? None of these questions have yet been answered.
So everything we do is actually about generalization. And we've been insisting on general scenes to solve generalizations, instead of doing some sort of drawings in one of these scenes.
Gene.
Founder: You also mentioned in your speech two concepts, “conscious learning” and “intentional learning”, which are key to defining Orca as “learning like human beings”.
Wang Peng Wai: Yes. How do we learn the world?
We believe that the learning process of babies before they learn to speak is to sense change throughout the world through continuous observation, in which no one clearly tells him that this is the law of physics. That's unconscious learning. And as human beings grow up and acquire language skills, they learn from the physical laws of a large number of condensed languages, that is, conscious learning. Orca is a model of the two paradigms that follow.
Ignorant learning is learning from viideo, which, like most of the brain now, gives current observations of pure video, allowing models to learn to shift at the Latent level. There is no explicit language guidance, no obvious physical patterns, and Orca is a pure simulation of unconscious learning to learn the most natural natural patterns.
Conscious learning, on the other hand, is a shift in the state of learning under the intervention conditions, with “causes push” and “retroactive causes”, from the current state to predict the state before and after. Conscious learning is more concerned with event-level, semantic, thin state shifts than with the prediction of neighbouring frameworks in unconscious learning.
At present, the awareness learning paradigm is extremely lacking in training data sets. We need to build a large number of these keyevent data sets that include action and action-induced state shifts.
Founder: What is the current generalization of VLA?
Wang Peng Wei: So far, the VLA route has not seen a particularly clear path of breakthrough in terms of its ability to be generalized.
The essence of VLA is to generate action using the idea of multimodular LLM, which is not very efficient.
In fact, VLA's data utilization rate is already the highest in terms of some limited scenarios success rate. Only VLA has little effect on generalization.
If the world model is compared to the VLA, it is possible that the world model will be better than VLA in some missions using data efficiency and implementing success rate. But the world model is actually exploring a new approach to generalization rather than pursuing a particular task, success rate.
Founder: So VLA can only be used to solve tasks in a given scenario, and the world model can solve the problem of universalization?
Wang Peng Wei: In fact, the relationship between VLA and the world model reminds me of two cases.
One is Bert, a pre-trained language model developed by Google in 2018. At the time, Bert, it was basically possible to compete with expert models in all fields. Individual areas may be weaker than expert models, but the value of Bert lies in providing a uniform standard that can defeat expert models only if it is optimized vertically;
The other is ChatGPT, which has been trying to solve all problems with the same set of models before the fire, rather than seeking to defeat the expert model on a specific mission. So when ChatGPT first came out, the most beneficial task was the long ones, those expert models didn't cover.
Now if VLA were to be seen as an expert model, the world model would be Bert and ChatGPT.
Founder: How do you see the end of VLA and the world model?
Wang Peng Wei: I think they're going to be together for a while, but it's hard to say what the last paradigm will be. But I'm going to look at the world model better, because it's actually solving more of a problem.
And in the future, the world model doesn't mean that VLA is completely replaced.
There's some scenes and tasks, like the inflexible sorting of industrial scenes. Such tasks, which do not need to be generalized, can be handled with low-cost data mapping and VLA is very applicable. And the world model is better suited to address a scenario that demands more generalization.
Founder: In your speech at the Genius Congress, you mentioned some of the problems of the current modeling routes of the world. Would you like to start again?
Wang Peng Wei: Yes, we divide the current model of the world into four categories, one based on language, pixels, three-dimensional structure and visual representation.
Language-centred world models, such as the Big Language model, are also essentially a world model. This model understands the world in terms of language, the only information that is entered is language and the missing information necessary for vision, hearing, touch, etc., is one-sided.
World models centred on pixels, such as video generation models. The problem with it is that every pixel needs to be restored when the video is generated. But in fact, humans do not need to consider every pixel point so clearly when they are doing their moves or predicting their state, because most of them are useless.
A world model centred on three-dimensional structures, including 3D reconstruction and the World Labs Marble model of the Li Fei Fei team, although the model ' s reconstruction of 3D space does not amount to an understanding of the world and geometry does not represent physical state.
World models centred on visual expression, such as JEPA, are actually learning from video, and it's essentially learning data distribution. No one can tell you what the physical law behind this expression is.
The problem with these four world models is that it's just that Focus does modeling on single points and individual models. This is also the first key issue of the world model that I have mentioned — the one-sided world only.
The second question you mentioned in your speech was that the current world model is “passive mission driven”?
Wang Peng Wei: Yes, all data collection and mission learning that are now smart require manual deployment, passive implementation of models, passive collection of tasks and passive generation of data, which is highly detrimental to the physical AGI.
How can smart learning processes become proactive? To allow itself to explore in the environment, the most powerful voice in the industry today is the deployment of hundreds of robots in a variety of scenarios.
But the whole industry has not yet tested the idea on a large scale, not because there are fewer robots, but because the current brain capacity is so poor that often the result of exploration is the majority of failed data. The use of manual interventions to improve accuracy, on the other hand, can significantly reduce the efficiency of exploration and make it difficult to scaling.
Founder: What is Orca's solution?
Wang Peng Wei: The nature of the problem is not smart enough for its brain, and its behavioral physics is poor.
At the heart of Orca's solution is that we have adopted a unique model: Next State Production: Learning from a multi-modular world signal to the current state, and predicting pre- and post-state evolution by “causing” and “retroactive”. The core idea is to understand the semantics of big language models, to visual predictions of video generation, to action predictions of action generation, and ultimately to model the real world.
In other words, Orca predicts a wide range of future scenarios and decides which one to choose rather than passively implement a segment.
Founder: You also mentioned in your speech two concepts, “conscious learning” and “intentional learning”, which are key to defining Orca as “learning like human beings”.
Wang Peng Wai: Yes. How do we learn the world?
We believe that the learning process of babies before they learn to speak is to sense change throughout the world through continuous observation, in which no one clearly tells him that this is the law of physics. That's unconscious learning. And as human beings grow up and acquire language skills, they learn from the physical laws of a large number of condensed languages, that is, conscious learning. Orca is a model of the two paradigms that follow.
Unintended learning is learning from visual, giving current observations of pure video, as is currently the case with most visual expression studies, to allow models to learn to shift at the Latent level. There is no explicit language guidance, no obvious physical patterns, and Orca is a pure simulation of unconscious learning to learn the most natural natural patterns.
Conscious learning, on the other hand, is a shift in the state of learning under the intervention conditions, with “causes push” and “retroactive causes”, from the current state to predict the state before and after. Conscious learning is more concerned with event-level, semantic, thin state shifts than with the prediction of neighbouring frameworks in unconscious learning.
At present, the awareness learning paradigm is extremely lacking in training data sets. We need to build a large number of these keyevent data sets that include action and action-induced state shifts.
Founder: What kind of data set does Orca need for these two learning models?
Wang Peng Wei: We need data in three parts:
In the first part, a large amount of Web data, approximately 125,000 hours, was used on the Internet. Data formation consists of first perspective, third perspective, robotic perspective and other video data.
In the second part, to support conscious learning, we used a large number of expert models to mark some of the event-level shift data, which are about 130 m to support conscious learning.
In the third part, we wanted the world model to have some outputs at the linguistic level, so we built VQA data, including cause and effect VQA and generic VQA data.
Founder: What's the point of this?
Wang Peng Wei: This is where I'm special, but also what we expected.
At the current rate of growth, action data will take hundreds of years to reach GPT3. And if our model structure, the way in which data are consumed, is fixed, then the essence of the broader question becomes: what kind of data is really Scaling?
Now there is a lot of interactive data in the whole webdata, and models should learn from these broader interactive data. On the other hand, training a world foundation model would necessarily consume a large amount of data, and that point of action would inevitably be flooded with other big data.
So we defined from the beginning: the world model must be generalized with action free data, that is, without any action data.
Founder: Are there consensus in the industry on the current assessment of world modelling capacity?
Wang Peng Wei: In fact, now that the industry is actually a Benchmark model of the world of state transfer, the assessment of the world model, especially the world model of state transition, is still a blue sea.
Because most VGM assessments measure the consistency of pixels generated by model predictions with target images or videos. It makes no sense to assess the world model for state transfer in this way. This includes extensive testing, and the current methodology in the industry is not perfect.
So we've now built an evaluation of the state transfer world model. A key event will be cut out based on existing video and the consistency of model predictions with real future status.
So, how long do you think it's gonna take for the brain to make a breakthrough in generic zero?
Wang Peng Wei: I think within a year there will be some breakthroughs of zero percent. But to be truly accessible, over 90 per cent must be achieved, which is a long path.
The biggest problem now is that there is no single route to see the possibility of a breakthrough. This is the most anxious place in the industry.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.