本文由 AI 分析生成
建立時間： 2026-05-30 來源： https://x.com/junfanzhu98/status/2050834699275383008
Summary
Research review talk by Tongzhou Mu (Rhoda AI) at Reading Club Session 06, providing the most systematic taxonomy of video world models in robotics to date. Video models serve as simulator (data synthesis, inference-time planning, policy evaluation) or as policy (joint video+action generation, feature extraction, open/closed-loop generation + IDM translation). The talk introduces the DVA (Direct Video-Action) closed-loop framework and challenges the assumption that video model capability is the bottleneck — the actual bottleneck is grounding generated predictions to physical actions.
Tongzhou Mu 提出迄今最系統的機器人視頻世界模型分類法，分為用作模擬器和用作策略兩大類。核心洞察是視頻模型能力並非瓶頸，真正的瓶頸是如何將預測的未來幀轉化為可執行的物理動作。
Key Points
- Video-as-simulator: data synthesis (GR00T/DreamGen pipeline: finetune→rollout→IDM label→train policy), inference-time planning (action-conditioned model + candidate scoring), policy evaluation (Veo Robotics, offline trajectory scoring)
- Video-as-policy: 4 paradigms — joint video+action diffusion, feature extraction for action head, open-loop generation+IDM, closed-loop generation+IDM (DVA)
- IDM (Inverse Dynamics Model) is NOT the bottleneck — forward world modeling is the hard problem; IDM generalizes across similar motions across embodiments
- DVA: closed-loop video → IDM → actions, with KV cache and real-observation grounding at each step to prevent hallucination accumulation
- RL warning: learned simulators are NOT ground truth — RL will exploit model bias and hack the simulator
- Latent space dilemma: latent is efficient but may discard task-relevant info; pixel-space preserves everything but is expensive; “we don’t know the correct latent space”
Insights
The philosophical takeaway from Tongzhou: if a system cannot do something, it is usually NOT a video-model problem — it is a translation/constraint/IDM problem. This inverts the common intuition that better video generation = better robot. The deeper point is that video models and policies are now separate concerns: video models propose futures, and action selection emerges via scoring/translation. The analogy to “Robotics is no longer about learning actions — it is about selecting actions from predicted futures” marks a fundamental conceptual shift in how the field should frame the problem.
Connections
Raw Excerpt
Video world models have evolved into a unified paradigm that can act as simulator, policy, or value function. The bottleneck has shifted from modeling the world to grounding decisions in it.