本文由 AI 分析生成
建立時間： 2026-05-03 來源： https://x.com/junfanzhu98/status/2050834699275383008
Summary
Tongzhou Mu (Rhoda AI) presented a taxonomy of how video world models are used in robotics: as learned simulators (data synthesis, inference-time planning, policy evaluation) and as direct policies (joint video-action generation, feature extraction, open-loop/closed-loop translation). The key insight is that video world models are a unified computational primitive that can simultaneously act as simulator, policy, and value function. The bottleneck has shifted from modeling the world to grounding decisions in predicted futures.
Tongzhou Mu 的報告將視頻世界模型的機器人應用分為兩大類：作為模擬器（數據合成、推理時規劃、策略評估）與作為策略本身（聯合生成、特徵萃取、開/閉環翻譯）。核心觀點：視頻世界模型是統一的計算原語，可同時充當模擬器、策略與價值函數。機器人學的瓶頸已從「如何建模世界」轉移到「如何將決策紮根於預測的未來」。
Key Points
- Four paradigms for video-as-policy: (1) joint video+action generation, (2) extract visual representation for separate action model, (3) open-loop generation + IDM translation, (4) closed-loop generation + IDM translation (DVA)
- IDM (Inverse Dynamics Model) is NOT the bottleneck — forward world modeling is the hard problem; IDM generalizes across embodiments
- Closed-loop DVA avoids hallucination by replacing generated frames with real observations each step + KV cache
- RL warning: learned video simulators are not ground truth — RL will find and exploit model bias
- Latent space is the unresolved tension: latent is efficient but may discard task-relevant information; pixel space preserves everything but is expensive
- Video models are versatile: simulator, on-policy data generator, value function, policy — no need to constrain to single use case
Insights
The failure diagnosis principle from DVA is significant: “if the system cannot do something, it is usually NOT a video-model problem — it is a translation/constraint/IDM problem.” This inverts the common assumption that video quality is the bottleneck. The observation that tactile sensing lacks internet-scale pretraining data (unlike video) explains why video dominates despite tactile being information-rich — it is a dataset scaling problem, not a sensing quality problem.
Connections
Raw Excerpt
“Robotics is no longer about learning actions — it is about selecting actions from predicted futures. The bottleneck has shifted from modeling the world to grounding decisions in it.”