Quartz 5

141. Robotics World Model Reading Club 06: Evo of Video World Models, Tongzhou (RhodaAI) —SF0502

Properties2
descriptionInsights from Tongzhou Mu (Rhoda AI)’s Research Review Talk, May 2, 2026Reading Club 06: Evolution of Video World Models for Robotics@tongzh...
tagsclippings

10 min read

圖片

Insights from Tongzhou Mu (Rhoda AI)’s Research Review Talk, May 2, 2026

圖片

Reading Club 06: Evolution of Video World Models for Robotics

@tongzhou_mu (@RhodaAI)

圖片

Luma Page: https://luma.com/sdrd4zwr

Following the recent surge in world modeling for robotics, this session analyzes the two major frameworks grounding video models in robot control: their use as learned simulators and their integration as direct decision-making policies. We will discuss the trade-offs of these approaches, focusing on the challenge of grounding digital predictions in physical reality. Central to this discussion is a case study of the Direct Video-Action (DVA) model, which enables reliable robot control by reducing control to a problem of real-time video generation.

圖片

Pre-Readings

  • Causal Video Models Are Data-Efficient Robot Policy Learners (2026)
  • DreamGen: Unlocking Generalization in Robot Learning through Video World Models (2025)
  • V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025)
  • Evaluating Gemini Robotics Policies in a Veo World Simulator (2025)
  • DreamZero: World Action Models are Zero-shot Policies (2026)
  • Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations (2024)
  • Learning Universal Policies via Text-Guided Video Generation (2023)

圖片

Video world models have rapidly become one of the most discussed concepts in robotics. Spectrum of perceptions: from “Who needs a complex ‘World Model’? Just hard-code the arm to move to point (x,y,z). Simple.” to the excited researcher proclaiming “a profound, high-dimensional representation embodying the robot’s predictive understanding of affordances, object permanence, and temporal dynamics… truly revolutionary!” to the pragmatic engineer viewing it as “basically just a real-time, internalized physics engine for the robot’s mind to predict future states. Useful tool.”

1. Video Model as Simulator (vs. Traditional Isaac Sim / MuJoCo)

Video world models act as learned simulators that can be trained on data, generate their own training data, and serve downstream tasks—without manually programming every physical law. The unifying goal, whether as simulator or policy, is the same: provide robots with “physical common sense” by leveraging implicit physics embedded in billions of web videos.

圖片

1.1 Synthesizing Training Data for Policy Training

Papers: GR00T N1 [NVIDIA] / DreamGen [NVIDIA]

Exact Pipeline (retrieval-augmented generation):

  1. Finetune Video World Model on human teleoperation data.
  2. Prompt the model to solve the task → retrieve relevant robotic videos (RAG-style hybrid, not pure generation) → rollout to generate synthetic videos for new tasks (e.g., “Water the flowers,” “Pick up the tangerine”).
  3. Label Pseudo Actions automatically via Inverse Dynamics Model (IDM) → paired video-action data (a^1:H \hat{a}_{1:H} a^1:H, a^H:2H \hat{a}_{H:2H} a^H:2H, …).
  4. Visuomotor Policy Training on the pseudo-labeled neural trajectories.

Pros: Massive “synthetic experience” at low cost; safely simulates rare/dangerous edge cases.

Cons: Open-loop rollout leads to hallucination and error accumulation.

Critical nuance (speaker): IDM is not the bottleneck of the system (inverse mapping is easy; forward world modeling is the true hard problem). It is data-efficient—works with evaluation data, teleop, or even random data. Different robots’ data can be merged because IDM generalizes across similar motions (e.g., picking up a pen). Occlusion in dexterous manipulation is handled by engineering (wrist cameras); non-static backgrounds and external pushes do not break supervision if the dataset is diverse. If jointly diffuse together, occlusion shouldn’t be a problem? Not necessarily. Modern modalities are always input; you can attend to those tokens and the information received is the same, but when denoising we do not denoise at the same time—still an open question.

圖片

1.2 Inference-time Planning

Papers: V-JEPA 2 [Meta] / CLASP [23] / GPC [38]

Core mechanism: Train an action-conditioned video model. At test time:

  • Current observation → Encoder E_θ.
  • Candidate actions a1, at → probability maps P_ϕ (probabilistic modeling / likelihood scoring in latent space).
  • Forward rollout → compute loss L1 against goal image latent.
  • Select the action sequence with the highest likelihood / lowest error. Repeat the process (continuous replanning, akin to receding-horizon control).

Pros: Test-time scaling—sample more actions and longer horizons for better performance (like reasoning models).

Cons: Heavy compute; challenging to fit real-time control budgets.

Positioning: Policy evaluation (1.3) explicitly precedes this step: it prunes the action space and ranks candidates before expensive planning.

圖片

圖片

1.3 Policy Evaluation

Papers: Veo Robotics [Google DeepMind] / Ctrl-World [Stanford & Tsinghua] / IRASim [HKUST & ByteDance]

Role: Use the video world model as a learned simulator to evaluate policy performance across nominal / OOD / safety scenarios. Roll out policy inside the model, compare generated frames vs. predictions, and score trajectories (visualized with “Close the laptop lid” unsafe example marked ❌).

Key positioning (often missed):

  • Acts as action filter / value function proxy.
  • Used for pruning action space + ranking candidates before inference-time planning.
  • Not real-time rollout—pure offline evaluation.

Pros: Unlimited rollouts (traditional simulators cannot afford this at customer-deployment scale).

Cons: Less accurate than hand-engineered simulators.

Fine-tuning note: Models can be further fine-tuned on robot data.

圖片

圖片

2. Video Model as Policy (VLA backbones: language or video)

Four distinct paradigms have crystallized.

2.1 Generating Video and Action Jointly

Papers: DreamZero [NVIDIA] / GR1 [ByteDance] / GR2 [ByteDance] / PAD [Tsinghua]

Training: Joint Video-Action Flow Matching / Diffusion. Inputs: Video + Action + Proprioception + Language. Architecture: VAE Encoder → Joint Video-Action DiT (Causal DiT Blocks) with cross-attention + teacher forcing.

Inference: Closed-loop with KV Cache, autoregressive flow sampling, asynchronous real-world execution, and real-observation updates.

Pros: Leverages rich pretrained priors (object motion, human interaction) with minimal architecture change.

Cons / open question: Video models were not pretrained to predict actions; unclear if adding an action head fully utilizes the backbone.

圖片

圖片

2.2 Extract Visual Representation for Action Generation

Papers: VPP [22] / VPDD [21] / Video Policy [25] / DiT4DiT [26]

Key design: Video model only extracts features (Stage 1: CLIP → tokenized/stack + time-position embeddings → Video Former). A smaller separate action model (Stage 2: DiT Diffusion Policy) maps representations to actions—no full denoising on the giant backbone.

Partial-denoising spectrum (critical continuum):

  • No denoising → action model does everything (must make perfect decisions).
  • Partial denoising → control authority allocation between world model and policy head
  • Full denoising → video model dominates.

Pros: Dramatically faster real-time inference.

Cons: Unclear if full potential of video model is utilized (“no free lunch”).

Additional engineering reality: Loss for manipulation backbones is hard to design. Real-time robotics is special—you need to consider it carefully.

圖片

圖片

2.3 Open-loop Video Generation + Video-to-Action Translation

Papers: UniPi [MIT/Google/Berkeley] / This&That [Michigan] / TesserAct [UMass Amherst]

UniPi: Text → Tiling → Video Diffusion → Temporal Super-Resolution → Inverse Dynamics Model translates frames to actions (joint deltas, rotations, etc.).

System-level distinction (speaker): The system itself is the policy—no separate policy is trained. Video generation is slow; open-loop means the plan is fixed once generated.

Pros: Video model does exactly what it was pretrained for.

Cons: Open-loop → fixed plan, no reaction, frequent hallucinations.

圖片

圖片

2.4 Closed-loop Video Generation + Video-to-Action Translation

Papers: DVA [36] / LingBot-VA [35] / mimic-video [34]

DVA Pipeline: Video Context → Causal Video Model → Generated Video → Inverse Dynamics → Actions. Real observations replace generated frames every step + KV Cache + heavy inference optimization.

Pros: Avoids hallucination via constant grounding.

Cons: Requires causal modeling and nontrivial engineering for real-time.

Failure diagnosis (philosophical takeaway): If the system cannot do something, it is usually NOT a video-model problem—it is a translation / constraint / IDM problem. The video model is capable; the bottleneck is grounding actions to decisions. Video models propose futures; the decision of what action to take emerges via scoring / translation. Some videos lead to actions. Video model ≠ policy.

Actions for translation (task-dependent): Position space or other spaces depends on the task and how objects move in the video. For tasks that need feedback, the entire system must include teleop and be folded into the policy—more of a system-level design.

圖片

圖片

Q&A

  1. Closed-loop latency reduction: “For closed-loop video generation, we need to reduce inference time… partial denoising is one solution. Are there other methods?” Answer: Model-side: less precision, straighter diffusion denoising ODE trajectories. Engineering: kernel fusion and optimizations. People underestimate Infra—giant models can run fast.
  2. Most beneficial use case? (Policy evaluation / Data synthesizing / Policy guidance) Answer: The benefit of video models is so versatile. They can be simulator, on-policy, value function, or something else. No need to constrain to a single use case—they can do everything. They are a unified computational primitive.
  3. Systematic evaluation of world models? (Video quality feels too strict; policy scaffolding matters more.) Answer: Eval pretrained video model by building metrics for performance. Naive but effective: post-train the video model for the downstream task you care about.

圖片

Overarching Takeaways & Deployment Realities

  • Manually programming every physical law → leverage implicit physics in web-scale video → robots gain physical common sense.
  • Latent-space tension (unresolved): Most diffusion models operate in latent space (VAE, JEPA, etc.). Latent is efficient but may discard task-relevant information (latent choice defines what the robot can and cannot perceive → directly caps policy performance); pixel-space reconstruction preserves everything but is expensive. “We don’t know the correct latent space.”
  • RL warning: Learned simulators (e.g., UniSim) can initialize RL agents that already perform well, but RL is good at hacking your simulator—learned simulators are not ground truth and RL will exploit model bias. World model ≠ safe environment.
  • Combining video models with VLA: Generative models can generate assets for traditional simulators, but clever ways of integration are not yet widely known.
  • Distillation of two-stage pipelines: Distill 2-stage pipeline? Usually no, be cautious—it does not always preserve the same performance.
  • Perception engineering is a system-level bottleneck: Wrist cameras suffice for many close-up tasks (rich spatial info), but precision tasks require optimizing camera latency, resolution, and depth. Depth helps but is only one piece.
  • Data collection uncertainty: What data to collect has not converged yet; umi and egocentric data are promising.
  • Scaling & infra realities: 100 robots 1000 GPUs? Edge compute vs. cloud compute and transfer questions remain critical. Bottleneck remains infra, data collection, full-system optimization, model infra, and pipeline. Customer deployment is critical to many customers—deployment time must be squeezed.
  • Tactile / multi-modal: Easy to add (train video + inverse model), but lacks internet-scale pretraining data. Video dominates because of scale; “no evidence tactile is better—problem is how we scale up the dataset first.”
  • Speed & future trends: Video generation can be accelerated (diffusion distillation helps) but not yet 5× human speed. Auto-regressive diffusion Transformers (similar to StepFun) are promising. 3DGS is hard to pretrain like internet-scale video data.

圖片

In summary, video world models have evolved into a unified paradigm that can act as simulator, policy, or value function. By explicitly separating concerns (retrieval-augmented synthesis, probabilistic planning, action pruning, responsibility spectra in denoising, translation vs. video capability, and task-dependent action spaces), the field is delivering the physical common sense robots need. The 2026 timeline already shows production-viable components—the future of robotics is video-native. Robotics is no longer about learning actions—it is about selecting actions from predicted futures. The bottleneck has shifted from modeling the world to grounding decisions in it.

圖片

References

Join Discord Community​

https://discord.gg/WH7DrTHRXK

​​Follow Saturday Robotics​

https://x.com/saturdayrobotic

Follow YouTube

https://www.youtube.com/@saturdayrobotic

圖片

Future events

#reading-club-07-0509: Learning to Dream: World Models, Imagination, Path to Foundation Models for Control#reading

Past events

#reading-club-06-0502: Evolution of Video World Models for Robotics

#reading-club-05-0425: World Models for Physical Intelligence: From Predictive Brains to Embodied Robots#reading

#⁠reading-club-04-0418: Abstractions of the Physical World for Decision-Making

#⁠reading-club-03-0411: Robotic Policy Adaptation

#⁠reading-club-02-0404: JEPA Zoo

⁠reading-club-01-0328**

Graph View

Backlinks