Abstract
Understanding and predicting the dynamics of the physical world is fundamental to enabling robots to plan and act effectively. We propose a 4D video generation model that enforces multi-view 3D consistency of generated videos by supervising the model with cross-view pointmap alignment during training. This enables the model to generate spatio-temporally aligned future video sequences from novel viewpoints given a single RGB-D image per view, without relying on camera poses as input. The generated 4D videos can be used to recover robot end-effector trajectories using an off-the-shelf 6DoF pose tracker, yielding robot manipulation policies that generalize well to novel camera viewpoints.
Introduction
The paper addresses two critical challenges in video generation for robotics: maintaining temporal coherence and preserving 3D geometric consistency across camera views. Traditional pixel-based models excel at short-term motion but create artifacts like flickering or object deformation. 3D-aware approaches enforce geometric constraints but struggle with realistic multi-object scenarios.
The authors propose unifying temporal modeling with geometric consistency through cross-view pointmap alignment inspired by DUSt3R. This approach leverages pretrained video diffusion models (Stable Video Diffusion) while enforcing spatial alignment through supervised pointmap prediction, enabling robust generalization to novel viewpoints.
Published at ICLR 2026. Project: robot4dgen.github.io
Method
Video Generation Backbone: Built on Stable Video Diffusion (SVD), encoding historical frames into latent space, predicting future latents via diffusion, and decoding back to RGB sequences.
Geometry-Consistent Supervision: The model predicts 3D pointmaps (per-pixel 3D coordinates) from two camera views. One view generates pointmaps in its native frame; the second generates them projected into the reference view’s coordinate frame. Both are supervised with diffusion losses, enabling shared 3D scene representation.
Multi-View Cross-Attention: Two separate decoders with cross-attention layers enable information transfer between views, helping learn 3D correspondences.
Joint Optimization: Training combines RGB diffusion loss with pointmap-based 3D consistency loss (λ=1), with loss re-weighting emphasizing gripper regions critical for action extraction.
Robot Pose Estimation: Predicted RGB-D sequences feed into FoundationPose (off-the-shelf 6DoF pose tracker) to extract end-effector trajectories without camera calibration at test time.
Experiments
Tasks: Three simulated manipulation tasks (StoreCerealBoxUnderShelf, PutSpatulaOnTable, PlaceAppleFromBowlIntoBin) plus one real-world task. 16-20 demonstrations per task from diverse camera viewpoints.
Baselines: SVD variants, Dreamitate, Diffusion Policy, 4D Gaussian approaches.
Key Results:
- Multi-view cross-attention: mIoU increased from 0.41 → 0.70 on Task 1
- Average success rate: 64% vs Dreamitate 9% vs Diffusion Policy 12%
- Generalized effectively to unseen camera viewpoints
Limitations
- Requires multi-view RGB-D datasets with varying camera angles — straightforward in simulation, challenging in real-world due to hardware/calibration demands
- Inference speed ~30 seconds per 10 steps (slow for real-time deployment)
- Potential improvements via flow matching and autoregressive transformer approaches
Conclusion
Enforcing geometric consistency through cross-view pointmap alignment produces spatio-temporally consistent RGB-D sequences suitable for robotic manipulation. The approach enables flexible camera placement and eliminates runtime pose calibration, supporting practical robot deployment across novel viewing configurations.