Abstract
TesserAct presents an approach for learning 4D embodied world models that predict the dynamic evolution of 3D scenes over time in response to an embodied agent’s actions. The key innovation is training on “RGB-DN videos” (RGB + Depth + Normal data) rather than traditional 2D pixels, using a fine-tuned CogVideoX architecture with separate projectors for each modality.
Affiliations: UMass Amherst, HKUST, Harvard University Project: tesseractworld.github.io
Core Method
RGB-DN Representation: Captures appearance (RGB), geometry (depth), and surface orientation (normals) — more efficient than explicit 3D while encoding more geometric information than 2D video.
Architecture: Fine-tuned CogVideoX with separate modal projectors. Two novel loss functions (consistency + regularization) ensure temporal-spatial coherence across frames.
4D Reconstruction Pipeline: Optical flow distinguishes static backgrounds from dynamic regions; depth maps optimized using geometric constraints from normal maps via normal integration.
Dataset: ~285k videos total — RLBench synthetic (80k with GT depth/normals) + RT1 Fractal (80k), Bridge (25k), SomethingSomethingV2 (100k) with estimated depth/normals.
Results
4D Scene Prediction: Lowest Chamfer distance on both real and synthetic datasets; outperforms OpenSora, CogVideoX, 4D Point-E baselines.
Embodied Planning (RLBench): 41-88% success across 9 manipulation tasks; outperforms image-only BC and video-based planning.
Novel View Synthesis: CLIP Score 83.02; ~1 min inference vs. ~2 hours for competing methods.
Limitations
- Captures only a single surface of the world (no multi-view, no occluded geometry)
- Future work needed for multi-view RGB-DN generation
Key Insight
By adding depth and normal channels to video prediction, TesserAct gives the world model metric geometric grounding without requiring explicit 3D optimization — a practical middle ground between 2D video models and full 3D Gaussian approaches.