arXiv: https://arxiv.org/abs/2601.03782 | cs.RO | 2025-01-07 Affiliations: Stanford University, NVIDIA
Abstract
PointWorld is a large pre-trained 3D world model that forecasts how physical environments respond to robotic actions. The system represents both state and action as 3D point flows, predicting per-pixel displacements in three-dimensional space from RGB-D images and low-level robot commands across diverse robot embodiments.
The team assembled a dataset spanning approximately 2 million trajectories and 500 hours of interaction data, combining real-world robotic manipulation (DROID) with simulated environments (BEHAVIOR-1K). A single pre-trained model enables real-world Franka robots to perform diverse manipulation tasks without task-specific demonstrations or fine-tuning.
Key Contributions
- Novel unified representation: Uses 3D point flows to simultaneously represent environment state and robot actions in a shared modality, enabling embodiment-agnostic learning across different robotic systems.
- Large-scale annotated dataset: ~2M trajectories with high-quality 3D annotations via metric depth estimation, camera pose refinement, and dense point tracking.
- Empirical design principles: Rigorous investigation of scaling laws, architectural choices, learning objectives, and domain transfer for 3D dynamics modeling.
- Real-world deployment: Single pre-trained checkpoint successfully performs diverse manipulation tasks on physical hardware via model-predictive control.
Methodology
State and Action Representation
Environmental state: 3D points from calibrated RGB-D captures with DINOv3-derived feature embeddings. Robot actions: Temporal sequences of 3D points sampled from gripper surfaces via forward kinematics from URDF files.
Design philosophy: “contact and geometry rather than appearance” — naturally accommodates heterogeneous embodiments without embodiment-specific modifications.
Learning Framework
- Architecture: PointTransformerV3 backbone processing concatenated scene and robot point clouds
- Prediction formulation: Chunked prediction — forecasts 10 timesteps (1 second) in a single forward pass (superior to both autoregressive and teacher-forcing)
- Training objectives:
- Movement weighting: focuses supervision on moving points (typically only 1–5% of scene)
- Uncertainty regularization: per-point confidence scores for noisy real-world data
- Robust loss: Huber loss on 3D displacement residuals
Planning (Deployment)
Model-Predictive Path Integral (MPPI) control. Given RGB-D input and point-level goal specification, samples candidate action sequences, rolls them through PointWorld, and optimizes trajectories via accumulated task costs.
Dataset Construction
Real-World (DROID)
Three-stage annotation pipeline:
- Replace sensor depth with FoundationStereo stereo-estimated depth
- Refine camera poses by aligning robot depth with known robot meshes
- Apply CoTracker3 2D point tracking + lift to 3D
Result: ~200 hours of annotated data (60% of DROID), median calibration error 1.8 cm translation / 1.9° rotation.
Simulation (BEHAVIOR-1K)
~1100 hours of simulated bimanual/whole-body interaction; filtered to sequences with active robot-object contacts and measurable object motion.
Experimental Findings
Scaling Laws
Prediction error decreases approximately log-linearly with model capacity (50M → 1B parameters) and training data volume — consistent with scaling observations from language and vision modeling.
Architecture
PointTransformerV3 substantially outperforms GBND, PointNet variants, and sparse convolutions. Inference latency: ~100 ms.
Generalization
- In-domain: sub-centimeter accuracy on held-out test splits
- Cross-domain zero-shot: moderate transfer between real and simulated data; efficient finetuning recovers performance with only 5% of original training iterations
- Held-out environments: zero-shot generalization to unseen real-world scenes matches specialist models
Real-World Tasks (Franka)
- Rigid-body pushing
- Deformable object manipulation (cloth folding, pillow placement)
- Articulated object interaction (microwave, drawer)
- Tool use (sweeping with duster/broom)
Key Design Insights
Action representation: Gripper-only point flows (300–500 points per gripper) >> whole-body flows >> low-dimensional joint positions. Balances contact reasoning with efficiency and enables positive cross-embodiment transfer.
Prediction formulation: Chunked prediction >> autoregressive self-feeding and teacher-forcing. Reduces error drift and computational cost simultaneously.
Observability robustness: Training with randomly varying camera counts achieves best generalization across test-time observability levels.
Limitations
- Assumes static initial world states
- Requires explicit reward/cost specification for planning
- Struggles with fine-scale objects and calibration noise
- Cannot distinguish correlation from causation
- Omits photometric effects
- Assumes rigid-body robot structure
- Requires accurate end-effector tracking and control
- Lacks explicit physics priors