Abstract
GWM proposes a novel world model architecture that reconstructs future robot scenes by inferring the propagation of 3D Gaussian primitives under the effect of robot actions. Combines a Diffusion Transformer (DiT) with a 3D variational autoencoder to enable scene-level prediction using Gaussian Splatting.
Venue: ICCV 2025 Project: gaussian-world-model.github.io
Core Method
Step 1 — World State Encoding:
- Feed-forward 3DGS via Splatt3R converts images to 3D Gaussian representations
- 3D Gaussian VAE: cross-attention encoder compresses variable-size Gaussian sets to fixed-length latent; Transformer decoder reconstructs
- Training: Chamfer loss + rendering loss
Step 2 — Diffusion-based Dynamics:
- DiT in latent space learns p(future state | history, actions)
- EDM preconditioning for training stability
- Action conditioning via cross-attention
Step 3 — Policy Integration:
- Imitation Learning: first denoising step features used as state encoder
- Model-based RL: integrated with MBPO framework
Results
| Setting | GWM | Baseline | Improvement |
|---|---|---|---|
| Meta-World FVD | 73.0 | 75.0 | Better prediction quality |
| Imitation (50 demos) | +10.5% | — | Avg across 24 tasks |
| Imitation (3k demos) | +7.6% | — | Avg across 24 tasks |
| MBRL convergence | 2× faster | iVideoGPT | — |
| Real-world (Franka) | 65% | 35% (DP) | +30% success |
| Novel distractor generalization | 60% | 0% | — |
Ablation: 3DGS alone: 18% → +3D VAE: 24% → full GWM: 65%
Evaluated on: Meta-World (50 tasks), RoboCasa (24 kitchen tasks), Franka FR3 real robot.
Limitations
- Relies on feed-forward Gaussian reconstruction quality (Splatt3R)
- Real-world validation limited to single task variant
- Computational cost not thoroughly benchmarked vs. baselines
- Unposed image inputs; potential camera calibration challenges
Key Insight
GWM is the first work to use 3D Gaussian Splatting as the internal representation of a world model for policy learning. Operating in 3D Gaussian space means the model’s predictions are geometrically grounded — enabling rendering from any viewpoint and, in principle, analytic collision queries on predicted future scenes.