3D and 4D World Modeling: A Survey

Abstract

This comprehensive survey addresses the rapidly expanding field of native 3D and 4D world modeling. Prior work largely emphasizes generative methods for 2D image and video data, overlooking approaches using occupancy grids and LiDAR point clouds. The survey establishes definitions, proposes a taxonomy across VideoGen, OccGen, and LiDARGen approaches, and systematizes datasets and evaluation metrics for these native 3D/4D modalities.

• Length: 50 pages, 10 figures, 14 tables
• Submitted: September 4, 2025; updated December 3, 2025 (v3)
• Categories: cs.CV, cs.RO
• GitHub: systematic literature repository included

Core Representations

Four primary 3D/4D representation types:

• Video Streams: Temporal sequences emphasizing geometric coherence and temporal consistency
• Occupancy Grids: Voxelized spaces where cells indicate occupation status
• LiDAR Point Clouds: Direct geometric measurements robust to texture, lighting, or weather variations
• Neural Representations: Implicit encodings including NeRF and Gaussian splatting

Conditioning Signals

Three categories guide generation:

• 𝒞_geo (Geometric): Camera poses, depth maps, HD maps
• 𝒞_act (Action-based): Trajectories, control commands, navigation goals
• 𝒞_sem (Semantic): Text prompts, scene graphs, object labels

Taxonomy: Four Functional Types

1. Data Engines — Diverse 3D/4D scene synthesis for augmentation and scenario creation
2. Action Interpreters — Forecast future states under action conditions
3. Neural Simulators — Closed-loop agent-environment interactions
4. Scene Reconstructors — Recovery of complete scenes from partial observations

VideoGen Methods

Notable approaches:

• MagicDrive series — BEV and geometry-conditioned driving scene generation
• DriveDreamer variants — Action-guided video prediction with corner case synthesis
• DriveArena — First closed-loop framework combining traffic synthesis with autoregressive generation
• StreetGaussian — NeRF/Gaussian splatting-based reconstruction

OccGen Methods (4D Occupancy)

• SSD, XCube — Latent diffusion models for 3D occupancy
• FF4D, OccWorld — 4D temporal occupancy forecasting
• WoVoGen — Hybrid occupancy + video generation
• OccLLaMA, OccSora — Large-model approaches with transformer architectures

Evaluation Framework

• Generation Quality (fidelity, diversity)
• Forecasting Quality (prediction accuracy)
• Planning-Centric Quality (downstream task performance)
• Reconstruction-Centric Quality (geometric accuracy)
• Downstream Evaluation (real-world success)

Applications

1. Autonomous Driving — simulation, data augmentation, planning
2. Robotics — interactive training and sim-to-real transfer
3. Video Games & XR — immersive environment generation
4. Digital Twins — large-scale environment modeling

Key Insight from Authors

Native 3D/4D signals “encode metric geometry, visibility, and motion in the coordinates where physics acts,” providing safety-critical advantages over 2D projections. This is the fundamental argument for using 3D/4D reconstruction rather than latent video models for safety evaluation.

Open Challenges

• Standardized benchmarking across modalities
• Quality in long-horizon generation
• Physical fidelity and controllability
• Real-time computational efficiency
• Cross-modal generation coherence