World Models: 3D Representation Approaches
Sources: Session notes, 2026-04-06; 2026-04-05; SF Robotics World Model Reading Club sessions 01–06, 2026-03 to 2026-05; Yann LeCun keynote notes, 2026-05; 3D/4D World Modeling Survey arXiv:2509.07996, 2025-09; Zhen et al. (UMass/HKUST/Harvard), arXiv:2504.20995, 2025-04; Lu et al. (GWM team), arXiv:2508.17600, ICCV 2025; Liu et al. (Stanford/TRI), arXiv:2507.01099, ICLR 2026; Karcini et al. (Motoniq.ai/Stanford/ETH/UCL), arXiv:2606.06556, 2026-06 Raw: ParticleFormer Architecture; 4D Video Generation; LeWM JEPA; [Reading Club 01](../../raw/clippings/130. Robotics World Model Reading Club 01—San Francisco 20260328 Explicit 3D Backbone, Unified WM.md); [Reading Club 02](../../raw/clippings/132. Robotics World Model Reading Club 02 JEPA Zoo, WAMs & Unified Latent Representation—SF0404.md); [Reading Club 04](../../raw/clippings/134. Robotics World Model Reading Club 04 Abstract of Physical World for Decision Making—SF0418.md); [Reading Club 06](../../raw/clippings/141. Robotics World Model Reading Club 06 Evo of Video World Models, Tongzhou (RhodaAI) —SF0502.md); [Yann LeCun](../../raw/clippings/139. Yann LeCun on the Path to Grounded Intelligence World Models Beyond Next-Token Prediction.md); 3D/4D WM Survey; World-VLA-Loop; TesserAct; GWM; Geometry-Aware 4D Video; Robots Need More Updated: 2026-08-12
Overview
Three distinct approaches have emerged for 3D/4D world models for robot manipulation: 3D point cloud dynamics (ParticleFormer, PointWorld), geometry-aware 4D video generation, and JEPA-style latent prediction (LeWM). Each prioritizes a different dimension — physical accuracy, generalization breadth, or view-point invariance — and none currently solves all three simultaneously.
The 2026 field is undergoing a conceptual shift from policy-centric (VLA: observation → action) to world-model-centric (WAM: latent world → future trajectories → actions). GR00T N2 is cited as the clearest current WAM exemplar. The missing piece: a unified latent interface aligning perception, geometry, physics, and action — these remain fragmented across separate pipelines.
The WAM Paradigm and Representation Crisis
From VLA to WAM (SF Robotics World Model Reading Club, 2026): the field is shifting from learning direct observation→action mappings (VLA) to learning World Action Models — systems that build a latent world model first, then derive trajectories and controllable actions from it.
GR00T N2 (NVIDIA, 7B params): the strongest current end-to-end WAM example — memory-efficient, running on Thor hardware.
The representation crisis: pixel-based representations are fundamentally inefficient for robotics — high redundancy, not geometry-aware, weakly grounded in physical interaction. Three emerging directions:
- Explicit 3D backbone (point clouds, meshes, object-centric representations with geometry-aware selective tracking)
- Unified latent field (D4RT-style: joint inference of depth, spatio-temporal correspondence, camera parameters → 3D point tracks in world coordinates; up to 300× speedup, enabling real-time applications)
- Decoupled world model: split latent into static (scene, objects, geometry, 3D) and dynamic (motion, interaction, forces) components — avoids entanglement between planning and grounding objectives
Current gap: no unified interface. The required chain pixel → geometry → semantics → action → physics is fragmented across separate subsystems in all current methods.
Physics gap vs. visual gap: the main sim2real failure is physical (discontinuous contact dynamics, deformable objects, non-differentiable friction) — not visual. Even photorealistic simulations fail because their contact models are brittle.
JEPA Family (2026)
The Joint Embedding Predictive Architecture (JEPA) family implements LeCun’s principle: predict future representations in latent space rather than reconstructing pixels directly. Let z_t be the current state representation; the predictor learns z_{t+1} = f_θ(z_t, a_t) — explicitly action-conditioned, turning JEPA into a true world model.
Key insight (LeCun 2026): good representations must discard unpredictable information (random leaf motion, illumination noise) while preserving variables that are predictive under action. Next-token prediction over continuous sensory streams is inefficient — the objective must focus on invariants and predictable subspaces.
V-JEPA 2 (Meta): two-stage pipeline — action-free pretraining on >1 million hours of internet video + 1 million images, then action-conditioned post-training on <62 hours of DROID robot trajectories. Enables zero-shot deployment on Franka arms for image-goal pick-and-place with no environment data, no task-specific training, and no reward. Hot take: web-scale video + minimal robot data suffices to unlock physical planning.
V-JEPA 2.1: advances over V-JEPA 2 with dense predictive loss (both visible and masked tokens contribute for explicit spatial/temporal grounding), deep self-supervision across multiple encoder layers, and multi-modal tokenizers.
Act-JEPA: integrates JEPA into imitation learning by operating in policy latent space — ~40% better world understanding and ~10% higher task success.
Causal-JEPA: object-centric — masks out an object in latent space and requires the model to predict its counterfactual state from other objects. No explicit causal graph; causal inductive bias emerges from the objective. A key step toward genuine causal world models.
ThinkJEPA: dual-temporal pathways — dense JEPA for fine-grained motion/interaction + VLM branch at larger temporal strides for long-horizon semantic guidance. Hierarchical pyramid representation extraction compensates for JEPA’s short-horizon limitations in hand manipulation.
LeJEPA / LeWM (see LeWM section below): the minimalist end — trains from raw pixels with only next-embedding prediction + SIGReg. The hot take: trajectory straightening degrades latent semantic richness; task progress estimation is a better objective than trajectory straightening.
Reconstruction vs. planning conflict: these objectives are inherently in tension. Reconstruction needs semantically rich latents for recoverability. Planning needs minimal sufficient dynamics representations. Practical resolution: do not force reconstruction during planning; use it only as a 2nd-stage diagnostic probe.
3D/4D World Modeling: Survey Taxonomy (arXiv:2509.07996)
Kong et al. 2025 survey (50 pages) provides the first formal taxonomy of native 3D/4D world modeling approaches (VideoGen, OccGen, LiDARGen paradigms):
Representation types:
- Video Streams: temporal sequences with geometric coherence
- Occupancy Grids: voxelized spaces indicating occupation status
- LiDAR Point Clouds: direct geometric measurements, robust to texture/lighting
- Neural Representations: NeRF, Gaussian Splatting
Conditioning signals:
- 𝒞_geo (geometric): camera poses, depth maps, HD maps
- 𝒞_act (action-based): trajectories, control commands, navigation goals
- 𝒞_sem (semantic): text prompts, scene graphs, object labels
Functional taxonomy (four types):
- Data Engines — diverse 3D/4D scene synthesis for augmentation
- Action Interpreters — forecast future states under action conditions
- Neural Simulators — closed-loop agent-environment interaction
- Scene Reconstructors — recover complete scenes from partial observations
World-VLA-Loop: Co-Evolving World Model and Policy
World-VLA-Loop (arXiv:2602.06508, NUS Show Lab): closes the loop between video world model training and VLA policy optimization via RL.
The core problem: existing video world models have poor action-following precision — they hallucinate successful outcomes even with erroneous actions, relying on visual priors over actual physical dynamics. This makes them unreliable as reward functions for RL.
The SANS dataset (Success and Near-Success Dataset): near-success trajectories that fail due to minor end-effector positioning inaccuracies. These are hard to distinguish from successes, forcing the world model to focus on fine-grained spatial dynamics. Curated across ManiSkill, LIBERO, and real-world settings.
Co-evolving cycle:
- Train state-aware video world model (joint reward + video supervision on diffusion latents) on SANS
- Run RL post-training of VLA policy within the virtual environment
- Feed VLA failure rollouts back to refine the world model → more robust action-following
- Repeat: the updated policy generates new failure modes; the world model learns to cover them
Result: VLA performance significantly improves with minimal physical interaction — eliminates thousands of physical rollouts needed for real-world RL.
Video World Models as Simulators vs. Policies (Reading Club 06)
Two major frameworks for video world models in robot control (Tongzhou Mu, Rhoda AI, 2026):
As learned simulators (GR00T N1 / DreamGen pipeline):
- Fine-tune video world model on human teleoperation data
- Prompt the model → retrieve relevant robot videos (RAG-style) → rollout to generate synthetic videos for new tasks
- Label pseudo-actions via Inverse Dynamics Model (IDM)
- Train visuomotor policy on pseudo-labeled data
Critical nuance: IDM is not the bottleneck — inverse mapping is easy. Forward world modeling (predicting what happens) is the hard problem. IDM generalizes across similar motions from different robots.
Direct Video-Action (DVA) model: reduces robot control to a real-time video generation problem. The policy generates the next video frame, then acts to achieve it.
Trade-off: simulators enable massive synthetic experience at low cost + safe simulation of rare edge cases. Weakness: open-loop rollout causes hallucination and error accumulation.
ParticleFormer: Transformer-Based Particle Dynamics
Problem addressed: GNN-based particle simulators require hand-tuning graph topology hyperparameters (TopK neighbor counts) per material type — no single setting generalizes across rigid, deformable, and granular materials. Transformers learn interaction structure implicitly from data.
Architecture: Point cloud input → Transformer (self-attention over particles) → predicted next-frame particle positions
Loss design — Chamfer Distance + Hausdorff Distance:
- CD measures overall shape fidelity (average distance to nearest neighbor)
- HD measures worst-case deviation (maximum distance to nearest neighbor)
- Combined: local contact accuracy + global shape preservation simultaneously
Planning integration: MPPI uses ParticleFormer as the world model — samples N random action sequences, rolls out each in simulation, selects weighted-best action. World model value: replaces expensive real-robot trials with virtual rollouts.
Perception fragility: The entire pipeline depends on GroundingDINO + SAM for object segmentation. Perception failure at this stage cascades and corrupts all downstream dynamics prediction.
Training mode: Per-scene training (not generalizable across environments out of the box). This is the key limitation relative to PointWorld.
PointWorld: Scaling 3D Point Flow World Models
Philosophy: Generalization breadth first. 3D point flow simultaneously represents scene state and robot action in metric 3D space — interpretable, cross-embodiment by design.
Scale: 2M trajectories, 1B parameters. In-the-wild training across diverse real environments.
Advantages over 2D pixel world models:
- Geometric inductive bias: objects don’t teleport, spatial relationships are explicit
- Cross-embodiment: point cloud representation is robot-agnostic
- Zero-shot deployment to novel environments
Deployment requirements: RGB-D cameras for point cloud reconstruction; accurate end-effector tracking for action labeling.
Geometry-Aware 4D Video Generation (ICLR 2026, arXiv:2507.01099)
Problem addressed: Imitation learning models trained on 2D pixels fail when camera angle changes. Solution: learn implicit 3D geometry rather than 2D patterns directly.
Core mechanism — Pointmaps: Each pixel stores (x, y, z) in world coordinates, bridging 2D video generation with 3D scene understanding (inspired by DUSt3R). Cross-view pointmap alignment provides geometric supervision: two camera views’ pointmaps must project to the same 3D coordinate frame.
Cross-attention between decoders: Decoder A’s Q queries Decoder B’s K/V, enabling views to share information during generation. Ablation: mIoU improves from 0.41 to 0.70 with cross-attention.
Action extraction pipeline: Generate future scene video → track gripper pose with FoundationPose → extract trajectory. Video generation learns physics; pose tracking learns geometry; combined, these are camera-agnostic by construction.
No camera calibration at inference: Geometric alignment during training makes the model implicitly geometry-aware; deployment does not require camera parameter registration.
Results: 64% average success rate vs. Dreamitate 9% vs. Diffusion Policy 12% on simulated manipulation tasks; generalizes effectively to unseen camera viewpoints.
Limitations:
- Inference speed: ~30 seconds per 10-step rollout — too slow for real-time use
- 64% is on relatively simple manipulation tasks; complex tasks untested
- Requires multi-view RGB-D capture hardware for training data
LeWM: JEPA with Statistical Regularization (arXiv:2603.19312)
Representation collapse in JEPA: The encoder pushes all inputs toward the same vector (prediction error → 0 but meaningless). Original JEPA uses EMA teacher networks as a heuristic workaround.
SIGReg solution: Forces the latent distribution toward a Gaussian. Physical meaning: ensures similar states have appropriate distances and prediction gradients are meaningful — MPC can navigate the latent space effectively. The latent space becomes a uniformly navigable map.
LeWM vs. PointWorld:
| Dimension | LeWM | PointWorld |
|---|---|---|
| Input | 2D RGB | 3D point cloud |
| Scale | 15M parameters | 1B parameters |
| Planning speed | 48× faster than DINO-WM | Standard |
| Interpretability | Black-box latent | Geometric, interpretable |
| Generalization | Per-task training | In-the-wild, cross-embodiment |
| Hardware requirement | Standard RGB cameras | RGB-D |
DINO-WM comparison: Pre-trained features avoid collapse without SIGReg, but 12,288-dimensional output (vs. LeWM’s 192) makes planning 48× slower; features are not tuned for the specific task.
Emergent physics: LeWM learns “objects don’t teleport” without any physical law supervision — emergent from pure prediction. Supports latent prediction as a framework for general physical understanding.
SIGReg boundary: Two-Room navigation (low-dimensional, simple state space) performs worse with SIGReg — Gaussian constraint is over-constraining in low-intrinsic-dimensionality environments. Best suited for high-dimensional complex manipulation.
TesserAct: RGB-DN 4D Embodied World Model (arXiv:2504.20995)
TesserAct (Zhen et al., UMass Amherst / HKUST / Harvard, April 2025): a 4D embodied world model that predicts dynamic 3D scene evolution under agent actions.
Key innovation — RGB-DN representation: instead of standard 2D pixels, trains on RGB + Depth + Normal (RGB-DN) videos. This encodes appearance, geometry, and surface orientation simultaneously — a practical middle ground between 2D video models (no geometry) and full 3D Gaussian approaches (expensive optimization).
Architecture: fine-tuned CogVideoX with separate modal projectors for each channel (RGB, depth, normals). Two novel loss functions — consistency and regularization — enforce temporal-spatial coherence across frames. Optical flow distinguishes static backgrounds from dynamic regions; depth maps are optimized using geometric constraints from normal maps via normal integration.
Training dataset: ~285k videos total — RLBench synthetic (80k with GT depth/normals) + RT1 Fractal (80k) + Bridge (25k) + SomethingSomethingV2 (100k) with estimated depth/normals.
Results:
- 4D scene prediction: lowest Chamfer distance on real and synthetic datasets; outperforms OpenSora, CogVideoX, 4D Point-E baselines
- Embodied planning (RLBench): 41–88% success across 9 manipulation tasks; outperforms image-only BC and video-based planning
- Novel view synthesis: CLIP Score 83.02; ~1 min inference vs. ~2 hours for competing methods
Limitations: captures only a single surface (no multi-view, no occluded geometry). Future work needed for multi-view RGB-DN generation.
Position in the taxonomy: sits between LeWM (pure 2D latent) and PointWorld (full 3D). Geometric grounding without explicit 3D optimization.
GWM: Gaussian World Models for Manipulation (arXiv:2508.17600, ICCV 2025)
GWM (Lu et al.): first world model to use 3D Gaussian Splatting as its internal scene representation, combined with a Diffusion Transformer (DiT) to predict how Gaussian primitives propagate under robot actions.
Architecture (three stages):
- World state encoding: Splatt3R converts images to 3D Gaussian sets; a 3D Gaussian VAE (cross-attention encoder + Transformer decoder) compresses variable-size Gaussian sets to fixed-length latents. Training: Chamfer loss + rendering loss.
- Diffusion dynamics: DiT learns
p(future state | history, actions)in latent space; action conditioning via cross-attention; EDM preconditioning for stability. - Policy integration: first denoising step features serve as the state encoder for imitation learning; MBPO framework for model-based RL.
Results:
| Setting | GWM | Baseline | Δ |
|---|---|---|---|
| Real-world Franka | 65% | 35% (DP) | +30% |
| Novel distractor | 60% | 0% | — |
| IL improvement (50 demos, 24 tasks) | +10.5% | — | vs. no world model |
| MBRL convergence | 2× faster | iVideoGPT | — |
Ablation: 3DGS alone: 18% → +3D VAE: 24% → full GWM: 65% — validates each component’s contribution.
Key insight: operating in 3D Gaussian space means predictions are geometrically grounded — enabling viewpoint-arbitrary rendering and, in principle, analytic collision queries on predicted future scenes. Novel distractor generalization (60% vs. 0%) demonstrates that geometric grounding produces substantially better out-of-distribution robustness than pixel-based baselines.
Grounding Gap: What VLAs and World Models Miss (arXiv:2606.06556)
“Robots Need More Than VLAs & World Models” (Karcini et al., Motoniq.ai / Stanford / IIT / ETH Zurich / TU Darmstadt / UCL, June 2026): a position paper reframing the robotics scaling problem as a grounding problem rather than a policy-learning problem.
Core thesis: “The central bottleneck is not only policy learning, but the absence of mechanisms that convert the world’s abundant unstructured behavioral data into grounded robot supervision.” VLAs depend on upstream mechanisms for data transformation that the field has not yet systematized.
Four missing components:
-
Physical Data Engines and Embodied Autolabeling: recovering hidden labels automatically from heterogeneous inputs (video, wearable sensors, tactile, logs) — structured supervision including object states, contacts, task phases, latent actions, rewards, and success indicators.
-
Task-Preserving Retargeting: mapping latent physical actions across embodiments while “preserving the intended effect on the world” rather than merely copying human kinematics. Maintaining goal-relevant physical outcomes despite morphological differences.
-
Physics-Grounded World Models: models that predict “not only what the world may look like after an action, but what physically changes and why” — reasoning about geometry, contact, force, constraints, and material properties, not only visual plausibility.
-
Self-Improving Deployment Loops: converting real-world execution traces into structured supervision through task-conditioned reward grounding, enabling systems to learn from successes, failures, and human corrections in closed-loop fashion.
Gap analysis: existing cross-embodiment datasets (Open X-Embodiment, DROID, RH20T) and foundation models (RT-1, RT-2, OpenVLA, π₀) still require experience in robot-compatible formats with explicit actions and task labels. Latent-action approaches (LAPA, UniVLA) and task-progress signals (PROGRESSOR, ReWiND) demonstrate that passive video contains behavioral information, but this signal remains “not yet robot supervision” without proper grounding.
Implication for world model research: the physics-grounded world model component (component 3) is the gap that most current world model work leaves open — current approaches (including PointWorld, TesserAct, World-VLA-Loop) optimize for visual plausibility or trajectory success, not for physics reasoning (contact, force, constraints, deformable dynamics).
Three-Way Comparison
| Approach | Primary Priority | Training Mode | Planning | Real-World Data Req. |
|---|---|---|---|---|
| ParticleFormer | Physical accuracy | Per-scene | MPPI rollout | RGB-D + segmentation |
| PointWorld | Generalization breadth | Large-scale cross-env | Action flow | RGB-D + tracking |
| 4D Video Gen | View-point invariance | Cross-camera training | Post-hoc extraction | Multi-view RGB-D |
| LeWM | Representation theory | Per-task | Latent MPC | RGB cameras |
| GWM | Geometric grounding | Multi-task (Meta-World, RoboCasa) | IL/MBRL state encoder | Multi-view RGB |
Fundamental tension: No current approach simultaneously achieves (1) physical accuracy, (2) generalization across scenes and embodiments, and (3) view-point robustness. Future direction: 3D point cloud input + JEPA-style latent prediction + SIGReg may combine geometric interpretability with training stability guarantees.
Open Questions
- Can ParticleFormer be made cross-scene by training on diverse synthetic data, or does per-scene fine-tuning remain necessary?
- Can LeWM’s latent planning scale to contact-rich manipulation with discontinuous dynamics (grasping transitions, slipping)?
- The 4D video generation approach is too slow for real-time use. Can distillation or amortized inference close the 30s gap?
- PointWorld uses RGB-D; most existing demonstration datasets are RGB-only. How much does depth matter for point cloud quality in practice?
- Can the decoupled world model (static + dynamic latent) be trained end-to-end, or does it require explicit architectural separation?
- V-JEPA 2 achieves zero-shot pick-and-place on Franka with <62h of robot data. Does this generalize to contact-rich manipulation or only spatial-displacement tasks?
- World-VLA-Loop’s co-evolving cycle improves RL efficiency — but does it generalize beyond LIBERO’s tabletop tasks to in-the-wild manipulation?
- If IDM generalizes across embodiments (picking up a pen on any robot), what is the actual bottleneck for video-based synthetic data for policy training?