Robotics World Model Reading Club 01:
1. From Policies to World Models
Robotics is undergoing a fundamental shift: from learning policies to learning world models as the backbone of intelligence.
- VLA (Vision-Language-Action) learns direct mappings: observation → action.
- WAM (World Action Model) learns: latent world → future trajectories → controllable actions.
The paradigm shift is from reactive policies to controllable simulation. Gr00t N2 is essentially kind of a version of DreamDojo (world action model architecture). NVIDIA’s Gr00t (7B params with exceptionally strong memory utilization) running on Thor hardware is currently the strongest end-to-end example.
This shift is driven by one missing piece: there is no unified interface aligning perception, geometry, physics, and action under real-world constraints.
2. Representation Crisis: Explicit 3D vs Pixel Space
Pixel-based representations are fundamentally inefficient for robotics: high redundancy, not geometry-aware, and weakly grounded in interaction.
Explicit 3D as Backbone Emerging direction includes point clouds/meshes, object-centric & sub-object representations, and geometry-aware selective tracking. Instead of dense pixels, systems selectively track salient geometric primitives (contact points, affordances, articulated parts).
Point Flow Pipeline (Columbia-style) Representative pipeline:
- Detect objects / sub-objects
- Sample keypoints / surface points (shape of object)
- Track point flow across time
- Build object-centric dynamic graphs
Key challenge: Which points to track? How much downsampling? Solutions involve motion saliency, affordance-driven attention, and contact prediction. For geometric representations, you can pick and choose which objects/sub-objects to follow.
Current systems remain fragmented: few layers, different layer paths, no unified representation hierarchy.
Representation Density Tradeoff
3. 4D Reconstruction and Unified Latent Space
Systems such as D4RT (Dynamic 4D Reconstruction and Tracking) encode dynamic scenes into a shared latent space via a unified feedforward transformer architecture. It jointly infers depth, spatio-temporal correspondence, and camera parameters from video, producing outputs including point clouds, 3D point tracks (all pixels tracking in world coordinates), and full-scene reconstruction with temporal consistency. This delivers up to 300× speedup, enabling real-time applications in robotics.
Key insight: D4RT-style systems encode dynamic scenes as temporally consistent latent fields that jointly capture geometry, motion, and visibility, unifying perception, memory, and dynamics into one space.
Missing Piece: Unified Latent Interface Current systems are fragmented (vision latent / geometry latent / action latent / physics latent). They are not aligned.
Required Interface
pixel → geometry → semantics → action → physics
Robotics still lacks this shared latent interface.
4. Physics Gap: Why Sim2Real Fails
The main domain gap is not visual — it is physical. Core challenges include discontinuous contact dynamics, deformable objects with high/infinite DoF, and non-differentiable friction.
Engineering Reality Collision meshes remain brittle, deformable simulation is unreliable, and contact modeling is unstable. “Simulation-ready” data is not defined (e.g., one-object $12k still leave geometry fidelity, physics consistency, and interaction completeness undefined).
Emerging Solutions Learned physics proxies, hybrid pipelines (human data + semantic labels), and geometry decomposition.
Example: Google/NVIDIA work on convex decomposition via learned feature fields produces high-quality open-world decompositions of 3D shapes into unions of convex bodies. Convex decomposition serves as a bridge from geometric reconstruction to physics simulation by enabling efficient collision proxies, accelerating collision detection (~5×) in simulators.
This is exactly why “we still need World Models” for jointly modeling visual quality and physics.
5. Video Pretraining: Powerful but Incomplete
Human video is a massively underrated World Model dataset: rich visual priors, diverse behaviors, and scalable pretraining. But it fundamentally lacks counterfactual interaction — it tells us “what happened” but not “what would happen under different actions.”
Missing Modalities for Robotics Force, contact, depth, proprioception/tactile.
Systems Like Rhoda Internet-scale video pretraining + predictive modeling, but still exhibit clear gaps in physical grounding and real-world control.
Omni-Embodiment Data Gap: Full-body UMI should not be manipulation-only — it must extend to humanoid whole-body intelligence (locomotion + manipulation + gaze + coordination). Motion capture is lab-limited; the real bottleneck is exocentric in-the-wild data (how to collect it in factories remains unsolved). Cross-embodiment learning requires aligning coordinate systems, action spaces, and kinematic constraints across different morphologies.
6. Latency and Control: Bridging Inference and Physics
The real world operates at high frequency. Challenges: Transformer inference latency, long-horizon instability, and control loop mismatch.
Core Techniques
- Action chunking + action prediction — predict sequences to shrink the decision gap and reduce frequency.
- Latent actions.
- Decoupled Training / Inference — FastWAM retains video co-training during training but skips future prediction (rollout) at test time. FastWAM works because control only requires selecting a feasible trajectory, not fully modeling the future distribution at inference time, enabling massive latency reduction while preserving performance.
AutoGaze optimizes long-video understanding via KV-cache techniques. Long-range dependency stability requires engram-style dynamic quality clipping.
Key Insight: Inference ≠ control. Models must behave as control systems.
7. System Gap: Edge vs Cloud
Fundamental bottleneck: Cloud offers high compute but high latency (Gr00t 7B); Edge demands low latency with limited compute. NVIDIA Thor is the best hardware platform so far. Scaling alone does not transfer to deployment — robotics is not the LLM scaling path.
8. Data Bottleneck: No Robotics Internet
Data sources are fragmented (simulation, human video, teleoperation, factory logs from MineRobotics (Rivian spin-out), motion capture). Core problems: fragmented distributions, inconsistent modalities, no unified labeling, no quality metrics, and 100,000-year data gaps in robotics.
Industrial Reality Pick-and-place offers high reliability and throughput; factories often use fixed motion primitives (generalization often unnecessary). Real systems use a mix of both (fixed + learned).
Bitter Lesson: Cascading pipelines are easy, but big data will eventually crush hand-crafted engineering. Robotics lacks a true data flywheel. NVIDIA can provide infra + model but cannot own the full data chain (no data tentacles). Big companies cannot win robotics purely with capital advantage — even simple tasks like screwing different bolts vary across instances.
9. Embodiment: Toward Full-Body Intelligence
Current systems are manipulation-centric. Future intelligence requires locomotion, manipulation, gaze, and social coordination. We need humanoid full-body transfer and cross-embodiment alignment.
10. Sim2Real Pipeline (Practical View)
Human data → semantic labeling → geometry reconstruction → collision proxy generation (via convex decomposition) → simulation → real-world fine-tuning. Open problems remain in deformable objects, contact stability, long-horizon consistency, and jointly modeling visual quality + physics.
11. Model Architecture Evolution
Compression & Alignment
- VQVAE for discrete latent smoothing
- VL-JEPA for multimodal predictive alignment (good pretraining allows much more intelligence compressed at smaller scales)
Token Efficiency: Learned information/visual token pruning — let the model learn to compress itself.
Recursive Models: Repeated module reuse for higher intelligence density at smaller scale.
GRPO / Multi-path: Explore multiple trajectories and cherry-pick the meaningful path (SpatialBoost, ActionPlan in this context).
Additional systems: RoboForge (text-to-embodied motion), ColaVLA (latent reasoning for planning), V-Retrver (retrieval-augmented perception), AlphaDreams (Causal Diffusion Transformer for autonomous driving).
12. Infrastructure & Toward SLM
Robotics lacks native infra (existing stacks like SGLang/vLLM are not real-time and need to be much faster). The pipeline must unify vision + speech + text into one domain.
World models must compress into SLMs due to real-time control constraints, where inference latency directly impacts physical stability. Future: smaller models + vertical specialization + domain grounding. RoboOmni is still LLM-style; the open question is how to convert it into a true WAM for joint multi-scale modeling (chat while acting; speech for better emotion understanding).
13. Additional Systems & Directions
- OpenClaw-RL, Psi0
- VLM4VLA: high success rate, showing video DiT can transfer to WAM
- Cosmos-RL: the only current World Model finetuning
Next week’s theme: NVIDIA Sonic and SOMA — unifying parametric human body models into one framework + RGBD depth cameras → VGGT 3D modeling.
14. Alternative Compute Paradigms
Spiking Neural Networks (2017): event-driven, energy-efficient, biologically inspired. Simulate neural pulses via DSP. A fruit-fly mapping is already this intelligent; biological neurons are far richer — a potential edge-side control paradigm beyond Transformers.
15. Deployment Reality
Even highly engineered systems struggle: Boston Dynamics programmed robots remain difficult to deploy.
Industrial Example: Ambi Robotics suction-based manipulation — car tire replacement only requires alignment + teleoperation. Simple tasks can scale today with minimal generalization.
16. Final Synthesis
Robotics is not bottlenecked by model scale. It is bottlenecked by:
- Lack of unified representation (Explicit 3D + shared latent)
- Lack of data flywheel
- Mismatch between inference and control
- Unresolved physics (Sim2Real)
- Fragmented embodiment
Convergence: World models for generalization + vertical domain-specialized SLMs for deployment.
Final Insight Reality cannot be scraped like the internet. It must be sensed, interacted with, and simulated.
The Core Challenge Build a system where representation, simulation, and action are jointly optimized under physical constraints. Only then will world models move from abstraction to true embodiment.
Questions for Robotics World Model Reading Club 02 Discussion
- What is the minimal sufficient representation for robotics World Models?
- Can video DiT truly become WAM, or is the interaction prior fundamentally missing?
- Is robotics doomed to be vertical (domain-specific SLMs)?
- What is the “ImageNet moment” for robotics?
@BostonDynamics, @Stanford, @AGIBOTofficial, @intbotai, @BytedanceTalk, @Google ,@moonlake, @Rivian, @Meta, @Samsung, @UCBerkeley, @Cruise, @encord_team, @ManycoreTech, @OpenGraph_Labs, @neuralmotion, @AMD, @nvidia, @oysterecosystem, @Zoom, @FusionFundVC, @BoostVC, @yzilabs…
References
[1] 📖 130. Robotics World Model Reading Club 01—San Francisco 20260328 Recap: Explicit 3D as the Backbone: Toward Unified World Models for Robotics https://www.linkedin.com/pulse/130-robotics-world-model-reading-club-01san-francisco-20260328-glvtc