134. Robotics World Model Reading Club 04: Abstract of Physical World for Decision Making—SF0418

World Models in Robotics: Abstractions, Dynamics, Modularity, and the Path from End-to-End Dreams to Practical Systems

Three foundational hypotheses framed the entire discussion on world models for robotics:

• All world models are abstractions; some are useful.
• Different representations preserve different useful structure.
• Robotics may ultimately need multiple kinds of world models, not just one.

The emphasis throughout was on “the” world model—yet the hypotheses explicitly rejected a single, universal solution. Instead, world modeling is best understood as the art of constructing a small, useful, task-relevant world for reasoning under uncertainty, in the spirit of Leonard Savage’s The Foundations of Statistics and Oliver Brock’s work. A memorable visual contrasted a photorealistic sleeping cat with a cartoon cat dangling from a windowsill: both are valid abstractions, but they preserve entirely different structures for different downstream tasks.

1. World Model

Define world model formally as dynamics over a chosen state:

$$st+1\sim p(st+1\mid st,at)s_{t+1}\sim p(s_{t+1}\mid s_t,a_t)$$

The central question became: What lives in the state s_t?

Concrete examples—pixel-level video frame prediction, 3D skeletal poses of humans, dense robot pose grids, and simulated robotic arm trajectories—showing how each representation trades off semantics, kinematics, physics fidelity, and data availability. A later comparison table made this explicit:

Recent methods try to combine these paths to strengthen the model across multiple axes. Crucially, a world model is useful only relative to the downstream structure it preserves.

2. Practical Architectures Spotlighted

π0.7: Steerable Generalist Robotic Foundation Model The talk dissected π0.7 in detail as a Vision-Language-Action (VLA) model pre-trained on SigLIP (400M) + Gemma (4B). The architecture is explicitly modular:

• Observation memory + task instruction feed a High-Level Policy (pre-trained SigLIP 400M + Gemma 4B).
• A separate World Model branch (pre-trained BAGEL 14B) takes current observation + subtask instruction (“pick up the knife”).
• Subgoal images and metadata (star ratings) condition both branches.
• An action expert (860M parameters) outputs discretized actions with added noise.

Question: whether π0.7 (and similar π0-style scaling) was scaling the wrong thing or suffering from architectural limitations? Recent papers show VQA (visual question answering) sometimes fails entirely when systems rely too heavily on language, and that pure action labels miss affordances. Egocentric data is often misused—collected for geometry but labeled only with actions—whereas it should capture richer structural priors.

3. Dream to Fly: Model-Based RL for Vision-Based Drone Flight

A full pipeline was presented for learning dynamics in a compact latent space. The latent state is s_k = (h_k, z_k), with transition probability:

$$P(sk+1\mid sk,ak)P(s_{k+1}\mid s_k,a_k)$$

Training proceeds in three stages:

1. Data collection in simulation using the current actor-critic policy, storing (observation, action, reward, termination) tuples in a replay buffer.
2. World Model training: encoder/decoder, RSSM, dynamics model, reward model, and continue predictor—all trained on the replay buffer.
3. Actor-Critic training inside the imagined world model to maximize expected returns; the improved policy then collects fresh data.

The speaker repeatedly returned to evaluation: data augmentation and latent regularization are both essential. A world model acts as a strong prior that helps training, enables subgoal prediction, and supports evaluation. Success is measured in simulation, but everything ultimately depends on the sim2real gap.

4. Additional Modalities and Challenges

4.1 Tactile

World models contain “so much noise.” The talk moved into multimodal extensions, especially tactile sensing. Conventional gel+vision sensors rely on deformation to approximate feedback but suffer from air bubbles inside the sensor, making stable data extremely hard to collect. An alternative explored was microphone-based tactile sensing: microphones embedded in the gripper are far cheaper than cameras. By listening to the sound of grasping a sheet of paper or using IMU signals to detect drops, one can infer contact states. However, the room is loud, data is noisy, and long-horizon tasks remain under-constrained.

4.2 Video-Based and Causal Models

• Genie was referenced as a generative world model.
• Causal video models and Inverse Dynamics Models (IDM) predict both future video frames and actions. Helix combines a world model with IDM; the speaker noted that IDMs converge extremely fast regardless of data quality, yet they are slow (1× generates an entire video). In contrast, Rhoda runs synchronously.
• Cosmos modifies policies directly rather than building full world models.
• Action chunking was highlighted: combining diffusion models with sampling-based planning. A neural network predicts dynamics of an optimization-based model (predicting barrier inputs rather than full trajectories—an older but still relevant idea). Torque-based control closes the loop.

4.3 3D Reconstruction and Simulation Fidelity

“Reconstructing Hand-Held Objects in 3D from Images and Videos.” The pipeline:

1. GPT-4(V) + model retrieval.
2. RGB and XYZ encoders (E_{RGB}, E_{XYZ}) produce 3D hand point clouds (MCC-HO).
3. Decoder + retrieval-augmented reconstruction yields a 3D object model.
4. Temporal alignment in a discretized pose state space q_j, followed by ICP rigid alignment of the template mesh to network-inferred geometry.

GaussGym was analyzed in depth for sim2real implications. Video reconstruction first builds 3D Gaussian Splatting (3DGS), then converts Gaussians to meshes. Mesh precision is task-dependent: too coarse and the simulation is inaccurate; too fine and the simulator cannot compute dynamics in real time. This conversion is critical when placing many meshes into a physics simulator for robot training. real2sim2real loops and noted that 3D unstructured data differs fundamentally from LLM-style auto-regressive training—structured information must be retrieved via databases during collection.

4.4 Physics Simulators and Soft Bodies

• A physics dynamic system is a world model: it outputs future states exactly like MuJoCo.
• PhysTwin enables deformable objects and soft bodies inside simulators.
• NVIDIA’s continuous fluid dynamics (CFD) provides pure physics but lacks visual rendering and material information. The speaker asked: can we go faster by using depth only? What unique information does RGB provide? Friction modeling, for example, likely requires RGB to infer material properties that depth alone cannot reveal.

4.5 Sim2Real RL with Physics Simulator (RMA – Rapid Motor Adaptation for Legged Robots)

Two phases in simulation:

• Phase 1: Environment factor encoder μ (mass, CoM, friction, terrain height, motor strength) produces latent z_t that trains a base policy π.
• Phase 2: Adaptation module ϕ (trainable, red in diagram) regresses from history to refined \hat{z}_t.

At deployment the adaptation module runs at 10 Hz while the base policy runs at 100 Hz. The diagram explicitly marks trainable modules in red.

5. Hypothesis: From End-to-End Dream to Modular Structure?

Provocative hypothesis: as systems mature, they often become more modular (MoE, tokenizers, encoders; high-level policy + planner + low-level policy; pre-training / mid-training / post-training). A detailed diagram illustrated emerging properties in unified multimodal pretraining, showing separate experts for language response (next-token prediction via FFN + text tokenizer + und. encoder) and image/video/velocity prediction (gen. encoder + gen. expert), all sharing multi-modal self-attention (QKV). The π0.7 VLA architecture was revisited as evidence of this modularity trend.

Robotics may need multiple world models, each preserving the structure required by its downstream task—whether pixel-level for semantics, 3D for kinematics, physics-augmented for control, or multimodal (including imagined tactile) for rich interaction. Evaluation must combine simulation success with real-world validation, always mindful of the sim2real gap, latent regularization, data augmentation, and the strong prior a well-designed world model provides.

Moving beyond the “end-to-end dream” toward principled, modular, multi-representation world modeling that actually works on real robots.

References

[1] 📚 134. Robotics World Model Reading Club 04: Abstractions of the Physical World for Decision-Making — SF 20260419 https://www.linkedin.com/pulse/134-robotics-world-model-reading-club-04-abstractions-physical-iss1c

Join Discord Community​

https://discord.gg/WH7DrTHRXK

​​Follow Saturday Robotics​

https://x.com/saturdayrobotic

Follow YouTube

https://www.youtube.com/@saturdayrobotic

Future events

#reading-club-07-0509: Learning to Dream: World Models, Imagination, Path to Foundation Models for Control

• Session 07 Luma: https://luma.com/srhe0vuo

#reading-club-06-0502: Evolution of Video World Models for Robotics

• Session 06 Luma: https://luma.com/sdrd4zwr

#reading-club-05-0425: World Models for Physical Intelligence: From Predictive Brains to Embodied Robots

• Session 05 Luma: https://luma.com/p7zvpyvg

Past events

#⁠reading-club-04-0418: Abstractions of the Physical World for Decision-Making

• Session 04 Luma: https://luma.com/atv7bm3i
• ​Reading Club 04 Review: https://x.com/junfanzhu98/status/2045770010979905862
• ​YouTube Recording: https://www.youtube.com/@saturdayrobotic

#⁠reading-club-03-0411: Robotic Policy Adaptation

• Session 03 Luma: https://luma.com/561xgirg
• ​Reading Club 03 Review: https://x.com/junfanzhu98/status/2043243484568768519?s=20
• ​YouTube Recording: https://www.youtube.com/@saturdayrobotic

#⁠reading-club-02-0404: JEPA Zoo

• ​Session 02 Luma: https://luma.com/g3qrrti0
• ​Reading Club 02 Review (liked by Yann LeCun on X): https://x.com/junfanzhu98/status/2040716119259164673?s=20

⁠reading-club-01-0328**

• ​​Session 01 Luma: https://luma.com/8s4w1wu6
• ​​Reading Club 01 Review (liked by Yann LeCun on X): https://x.com/junfanzhu98/status/2038153945219305812