Robotics & World Model Reading Club 02 Hot🔥Takes: cohost with @aurorafeng_01, JEPA zoo keynote by @JulianSaks, next week keynote @t641769919
#reading-club-02-0404
- Session 02 Luma: https://luma.com/g3qrrti0
- Reading Club 02 Review (liked by Yann LeCun on X): https://x.com/junfanzhu98/status/2040716119259164673?s=20
- Event photos: https://x.com/junfanzhu98/status/2040717084972245341?s=20
- LinkedIn more photos: https://www.linkedin.com/posts/junfan-zhu_robotics-world-model-reading-club-02-hot-ugcPost-7446465600723509248-1qbj?utm_source=share&utm_medium=member_desktop&rcm=ACoAABxP-p0BpUNGDf347aKh_1uJAPzG4er0As8
- Session 01 Luma: https://luma.com/8s4w1wu6
- Reading Club 01 Review: https://x.com/junfanzhu98/status/2038153945219305812
- Event photos (liked by Yann LeCun on X): https://x.com/junfanzhu98/status/2038161288090779985
Current world models are often misaligned with the physical reality they aim to capture. The issue lies not in scale, but in flawed inductive biases around representation, dynamics, and control. The JEPA (Joint Embedding Predictive Architecture) family provides one of the strongest directions forward by moving from pixel prediction to latent prediction in a non-generative setup.
1. JEPA Zoo
V-JEPA 2 features a two-stage pipeline: action-free pretraining on >1 million hours of internet video plus 1 million images (encoder compresses frames into latent representations; predictor forecasts masked targets), followed by action-conditioned post-training on <62 hours of DROID robot trajectories. It enables impressive zero-shot deployment on Franka arms in two labs for image-goal pick-and-place — with no environment data, no task-specific training, and no reward.
🔥 Hot take: Web-scale video combined with minimal robot interaction data suffices to unlock physical planning capabilities.
V-JEPA 2.1 advances dense features through four innovations: dense predictive loss (both visible and masked tokens contribute for explicit spatial/temporal grounding), deep self-supervision (self-supervised objectives applied hierarchically across multiple intermediate encoder layers), multi-modal tokenizers, and scaling of model capacity + data.
Act-JEPA integrates JEPA into imitation learning by operating directly in policy latent space via embedding prediction, delivering ~40% better world understanding and ~10% higher task success.
Causal-JEPA is object-centric: randomly masks out an object in latent space and requires the model to predict its counterfactual “what if” state from other objects through latent intervention. No explicit causal graph is needed — the causal inductive bias emerges from the objective. This marks a key step toward genuine causal world models.
ThinkJEPA (linked to ego-TL “think out loud” data) adds a VLM “thinker” branch with “think-out-loud” style reasoning traces. It uses dual-temporal pathways — dense JEPA for fine-grained motion/interaction and a VLM branch at larger temporal strides for long-horizon semantic guidance — with hierarchical pyramid representation extraction from multi-layer VLM features. This compensates for JEPA’s short-horizon limitations in hand-manipulation trajectory prediction.
LeJEPA / LeWorldModel (LeWM) (& GitHub eb_jepa) is the most minimalist and debated. It trains end-to-end from raw pixels with only two losses: next-embedding prediction and SIGReg (Sketched-Isotropic-Gaussian Regularizer) to enforce isotropic Gaussian distributions and prevent collapse. Training flow: two images/frames (context + target) passed to encoders producing latent representations; each frame converted to a single token for efficient modeling. No EMA, no teacher-student tricks, no pre-trained encoder; ~15M parameters, trainable on a single GPU. DINO / DINOv2 encoders serve as baselines or for reward evaluation.
🔥 Hot Take on LeWorldModel: It is too extreme. It successfully escapes pixel-level reconstruction, but cannot push latent-level compression too far either. Trajectory straightening causes latent semantic degradation, destroying temporal richness, interaction semantics, and affordance structure. Why not learn task progress estimation instead of blindly straightening trajectories?
2. Conflicts in Latent Design
Reconstruction and planning are inherently conflicting. Reconstruction requires the latent to retain rich semantics for visual/state recoverability. Planning requires a minimal sufficient dynamics representation. Forcing both jointly is problematic. Practical solution: Do not force reconstruction during planning. Use it only as a 2nd-stage post-training probe / diagnostic tool. With little data it succeeds in proving the latent carries verifiable semantics. Reconstruction is not the goal — it is a potential signal and verifiability check.
This naturally leads to the decoupled world model proposal: split the latent state into two parts — static (scene, objects, geometry, 3D) and dynamic (motion, interaction, forces). The latent space is explicitly decoupled so each component carries distinct structured information, avoiding entanglement while enabling better planning (dynamic) and grounding/reconstruction (static).
3. Continuous Dynamics and Sim2Real Alignment
The physical world is continuous. Most models rely on discrete next-step prediction with fixed sampling, mismatching real data’s irregular sampling frequencies and the discrete manner of dynamics models. World models must learn continuous dynamics — not only next-step but full continuous evolution — to align with physics. Explicit 3D representations are critical to close the sim2real gap. Ongoing debate: Is pixel input enough? Pixels are rich but noisy; latent is abstract but may lose critical details.
4. Data Paradigm Shift and Deployment
Reliance on buying third-party datasets suffers from distribution shift and will become obsolete. Autonomous exploration will dwarf third-party datasets. The future paradigm: deploy robots, collect click data, and perform on-policy learning with no distribution shift, delivering true real-world grounding (reference to AV data engines and Skild). Supporting elements include GoPro egocentric videos and FPV lab for pretraining phase, gloves for mid-training precise interaction signals, humanoid extra data scale, resource collection (“world models on other worlds”), 3D AMR (Autonomous Mobile Robotics), models deployed as proprietary services, and the virtual robot seesaw benchmark for physical stability testing.
5. Reward Modeling and Control Awareness
Video foundation models are not control-aware: simply measuring pixels to “move forward toward target” makes reward spike immediately, even when the robot fails to grab the object. We need adjustment for controllability. Training-free reward adjustment is favored: do not train the reward model; instead train the policy (reward finetuning / policy finetuning agnostic). Methods include success over failure contrastive signals to adjust the world model, and DINOv2 encoder to encode goal state with cosine similarity as reward (not a full neural network head).
The paper Planning in Latent Dynamics proposes a reward-free approach: learn implicit reward as cost from latent geometry. The distance between next state and current state in latent space is proportional to temporal/causal progression — if the robot takes 100 steps, current and future representations remain proportionally related. LeWorldModel uses a similar concept to drive robot exploration. Hot take: Reward should emerge from the geometry of the latent manifold rather than a trained head, and it must be control-aware.
6. Additional Techniques
RoboWheel: Extracts universal, embodiment-agnostic motion representations from real human hand-object interaction (HOI) videos using diffusion-based editing/retargeting for cross-embodiment transfer (arms, dexterous hands, humanoids) with monocular RGB(D) and a helical data engine.
Cross-domain data editing (xTED): Diffusion Transformer (DiT)-based trajectory editing; optimal editing ratio κ ≈ 0.5 for adapting source trajectories to target domains while solving domain gaps in third-party data.
7. Open Challenges
How to best provide action representation in world models remains an open question. Policy adaptation and continual adaptation are critical — robots easily “forget” old skills in non-stationary environments.
8. Core Hot Takes from the Discussion
- Reconstruction vs planning conflict: Reconstruction needs semantic-rich latent for recoverability; they are incompatible during planning. Use reconstruction only as post-training probe/diagnostic (succeeds with little data, proves semantics, serves as potential/verifiability signal; not the goal).
- LeWorldModel is too extreme: Not pixel-level but cannot be latent-level too extreme. Trajectory straightening causes latent semantic degradation. Prefer task progress estimation.
- Decoupled world model (static + dynamic) is essential for structured latent spaces.
- Continuous dynamics (not only next-step) + explicit 3D are necessary to align with the continuous physical world and handle varying sampling frequencies.
- Autonomous on-policy exploration will dwarf third-party dataset purchasing; the buying paradigm ends with deploy-click-on-policy real-world data (AV-style).
- Reward emerges from latent geometry (proportional distance, 100-step proportionality) and must be control-aware, not a separate head.
- World models > VLA because prediction forces physical understanding and enables better action generation (VLA is merely a context machine).
- Persistent challenges: pixel vs latent tradeoff, action representation, and continual adaptation (robot forgetting).
We concluded that future progress hinges on better-designed inductive biases — decoupled latents, continuous dynamics, autonomous closed-loop data engines, and geometry-driven rewards — rather than scale alone. JEPA variants currently lead this direction, while there’re some other elegant solutions too.
References
[1] 📖 132. Robotics World Model Reading Club 02: JEPA Zoo, WAMs & Unified Latent Representation—SF 20260404 https://www.linkedin.com/pulse/132-robotics-world-model-reading-club-02-jepa-zoo-wams-unified-ntlcc/