本文由 AI 分析生成
建立時間： 2026-05-01 來源： https://x.com/vai_viswanathan/status/2050177504392998932
Summary
A thorough conceptual survey of “world models” — a term covering four distinct architectural approaches (generative, latent dynamics, JEPA, 3D neural) — grounded in LeCun’s formal definition of world modeling as predicting future states s(t+1) from observations, actions, and latent uncertainty. The piece then maps five robotics use cases — data synthesis, training gym, evaluation, direct planning, and world-action models — onto the full ML development lifecycle.
關於「世界模型」的全面概念調查。該術語涵蓋四種截然不同的架構（生成式、潛在動態、JEPA、3D 神經），以 LeCun 的形式定義為基礎，並將五種機器人應用（資料合成、訓練環境、評估、直接規劃、世界-行動模型）對應到完整的 ML 開發生命週期。
Prerequisites
- Transformer/diffusion model basics
- Reinforcement learning fundamentals (actor-critic, MPC, Dyna)
- Basic robotics concepts (VLA, policy, embodiment)
Core Idea
World modeling is a problem statement — build a system that predicts future world states given current observation, current state, action, and sampled latent uncertainty — not a specific architecture. Four current paradigms disagree on whether to render pixels (Dec()), whether to use reward, and whether to commit to 3D geometry as the representation.
Results
- Latent dynamics models (DreamerV3/Dreamer 4) and JEPA (V-JEPA 2) best fit LeCun’s formal definition
- Generative models (Sora, Genie 3) conflate state and pixels, wasting capacity on rendering
- 3D neural models (NeRF, Gaussian Splatting) have strong Enc() but weak Pred() — better called world representations
- WorldGym achieves r=0.78 correlation with real-world success rates for policy evaluation; DreamDojo achieves r=0.995
- World-Gymnast TTT pushes “close drawer” success from 62% to 100% with no real-world rollouts
Limitations
- Author-stated: “We’re at the beginning, not the end”; which approaches scale and become standard infrastructure is still unknown
- Unstated: the survey is primarily theoretical; no original experiments; coverage skews toward papers available mid-2025
Reproducibility
- Code: N/A (conceptual piece)
- Datasets: N/A
- Compute: N/A
Insights
The most underappreciated claim is that iterative world model updates turn every evaluation run into a data-collection run for the model that powers the next training cycle — collapsing the offline/online distinction. Test-time training (TTT) inside the world model is a qualitatively new capability: a robot can practice a novel task in imagination before touching hardware.
Connections
Raw Excerpt
“A world model isn’t a single tool you reach for at one moment — it’s a substrate that shows up across the entire robotics ML pipeline: Before you train, the world model generates data. During training, it’s the environment. After training, it’s the grader and the adversary probing for safety failures. At deployment, it’s the planner — or, increasingly, it is the policy.”