本文由 AI 分析生成
建立時間： 2026-04-19 來源： https://x.com/junfanzhu98/status/2045770010979905862
Summary
Session 04 of the SF Robotics World Model Reading Club examined world models as task-relative abstractions, covering π0.7 architecture dissection, model-based RL for drone flight (Dream to Fly), tactile sensing challenges, 3D reconstruction pipelines (GaussGym, real2sim2real), and physics simulators for soft bodies. The central thesis: robotics likely needs multiple heterogeneous world models preserving different downstream-relevant structures, not a single universal model.
第四期讀書會以「有用的抽象即世界模型」為核心，解剖了 π0.7 的模組化架構（SigLIP + Gemma + BAGEL），並討論了觸覺感測、3D重建（3DGS→mesh）與物理模擬器（PhysTwin）的挑戰。最終結論是機器人需要多種異質世界模型，而非單一通用模型。
Key Points
- World model defined formally as dynamics over chosen state: s_{t+1} ~ p(s_{t+1} | s_t, a_t); the key question is what goes in the state
- π0.7 is explicitly modular: SigLIP (400M) + Gemma (4B) for high-level policy, BAGEL (14B) as world model branch, 860M action expert
- GaussGym pipeline: 3DGS video reconstruction → mesh conversion; mesh precision is task-dependent and critical for sim fidelity
- RMA (Rapid Motor Adaptation) for legged robots uses a two-phase sim approach: environment factor encoder + adaptation module at 10 Hz vs base policy at 100 Hz
- Microphone-based tactile sensing as cheap alternative to camera-based gel sensors
- Hypothesis: mature robotic systems will become more modular (MoE, tokenizers, hierarchical policies) not end-to-end
Insights
The contrast between photorealistic vs. cartoon cat abstractions is a vivid illustration of why a single world model cannot serve all robotics tasks — different representations preserve different useful structure. The observation that egocentric data is “collected for geometry but labeled only with actions” while it should capture richer affordance priors identifies a systematic data labeling gap in current robot learning pipelines.
Connections
Raw Excerpt
“All world models are abstractions; some are useful. Different representations preserve different useful structure. Robotics may ultimately need multiple kinds of world models, not just one.”