本文由 AI 分析生成
建立時間： 2026-08-18 來源： https://arxiv.org/abs/2504.16680
Summary
ETH Zurich’s RWM-U extends autoregressive robotic world models with ensemble-based epistemic uncertainty estimation, enabling temporally consistent uncertainty propagation over long imagined rollouts. Combined with MOPO-PPO (uncertainty-penalized policy optimization), it achieves effective policy training entirely from offline datasets and transfers to real quadruped and humanoid robots without online interaction.
ETH Zurich 的 RWM-U 為自迴歸機器人世界模型加入集成式認識論不確定性估計，透過 MOPO-PPO 懲罰高不確定性的想象軌跡，實現完全離線訓練並成功遷移至真實四足與人形機器人——首個在離線 MBRL 中針對真實機器人硬體進行完整驗證的世界模型框架（ETH Zurich, 2025）。
Prerequisites
- Offline RL / MOPO — training RL agents from fixed datasets without environment interaction; MOPO specifically uses world model uncertainty as a conservative penalty to avoid distribution shift.
- Autoregressive world models — the model predicts future states sequentially; compounding errors over long rollouts are a known failure mode that epistemic uncertainty estimation helps diagnose.
- Epistemic vs. aleatoric uncertainty — epistemic uncertainty (model uncertainty, reducible with more data) is what ensemble disagreement estimates; this is the safety-relevant signal.
Core Idea
Offline RL with world models fails in practice because the learned model is only reliable within the support of the offline dataset — outside that support, the model confidently generates hallucinated trajectories that lead the policy into unsafe or unreachable states. RWM-U addresses this by training an ensemble of world models: when the ensemble members disagree significantly about the next state, epistemic uncertainty is high, signaling out-of-distribution extrapolation. MOPO-PPO then penalizes the policy for entering high-uncertainty regions, making it conservative in areas where the world model is unreliable. This is a principled safety mechanism: the agent is uncertain → it behaves cautiously → it stays within the data distribution → the world model is reliable → the policy is safe.
Results
| Task / Benchmark | RWM-U | Baselines | Delta |
|---|---|---|---|
| Manipulation (sim) | Outperforms model-free and uncertainty-unaware | Model-free, no-uncertainty WM | Superior |
| Locomotion (sim) | Outperforms baselines | Same | Superior |
| Real quadruped transfer | Successful | Not demonstrated by baselines | First real-robot offline MBRL result |
| Sim + real data fusion | Further improves robustness | Real-only | Additional improvement |
Limitations
- Author-stated: Ensemble uncertainty estimation requires training multiple world models; computational cost scales with ensemble size.
- Unstated: Ensemble disagreement measures epistemic uncertainty but is an approximation; in high-dimensional observation spaces (e.g., vision), ensemble members may agree while still being wrong. The sim+real data fusion benefit suggests real-world data remains important even with this method.
Reproducibility
- Code: GitHub:
leggedrobotics/robotic_world_model(public) - Datasets: Offline datasets from simulation + real robot data
- Compute: Ensemble training (N × single model compute)
Insights
RWM-U makes a clean argument that uncertainty quantification is the missing piece for offline MBRL on real robots. The ensemble approach is well-understood in Bayesian deep learning but applying it to autoregressive world models with consistent uncertainty propagation over multi-step rollouts is non-trivial — each step’s uncertainty must compose correctly rather than independently accumulate (which would make the signal useless after a few steps).
The sim+real data fusion result is particularly interesting for lab settings: pure real-robot data collection is expensive and slow, but pure simulation suffers from sim-to-real gap. Offline MBRL with mixed datasets could dramatically reduce real-robot data requirements for locomotion and manipulation.
Connections
- safedreamer-safe-reinforcement-learning-world-models — SafeDreamer also uses world model imagination for safety; complementary (constraint optimization vs. uncertainty penalization)
- world-models-robot-safety — broader world model safety survey
- 2026-05-16-3d-4d-reconstruction-safety-vault-survey — 3D/safety survey context
Raw Excerpt
“Uncertainty signal penalizes high-risk imagined transitions during policy learning… Fusing real-world data with simulation further improves robustness.”