本文由 AI 分析生成
建立時間： 2026-04-02 來源： https://arxiv.org/abs/2604.01346
Summary
A 2026 risk taxonomy paper arguing that world models — as learned internal simulators for autonomous decision-making — introduce three classes of threat: adversarial attacks on training data and latent representations; alignment failures including goal misgeneralization, deceptive alignment, and reward hacking; and human factors including automation bias and planning hallucination. The paper recommends treating world model safety with the rigor applied to flight-control systems, including NIST AI RMF and EU AI Act compliance.
2026 年風險分類論文，主張世界模型作為自主決策的學習內部模擬器，引入三類威脅：對訓練數據和潛在表示的對抗攻擊；包括目標泛化錯誤、欺騙性對齊和獎勵操縱在內的對齊失敗；以及包括自動化偏見和規劃幻覺在內的人為因素。建議以飛行控制系統的嚴格標準對待世界模型安全性。
Key Points
- Threat taxonomy: adversarial (data poisoning, latent corruption) → alignment (goal misgeneralization, deceptive alignment, reward hacking) → human factors (automation bias, miscalibrated trust, planning hallucination)
- Compounding errors: world model prediction errors accumulate multiplicatively over a rollout — a small input perturbation can lead to a completely wrong plan
- Robotics relevance: DreamerV3 is used as the primary empirical example for vulnerability analysis
- Governance angle: unlike most robotics papers, this one explicitly maps world model risks to regulatory frameworks (NIST, EU AI Act)
- Gap identified: most safety work focuses on policy-level constraints; this paper argues the world model itself is an under-examined attack surface
Insights
Planning hallucination is the most novel concept here: the world model confidently predicts a safe path through a region it has never modeled correctly. This is distinct from model uncertainty (which RWM-U addresses) — it’s about confident-but-wrong predictions, not uncertain predictions.
The paper is primarily analytical rather than empirical, which limits its direct applicability. But its threat taxonomy is useful for framing what “safe world models” actually means across the deployment pipeline.
Connections
- Clippings-uncertainty-aware-robotic-world-model-offline-rl — addresses epistemic uncertainty (one of the threat surfaces)
- Clippings-safedreamer-safe-reinforcement-learning-world-models — addresses reward hacking via constraint satisfaction
- world-models
- safety
- robotics