bot_vault

SafeDreamer: Safe Reinforcement Learning with World Models

2 min read

Summary

SafeDreamer (ICLR 2024) is the first method to integrate Lagrangian-based constraint satisfaction directly into world model planning within the DreamerV3 framework. By imagining safety-cost rollouts in latent space and applying dual-variable optimization, the agent learns to maximize reward while satisfying hard safety constraints. It achieves near-zero constraint violations on the Safety-Gymnasium benchmark — across both low-dimensional state inputs and vision-only tasks where prior SafeRL methods consistently fail.

SafeDreamer（ICLR 2024）是首個將 Lagrangian 約束最佳化整合進世界模型規劃的安全強化學習方法。在潛在空間中模擬成本軌跡並施以對偶變數最佳化，使 agent 在最大化獎勵的同時滿足安全約束。在 Safety-Gymnasium 基準測試中實現接近零的約束違反率，包括低維狀態與純視覺任務。

Key Points

  • Architecture: DreamerV3 RSSM world model + Lagrangian penalty applied to imagined cost rollouts
  • Two-level optimization: outer loop updates Lagrange multiplier (safety constraint tightness); inner loop trains actor to minimize penalized reward
  • Vision-only capability: world model compresses image observations into latent states, enabling constraint satisfaction without ground-truth state access
  • Zero-cost goal: prior methods (CPO, PCPO, PPO-Lag) trade off safety for reward; SafeDreamer achieves both near-zero violations and competitive reward
  • Sample efficiency: imagination rollouts in world model dramatically reduce environment interactions needed for safe policy learning

Insights

The core insight is that safety constraints are easier to enforce in latent space than in pixel space — the world model creates a differentiable simulator where Lagrangian penalties can be back-propagated through imagined trajectories. This separates the safety enforcement problem from the perception problem.

The ICLR 2024 acceptance signals that world model + safe RL is considered a mature research direction. The remaining limitation is that the world model itself may be inaccurate, so safety guarantees are only as strong as the model’s predictive fidelity.

Connections

Graph View

Backlinks