本文由 AI 分析生成
建立時間： 2026-05-30 來源： https://lilianweng.github.io/posts/2018-04-08-policy-gradient/
Summary
Lilian Weng’s comprehensive guide covering 18+ policy gradient algorithms: from REINFORCE through Actor-Critic, A3C/A2C, DPG/DDPG, TRPO, PPO, SAC, TD3, MADDPG, and IMPALA. The post explains the Policy Gradient Theorem, then categorizes algorithms by their approach to reducing variance, enabling off-policy learning, stabilizing training, and scaling to many parallel actors.
Lilian Weng 的 Policy Gradient 深度指南，涵蓋 18+ 演算法。核心定理說明梯度計算只需 Q 值乘以 log-probability 梯度的期望值，不需計算狀態分佈梯度。文章系統整理了各種降低 Variance、穩定訓練、提升樣本效率的技術路徑，是理解現代 Robot Learning RL 方法的核心參考。
Key Points
- Policy Gradient Theorem: ∇J(θ) = E[Q(s,a) · ∇log π(a|s)] — no state distribution gradient needed
- REINFORCE: high variance Monte Carlo baseline; Actor-Critic: TD error as baseline, online updates
- PPO: TRPO simplified with clipping instead of KL constraint — currently most widely used for locomotion
- SAC: entropy maximization + off-policy = sample-efficient + robust to hyperparameters
- TD3: double Q-learning + delayed actor update + target policy smoothing to fix DDPG overestimation
- IMPALA: decoupled acting/learning with V-trace off-policy correction, scales to thousands of actors
- Six recurring design principles: variance reduction, off-policy learning, experience replay, target networks, entropy regularization, policy constraint
Insights
The survey reveals a clear progression: each algorithm addresses a specific failure mode of its predecessor. DDPG addresses discrete action space limitations of REINFORCE; TRPO/PPO address DDPG’s instability; SAC addresses PPO’s sample inefficiency; TD3 addresses DDPG/SAC’s Q-overestimation. Understanding this chain makes each algorithm’s design choices feel inevitable rather than arbitrary.
For robotics specifically: SAC’s entropy maximization is particularly valuable in sim-to-real transfer because diverse policies in simulation are more likely to generalize to real-world variations than greedy policies.
Connections
Raw Excerpt
“Policy Gradient methods directly optimize Policy parameters via Gradient Ascent to maximize expected return. Unlike Value-based methods, they naturally handle continuous action spaces — essential for robot control.”