本文由 AI 分析生成
建立時間： 2026-05-30 來源： https://lilianweng.github.io/posts/2018-02-19-rl-overview/
Summary
Lilian Weng’s foundational RL overview covering MDP formalism, Bellman equations, Dynamic Programming, Monte Carlo methods, TD Learning (SARSA, Q-Learning, DQN), Policy Gradient, and Evolution Strategies. Uses AlphaGo Zero as a case study synthesizing many of these ideas. Intended as a first deep-read after introductory RL material.
Lilian Weng 的 RL 基礎全覽，系統介紹 MDP、Bellman 方程、動態規劃、蒙地卡羅、TD Learning、Q-Learning、DQN、Policy Gradient 及演化策略。以 AlphaGo Zero 作為整合案例研究，是學習 RL 基礎的最佳起點之一。
Key Points
- MDP: (States, Actions, Transition Probabilities, Rewards, Discount Factor) — Markov property means future depends only on current state
- Bellman equation: V(s) = R(s) + γ·max_a Σ P(s’|s,a)·V(s’) — recursive decomposition enabling DP solutions
- Deadly triad: off-policy + nonlinear function approximation + bootstrapping → instability (DQN mitigates with experience replay + target network)
- SARSA vs Q-Learning: on-policy vs off-policy TD control — Q-Learning learns optimal policy regardless of exploration behavior
- DQN: adds neural network Q-function approximation to Q-Learning; experience replay breaks correlation, target network stabilizes training
- Evolution Strategies: gradient-free, trivially parallelizable, no backprop needed — useful for non-differentiable environments
Insights
The “Deadly Triad” framing is crucial for understanding why deep RL is unstable: any two of the three components is fine, but combining all three creates divergence. This explains why PPO (on-policy) and SAC (off-policy but with specific stabilization) became the workhorses — they carefully navigate this triad.
The AlphaGo Zero case study is pedagogically excellent because it shows how self-play generates its own curriculum: the policy improves exactly as fast as the opponent improves, maintaining the right difficulty level throughout training. This principle applies broadly to robot learning in simulation.
Connections
Raw Excerpt
“Q-Learning is off-policy: it can learn the optimal policy while following an exploratory behavior policy. The key insight is that the greedy update V*(s) = max_a Q*(s,a) is independent of the exploration strategy.”