Download as PDF, PPTX
![問題設定
▶ Markov Decision Process
▶ 状態 st ∈ RNS
▶ 行動 at ∈ RNA
▶ 初期状態分布 s0 ∼ p0(·)
▶ 遷移分布 st+1 ∼ p(·|st, at)
▶ st
at
確率的 st+1
決
▶ 報酬関数 rt = r(st, at, t)（時間依存）
▶ 求
▶ （確率的 ）policy at ∼ p(·|st; θ)
▶ st
確率的 at
決
▶ 最大化
▶ 報酬 和 期待値 J(θ) = E[
∑T
t=0 γtrt|θ]
▶ γ ∈ [0, 1] 割引率](https://image.slidesharecdn.com/slides-160120050539/85/Learning-Continuous-Control-Policies-by-Stochastic-Value-Gradients-5-320.jpg)
![価値関数
▶ 状態行動価値関数
Qt
(s, a) = E[
∑
τ=t γτ−t
rτ
|st
= s, at
= a, θ]
▶ 状態価値関数 V t
(s) = E[
∑
τ=t γτ−t
rτ
|st
= s, θ]
▶ （確率的）Bellman 方程式
V (s) = Ea∼p(·|s)[r(s, a) + γEs′∼p(·|s,a)[V ′
(s′
)]]
= Ea∼p(·|s)[Q(s, a)]
▶ ′ 次 時間 表 使](https://image.slidesharecdn.com/slides-160120050539/85/Learning-Continuous-Control-Policies-by-Stochastic-Value-Gradients-6-320.jpg)
![表記 関 注意
▶ 下付 文字 偏微分 表
▶ πθ = ∂π
∂θ
▶ 「 θ 表 π」
▶ （ 1 箇所 πθ 後者 意味 使 場所 …）
▶ 上付 文字 時間 指数
▶ 報酬 和 期待値（再掲） J(θ) = E[
∑T
t=0 γtrt|θ]
▶ rt
時間 t 報酬
▶ γt
γ t 乗
▶ 時間依存 判断 …](https://image.slidesharecdn.com/slides-160120050539/85/Learning-Continuous-Control-Policies-by-Stochastic-Value-Gradients-7-320.jpg)
![行動 連続値
▶ 「DQN 駄目 ？」
▶ DQN [Mnih et al. 2013; Mnih et al. 2015] 状態行動価
値 Q(s, a; θ) 学習 ，行動 arg maxa Q(s, a; θ) 選択
▶ a 連続値 arg max 求 ！
▶ policy 直接 （NN ）表
▶ at ∼ p(·|st; θ)
▶ 行動 選 際
▶ θ 更新 方法 ， 論文 policy
gradient methods 種類 方法 扱](https://image.slidesharecdn.com/slides-160120050539/85/Learning-Continuous-Control-Policies-by-Stochastic-Value-Gradients-8-320.jpg)
![Policy Gradient Methods
▶ 目標：J(θ) = E[
∑T
t=0 γt
rt
|θ] 最大化 policy
θ 求
▶ ∇θJ(θ)（policy gradient） 求
▶ 求 勾配法 policy 最適化 （policy
gradient methods）
▶ 求 ？
▶ likelihood ratio methods
▶ value gradient methods](https://image.slidesharecdn.com/slides-160120050539/85/Learning-Continuous-Control-Policies-by-Stochastic-Value-Gradients-9-320.jpg)
![Likelihood Ratio Methods (1)
▶ 分布 p(y|x) 上 ， 関数 g(y) 期待値 勾配
∇x Ep(y|x)g(y) 求
▶ 関数 ∇x log p(y|x) 使
∇x Ep(y|x)g(y) = Ep(y|x)[g(y)∇x log p(y|x)]
≈
1
M
M∑
i=0
g(yi
)∇x log p(yi
|x), yi
∼ p(yi
|x)
▶ ∇θEp(y;θ)g(y) 同様 求
▶ likelihood ratio methods, score function estimators,
REINFORCE 様々 呼](https://image.slidesharecdn.com/slides-160120050539/85/Learning-Continuous-Control-Policies-by-Stochastic-Value-Gradients-10-320.jpg)
![Likelihood Ratio Methods (2)
▶ 使 ∇θJ(θ) 推定
[Williams 1992; Sutton et al. 1999]
∇θJ(θ) = Es∼ρπ,a∼p(·|s;θ)[Q(s, a)∇θ log p(a|s; θ)]
▶ policy gradient 求 方法 広 使
▶ 欠点
▶ Q(s, a) 勾配情報 使
▶ variance 大](https://image.slidesharecdn.com/slides-160120050539/85/Learning-Continuous-Control-Policies-by-Stochastic-Value-Gradients-11-320.jpg)
![Stochastic Value Gradients
▶ 遷移分布 s′
= f (s, a, ξ)，policy a = π(s, η; θ)
reparameterize
Vs = Eρ(η)[rs + raπs + γEρ(ξ)V ′
s′ (fs + faπs)] (7)
Vθ = Eρ(η)[raπθ + γEρ(ξ)[V ′
s′ faπθ + γV ′
θ]] (8)
= Eρ(η)[Qaπθ + γV ′
θ]
▶ MDP 確率的 ，policy 確率的 ，value
gradient 求 ！（stochastic value gradient）](https://image.slidesharecdn.com/slides-160120050539/85/Learning-Continuous-Control-Policies-by-Stochastic-Value-Gradients-16-320.jpg)
![評価
▶ AC [Wawrzynski 2009]，DPG [Silver et al. 2014] 既存
手法（ policy value function 学習 ）
▶ SVG(1)-ER 総 良](https://image.slidesharecdn.com/slides-160120050539/85/Learning-Continuous-Control-Policies-by-Stochastic-Value-Gradients-22-320.jpg)
![参考文献 I
[1] Volodymyr Mnih et al. “Human-level control through deep reinforcement
learning”. In: Nature 518.7540 (2015), pp. 529–533.
[2] Volodymyr Mnih et al. “Playing Atari with Deep Reinforcement Learning”. In:
NIPS 2014 Deep Learning Workshop. 2013, pp. 1–9. arXiv:
arXiv:1312.5602v1.
[3] David Silver et al. “Deterministic Policy Gradient Algorithms”. In: ICML 2014.
2014, pp. 387–395.
[4] Richard S. Sutton et al. “Policy Gradient Methods for Reinforcement Learning
with Function Approximation”. In: In Advances in Neural Information
Processing Systems 12. 1999, pp. 1057–1063.
[5] Pawel Wawrzynski. “Real-time reinforcement learning by sequential
Actor-Critics and experience replay”. In: Neural Networks 22.10 (2009),
pp. 1484–1497.
[6] RJ Williams. “Simple statistical gradient-following algorithms for connectionist
reinforcement learning”. In: Reinforcement Learning 8.3-4 (1992), pp. 229–256.](https://image.slidesharecdn.com/slides-160120050539/85/Learning-Continuous-Control-Policies-by-Stochastic-Value-Gradients-27-320.jpg)
More Related Content
![[Dl輪読会]introduction of reinforcement learning](https://cdn.slidesharecdn.com/ss_thumbnails/dlintroductionofreinforcementlearning-161121061444-thumbnail.jpg?width=640&height=640&fit=bounds)
![SSII2021 [TS2] 深層強化学習 〜 強化学習の基礎から応用まで 〜](https://cdn.slidesharecdn.com/ss_thumbnails/ts2-01-210607042910-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DL輪読会]Understanding Black-box Predictions via Influence Functions](https://cdn.slidesharecdn.com/ss_thumbnails/dlhacksinffunc-170822055634-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DL輪読会]Control as Inferenceと発展](https://cdn.slidesharecdn.com/ss_thumbnails/20191004-191204055019-thumbnail.jpg?width=640&height=640&fit=bounds)
![SSII2021 [OS2-01] 転移学習の基礎：異なるタスクの知識を利用するための機械学習の方法](https://cdn.slidesharecdn.com/ss_thumbnails/os2-02final-210610091211-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DL輪読会]Learning to Act by Predicting the Future](https://cdn.slidesharecdn.com/ss_thumbnails/20171113dl-171114045641-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DL輪読会]Convolutional Conditional Neural Processesと Neural Processes Familyの紹介](https://cdn.slidesharecdn.com/ss_thumbnails/20191220readingpaperconvcnp-191220034420-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Learning Continuous Control Policies by Stochastic Value Gradients
![[DL輪読会]What Matters In On-Policy Reinforcement Learning? A Large-Scale Empiri...](https://cdn.slidesharecdn.com/ss_thumbnails/20200619misono-200630053718-thumbnail.jpg?width=640&height=640&fit=bounds)
Learning Continuous Control Policies by Stochastic Value Gradients
- 1. Learning Continuous ControlPolicies by Stochastic Value Gradients NIPS2015 読 会 藤田康博 Preferred Networks Inc. January 20, 2016
- 2. 話 人 ▶ 藤田康博 ▶Preferred Networks Inc. ▶ Twitter: @mooopan ▶ GitHub: muupan ▶ 強化学習・ AI 興味 ▶ 最近 仕事（ 発表 関係 ） (https://twitter.com/hillbig/status/684813252484698112)
- 3. 読 論文 ▶ LearningContinuous Control Policies by Stochastic Value Gradients ▶ Nicolas Heess, Greg Wayne, David Silver, Timothy Lillicrap, Yuval Tassa, Tom Erez (Google DeepMind) ▶ 強化学習 状態・行動 連続値 取 確率的 制御問題 扱 提案 ▶ ・価値関数・policy NN 表 ▶ reparameterization trick 使
- 4.
- 5. 問題設定 ▶ Markov DecisionProcess ▶ 状態 st ∈ RNS ▶ 行動 at ∈ RNA ▶ 初期状態分布 s0 ∼ p0(·) ▶ 遷移分布 st+1 ∼ p(·|st, at) ▶ st at 確率的 st+1 決 ▶ 報酬関数 rt = r(st, at, t)（時間依存） ▶ 求 ▶ （確率的 ）policy at ∼ p(·|st; θ) ▶ st 確率的 at 決 ▶ 最大化 ▶ 報酬 和 期待値 J(θ) = E[ ∑T t=0 γtrt|θ] ▶ γ ∈ [0, 1] 割引率
- 6. 価値関数 ▶ 状態行動価値関数 Qt (s, a)= E[ ∑ τ=t γτ−t rτ |st = s, at = a, θ] ▶ 状態価値関数 V t (s) = E[ ∑ τ=t γτ−t rτ |st = s, θ] ▶ （確率的）Bellman 方程式 V (s) = Ea∼p(·|s)[r(s, a) + γEs′∼p(·|s,a)[V ′ (s′ )]] = Ea∼p(·|s)[Q(s, a)] ▶ ′ 次 時間 表 使
- 7. 表記 関 注意 ▶下付 文字 偏微分 表 ▶ πθ = ∂π ∂θ ▶ 「 θ 表 π」 ▶ （ 1 箇所 πθ 後者 意味 使 場所 …） ▶ 上付 文字 時間 指数 ▶ 報酬 和 期待値（再掲） J(θ) = E[ ∑T t=0 γtrt|θ] ▶ rt 時間 t 報酬 ▶ γt γ t 乗 ▶ 時間依存 判断 …
- 8. 行動 連続値 ▶ 「DQN駄目 ？」 ▶ DQN [Mnih et al. 2013; Mnih et al. 2015] 状態行動価 値 Q(s, a; θ) 学習 ，行動 arg maxa Q(s, a; θ) 選択 ▶ a 連続値 arg max 求 ！ ▶ policy 直接 （NN ）表 ▶ at ∼ p(·|st; θ) ▶ 行動 選 際 ▶ θ 更新 方法 ， 論文 policy gradient methods 種類 方法 扱
- 9. Policy Gradient Methods ▶目標：J(θ) = E[ ∑T t=0 γt rt |θ] 最大化 policy θ 求 ▶ ∇θJ(θ)（policy gradient） 求 ▶ 求 勾配法 policy 最適化 （policy gradient methods） ▶ 求 ？ ▶ likelihood ratio methods ▶ value gradient methods
- 10. Likelihood Ratio Methods(1) ▶ 分布 p(y|x) 上 ， 関数 g(y) 期待値 勾配 ∇x Ep(y|x)g(y) 求 ▶ 関数 ∇x log p(y|x) 使 ∇x Ep(y|x)g(y) = Ep(y|x)[g(y)∇x log p(y|x)] ≈ 1 M M∑ i=0 g(yi )∇x log p(yi |x), yi ∼ p(yi |x) ▶ ∇θEp(y;θ)g(y) 同様 求 ▶ likelihood ratio methods, score function estimators, REINFORCE 様々 呼
- 11. Likelihood Ratio Methods(2) ▶ 使 ∇θJ(θ) 推定 [Williams 1992; Sutton et al. 1999] ∇θJ(θ) = Es∼ρπ,a∼p(·|s;θ)[Q(s, a)∇θ log p(a|s; θ)] ▶ policy gradient 求 方法 広 使 ▶ 欠点 ▶ Q(s, a) 勾配情報 使 ▶ variance 大
- 12. Deterministic Value Gradients(1) ▶ Backpropagation 価値関数 勾配（value gradient） 求 （value gradient methods） ▶ J(θ) = Es0∼p0 V 0(s0) V 0 θ 計算 良 ▶ MDP policy 決定的（s′ = f (s, a), a = π(s)） ， 決定的 Bellman 方程式 V (s) = r(s, π(s)) + γV ′ (f (s, π(s))) 微分 value gradient 計算 Vs = rs + raπs + γV ′ s′ (fs + faπs) (3) Vθ = raπθ + γV ′ s′ faπθ + γV ′ θ (4) = Qaπθ + γV ′ θ
- 13. Deterministic Value Gradients(2) ▶ 式 (3)，(4) 系列 (s0 , a0 , s1 , a1 , . . . ) V 0 θ (s0 ) RNN 計算
- 14. Deterministic Value Gradients(3) ▶ 欠点 ▶ 確率的 MDP policy 扱 ▶ 異 区別 状態 （state aliasing） 確 率的 policy 必要 ▶ 例：灰色 状態 区別 場合，決 定的 policy 開始地点 一生 金 辿 ▶ reparameterization trick 解決
- 15. Reparameterization Trick ▶ ∇xEp(y|x)g(y) 求 別 方法 ▶ p(y|x) 決定的 関数 f 変数 ξ 使 書 ：y = f (x, ξ), ξ ∼ ρ(·) ▶ 例：p(y|x) = N(µ(x), σ2(x)) y = µ(x) + σ(x)ξ, ξ ∼ N(0, 1) ▶ 微分 ∇x Ep(y|x)g(y) = Eρ(ξ)gy fx ≈ 1 M M∑ i=0 gy fx |ξ=ξi (5) ▶ likelihood ratio methods 異 g 勾配 情報 使 ，variance 低
- 16. Stochastic Value Gradients ▶遷移分布 s′ = f (s, a, ξ)，policy a = π(s, η; θ) reparameterize Vs = Eρ(η)[rs + raπs + γEρ(ξ)V ′ s′ (fs + faπs)] (7) Vθ = Eρ(η)[raπθ + γEρ(ξ)[V ′ s′ faπθ + γV ′ θ]] (8) = Eρ(η)[Qaπθ + γV ′ θ] ▶ MDP 確率的 ，policy 確率的 ，value gradient 求 ！（stochastic value gradient）
- 17. 復元 ▶ 遷移関数 f実際 未知 学習 ，ˆs′ = ˆf (s, a, ξ) 実際 観測 s′ 使 勾配 計算 ▶ f 予測誤差 影響 抑 ▶ 昔 θk 使 選 ak = π(sk , η; θk ) 使 ，今 θt 勾配 計算 ▶ experience replay（経験 再利用） 可能 ▶ 結果 復元 必要 ξ ∼ p(ξ|s, a, s′ ), η ∼ p(η|s, a) ▶ Gaussian 場合 η = (ak − µ(sk))/σ(sk) 求 （著者 確認 ）
- 18. 3種類 ▶ value gradient求 方 異 3 種類 提案 ▶ SVG(∞) ▶ SVG(1) ▶ SVG(0)
- 19. SVG(∞) ▶ 遷移関数 ˆf(s, a, ξ) policy π(s, η) 一緒 学習
- 20. SVG(1) ▶ 遷移関数 ˆf(s, a, ξ) policy π(s, η) ˆV (s) 一緒 学習 ▶ ˆf 1 使 残 ˆV 使 ▶ experience replay 使 場合 特 SVG(1)-ER 表記
- 21. SVG(0) ▶ policy π(s,η) ˆQ(s, a) 一緒 学習 ▶ 遷移関数 使
- 22. 評価 ▶ AC [Wawrzynski2009]，DPG [Silver et al. 2014] 既存 手法（ policy value function 学習 ） ▶ SVG(1)-ER 総 良
- 23. 悪化 場合 ▶ ˆf隠 層 次元数 減 評価 ▶ SVG(∞) 性能 大 劣化 ，SVG(1) 変
- 24. 価値関数 悪化 場合 ▶価値関数 隠 層 次元数 減 評価 ▶ DPG 性能 大 劣化 ，SVG(1) 影響
- 25. ▶ likelihood ratiomethods 代 reparameterization trick 使 ▶ 確率的 MDP，確率的 policy 対 value gradient 計算 （stochastic value gradients） ▶ 提案 実験 SVG(1)-ER 良 性能
- 26. 感想 ▶ reparameterization trick便利 ▶ likelihood ratio methods 代 使 使 ▶ 行動 離散的 reparameterization trick 使 likelihood ratio methods 頼 無 ？ ▶ SVG(0)-ER 評価 気 ▶ experience replay 重要
- 27. 参考文献 I [1] VolodymyrMnih et al. “Human-level control through deep reinforcement learning”. In: Nature 518.7540 (2015), pp. 529–533. [2] Volodymyr Mnih et al. “Playing Atari with Deep Reinforcement Learning”. In: NIPS 2014 Deep Learning Workshop. 2013, pp. 1–9. arXiv: arXiv:1312.5602v1. [3] David Silver et al. “Deterministic Policy Gradient Algorithms”. In: ICML 2014. 2014, pp. 387–395. [4] Richard S. Sutton et al. “Policy Gradient Methods for Reinforcement Learning with Function Approximation”. In: In Advances in Neural Information Processing Systems 12. 1999, pp. 1057–1063. [5] Pawel Wawrzynski. “Real-time reinforcement learning by sequential Actor-Critics and experience replay”. In: Neural Networks 22.10 (2009), pp. 1484–1497. [6] RJ Williams. “Simple statistical gradient-following algorithms for connectionist reinforcement learning”. In: Reinforcement Learning 8.3-4 (1992), pp. 229–256.