This document appears to be a presentation on reinforcement learning. It includes sections on reinforcement learning concepts like the reinforcement learning loop, Markov decision processes, value functions, optimal policies, Q-learning, SARSA, and temporal difference learning. Diagrams and mathematical equations are provided to explain these concepts. Examples of reinforcement learning applications in Python and popular algorithms like Deep Q-Network and AlphaGo are also mentioned.

1. Ryohei Saito
May 22, 2017
レイ・フロンティア株式会社

2. Contents
1
2
3
4
Ryohei Saito May 22, 2017 2 / 21

3. (Reinforcement Learning, RL)
Figure:
Ryohei Saito May 22, 2017 3 / 21

4. (Supervised Learning)
(Unsupervised Learning)
etc...
Ryohei Saito May 22, 2017 4 / 21

5. John Broadus Watson
Ryohei Saito May 22, 2017 5 / 21

6. Ryohei Saito May 22, 2017 6 / 21

7. Ryohei Saito May 22, 2017 7 / 21

8. Ryohei Saito May 22, 2017 8 / 21

9. 3
Ryohei Saito May 22, 2017 9 / 21

10. Ryohei Saito May 22, 2017 10 / 21

11. (Agent)
(Environment) (Aciton)at
(Reward)rt+1
(State) st st+1
· · · · · ·
Ryohei Saito May 22, 2017 11 / 21

12. (Markov
Decision Process: MDP)
S = {s1, s2, . . . , sN }
A(s) = {a1, a2, . . . , aM }
P0
P(s′|s, a)
r(s, a, s′)
Ryohei Saito May 22, 2017 12 / 21

13. s a π(s|a) (policy)
(return, income)
Gt =
T−1
τ=0
rt+1+τ
Gt = lim
T→∞
1
T
T−1
τ=0
rt+1+τ
Gt =
∞
τ=0
γτ
rt+1+τ
Ryohei Saito May 22, 2017 13 / 21

14. V π
(s) = Eπ
[Gt|st = s]
Qπ
(s, a) = Eπ
[Gt|st = s, at = a]
V π Qπ s, a π∗
(optimal policy)
( ) ( )
V ∗
(s) = V π∗
(s) = max
π
V π
(s) , ∀
s ∈ S
Q∗
(s, a) = Qπ∗
(s, a) = max
π
Qπ
(s, a) , ∀
s ∈ S , ∀
a ∈ A
Ryohei Saito May 22, 2017 14 / 21

15. Q∗(s, a)
a r s
s, a, r
3
Q∗(s, a) π∗
Ryohei Saito May 22, 2017 15 / 21

16. greedy
π(a|s) =
⎧
⎨
⎩
1 (a = argmax
a∈A
Q(s, a))
0 (otherwise)
Q
( ,
exploration-exploitation tradeoff
Ryohei Saito May 22, 2017 16 / 21

17. ε-greedy
π(a|s) =
⎧
⎨
⎩
1 − ε + ε
|A(s)| (a = argmax
a∈A
Q(s, a))
ε
|A(s)| (otherwise)
soft-max
π(a|s) =
exp(Q(s, a)/T)
b∈A exp(Q(s, b)/T)
Ryohei Saito May 22, 2017 17 / 21

18. Bellman
V π, Qπ
Bellman
V π
(s) =
a∈A(s)
π(a|s)
s′∈S
P(s′
|s, a)(r(s, a, s′
) + γV π
(s′
))
Qπ
(s, a) =
s′∈S
P(s′
|s, a)
⎛
⎝r(s, a, s′
) +
a′∈A(s)
γπ(a′
|s′
)Qπ
(s′
, a′
)
⎞
⎠
SARSA
Q(st, at) ← (1 − α)Q(st, at) + α(rt+1 + γQ(st+1, at+1))
Bellman
Ryohei Saito May 22, 2017 18 / 21

19. Bellman Q-Learning
Q∗
Bellman
Q∗
(s, a) =
s′∈S
P(s′
|s, a) r(s, a, s′
) + γ max
a′∈A
Q∗
(s′
, a′
)
Q(st, at) ← (1 − α)Q(st, at) + α rt+1 + γ max
a′∈A
Q(st+1, a′
)
Q-Learning
Q-Learning
Ryohei Saito May 22, 2017 19 / 21

20. Q(λ),TD(λ), actor-critic, . . .
python Open AI Gym
Ng
Deep Q-Network, Alpha Go
Ryohei Saito May 22, 2017 20 / 21

21. ,
, 2008.
, ,
, 2016.
M. Mohri, A. Rostamizadeh, A. Talwalkar
Foundations of Machine Learning
The MIT Press, 2012.
R. S. Sutton, A. G. Barto, ,
, 2000.
Ryohei Saito May 22, 2017 21 / 21