東京大学松尾研究室 強化学習サマースクール2020 第5回 https://deeplearning.jp/reinforcement_cource-2020s/

1. TA
Control as Inference
5

2. Control as Inference
(POMDP)

3. …

4. …

5. …
???
???
???

6. etc.

8. …

9. ‣
‣ MDP (POMDP)

10. Control as Inference
(POMDP)

12. x1, …, xN ∼ p (X)

13. p (X) θ p (X ∣ θ)
p (X = k ∣ θ) = μk
θ(1 − μθ)1−k
μθ 1 − μθ
μθ

14. 1.
e.g.,
2.
e.g., 0.5
➡ …
p (X ∣ θ)
μθ

15. 1.
e.g.,
2.
e.g., 0.5
➡ …
p (X ∣ θ)
μθ

16. θ
N
x

17. N
x
y θ
/
N
z
x
θ

18. N
x
y θ
/
N
z
x
θ

19. DNN
Y
p (Y ∣ X, θ) = Normal (fθ (X), Σ)
fθ
N
x
y θ

20. DNN
Y
p (Y = k ∣ X, θ) =
exp (fθ (X)[k])
∑
K
k′=1
exp (fθ (X)[k′])
fθ
N
x
y θ

21. VAE ( )
Z
p (X, Z ∣ θ) = p (Z ∣ θ) p (X ∣ Z, θ)
θ Z
N
z
x
θ

23. Maximum Likelihood Estimation (MLE)
̂θ = argmax
θ
N
∏
i=1
p (X = xi ∣ θ)

24. Maximum a Posteriori Estimation (MAP)
MLE
p (θ)
̂θ = argmax
θ
p (θ ∣ X = x1, …, xN)
= argmax
θ
p (θ)
N
∏
i=1
p (X = xi ∣ θ)
p (θ) = const .

25. Bayesian Inference
1
p (X ∣ x1, …, xN) = 𝔼p(θ ∣ X = x1, …, xN) [p (X ∣ θ)]

26. MLE/MAP
exp
−log p (x, θ)
θ
p (θ ∣ x)
θ

27. 1 MLE/MAP

28. p (θ ∣ X = x1, …, xN)

30. x1, …, xN x
p (X, θ) = p (θ) p (X ∣ θ)
p (θ ∣ X = x) p (θ ∣ x)

31. (MCMC)
qϕ (θ)
p (θ ∣ x)
p (θ ∣ x)

32. Variational Inference
Kullback–Leibler divergenceqϕ (θ) p (θ ∣ x)
p (θ ∣ x) ≈ ̂qϕ (θ) = argmin
qϕ
KL (qϕ (θ) ∥ p (θ ∣ x))
qϕ (θ) Normal
(
μϕ, diag (σ2
ϕ))
ϕ = {μϕ, σ2
ϕ}

33. Variational Inference
( )
KL (qϕ (θ) ∥ p (θ ∣ x)) =
∫
qϕ (θ) log
qϕ (θ)
p (θ ∣ x)
dΘ
= 𝔼qϕ
[
log
qϕ (θ)
p (x, θ) ]
+ log p (x)
log p (x) qϕ ℒϕ (x)
ℒϕ (x) ℒϕ (x) ≤ log p (x)
−ℒϕ (x)

34. Reparameterization Gradient
ℒϕ (x) ϕ
qϕ
∇ϕℒϕ (x) = − ∇ϕ 𝔼qϕ
[
log
qϕ (θ)
p (x, θ) ]

35. Reparameterization Gradient
qϕ (θ) = Normal
(
μϕ, diag (σ2
ϕ))
𝔼qϕ
[
log
qϕ (θ)
p (x, θ) ]
= 𝔼p(ϵ) log
qϕ (θ)
p (x, θ)
θ=f(ϵ, ϕ)
p (ϵ) = Normal (0, I), f (ϵ, ϕ) = μϕ + σϕ ⊙ ϵ

36. Reparameterization Gradient
∇ϕ 𝔼qϕ
[
log
qϕ (θ)
p (x, θ) ]
= 𝔼p(ϵ) ∇ϕlog
qϕ (θ)
p (x, θ)
Θ=f(ϵ, ϕ)
≈
1
L
L
∑
l=1
∇ϕlog
qϕ (θ)
p (x, θ)
θ=f(ϵ(l), ϕ)
ϵ(1)
, ⋯, ϵ(L)
∼ p (ϵ)

37. Reparameterization Gradient
1.
‣
‣ ( ) http://blog.shakirm.com/2015/10/machine-learning-trick-of-the-day-4-
reparameterisation-tricks/
2.
qϕ f
ϕ

38. MLE/MAP
MAP
0
MLE
δ (θ − μϕ)
δ (θ − μϕ) = lim
σ2
→0
Normal (μϕ, diag (σ2
))
p (θ) = const .
δ (x)
https://commons.wikimedia.org/wiki/File:Dirac_distribution_PDF.png

39. Amortized Variational Inference
z1:N
qϕ (Z1:N) =
N
∏
i=1
qϕi (Zi)
N
zi
ϕi
N
z
x
θ

40. Amortized Variational Inference
qϕ (Z1:N) =
N
∏
i=1
qϕ (Zi ∣ fϕ (xi))
xi fϕ
ϕ
z
N
z
x
θ

41. Amortized Variational Inference
DNN
qϕ (Z) =
N
∏
i=1
Normal
(
μϕ (xi), diag (σ2
ϕ (xi)))
μϕ, σ2
ϕ
N
z
x
θ

42. Variational Autoencoder (VAE)
DNN
Autoencoder
p (X ∣ z, θ) =
N
∏
i=1
Normal
(
μθ (zi), diag (σ2
θ (zi)))
qϕ (Z) =
N
∏
i=1
Normal
(
μϕ (xi), diag (σ2
ϕ (xi)))
μθ, σ2
θ μϕ, σ2
ϕ
qϕ
N
z
x
θ

43. e.g.,

44. Markov Chain Monte Carlo (MCMC)
p (θ ∣ x)
p (θ ∣ x) ≈
1
T
T
∑
T=1
δ (θ − θ(t)
)
θ(1)
, …, θ(T)
∼ p (θ ∣ x)

45. Markov Chain Monte Carlo (MCMC)
1.
2.
3. 2
θ(0)
θ(t+1)
∼ p (θ′ ∣ θ = θ(t)
)
T {θ(1)
, …, θ(T)
}

46. Langevin Dynamics
MCMC
pβ
(θ′ ∣ θ) = Normal
(
θ + η
∂
∂θ
log p (x, θ), 2ηβ−1
I
)
η → 0 pβ
(θ ∣ x) = (p (θ ∣ x))
β
β = 1 p (θ ∣ x)

47. Langevin Dynamics
https://upload.wikimedia.org/wikipedia/commons/0/0d/First_passage_time_in_double_well_potential_under_langevin_dynamics.gif
−log p (x, θ)

48. MLE/MAP
MAP
MLE
β → ∞
lim
β→∞
pβ
(θ′ ∣ θ) = δ
(
θ′−
(
θ + η
∂
∂θ
log p (x, θ)
))
p (θ) = const .

49. MCMC

50. •
• MCMC

51. Control as Inference
(POMDP)

53. st π at
st+1 r (st, at)
∞
∑
t=1
r (st, at) π
※

55. Action-Value Function (Q-function)
st at π
Qπ
(st, at) = r (st, at) + 𝔼π
[
∞
∑
k=1
r (st+k, at+k)
]
※

56. Optimal Action-Value Function (Optimal Q-function)
st at
Q* (st, at) = r (st, at) + max
a
∞
∑
k=1
r (st+k, at+k)
= max
π
Qπ
(st, at)
※

57. (State) Value Function
st π
Vπ
(st) = 𝔼π
[
∞
∑
k=0
r (st+k, at+k)
]
= 𝔼π [Qπ
(st, at)]
※

58. Optimal (State) Value Function
st
V* (st) = max
a
∞
∑
k=0
r (st+k, at+k)
= max
π
Vπ
(st)
= max
a
Q* (st, at)
※

59. Bellman Equation
Qπ
(st, at) = r (st, at) + Vπ
(st+1)
Vπ
(st) = 𝔼π [r (st, at)] + Vπ
(st+1)
※

60. Bellman Optimality Equation
Q* (st, at) = r (st, at) + V* (st+1)
V* (st) = max
a
r (st, a) + V* (st+1)
※

62. Q
Q-learning
(greedy )
Q (st, at) ← Q (st, at) + η
[
r (st, at) + max
a
Q (st+1, a) − Q (st, at)]
π (s) = argmax
a
Q (s, a)
※

63. Q +
Q-learning + Function Approximation
(e.g., )
DNN (e.g., DQN)
Qθ
θ ← θ − η∇θ 𝔼
[
r (st, at) + max
a
Qθ (st+1, a) − Qθ (st, at)
2
]
Qθ
※

64. Policy Gradient (REINFORCE)
DNN
πϕ (a ∣ s)
πϕ (a ∣ s) = Normal
(
μϕ (s), diag (σ2
ϕ (s)))
μϕ, σ2
ϕ
※

65. Policy Gradient (REINFORCE)
θ
ϕ ← ϕ + η∇ϕ 𝔼πϕ
[
T
∑
t=1
r (st, at)
]
∇ϕ 𝔼πϕ
[
T
∑
t=1
r (st, at)
]
= 𝔼πϕ
[
T
∑
t=1
r (st, at)
T
∑
t=1
∇ϕlog πϕ (at ∣ st)
]
※

66. Actor-Critic
Q
πϕ
θ
πϕ
ϕ ← ϕ + ηϕ ∇ϕ 𝔼πϕ [Q
πϕ
θ
(s, a)]
θ ← θ − ηθ ∇θ 𝔼
[
r (st, at) + V
πϕ
θ (st+1) − Q
πϕ
θ (st, at)
2]
V
πϕ
θ
(s) = 𝔼πϕ [Q
πϕ
θ
(s, a)]
※

67. Q Q
or Actor-Critic
※
(e.g., Qt-opt Q )

68. vs
On-policy vs Off-policy
(on-policy)
(e.g. , )
(off-policy)
(e.g., Q )

70. Maximum Entropy Reinforcement Learning (MERL)
∞
∑
t=1
r (st, at) + ℋ (π (at ∣ st))
※

71. Soft Actor-Critic
Actor-Critic
ϕ ← ϕ + ηϕϕ
∇ϕ 𝔼πϕ [Q
πϕ
θ
(s, a)−log πϕ (a ∣ s)]
θ ← θ − ηθ ∇θ 𝔼
[
r (st, at) + V
πϕ
θ (st+1) − Q
πϕ
θ (st, at)
2]
V
πϕ
θ
(s) = 𝔼π [Q
πϕ
θ
(s, a)−log πϕ (at ∣ st)]
※
https://arxiv.org/abs/1801.01290

72. Soft Actor-Critic
Actor-Critic
➡ Actor-Critic on-policy
πϕ 𝔼πϕ [Q
πϕ
θ (st, at)]
πϕ

73. Soft Actor-Critic
SAC
KL divergence
➡ SAC off-policy
𝔼πϕ [Q
πϕ
θ
(s, a)−log πϕ (a ∣ s)]
πϕ ̂π (a ∣ s) ∝ exp (Qπ
ϕ (s, a))
KL (πϕ ∥ ̂π) = − 𝔼πϕ [Q
πϕ
θ
(s, a)−log πϕ (a ∣ s)] + log
∫
exp (Q
πϕ
θ
(s, a)) da

74. • Q
•
• Actor-Critic

75. Control as Inference
(POMDP)

76. Control as Inference

78. Markov Decision Process (MDP)
N
st st+1
at
rt
at+1
rt+1
••••••

79. Markov Decision Process (MDP) + Optimality Variables
N
st st+1
ot
at
rt
ot+1
at+1
rt+1
••••••

80. Optimality Variable
‣
s a
O = 1 O = 0
r O
p (O = 1 ∣ r) ∝ exp (r (s, a))

81. 2
1.
2.
p (s1:T, a1:T ∣ O1:T = 1)
s1:T, a1:T
p (at ∣ st, O≥t = 1)

82. ➡
➡ p (s1:T, a1:T ∣ O1:T = 1) p (at ∣ st, O≥t = 1)
Ot = 1 ot

83. p (at ∣ st, o≥t) ∝ p (at ∣ st) p (o≥t ∣ st, at)
p (at ∣ st)
p (at ∣ st, o≥t) ∝ p (o≥t ∣ st, at)

84. Q* (st, at) = log p (o≥t ∣ st, at), V* (st) = log p (o≥t ∣ st)
Q* (st, at) = log p (ot ∣ st, at) + log p (o≥t+1 ∣ st, at)
= r (st, at) + log
∫
p (st+1 ∣ st, at) p (o≥t+1 ∣ st+1) dst+1
= r (st, at) + log 𝔼p(st+1 ∣ st, at) [
exp (V* (st+1))]
※

85. i.e.,
Q* (st, at) = r (st, at) + log 𝔼p(st+1 ∣ st, at) [
exp (V* (st+1))]
p (st+1 ∣ st, at) = δ (st+1 − f (st, at))
Q* (st, at) = r (st, at) + V* (st+1)
V* (s) = log
∫
exp (Q* (s, a)) da ≠ max Q* (s, a)
※

86. p (s1:T, a1:T ∣ o1:T) p (at ∣ st, o≥t)
Q* (st, at) = log p (o≥t ∣ st, at),
V* (st) = log p (o≥t ∣ st)
Q* (st, at) = r (st, at) + log 𝔼p(st+1 ∣ st, at) [
exp (V* (st+1))]
※

87. 2
1.
2.
p (s1:T, a1:T ∣ o1:T)
s1:T, a1:T
p (at ∣ st, o≥t)

89. ➡
p (s1:T, a1:T ∣ o1:T) ∝ p (s1)
T
∏
t=1
p (st+1 ∣ st, at) exp (r (st, at))

90. qϕ (s1:T, a1:T) = p (s1)
T
∏
t=1
p (st+1 ∣ st, at) πϕ (at ∣ st)
πϕ (a ∣ s) = Normal
(
μϕ (s), diag (σ2
ϕ (s)))
μϕ, σ2
ϕ ϕ s

91. KL divergenceqϕ (s1:T, a1:T) p (s1:T, a1:T ∣ o1:T)
KL (qϕ (s1:T, a1:T) ∥ p (s1:T, a1:T ∣ o1:T))
= 𝔼qϕ
[
log
qϕ (s1:T, a1:T)
p (s1:T, a1:T ∣ o1:T) ]
= 𝔼qϕ
[
T
∑
t=1
log πϕ (at ∣ st) − r (st, at)
]
+ log p (o1:T)

92. ∇ϕ 𝔼qϕ
[
T
∑
t=1
r (st, at)
]
= 𝔼qϕ
[
T
∑
t=1
r (st, at)∇ϕlog qϕ (s1:T, a1:T)
]
= 𝔼qϕ
[
T
∑
t=1
r (st, at)
T
∑
t=1
∇ϕlog πϕ (at ∣ st)
]

93. ➡
𝔼qϕ
[
T
∑
t=1
log πϕ (at ∣ st) − r (st, at)
]

94. 2
1.
2.
p (s1:T, a1:T ∣ o1:T)
s1:T, a1:T
p (at ∣ st, o≥t)

96. p (at ∣ st, o≥t) ∝ exp (Q* (st, at)) Q*
p (at ∣ st, o≥t) =
exp (Q* (st, at))
∑a∈A
exp (Q* (st, a))

97. ➡
p (at ∣ st, o≥t) =
exp (Q* (st, at))
∫ exp (Q* (st, a)) da

98. p (at ∣ st, o≥t) πϕ (at ∣ st)
πϕ (a ∣ s) = Normal
(
μϕ (s), diag (σ2
ϕ (s)))
μϕ, σ2
ϕ ϕ s

99. KL divergenceπϕ (at ∣ st) p (at ∣ st, o≥t)
KL (πϕ (at ∣ st) ∥ p (at ∣ st, o≥t))
= 𝔼πϕ
[
log
πϕ (at ∣ st)
p (at ∣ st, o≥t)]
= 𝔼πϕ [log πϕ (at ∣ st) − Q* (st, at)] + V* (st)

100. KL (πϕ (at ∣ st) ∥ p (at ∣ st, o≥t))
= 𝔼πϕ [log πϕ (at ∣ st)−Q* (st, at)] + V* (st)
Q* (st, at)

101. Q* (st, at) = r (st, at) + log 𝔼p(st+1 ∣ st, at) [
exp (V* (st+1))]
V* (s) = log
∫
exp (Q* (s, a)) da
V*
※

102. 1.
Soft Q-learning
V* (s) = log 𝔼πϕ
[
exp (Q* (s, a))
πϕ (a ∣ s) ]
≈ log
1
L
L
∑
l=1
exp (Q* (s, a(l)
))
πϕ (a(l) ∣ s)
a1, …aL ∼ πϕ (a ∣ s)
L → ∞ V* (s)
※

103. 2.
Q* (st, at) = r (st, at) + log 𝔼p(st+1 ∣ st, at) [
exp (V* (st+1))]
≥ r (st, at) + log 𝔼p(st+1 ∣ st, at) [
exp (Vπϕ
(st+1))]
= Qπϕ
(st, at)
※

104. 2.
V*(s) = log 𝔼πϕ
[
exp (Q*(s, a))
πϕ(a ∣ s) ]
≥ 𝔼πϕ [Q*(s, a) − log πϕ(a ∣ s)]
≥ 𝔼πϕ [Qπϕ(s, a) − log πϕ(a ∣ s)]
= Vπϕ(s)
※

105. 2.
➡
Soft Actor-Critic
Qπϕ, Vπϕ Q*, V* πϕ (at ∣ st) = p (at ∣ st, o≥t)
Qπϕ, Vπϕ Q*, V*
※

106. Qπϕ Q
πϕ
θ
θ ← θ − ηθ ∇θ 𝔼
[
r (st, at) + V
πϕ
θ (st+1) − Q
πϕ
θ (st, at)
2]
V
πϕ
θ
(s) = 𝔼πϕ [Q
πϕ
θ
(s, a) − log πϕ (a ∣ s)]
V
πϕ
θ
πϕ
※

107. Soft Actor-Critic
πϕ (at, st) ̂π (a ∣ s) ∝ exp (Q
πϕ
θ
(s, a))
KL (πϕ (at ∣ st) ∥ ̂π (at ∣ st))
= 𝔼πϕ [log πϕ (at ∣ st) − Q
πϕ
θ (st, at)] + log
∫
exp (Q
πϕ
θ
(s, a)) da

108. Soft Actor-Critic (SAC)
SAC off-policy
1

109. On-policy Off-policy
➡ On-policy
➡ Off-policy
(st, at, rt, st+1)

110. Control as Inference
(POMDP)

112. MDP
MDP
MDP
s
?
?
CartPole

113. MDP
DQN MDP
‣ 4
➡
Partially Observable Markov Decision Process (POMDP)

114. Partially Observable Markov Decision Process (POMDP)
N
xt
at
rt
••••••
st
xt+1
at+1
rt+1
st+1

115. POMDP + Optimality Variables
N
xt
ot
at
rt
••••••
st
xt+1
ot+1
at+1
rt+1
st+1

116. POMDP
POMDP p (at ∣ st, o≥t)
x s p (st ∣ xt, st−1, at−1)
p (s≤t, at ∣ x≤t, a<t, o≥t)
= p (at ∣ st, o≥t) p (s1 ∣ x1)
t
∏
τ=1
p (sτ+1 ∣ xτ+1, sτ, aτ)

117. p (s≤t, at ∣ x≤t, a<t, o≥t)
qϕ (s≤t, at ∣ x≤t, a<t)
= πϕ (at ∣ st) qϕ (s1 ∣ x1)
t
∏
τ=1
qϕ (sτ+1 ∣ xτ+1, sτ, aτ)

118. KL divergence
KL (qϕ (s≤t, at ∣ x≤t, a<t) ∥ p (s≤t, at ∣ x≤t, a<t, o≥t))
= 𝔼qϕ
[
log
qϕ (s≤t, at ∣ x≤t, a<t)
p (s≤t, at ∣ x≤t, a<t, o≥t)]
= 𝔼qϕ
[
log πϕ (at ∣ st) + log
qϕ (s1 ∣ x1)
p (x1, s1)
+
t
∑
τ=1
log
qϕ (sτ+1 ∣ xτ+1, sτ, aτ)
p (xτ+1, sτ+1 ∣ sτ, aτ)
− Q* (st, at)
]
+log p (x≤t ∣ a<t) + V* (st)
−ℒϕ (x≤t, a<t, o≥t)

119. KL divergence
KL (qϕ (s≤t, at ∣ x≤t, a<t) ∥ p (s≤t, at ∣ x≤t, a<t, o≥t))
= 𝔼qϕ
[
log
qϕ (s≤t, at ∣ x≤t, a<t)
p (s≤t, at ∣ x≤t, a<t, o≥t)]
= 𝔼qϕ
[
log πϕ (at ∣ st) + log
qϕ (s1 ∣ x1)
p (x1, s1)
+
t
∑
τ=1
log
qϕ (sτ+1 ∣ xτ+1, sτ, aτ)
p (xτ+1, sτ+1 ∣ sτ, aτ)
− Q* (st, at)
]
+log p (x≤t ∣ a<t) + V* (st)
−ℒϕ (x≤t, a<t, o≥t)

120. KL divergence
KL (qϕ (s≤t, at ∣ x≤t, a<t) ∥ pψ (s≤t, at ∣ x≤t, a<t, o≥t))
= 𝔼qϕ
[
log
qϕ (s≤t, at ∣ x≤t, a<t)
pψ (s≤t, at ∣ x≤t, a<t, o≥t) ]
= 𝔼qϕ
[
log πϕ (at ∣ st) + log
qϕ (s1 ∣ x1)
pψ (x1, s1)
+
t
∑
τ=1
log
qϕ (sτ+1 ∣ xτ+1, sτ, aτ)
pψ (xτ+1, sτ+1 ∣ sτ, aτ)
− Q* (st, at)
]
+log pψ (x≤t ∣ a<t) + V* (st)
➡
−ℒϕ,ψ (x≤t, a<t, o≥t)

121. log pψ (x≤t ∣ a<t) + V* (st) ≥ ℒϕ,ψ (x≤t, a<t, o≥t)
qϕ (s≤t, at ∣ x≤t, a<t) = pψ (s≤t, at ∣ x≤t, a<t, o≥t)

122. qϕ (s≤t, at ∣ x≤t, a<t) = pψ (s≤t, at ∣ x≤t, a<t, o≥t)
argmax
ψ
ℒϕ,ψ (x≤t, a<t, o≥t) = argmax
ψ
pψ (x≤t ∣ a<t)
ψ ℒϕ,ψ (x≤t, a<t, o≥t)

123. SAC
Q* (st, at) ≥ r (st, at) + log 𝔼p(st+1 ∣ st, at) [
exp (Vπϕ
(st+1))]
= Qπϕ
(st, at) ≈ Q
πϕ
θ (st, at)
V*(s) ≥ 𝔼πϕ [Qπϕ(s, a) − log πϕ(a ∣ s)]
= Vπϕ(s) ≈ V
πϕ
θ
(s)

124. Stochastic Latent Actor-Critic (SLAC)
̂θ = argmin 𝔼
[
r (st, at) + V
πϕ
θ (st+1) − Q
πϕ
θ (st, at)
2]
̂ϕ, ̂ψ = argmax
ϕ,ψ
ℒϕ,ψ (x≤t, a<t, o≥t)

125. POMDP
POMDP
Stochastic Latent Actor-Critic (SLAC) SAC POMDP
p (at ∣ st, o≥t)
p (st ∣ xt, st−1, at−1)
pψ (xt+1, st+1 ∣ st, at)

126. ➡ Control as Inference (or )
(Bayesian RL)

127. POMDP
➡

128. Control as Inference
(POMDP)

130. POMDP
➡
≒ POMDP (+ )
pψ (xt+1, st+1 ∣ st, at)

131. RL
1.
2.
3.
1 ~ 3
π D
D = {x1, a1, r1, …, xT, aT, rT}
D pψ
pψ (x1:T, r1:T ∣ a1:T)
π https://arxiv.org/abs/1903.00374

132. RL
1.
2.
➡

133. RL
1.
2.

134. RL
1.
2.
3.
1 ~ 3
π D
D = {x1, a1, r1, …, xT, aT, rT}
D pψ
pψ (x1:T, r1:T ∣ a1:T)
π https://arxiv.org/abs/1903.00374

136. Partially Observable Markov Decision Process
N
xt
at
rt
••••••
st
xt+1
at+1
rt+1
st+1

137. log pψ (x1:T, r1:T ∣ a1:T)
= log
∫
p (s1)
T
∏
t=1
pψ (st+1 ∣ st, at) pψ (rt ∣ st, at) pψ (xt ∣ st) ds1:T
= log 𝔼qϕ
[
pψ (s1)
qϕ (s1 ∣ x1)
T
∏
t=1
pψ (st+1 ∣ st, at) pψ (rt ∣ st, at) pψ (xt ∣ st)
qϕ (st+1 ∣ xt+1, rt, st, at) ]
≥ 𝔼qϕ
[
log
pψ (s1)
qϕ (s1 ∣ x1)
+
T
∑
t=1
log
pψ (st+1 ∣ st, at) pψ (rt ∣ st, at) pψ (xt ∣ st)
qϕ (st+1 ∣ xt+1, rt, st, at) ]
= ℒϕ,ψ (x1:T, r1:T, a1:T)

138. log pψ (x1:T, r1:T ∣ a1:T) ≥ ℒϕ,ψ (x1:T, r1:T, a1:T)
qϕ (s1:T ∣ x1:T, r1:T, a1:T) = pψ (s1:T ∣ x1:T, r1:T, a1:T)

139. qϕ (s1:T ∣ x1:T, r1:T, a1:T) = pψ (s1:T ∣ x1:T, r1:T, a1:T)
argmax
ψ
ℒϕ,ψ (x1:T, r1:T, a1:T) = argmax
ψ
pψ (x1:T, r1:T ∣ a1:T)
ψ ℒϕ,ψ (x1:T, r1:T, a1:T)

140. RL
1.
2.
3.
1 ~ 3
π D
D = {x1, a1, r1, …, xT, aT, rT}
D pψ
pψ (x1:T, r1:T ∣ a1:T)
π https://arxiv.org/abs/1903.00374

142. 1. (Model Predictive Control,MPC)
1.
2.
3.
a(1)
t:T
, a(2)
t:T
, ⋯, a(K)
t:T
R (a(k)
t:T ) = 𝔼pψ
[
T
∑
τ=t
rψ (sτ, a(k)
τ )]
at = a
̂k
t
(
̂k = argmax
k
R (a(k)
t:T ))

143. 1. (Model Predictive Control,MPC)
MPC 3
• Random-sample Shooting (RS)
MPC
• Cross Entropy Method (CEM)

144. 2.
ϕ ← ϕ + η∇ϕ 𝔼pψ,πϕ
[
T
∑
t=1
rψ (st, at)
]

145. 2.
rψ
∇ϕ 𝔼pψ,πϕ
[
T
∑
t=1
rψ (st, at)
]
= 𝔼p(ϵ)
[
T
∑
t=1
∇ϕrψ (st = fψ (st−1, at−1, ϵ), at = fϕ (st, ϵ))]

146. 2.
∇ϕ 𝔼pψ,πϕ
[
T
∑
t=1
rψ (st, at)
]
= 𝔼pψ,πϕ
[
T
∑
t=1
rψ (st, at)
T
∑
t=1
∇ϕlog πϕ (at ∣ st)
]

147. 3.Actor-Critic
ϕ ← ϕ + ηϕ ∇ϕ 𝔼pψ,πϕ [V
πϕ
θ
(s)]
θ ← θ − ηθ ∇θ 𝔼pψ,πϕ [
rψ (st, at) + V
πϕ
θ (st+1) − Q
πϕ
θ (st, at)
2]
V
πϕ
θ
(s) = 𝔼πϕ [Q
πϕ
θ
(s, a)]

149. World Models
[Ha and Schmidhuber,2018]
VAE + MDN-RNN
CMA-ES
https://www.slideshare.net/masa_s/ss-97848402
https://arxiv.org/abs/1803.10122
https://worldmodels.github.io/

150. [Hafner,et al.,2019]
Recurrent State Space Model ( )
CEM
PlaNet
DM Control Suite
https://arxiv.org/abs/1811.04551
https://planetrl.github.io/

151. Gaussian State Space Model
DNN
pψ (st+1 ∣ st, at)
= Normal
(
μψ (st, at), diag (σ2
ψ (st, at)))
μψ, σ2
ψ
xt
at
rt
st
xt+1
at+1
rt+1
st+1

152. Recurrent State Space Model (RSSM)
LSTM RNN
s h
z
ht+1 = fψ (ht, zt, at)
pψ (zt ∣ ht) = Normal
(
μψ (ht), diag (σ2
ψ (ht)))
fψ
xt
at
rt
zt
xt+1
at+1
rt+1
zt+1
ht ht+1

153. RSSM
Recurrent State Space Model (RSSM)

154. [Hafner,et al.,2019]
PlaNet
Actor-Critic
( )
PlaNet
λ
Dreamer
https://arxiv.org/abs/1912.01603
https://ai.googleblog.com/2020/03/introducing-dreamer-scalable.html

155. Vπ
(st) = 𝔼π [r (st, at)] + Vπ
(st+1)
n
Vπ
n (st) = 𝔼π
[
n−1
∑
k=1
r (st+k, at+k)
]
+ Vπ
(st+n)

156. 2
Vπ
n (st) = 𝔼π
[
n−1
∑
k=1
r (st+k, at+k)
]
+ Vπ
(st+n)
n = 1,…, ∞
¯Vπ
(st, λ) = (1 − λ)
∞
∑
n=1
λn−1
Vπ
n (st)
λ

157. Dreamer λ
θ ← θ − ηθ ∇θ 𝔼pψ,πϕ [
V
πϕ
θ (st) − ¯Vπ
(st, λ)
2]
H
¯Vπ
(st, λ) ≈ (1 − λ)
H−1
∑
n=1
λn−1
Vπ
n (st) + λH−1
Vπ
H (st)

158. λ
No value
λ H

159. ≒ POMDP (+ )
RL
DNN

160. https://www.kspub.co.jp/book/detail/1538320.html
https://www.kspub.co.jp/book/detail/5168707.html
https://www.coronasha.co.jp/np/isbn/9784339024623/

161. Control as Inference
Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
https://arxiv.org/abs/1805.00909
UC Berkeley Deep RL course ( 14 )
http://rail.eecs.berkeley.edu/deeprlcourse-fa19/

162. Control as Inference
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a
Stochastic Actor https://arxiv.org/abs/1801.01290
Reinforcement Learning with Deep Energy-Based Policies
https://arxiv.org/abs/1702.08165
Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable
Model https://arxiv.org/abs/1907.00953

163. World Models
https://arxiv.org/abs/1803.10122
Learning Latent Dynamics for Planning from Pixels
https://arxiv.org/abs/1811.04551
Dream to Control: Learning Behaviors by Latent Imagination
https://arxiv.org/abs/1912.01603

164. Dreamer