The document describes reinforcement learning algorithms. It defines the reinforcement learning framework using a 5-tuple (S, A, T, R, γ) representing states, actions, transitions, rewards, and discount factor. It presents the Bellman equations for calculating the optimal state-action value function Q* and state value function V* using recursive updates. It also describes the iterative policy evaluation algorithm which uses the Bellman equation to calculate the value function Vk at each iteration k.

6. (S, A, T, R, γ)
at = π(st)
st
T(s, a) = st+1
rt+1

7. Q*(s, a) =
∑
s′
Pa
(s, s′)
{
Ra
(s, s′) + max
a′
γQ*(s′, a′)
}
V*(s) = max
a ∑
s′
Pa
(s, s′){Ra
(s, s′) + γV*(s′)}

10. Q*(s, a) =
∑
s′
Pa
(s, s′)
{
Ra
(s, s′) + max
a′
γQ*(s′, a′)
}
V*(s) = max
a ∑
s′
Pa
(s, s′){Ra
(s, s′) + γV*(s′)}

11. Vk
(s) = max
a ∑
s′
Pa
(s, s′){Ra
(s, s′) + γVk−1
(s′)}
Vk−1
(s′)
Vk
(s)
Pa
(s, s′)
= max
a Ra
(s, s′)

12. ¯ri,j
vk
i,j
p(v)
a,i′−i,j′−j
p(r)
a,i′−i,j′−j
qk
a,i,j

17. pi,j
¯rin
i,j, ¯rout
i,j
¯rout
i,j = ¯rin
i,j

18. ¯ri,j + pi,j(vk−1
i′,j′ − ri,j) = pi,jvk−1
i′,j′ + ¯ri,j(1 − pi,j)