This document provides an overview of reinforcement learning and AlphaZero. It discusses the math behind reinforcement learning concepts like policy iteration, policy improvement, and policy evaluation. It then explains how AlphaZero uses these concepts along with a deep neural network and self-play to master the game of Go without human data. Key algorithms discussed include Monte Carlo tree search and how AlphaZero implements them in code to learn directly from games played between copies of itself.

1. [[ ] t0 3l0] t0 3l0αα00

2. OutlineOutline
1. The math behind Go
2. From Crazy Stone -> AlphaGO
3. AlphaGo vs AlphaZero
4. Policy Iteration
5. Policy Improvement (Math alert!)
6. Policy Evaluation
7. The deep side of AlphaZero
8. Code and demo

4. "For a true AI isn't measured by the size of its tree, but by
the precision of its moves." Filottete

6. Go is constructive
Humans describe more as intuititive game
possible states
possible games for each starting state
10
170
10
360

7. Adversarial
Fully observable
Deterministic

8. AI in GoAI in Go

9. "The mystery of Go, the ancient game that computers still
can't win" - Wired 2014

10. CrazyStoneCrazyStone

14. AlphaGo Zero vs AlphaZeroAlphaGo Zero vs AlphaZero

15. Reinforcement LearningReinforcement Learning

17. ExampleExample

20. NoticeNotice

21. The agent de nes the part of the world that wants to
explore
And it evaluates the goodness of its behaviors, based
on how much reward is getting

22. and:
π(a ∣ s) = P (a ∣ s) ∀s ∈ S
(s) = [ ∣ ]vπ Eπ ∑
t
γ
t
Rt St

23. def value(state):
"""
Black magic
"""
return v

24. def policy(state):
"""
White magic
"""
return reasonable_actions

25. Policy IterationPolicy Iteration

26. #TODO @Manuel: Add code here

28. Policy ImprovementPolicy Improvement

29. 1. Plan in the future
2. Try new actions

30. Monte-Carlo Tree SearchMonte-Carlo Tree Search

31. MCTS is an algorithm to perform sampling based
lookahead search.

34. With the backup operation we keep track of:
N(s,a) visit count
Q(s,a) mean action value

35. ExplorationExploration

36. Bandits
ϵ − greedy

37. Q(s, a) + cP (s, a)
N(s,b)∑
b
√
1+N(s,a)

38. Policy EvaluationPolicy Evaluation
Self PlaySelf Play

39. How well am I doing?

40. 1. Clone yourself and ght!
2. As the Yous battle, observe the ght
3. Use those experiences to improve further

42. How is it implemented in python?How is it implemented in python?
def play_against_yourself(game, player_mcts):
...
board = game.reset()
while not terminal:
act = player_mcts.pick_move(board)
board, r, terminal, opp_act = game.step(action)
training_samples.append((board, player_id, act))
training_samples.append((board, opp_id, opp_act))
return training_samples

43. To the code!To the code!
main: https://gist.github.com/manuel-
delverme/36f9fd220989903274c4badf83c0f880

44. The deeper side of RLThe deeper side of RL

46. In AlphaZero we want to classify cats nd the best
moves

47. π(s) and v(s)

48. The superstar of the newtorkThe superstar of the newtork

50. Deep Learning - where are theDeep Learning - where are the
layers? 1/523layers? 1/523

51. Deep Learning - where are theDeep Learning - where are the
layers? 2/523layers? 2/523

52. it's-going-to-take-a-while 3/523it's-going-to-take-a-while 3/523

53. it's-going-to-take-a-while 4/523it's-going-to-take-a-while 4/523

54. it's-going-to-take-a-while 4/523it's-going-to-take-a-while 4/523

55. lol joking/523lol joking/523
fast forwarding...

56. network heads/523network heads/523

57. value headvalue head

58. policy headpolicy head

59. Loss function - what makes theLoss function - what makes the
model happy?model happy?
(z − v(s) − π log p + c||θ||)
2

60. To the code!To the code!
train: https://gist.github.com/manuel-
delverme/a1b6b93bd5b4d607920b045b039fcb98

61. Q&AQ&A

62. ContactsContacts
manuel.delverme@gmail.com
simone.totaro@gmail.com

63. Thank you!Thank you!
github/mosc