the presentation of the article "Mastering the game of Go with deep neural networks and tree search" given at the Spring School of Combinatorics 2016 Notes: - All URLs are clickable. - All citations are clickable (when hovered over the "year" part of "[author year]"). - To download without a SlideShare account, use https://www.dropbox.com/s/4njuiaaou1po0y4/AlphaGo.pdf?dl=0 - The corresponding handout is available at http://www.slideshare.net/KarelHa1/mastering-the-game-of-go-with-deep-neural-networks-and-tree-search-handout - The video is available at https://youtu.be/Lso2kE58JrI - The source code is available at https://github.com/mathemage/AlphaGo-presentation

1. Mastering the game of Go
with deep neural networks and tree search
Karel Ha
article by Google DeepMind
Spring School of Combinatorics 2016

2. Why AI?

3. Applications of AI
spam filters
1

4. Applications of AI
spam filters
recommender systems (Netflix, YouTube)
1

5. Applications of AI
spam filters
recommender systems (Netflix, YouTube)
predictive text (Swiftkey)
1

6. Applications of AI
spam filters
recommender systems (Netflix, YouTube)
predictive text (Swiftkey)
audio recognition (Shazam, SoundHound)
1

7. Applications of AI
spam filters
recommender systems (Netflix, YouTube)
predictive text (Swiftkey)
audio recognition (Shazam, SoundHound)
music generation (DeepHear - Composing and harmonizing
music with neural networks)
1

8. Applications of AI
spam filters
recommender systems (Netflix, YouTube)
predictive text (Swiftkey)
audio recognition (Shazam, SoundHound)
music generation (DeepHear - Composing and harmonizing
music with neural networks)
self-driving cars
1

9. Auto Reply Feature of Google Inbox
Corrado 2015 2

10. Artistic-style Painting
[1] Gatys, Ecker, and Bethge 2015 [2] Li and Wand 2016 3

11. Artistic-style Painting
[1] Gatys, Ecker, and Bethge 2015 [2] Li and Wand 2016 3

12. Baby Names Generated Character by Character
Baby Killiel Saddie Char Ahbort With
Karpathy 2015 4

13. Baby Names Generated Character by Character
Baby Killiel Saddie Char Ahbort With
Rudi Levette Berice Lussa Hany Mareanne Chrestina Carissy
Karpathy 2015 4

14. Baby Names Generated Character by Character
Baby Killiel Saddie Char Ahbort With
Rudi Levette Berice Lussa Hany Mareanne Chrestina Carissy
Marylen Hammine Janye Marlise Jacacrie Hendred Romand
Charienna Nenotto Ette Dorane Wallen Marly Darine Salina
Elvyn Ersia Maralena Minoria Ellia Charmin Antley Nerille
Chelon Walmor Evena Jeryly Stachon Charisa Allisa Anatha
Cathanie Geetra Alexie Jerin Cassen Herbett Cossie Velen
Daurenge Robester Shermond Terisa Licia Roselen Ferine Jayn
Lusine Charyanne Sales Sanny Resa Wallon Martine Merus
Jelen Candica Wallin Tel Rachene Tarine Ozila Ketia Shanne
Arnande Karella Roselina Alessia Chasty Deland Berther
Geamar Jackein Mellisand Sagdy Nenc Lessie Rasemy Guen
Karpathy 2015 4

15. C code Generated Character by Character
Karpathy 2015 5

16. Algebraic Geometry Generated Character by Character
Karpathy 2015 6

17. DeepDrumpf
https://twitter.com/deepdrumpf
Hayes 2016 7

18. DeepDrumpf
https://twitter.com/deepdrumpf = a Twitter bot that has
learned the language of Donald Trump from his speeches
Hayes 2016 7

19. Atari Player by Google DeepMind
https://youtu.be/0X-NdPtFKq0?t=21m13s
Mnih et al. 2015 8

20. 8

21. Heads-up Limit Holdem Poker Is Solved!
Bowling et al. 2015 9

22. Heads-up Limit Holdem Poker Is Solved!
Cepheus http://poker.srv.ualberta.ca/
Bowling et al. 2015 9

23. Basics of Machine learning

24. Supervised versus Unsupervised Learning
Supervised learning:
10

25. Supervised versus Unsupervised Learning
Supervised learning:
data set must be labelled
10

26. Supervised versus Unsupervised Learning
Supervised learning:
data set must be labelled
e.g. which e-mail is regular/spam, which image is duck/face,
...
10

27. Supervised versus Unsupervised Learning
Supervised learning:
data set must be labelled
e.g. which e-mail is regular/spam, which image is duck/face,
...
10

28. Supervised versus Unsupervised Learning
Supervised learning:
data set must be labelled
e.g. which e-mail is regular/spam, which image is duck/face,
...
Unsupervised learning:
10

29. Supervised versus Unsupervised Learning
Supervised learning:
data set must be labelled
e.g. which e-mail is regular/spam, which image is duck/face,
...
Unsupervised learning:
data set is not labelled
10

30. Supervised versus Unsupervised Learning
Supervised learning:
data set must be labelled
e.g. which e-mail is regular/spam, which image is duck/face,
...
Unsupervised learning:
data set is not labelled
it can try to cluster the data into different groups
10

31. Supervised versus Unsupervised Learning
Supervised learning:
data set must be labelled
e.g. which e-mail is regular/spam, which image is duck/face,
...
Unsupervised learning:
data set is not labelled
it can try to cluster the data into different groups
e.g. grouping similar news, ...
10

32. Supervised Learning
http://www.nickgillian.com/ 11

33. Supervised Learning
1. data collection: Google Search, Facebook “Likes”, Siri, Netflix, YouTube views, LHC collisions, KGS Go
Server...
http://www.nickgillian.com/ 11

34. Supervised Learning
1. data collection: Google Search, Facebook “Likes”, Siri, Netflix, YouTube views, LHC collisions, KGS Go
Server...
2. training on training set
http://www.nickgillian.com/ 11

35. Supervised Learning
1. data collection: Google Search, Facebook “Likes”, Siri, Netflix, YouTube views, LHC collisions, KGS Go
Server...
2. training on training set
3. testing on testing set
http://www.nickgillian.com/ 11

36. Supervised Learning
1. data collection: Google Search, Facebook “Likes”, Siri, Netflix, YouTube views, LHC collisions, KGS Go
Server...
2. training on training set
3. testing on testing set
4. deployment
http://www.nickgillian.com/ 11

37. Supervised Learning
1. data collection: Google Search, Facebook “Likes”, Siri, Netflix, YouTube views, LHC collisions, KGS Go
Server...
2. training on training set
3. testing on testing set
4. deployment
http://www.nickgillian.com/ 11

38. Supervised Learning
1. data collection: Google Search, Facebook “Likes”, Siri, Netflix, YouTube views, LHC collisions, KGS Go
Server...
2. training on training set
3. testing on testing set
4. deployment
http://www.nickgillian.com/ 11

39. Regression
12

40. Regression
12

41. Mathematical Regression
https://thermanuals.wordpress.com/descriptive-analysis/sampling-and-regression/
13

42. Classification
https://kevinbinz.files.wordpress.com/2014/08/ml-svm-after-comparison.png 14

43. Underfitting and Overfitting
https://www.researchgate.net/post/How_to_Avoid_Overfitting 15

44. Underfitting and Overfitting
Beware of overfitting!
https://www.researchgate.net/post/How_to_Avoid_Overfitting 15

45. Underfitting and Overfitting
Beware of overfitting!
It is like learning for a math exam by memorizing proofs.
https://www.researchgate.net/post/How_to_Avoid_Overfitting 15

46. Reinforcement Learning
https://youtu.be/0X-NdPtFKq0?t=16m57s 16

47. Reinforcement Learning
Specially: games of self-play
https://youtu.be/0X-NdPtFKq0?t=16m57s 16

48. Monte Carlo Tree Search

49. Tree Search
Optimal value v∗(s) determines the outcome of the game:
Silver et al. 2016 17

50. Tree Search
Optimal value v∗(s) determines the outcome of the game:
from every board position or state s
Silver et al. 2016 17

51. Tree Search
Optimal value v∗(s) determines the outcome of the game:
from every board position or state s
under perfect play by all players.
Silver et al. 2016 17

52. Tree Search
Optimal value v∗(s) determines the outcome of the game:
from every board position or state s
under perfect play by all players.
Silver et al. 2016 17

53. Tree Search
Optimal value v∗(s) determines the outcome of the game:
from every board position or state s
under perfect play by all players.
It is computed by recursively traversing a search tree containing
approximately bd possible sequences of moves, where
Silver et al. 2016 17

54. Tree Search
Optimal value v∗(s) determines the outcome of the game:
from every board position or state s
under perfect play by all players.
It is computed by recursively traversing a search tree containing
approximately bd possible sequences of moves, where
b is the games breadth (number of legal moves per position)
Silver et al. 2016 17

55. Tree Search
Optimal value v∗(s) determines the outcome of the game:
from every board position or state s
under perfect play by all players.
It is computed by recursively traversing a search tree containing
approximately bd possible sequences of moves, where
b is the games breadth (number of legal moves per position)
d is its depth (game length)
Silver et al. 2016 17

56. Game tree of Go
Sizes of trees for various games:
chess: b ≈ 35, d ≈ 80
Go: b ≈ 250, d ≈ 150
Allis et al. 1994 18

57. Game tree of Go
Sizes of trees for various games:
chess: b ≈ 35, d ≈ 80
Go: b ≈ 250, d ≈ 150 ⇒ more positions than atoms in the
universe!
Allis et al. 1994 18

58. Game tree of Go
Sizes of trees for various games:
chess: b ≈ 35, d ≈ 80
Go: b ≈ 250, d ≈ 150 ⇒ more positions than atoms in the
universe!
That makes Go a googol
times more complex than
chess.
https://deepmind.com/alpha-go.html
Allis et al. 1994 18

59. Game tree of Go
Sizes of trees for various games:
chess: b ≈ 35, d ≈ 80
Go: b ≈ 250, d ≈ 150 ⇒ more positions than atoms in the
universe!
That makes Go a googol
times more complex than
chess.
https://deepmind.com/alpha-go.html
How to handle the size of the game tree?
Allis et al. 1994 18

60. Game tree of Go
Sizes of trees for various games:
chess: b ≈ 35, d ≈ 80
Go: b ≈ 250, d ≈ 150 ⇒ more positions than atoms in the
universe!
That makes Go a googol
times more complex than
chess.
https://deepmind.com/alpha-go.html
How to handle the size of the game tree?
for the breadth: a neural network to select moves
Allis et al. 1994 18

61. Game tree of Go
Sizes of trees for various games:
chess: b ≈ 35, d ≈ 80
Go: b ≈ 250, d ≈ 150 ⇒ more positions than atoms in the
universe!
That makes Go a googol
times more complex than
chess.
https://deepmind.com/alpha-go.html
How to handle the size of the game tree?
for the breadth: a neural network to select moves
for the depth: a neural network to evaluate current position
Allis et al. 1994 18

62. Game tree of Go
Sizes of trees for various games:
chess: b ≈ 35, d ≈ 80
Go: b ≈ 250, d ≈ 150 ⇒ more positions than atoms in the
universe!
That makes Go a googol
times more complex than
chess.
https://deepmind.com/alpha-go.html
How to handle the size of the game tree?
for the breadth: a neural network to select moves
for the depth: a neural network to evaluate current position
for the tree traverse: Monte Carlo tree search (MCTS)
Allis et al. 1994 18

63. Monte Carlo tree search
19

64. Neural networks

65. Neural Network: Inspiration
http://pages.cs.wisc.edu/~bolo/shipyard/neural/local.html 20

66. Neural Network: Inspiration
inspired by the neuronal structure of the mammalian cerebral
cortex
http://pages.cs.wisc.edu/~bolo/shipyard/neural/local.html 20

67. Neural Network: Inspiration
inspired by the neuronal structure of the mammalian cerebral
cortex
but on much smaller scales
http://pages.cs.wisc.edu/~bolo/shipyard/neural/local.html 20

68. Neural Network: Inspiration
inspired by the neuronal structure of the mammalian cerebral
cortex
but on much smaller scales
suitable to model systems with a high tolerance to error
http://pages.cs.wisc.edu/~bolo/shipyard/neural/local.html 20

69. Neural Network: Inspiration
inspired by the neuronal structure of the mammalian cerebral
cortex
but on much smaller scales
suitable to model systems with a high tolerance to error
e.g. audio or image recognition
http://pages.cs.wisc.edu/~bolo/shipyard/neural/local.html 20

70. Neural Network: Modes
Dieterle 2003 21

71. Neural Network: Modes
Two modes
Dieterle 2003 21

72. Neural Network: Modes
Two modes
feedforward for making predictions
Dieterle 2003 21

73. Neural Network: Modes
Two modes
feedforward for making predictions
backpropagation for learning
Dieterle 2003 21

74. Neural Network: an example of feedforward
http://stevenmiller888.github.io/mind-how-to-build-a-neural-network/ 22

75. Gradient Descent in Neural Networks
http://pages.cs.wisc.edu/~bolo/shipyard/neural/local.html 23

76. Gradient Descent in Neural Networks
Motto: ”Learn by mistakes!”
http://pages.cs.wisc.edu/~bolo/shipyard/neural/local.html 23

77. Gradient Descent in Neural Networks
Motto: ”Learn by mistakes!”
However, error functions are not necessarily convex or so “smooth”.
http://pages.cs.wisc.edu/~bolo/shipyard/neural/local.html 23

78. Deep Neural Network: Inspiration
The hierarchy of concepts is captured in the number of layers (the deep in “deep learning”)
http://pages.cs.wisc.edu/~bolo/shipyard/neural/local.html 24

79. Deep Neural Network: Inspiration
The hierarchy of concepts is captured in the number of layers (the deep in “deep learning”)
http://pages.cs.wisc.edu/~bolo/shipyard/neural/local.html 24

80. Convolutional Neural Network
http://pages.cs.wisc.edu/~bolo/shipyard/neural/local.html 25

81. Rules of Go

82. Classic games (1/2)
26

83. Classic games (1/2)
Backgammon: Man vs. Fate
26

84. Classic games (1/2)
Backgammon: Man vs. Fate
Chess: Man vs. Man
26

85. Classic games (2/2)
Go: Man vs. Self
Robert ˇS´amal (White) versus Karel Kr´al (Black), Spring School of Combinatorics 2016 27

86. Rules of Go
28

87. Rules of Go
Black versus White. Black starts the game.
28

88. Rules of Go
Black versus White. Black starts the game.
28

89. Rules of Go
Black versus White. Black starts the game.
the rule of liberty
28

90. Rules of Go
Black versus White. Black starts the game.
the rule of liberty
the “ko” rule
28

91. Rules of Go
Black versus White. Black starts the game.
the rule of liberty
the “ko” rule
Handicap for difference in ranks: Black can place 1 or more stones
in advance (compensation for White’s greater strength). 28

92. Scoring Rules: Area Scoring
https://en.wikipedia.org/wiki/Go_(game) 29

93. Scoring Rules: Area Scoring
A player’s score is:
the number of stones that the player has on the board
https://en.wikipedia.org/wiki/Go_(game) 29

94. Scoring Rules: Area Scoring
A player’s score is:
the number of stones that the player has on the board
plus the number of empty intersections surrounded by that
player’s stones
https://en.wikipedia.org/wiki/Go_(game) 29

95. Scoring Rules: Area Scoring
A player’s score is:
the number of stones that the player has on the board
plus the number of empty intersections surrounded by that
player’s stones
plus komi(dashi) points for the White player
which is a compensation for the first move advantage of the Black player
https://en.wikipedia.org/wiki/Go_(game) 29

96. Ranks of Players
Kyu and Dan ranks
https://en.wikipedia.org/wiki/Go_(game) 30

97. Ranks of Players
Kyu and Dan ranks
or alternatively, ELO ratings
https://en.wikipedia.org/wiki/Go_(game) 30

98. Chocolate micro-break
30

99. AlphaGo: Inside Out

100. Policy and Value Networks
Silver et al. 2016 31

101. Training the (Deep Convolutional) Neural Networks
Silver et al. 2016 32

102. SL Policy Networks (1/3)
13-layer deep convolutional neural network
Silver et al. 2016 33

103. SL Policy Networks (1/3)
13-layer deep convolutional neural network
goal: to predict expert human moves
Silver et al. 2016 33

104. SL Policy Networks (1/3)
13-layer deep convolutional neural network
goal: to predict expert human moves
task of classification
Silver et al. 2016 33

105. SL Policy Networks (1/3)
13-layer deep convolutional neural network
goal: to predict expert human moves
task of classification
trained from 30 millions positions from the KGS Go Server
Silver et al. 2016 33

106. SL Policy Networks (1/3)
13-layer deep convolutional neural network
goal: to predict expert human moves
task of classification
trained from 30 millions positions from the KGS Go Server
stochastic gradient ascent:
∆σ ∝
∂ log pσ(a|s)
∂σ
(to maximize the likelihood of the human move a selected in state s)
Silver et al. 2016 33

107. SL Policy Networks (1/3)
13-layer deep convolutional neural network
goal: to predict expert human moves
task of classification
trained from 30 millions positions from the KGS Go Server
stochastic gradient ascent:
∆σ ∝
∂ log pσ(a|s)
∂σ
(to maximize the likelihood of the human move a selected in state s)
Silver et al. 2016 33

108. SL Policy Networks (1/3)
13-layer deep convolutional neural network
goal: to predict expert human moves
task of classification
trained from 30 millions positions from the KGS Go Server
stochastic gradient ascent:
∆σ ∝
∂ log pσ(a|s)
∂σ
(to maximize the likelihood of the human move a selected in state s)
Results:
Silver et al. 2016 33

109. SL Policy Networks (1/3)
13-layer deep convolutional neural network
goal: to predict expert human moves
task of classification
trained from 30 millions positions from the KGS Go Server
stochastic gradient ascent:
∆σ ∝
∂ log pσ(a|s)
∂σ
(to maximize the likelihood of the human move a selected in state s)
Results:
44.4% accuracy (the state-of-the-art from other groups)
Silver et al. 2016 33

110. SL Policy Networks (1/3)
13-layer deep convolutional neural network
goal: to predict expert human moves
task of classification
trained from 30 millions positions from the KGS Go Server
stochastic gradient ascent:
∆σ ∝
∂ log pσ(a|s)
∂σ
(to maximize the likelihood of the human move a selected in state s)
Results:
44.4% accuracy (the state-of-the-art from other groups)
55.7% accuracy (raw board position + move history as input)
Silver et al. 2016 33

111. SL Policy Networks (1/3)
13-layer deep convolutional neural network
goal: to predict expert human moves
task of classification
trained from 30 millions positions from the KGS Go Server
stochastic gradient ascent:
∆σ ∝
∂ log pσ(a|s)
∂σ
(to maximize the likelihood of the human move a selected in state s)
Results:
44.4% accuracy (the state-of-the-art from other groups)
55.7% accuracy (raw board position + move history as input)
57.0% accuracy (all input features)
Silver et al. 2016 33

112. SL Policy Networks (2/3)
Small improvements in accuracy led to large improvements
in playing strength (see the next slide)
Silver et al. 2016 34

113. SL Policy Networks (3/3)
move probabilities taken directly from the SL policy network pσ (reported as a percentage if above 0.1%).
Silver et al. 2016 35

114. Training the (Deep Convolutional) Neural Networks
Silver et al. 2016 36

115. Rollout Policy
Rollout policy pπ(a|s) is faster but less accurate than SL
policy network.
Silver et al. 2016 37

116. Rollout Policy
Rollout policy pπ(a|s) is faster but less accurate than SL
policy network.
accuracy of 24.2%
Silver et al. 2016 37

117. Rollout Policy
Rollout policy pπ(a|s) is faster but less accurate than SL
policy network.
accuracy of 24.2%
It takes 2µs to select an action, compared to 3 ms in case
of SL policy network.
Silver et al. 2016 37

118. Training the (Deep Convolutional) Neural Networks
Silver et al. 2016 38

119. RL Policy Networks (1/2)
identical in structure to the SL policy network
Silver et al. 2016 39

120. RL Policy Networks (1/2)
identical in structure to the SL policy network
goal: to win in the games of self-play
Silver et al. 2016 39

121. RL Policy Networks (1/2)
identical in structure to the SL policy network
goal: to win in the games of self-play
task of classification
Silver et al. 2016 39

122. RL Policy Networks (1/2)
identical in structure to the SL policy network
goal: to win in the games of self-play
task of classification
weights ρ initialized to the same values, ρ := σ
Silver et al. 2016 39

123. RL Policy Networks (1/2)
identical in structure to the SL policy network
goal: to win in the games of self-play
task of classification
weights ρ initialized to the same values, ρ := σ
games of self-play
Silver et al. 2016 39

124. RL Policy Networks (1/2)
identical in structure to the SL policy network
goal: to win in the games of self-play
task of classification
weights ρ initialized to the same values, ρ := σ
games of self-play
between the current RL policy network and a randomly
selected previous iteration
Silver et al. 2016 39

125. RL Policy Networks (1/2)
identical in structure to the SL policy network
goal: to win in the games of self-play
task of classification
weights ρ initialized to the same values, ρ := σ
games of self-play
between the current RL policy network and a randomly
selected previous iteration
to prevent overfitting to the current policy
Silver et al. 2016 39

126. RL Policy Networks (1/2)
identical in structure to the SL policy network
goal: to win in the games of self-play
task of classification
weights ρ initialized to the same values, ρ := σ
games of self-play
between the current RL policy network and a randomly
selected previous iteration
to prevent overfitting to the current policy
stochastic gradient ascent:
∆ρ ∝
∂ log pρ(at|st)
∂ρ
zt
at time step t, where reward function zt is +1 for winning and −1 for losing.
Silver et al. 2016 39

127. RL Policy Networks (1/2)
identical in structure to the SL policy network
goal: to win in the games of self-play
task of classification
weights ρ initialized to the same values, ρ := σ
games of self-play
between the current RL policy network and a randomly
selected previous iteration
to prevent overfitting to the current policy
stochastic gradient ascent:
∆ρ ∝
∂ log pρ(at|st)
∂ρ
zt
at time step t, where reward function zt is +1 for winning and −1 for losing.
Silver et al. 2016 39

128. RL Policy Networks (1/2)
identical in structure to the SL policy network
goal: to win in the games of self-play
task of classification
weights ρ initialized to the same values, ρ := σ
games of self-play
between the current RL policy network and a randomly
selected previous iteration
to prevent overfitting to the current policy
stochastic gradient ascent:
∆ρ ∝
∂ log pρ(at|st)
∂ρ
zt
at time step t, where reward function zt is +1 for winning and −1 for losing.
Silver et al. 2016 39

129. RL Policy Networks (2/2)
Results (by sampling each move at ∼ pρ(·|st)):
Silver et al. 2016 40

130. RL Policy Networks (2/2)
Results (by sampling each move at ∼ pρ(·|st)):
80% of win rate against the SL policy network
Silver et al. 2016 40

131. RL Policy Networks (2/2)
Results (by sampling each move at ∼ pρ(·|st)):
80% of win rate against the SL policy network
85% of win rate against the strongest open-source Go
program, Pachi (Baudiˇs and Gailly 2011)
Silver et al. 2016 40

132. RL Policy Networks (2/2)
Results (by sampling each move at ∼ pρ(·|st)):
80% of win rate against the SL policy network
85% of win rate against the strongest open-source Go
program, Pachi (Baudiˇs and Gailly 2011)
The previous state-of-the-art, based only on SL of CNN:
Silver et al. 2016 40

133. RL Policy Networks (2/2)
Results (by sampling each move at ∼ pρ(·|st)):
80% of win rate against the SL policy network
85% of win rate against the strongest open-source Go
program, Pachi (Baudiˇs and Gailly 2011)
The previous state-of-the-art, based only on SL of CNN:
Silver et al. 2016 40

134. RL Policy Networks (2/2)
Results (by sampling each move at ∼ pρ(·|st)):
80% of win rate against the SL policy network
85% of win rate against the strongest open-source Go
program, Pachi (Baudiˇs and Gailly 2011)
The previous state-of-the-art, based only on SL of CNN:
11% of “win” rate against Pachi
Silver et al. 2016 40

135. Training the (Deep Convolutional) Neural Networks
Silver et al. 2016 41

136. Value Network (1/2)
similar architecture to the policy network, but outputs a single
prediction instead of a probability distribution
Silver et al. 2016 42

137. Value Network (1/2)
similar architecture to the policy network, but outputs a single
prediction instead of a probability distribution
goal: to estimate a value function
vp
(s) = E[zt|st = s, at...T ∼ p]
that predicts the outcome from position s (of games played
by using policy pρ)
Silver et al. 2016 42

138. Value Network (1/2)
similar architecture to the policy network, but outputs a single
prediction instead of a probability distribution
goal: to estimate a value function
vp
(s) = E[zt|st = s, at...T ∼ p]
that predicts the outcome from position s (of games played
by using policy pρ)
Specifically, vθ(s) ≈ vpρ (s) ≈ v∗(s).
Silver et al. 2016 42

139. Value Network (1/2)
similar architecture to the policy network, but outputs a single
prediction instead of a probability distribution
goal: to estimate a value function
vp
(s) = E[zt|st = s, at...T ∼ p]
that predicts the outcome from position s (of games played
by using policy pρ)
Specifically, vθ(s) ≈ vpρ (s) ≈ v∗(s).
task of regression
Silver et al. 2016 42

140. Value Network (1/2)
similar architecture to the policy network, but outputs a single
prediction instead of a probability distribution
goal: to estimate a value function
vp
(s) = E[zt|st = s, at...T ∼ p]
that predicts the outcome from position s (of games played
by using policy pρ)
Specifically, vθ(s) ≈ vpρ (s) ≈ v∗(s).
task of regression
stochastic gradient descent:
∆θ ∝
∂vθ(s)
∂θ
(z − vθ(s))
(to minimize the mean squared error (MSE) between the predicted vθ(s) and the true z)
Silver et al. 2016 42

141. Value Network (2/2)
Beware of overfitting!
Silver et al. 2016 43

142. Value Network (2/2)
Beware of overfitting!
Successive positions are strongly correlated.
Silver et al. 2016 43

143. Value Network (2/2)
Beware of overfitting!
Successive positions are strongly correlated.
Value network memorized the game outcomes, rather than
generalizing to new positions.
Silver et al. 2016 43

144. Value Network (2/2)
Beware of overfitting!
Successive positions are strongly correlated.
Value network memorized the game outcomes, rather than
generalizing to new positions.
Solution: generate 30 million (new) positions, each sampled
from a seperate game
Silver et al. 2016 43

145. Value Network (2/2)
Beware of overfitting!
Successive positions are strongly correlated.
Value network memorized the game outcomes, rather than
generalizing to new positions.
Solution: generate 30 million (new) positions, each sampled
from a seperate game
almost the accuracy of Monte Carlo rollouts (using pρ), but
15000 times less computation!
Silver et al. 2016 43

146. Selection of Moves by the Value Network
evaluation of all successors s of the root position s, using vθ(s)
Silver et al. 2016 44

147. Evaluation accuracy in various stages of a game
Move number is the number of moves that had been played in the given position.
Silver et al. 2016 45

148. Evaluation accuracy in various stages of a game
Move number is the number of moves that had been played in the given position.
Each position evaluated by:
forward pass of the value network vθ
Silver et al. 2016 45

149. Evaluation accuracy in various stages of a game
Move number is the number of moves that had been played in the given position.
Each position evaluated by:
forward pass of the value network vθ
100 rollouts, played out using the corresponding policy
Silver et al. 2016 45

150. Training the (Deep Convolutional) Neural Networks
Silver et al. 2016 46

151. ELO Ratings for Various Combinations of Networks
Silver et al. 2016 47

152. MCTS Algorithm
The next action is selected by lookahead search, using simulation:
Silver et al. 2016 48

153. MCTS Algorithm
The next action is selected by lookahead search, using simulation:
1. selection phase
Silver et al. 2016 48

154. MCTS Algorithm
The next action is selected by lookahead search, using simulation:
1. selection phase
2. expansion phase
Silver et al. 2016 48

155. MCTS Algorithm
The next action is selected by lookahead search, using simulation:
1. selection phase
2. expansion phase
3. evaluation phase
Silver et al. 2016 48

156. MCTS Algorithm
The next action is selected by lookahead search, using simulation:
1. selection phase
2. expansion phase
3. evaluation phase
4. backup phase (at end of simulation)
Silver et al. 2016 48

157. MCTS Algorithm
The next action is selected by lookahead search, using simulation:
1. selection phase
2. expansion phase
3. evaluation phase
4. backup phase (at end of simulation)
Silver et al. 2016 48

158. MCTS Algorithm
The next action is selected by lookahead search, using simulation:
1. selection phase
2. expansion phase
3. evaluation phase
4. backup phase (at end of simulation)
Each edge (s, a) keeps:
action value Q(s, a)
Silver et al. 2016 48

159. MCTS Algorithm
The next action is selected by lookahead search, using simulation:
1. selection phase
2. expansion phase
3. evaluation phase
4. backup phase (at end of simulation)
Each edge (s, a) keeps:
action value Q(s, a)
visit count N(s, a)
Silver et al. 2016 48

160. MCTS Algorithm
The next action is selected by lookahead search, using simulation:
1. selection phase
2. expansion phase
3. evaluation phase
4. backup phase (at end of simulation)
Each edge (s, a) keeps:
action value Q(s, a)
visit count N(s, a)
prior probability P(s, a) (from SL policy network pσ)
Silver et al. 2016 48

161. MCTS Algorithm
The next action is selected by lookahead search, using simulation:
1. selection phase
2. expansion phase
3. evaluation phase
4. backup phase (at end of simulation)
Each edge (s, a) keeps:
action value Q(s, a)
visit count N(s, a)
prior probability P(s, a) (from SL policy network pσ)
Silver et al. 2016 48

162. MCTS Algorithm
The next action is selected by lookahead search, using simulation:
1. selection phase
2. expansion phase
3. evaluation phase
4. backup phase (at end of simulation)
Each edge (s, a) keeps:
action value Q(s, a)
visit count N(s, a)
prior probability P(s, a) (from SL policy network pσ)
The tree is traversed by simulation (descending the tree) from the
root state.
Silver et al. 2016 48

163. MCTS Algorithm: Selection
Silver et al. 2016 49

164. MCTS Algorithm: Selection
At each time step t, an action at is selected from state st
at = arg max
a
(Q(st , a) + u(st , a))
Silver et al. 2016 49

165. MCTS Algorithm: Selection
At each time step t, an action at is selected from state st
at = arg max
a
(Q(st , a) + u(st , a))
where bonus
u(st , a) ∝
P(s, a)
1 + N(s, a)
Silver et al. 2016 49

166. MCTS Algorithm: Expansion
Silver et al. 2016 50

167. MCTS Algorithm: Expansion
A leaf position may be expanded (just once) by the SL policy network pσ.
Silver et al. 2016 50

168. MCTS Algorithm: Expansion
A leaf position may be expanded (just once) by the SL policy network pσ.
The output probabilities are stored as priors P(s, a) := pσ(a|s).
Silver et al. 2016 50

169. MCTS: Evaluation
Silver et al. 2016 51

170. MCTS: Evaluation
evaluation from the value network vθ(s)
Silver et al. 2016 51

171. MCTS: Evaluation
evaluation from the value network vθ(s)
evaluation by the outcome z using the fast rollout policy pπ until the end of game
Silver et al. 2016 51

172. MCTS: Evaluation
evaluation from the value network vθ(s)
evaluation by the outcome z using the fast rollout policy pπ until the end of game
Silver et al. 2016 51

173. MCTS: Evaluation
evaluation from the value network vθ(s)
evaluation by the outcome z using the fast rollout policy pπ until the end of game
Using a mixing parameter λ, the final leaf evaluation V (s) is
V (s) = (1 − λ)vθ(s) + λz
Silver et al. 2016 51

174. Tree Evaluation from Value Network
action values Q(s, a) for each tree-edge (s, a) from root position s (averaged over value network evaluations only)
Silver et al. 2016 52

175. Tree Evaluation from Rollouts
action values Q(s, a), averaged over rollout evaluations only
Silver et al. 2016 53

176. MCTS: Backup
At the end of simulation, each traversed edge is updated by accumulating:
the action values Q
Silver et al. 2016 54

177. MCTS: Backup
At the end of simulation, each traversed edge is updated by accumulating:
the action values Q
visit counts N
Silver et al. 2016 54

178. Once the search is complete, the algorithm
chooses the most visited move from the root
position.
Silver et al. 2016 54

179. Percentage of Simulations
percentage frequency with which actions were selected from the root during simulations
Silver et al. 2016 55

180. Principal Variation (Path with Maximum Visit Count)
The moves are presented in a numbered sequence.
Silver et al. 2016 56

181. Principal Variation (Path with Maximum Visit Count)
The moves are presented in a numbered sequence.
AlphaGo selected the move indicated by the red circle;
Silver et al. 2016 56

182. Principal Variation (Path with Maximum Visit Count)
The moves are presented in a numbered sequence.
AlphaGo selected the move indicated by the red circle;
Fan Hui responded with the move indicated by the white square;
Silver et al. 2016 56

183. Principal Variation (Path with Maximum Visit Count)
The moves are presented in a numbered sequence.
AlphaGo selected the move indicated by the red circle;
Fan Hui responded with the move indicated by the white square;
in his post-game commentary, he preferred the move (labelled 1) predicted by AlphaGo.
Silver et al. 2016 56

184. Scalability
asynchronous multi-threaded search
Silver et al. 2016 57

185. Scalability
asynchronous multi-threaded search
simulations on CPUs
Silver et al. 2016 57

186. Scalability
asynchronous multi-threaded search
simulations on CPUs
computation of neural networks on GPUs
Silver et al. 2016 57

187. Scalability
asynchronous multi-threaded search
simulations on CPUs
computation of neural networks on GPUs
Silver et al. 2016 57

188. Scalability
asynchronous multi-threaded search
simulations on CPUs
computation of neural networks on GPUs
AlphaGo:
40 search threads
Silver et al. 2016 57

189. Scalability
asynchronous multi-threaded search
simulations on CPUs
computation of neural networks on GPUs
AlphaGo:
40 search threads
40 CPUs
Silver et al. 2016 57

190. Scalability
asynchronous multi-threaded search
simulations on CPUs
computation of neural networks on GPUs
AlphaGo:
40 search threads
40 CPUs
8 GPUs
Silver et al. 2016 57

191. Scalability
asynchronous multi-threaded search
simulations on CPUs
computation of neural networks on GPUs
AlphaGo:
40 search threads
40 CPUs
8 GPUs
Silver et al. 2016 57

192. Scalability
asynchronous multi-threaded search
simulations on CPUs
computation of neural networks on GPUs
AlphaGo:
40 search threads
40 CPUs
8 GPUs
Distributed version of AlphaGo (on multiple machines):
40 search threads
Silver et al. 2016 57

193. Scalability
asynchronous multi-threaded search
simulations on CPUs
computation of neural networks on GPUs
AlphaGo:
40 search threads
40 CPUs
8 GPUs
Distributed version of AlphaGo (on multiple machines):
40 search threads
1202 CPUs
Silver et al. 2016 57

194. Scalability
asynchronous multi-threaded search
simulations on CPUs
computation of neural networks on GPUs
AlphaGo:
40 search threads
40 CPUs
8 GPUs
Distributed version of AlphaGo (on multiple machines):
40 search threads
1202 CPUs
176 GPUs
Silver et al. 2016 57

195. ELO Ratings for Various Combinations of Threads
Silver et al. 2016 58

196. Results: the strength of AlphaGo

197. Tournament with Other Go Programs
Silver et al. 2016 59

198. Fan Hui
https://en.wikipedia.org/wiki/Fan_Hui 60

199. Fan Hui
professional 2 dan
https://en.wikipedia.org/wiki/Fan_Hui 60

200. Fan Hui
professional 2 dan
European Go Champion in 2013, 2014 and 2015
https://en.wikipedia.org/wiki/Fan_Hui 60

201. Fan Hui
professional 2 dan
European Go Champion in 2013, 2014 and 2015
European Professional Go Champion in 2016
https://en.wikipedia.org/wiki/Fan_Hui 60

202. Fan Hui
professional 2 dan
European Go Champion in 2013, 2014 and 2015
European Professional Go Champion in 2016
biological neural network:
https://en.wikipedia.org/wiki/Fan_Hui 60

203. Fan Hui
professional 2 dan
European Go Champion in 2013, 2014 and 2015
European Professional Go Champion in 2016
biological neural network:
100 billion neurons
https://en.wikipedia.org/wiki/Fan_Hui 60

204. Fan Hui
professional 2 dan
European Go Champion in 2013, 2014 and 2015
European Professional Go Champion in 2016
biological neural network:
100 billion neurons
100 up to 1,000 trillion neuronal connections
https://en.wikipedia.org/wiki/Fan_Hui 60

205. AlphaGo versus Fan Hui
61

206. AlphaGo versus Fan Hui
AlphaGo won 5 - 0 in a formal match on October 2015.
61

207. AlphaGo versus Fan Hui
AlphaGo won 5 - 0 in a formal match on October 2015.
[AlphaGo] is very strong and stable, it seems
like a wall. ... I know AlphaGo is a computer,
but if no one told me, maybe I would think
the player was a little strange, but a very
strong player, a real person.
Fan Hui 61

208. Lee Sedol “The Strong Stone”
https://en.wikipedia.org/wiki/Lee_Sedol 62

209. Lee Sedol “The Strong Stone”
professional 9 dan
https://en.wikipedia.org/wiki/Lee_Sedol 62

210. Lee Sedol “The Strong Stone”
professional 9 dan
the 2nd in international titles
https://en.wikipedia.org/wiki/Lee_Sedol 62

211. Lee Sedol “The Strong Stone”
professional 9 dan
the 2nd in international titles
the 5th youngest (12 years 4 months) to become
a professional Go player in South Korean history
https://en.wikipedia.org/wiki/Lee_Sedol 62

212. Lee Sedol “The Strong Stone”
professional 9 dan
the 2nd in international titles
the 5th youngest (12 years 4 months) to become
a professional Go player in South Korean history
Lee Sedol would win 97 out of 100 games against Fan Hui.
https://en.wikipedia.org/wiki/Lee_Sedol 62

213. Lee Sedol “The Strong Stone”
professional 9 dan
the 2nd in international titles
the 5th youngest (12 years 4 months) to become
a professional Go player in South Korean history
Lee Sedol would win 97 out of 100 games against Fan Hui.
biological neural network, comparable to Fan Hui’s (in number
of neurons and connections)
https://en.wikipedia.org/wiki/Lee_Sedol 62

214. I heard Google DeepMind’s AI is surprisingly
strong and getting stronger, but I am
confident that I can win, at least this time.
Lee Sedol
62

215. I heard Google DeepMind’s AI is surprisingly
strong and getting stronger, but I am
confident that I can win, at least this time.
Lee Sedol
...even beating AlphaGo by 4-1 may allow
the Google DeepMind team to claim its de
facto victory and the defeat of him
[Lee Sedol], or even humankind.
interview in JTBC
Newsroom
62

216. I heard Google DeepMind’s AI is surprisingly
strong and getting stronger, but I am
confident that I can win, at least this time.
Lee Sedol
...even beating AlphaGo by 4-1 may allow
the Google DeepMind team to claim its de
facto victory and the defeat of him
[Lee Sedol], or even humankind.
interview in JTBC
Newsroom
62

217. AlphaGo versus Lee Sedol
https://en.wikipedia.org/wiki/AlphaGo_versus_Lee_Sedol 63

218. AlphaGo versus Lee Sedol
In March 2016 AlphaGo won 4-1 against the legendary Lee Sedol.
https://en.wikipedia.org/wiki/AlphaGo_versus_Lee_Sedol 63

219. AlphaGo versus Lee Sedol
In March 2016 AlphaGo won 4-1 against the legendary Lee Sedol.
AlphaGo won all but the 4th game; all games were won
by resignation.
https://en.wikipedia.org/wiki/AlphaGo_versus_Lee_Sedol 63

220. AlphaGo versus Lee Sedol
In March 2016 AlphaGo won 4-1 against the legendary Lee Sedol.
AlphaGo won all but the 4th game; all games were won
by resignation.
The winner of the match was slated to win $1 million.
https://en.wikipedia.org/wiki/AlphaGo_versus_Lee_Sedol 63

221. AlphaGo versus Lee Sedol
In March 2016 AlphaGo won 4-1 against the legendary Lee Sedol.
AlphaGo won all but the 4th game; all games were won
by resignation.
The winner of the match was slated to win $1 million.
Since AlphaGo won, Google DeepMind stated that the prize will be
donated to charities, including UNICEF, and Go organisations.
https://en.wikipedia.org/wiki/AlphaGo_versus_Lee_Sedol 63

222. AlphaGo versus Lee Sedol
In March 2016 AlphaGo won 4-1 against the legendary Lee Sedol.
AlphaGo won all but the 4th game; all games were won
by resignation.
The winner of the match was slated to win $1 million.
Since AlphaGo won, Google DeepMind stated that the prize will be
donated to charities, including UNICEF, and Go organisations.
Lee received $170,000 ($150,000 for participating in all the five
games, and an additional $20,000 for each game won).
https://en.wikipedia.org/wiki/AlphaGo_versus_Lee_Sedol 63

223. Conclusion

224. Difficulties of Go
challenging decision-making
Silver et al. 2016 64

225. Difficulties of Go
challenging decision-making
intractable search space
Silver et al. 2016 64

226. Difficulties of Go
challenging decision-making
intractable search space
complex optimal solution
It appears infeasible to directly approximate using a policy or value function!
Silver et al. 2016 64

227. AlphaGo: summary
Monte Carlo tree search
Silver et al. 2016 65

228. AlphaGo: summary
Monte Carlo tree search
effective move selection and position evaluation
Silver et al. 2016 65

229. AlphaGo: summary
Monte Carlo tree search
effective move selection and position evaluation
through deep convolutional neural networks
Silver et al. 2016 65

230. AlphaGo: summary
Monte Carlo tree search
effective move selection and position evaluation
through deep convolutional neural networks
trained by novel combination of supervised and reinforcement
learning
Silver et al. 2016 65

231. AlphaGo: summary
Monte Carlo tree search
effective move selection and position evaluation
through deep convolutional neural networks
trained by novel combination of supervised and reinforcement
learning
new search algorithm combining
Silver et al. 2016 65

232. AlphaGo: summary
Monte Carlo tree search
effective move selection and position evaluation
through deep convolutional neural networks
trained by novel combination of supervised and reinforcement
learning
new search algorithm combining
neural network evaluation
Silver et al. 2016 65

233. AlphaGo: summary
Monte Carlo tree search
effective move selection and position evaluation
through deep convolutional neural networks
trained by novel combination of supervised and reinforcement
learning
new search algorithm combining
neural network evaluation
Monte Carlo rollouts
Silver et al. 2016 65

234. AlphaGo: summary
Monte Carlo tree search
effective move selection and position evaluation
through deep convolutional neural networks
trained by novel combination of supervised and reinforcement
learning
new search algorithm combining
neural network evaluation
Monte Carlo rollouts
scalable implementation
Silver et al. 2016 65

235. AlphaGo: summary
Monte Carlo tree search
effective move selection and position evaluation
through deep convolutional neural networks
trained by novel combination of supervised and reinforcement
learning
new search algorithm combining
neural network evaluation
Monte Carlo rollouts
scalable implementation
multi-threaded simulations on CPUs
Silver et al. 2016 65

236. AlphaGo: summary
Monte Carlo tree search
effective move selection and position evaluation
through deep convolutional neural networks
trained by novel combination of supervised and reinforcement
learning
new search algorithm combining
neural network evaluation
Monte Carlo rollouts
scalable implementation
multi-threaded simulations on CPUs
parallel GPU computations
Silver et al. 2016 65

237. AlphaGo: summary
Monte Carlo tree search
effective move selection and position evaluation
through deep convolutional neural networks
trained by novel combination of supervised and reinforcement
learning
new search algorithm combining
neural network evaluation
Monte Carlo rollouts
scalable implementation
multi-threaded simulations on CPUs
parallel GPU computations
distributed version over multiple machines
Silver et al. 2016 65

238. Novel approach
Silver et al. 2016 66

239. Novel approach
During the match against Fan Hui, AlphaGo evaluated thousands
of times fewer positions than DeepBlue against Kasparov.
Silver et al. 2016 66

240. Novel approach
During the match against Fan Hui, AlphaGo evaluated thousands
of times fewer positions than DeepBlue against Kasparov.
It compensated this by:
selecting those positions more intelligently (policy network)
Silver et al. 2016 66

241. Novel approach
During the match against Fan Hui, AlphaGo evaluated thousands
of times fewer positions than DeepBlue against Kasparov.
It compensated this by:
selecting those positions more intelligently (policy network)
evaluating them more precisely (value network)
Silver et al. 2016 66

242. Novel approach
During the match against Fan Hui, AlphaGo evaluated thousands
of times fewer positions than DeepBlue against Kasparov.
It compensated this by:
selecting those positions more intelligently (policy network)
evaluating them more precisely (value network)
Silver et al. 2016 66

243. Novel approach
During the match against Fan Hui, AlphaGo evaluated thousands
of times fewer positions than DeepBlue against Kasparov.
It compensated this by:
selecting those positions more intelligently (policy network)
evaluating them more precisely (value network)
Deep Blue relied on a handcrafted evaluation function.
Silver et al. 2016 66

244. Novel approach
During the match against Fan Hui, AlphaGo evaluated thousands
of times fewer positions than DeepBlue against Kasparov.
It compensated this by:
selecting those positions more intelligently (policy network)
evaluating them more precisely (value network)
Deep Blue relied on a handcrafted evaluation function.
AlphaGo was trained directly and automatically from gameplay.
It used general-purpose learning.
Silver et al. 2016 66

245. Novel approach
During the match against Fan Hui, AlphaGo evaluated thousands
of times fewer positions than DeepBlue against Kasparov.
It compensated this by:
selecting those positions more intelligently (policy network)
evaluating them more precisely (value network)
Deep Blue relied on a handcrafted evaluation function.
AlphaGo was trained directly and automatically from gameplay.
It used general-purpose learning.
This approach is not specific to the game of Go. The algorithm
can be used for much wider class of (so far seemingly)
intractable problems in AI!
Silver et al. 2016 66

246. Thank you!
Questions?
66

247. Backup slides

248. Input features for rollout and tree policy
Silver et al. 2016

249. Results of a tournament between different Go programs
Silver et al. 2016

250. Results of a tournament between AlphaGo and distributed Al-
phaGo, testing scalability with hardware
Silver et al. 2016

251. AlphaGo versus Fan Hui: Game 1
Silver et al. 2016

252. AlphaGo versus Fan Hui: Game 2
Silver et al. 2016

253. AlphaGo versus Fan Hui: Game 3
Silver et al. 2016

254. AlphaGo versus Fan Hui: Game 4
Silver et al. 2016

255. AlphaGo versus Fan Hui: Game 5
Silver et al. 2016

256. AlphaGo versus Lee Sedol: Game 1
https://en.wikipedia.org/wiki/AlphaGo_versus_Lee_Sedol

257. AlphaGo versus Lee Sedol: Game 2 (1/2)
https://en.wikipedia.org/wiki/AlphaGo_versus_Lee_Sedol

258. AlphaGo versus Lee Sedol: Game 2 (2/2)
https://en.wikipedia.org/wiki/AlphaGo_versus_Lee_Sedol

259. AlphaGo versus Lee Sedol: Game 3
https://en.wikipedia.org/wiki/AlphaGo_versus_Lee_Sedol

260. AlphaGo versus Lee Sedol: Game 4
https://en.wikipedia.org/wiki/AlphaGo_versus_Lee_Sedol

261. AlphaGo versus Lee Sedol: Game 5 (1/2)
https://en.wikipedia.org/wiki/AlphaGo_versus_Lee_Sedol

262. AlphaGo versus Lee Sedol: Game 5 (2/2)
https://en.wikipedia.org/wiki/AlphaGo_versus_Lee_Sedol

263. Further Reading I
AlphaGo:
Google Research Blog
http://googleresearch.blogspot.cz/2016/01/alphago-mastering-ancient-game-of-go.html
an article in Nature
http://www.nature.com/news/google-ai-algorithm-masters-ancient-game-of-go-1.19234
a reddit article claiming that AlphaGo is even stronger than it appears to be:
“AlphaGo would rather win by less points, but with higher probability.”
https://www.reddit.com/r/baduk/comments/49y17z/the_true_strength_of_alphago/
Articles by Google DeepMind:
Atari player: a DeepRL system which combines Deep Neural Networks with Reinforcement Learning (Mnih
et al. 2015)
Neural Turing Machines (Graves, Wayne, and Danihelka 2014)
Artificial Intelligence:
Artificial Intelligence course at MIT
http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/
6-034-artificial-intelligence-fall-2010/index.htm
Introduction to Artificial Intelligence at Udacity
https://www.udacity.com/course/intro-to-artificial-intelligence--cs271

264. Further Reading II
General Game Playing course https://www.coursera.org/course/ggp
Singularity http://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html + Part 2
The Singularity Is Near (Kurzweil 2005)
Combinatorial Game Theory (founded by John H. Conway to study endgames in Go):
Combinatorial Game Theory course https://www.coursera.org/learn/combinatorial-game-theory
On Numbers and Games (Conway 1976)
Machine Learning:
Machine Learning course
https://youtu.be/hPKJBXkyTK://www.coursera.org/learn/machine-learning/
Reinforcement Learning http://reinforcementlearning.ai-depot.com/
Deep Learning (LeCun, Bengio, and Hinton 2015)
Deep Learning course https://www.udacity.com/course/deep-learning--ud730
Two Minute Papers https://www.youtube.com/user/keeroyz
Applications of Deep Learning https://youtu.be/hPKJBXkyTKM
Neuroscience:
http://www.brainfacts.org/

265. References I
Allis, Louis Victor et al. (1994). Searching for solutions in games and artificial intelligence. Ponsen & Looijen.
Baudiˇs, Petr and Jean-loup Gailly (2011). “Pachi: State of the art open source Go program”. In: Advances in
Computer Games. Springer, pp. 24–38.
Bowling, Michael et al. (2015). “Heads-up limit holdem poker is solved”. In: Science 347.6218, pp. 145–149. url:
http://poker.cs.ualberta.ca/15science.html.
Conway, John Horton (1976). “On Numbers and Games”. In: London Mathematical Society Monographs 6.
Corrado, Greg (2015). Computer, respond to this email. url:
http://googleresearch.blogspot.cz/2015/11/computer-respond-to-this-email.html#1 (visited on
03/31/2016).
Dieterle, Frank Jochen (2003). “Multianalyte quantifications by means of integration of artificial neural networks,
genetic algorithms and chemometrics for time-resolved analytical data”. PhD thesis. Universit¨at T¨ubingen.
Gatys, Leon A., Alexander S. Ecker, and Matthias Bethge (2015). “A Neural Algorithm of Artistic Style”. In:
CoRR abs/1508.06576. url: http://arxiv.org/abs/1508.06576.
Graves, Alex, Greg Wayne, and Ivo Danihelka (2014). “Neural turing machines”. In: arXiv preprint
arXiv:1410.5401.
Hayes, Bradley (2016). url: https://twitter.com/deepdrumpf.

266. References II
Karpathy, Andrej (2015). The Unreasonable Effectiveness of Recurrent Neural Networks. url:
http://karpathy.github.io/2015/05/21/rnn-effectiveness/ (visited on 04/01/2016).
Kurzweil, Ray (2005). The singularity is near: When humans transcend biology. Penguin.
LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton (2015). “Deep learning”. In: Nature 521.7553, pp. 436–444.
Li, Chuan and Michael Wand (2016). “Combining Markov Random Fields and Convolutional Neural Networks for
Image Synthesis”. In: CoRR abs/1601.04589. url: http://arxiv.org/abs/1601.04589.
Mnih, Volodymyr et al. (2015). “Human-level control through deep reinforcement learning”. In: Nature 518.7540,
pp. 529–533. url:
https://storage.googleapis.com/deepmind-data/assets/papers/DeepMindNature14236Paper.pdf.
Munroe, Randall. Game AIs. url: https://xkcd.com/1002/ (visited on 04/02/2016).
Silver, David et al. (2016). “Mastering the game of Go with deep neural networks and tree search”. In: Nature
529.7587, pp. 484–489.
Sun, Felix. DeepHear - Composing and harmonizing music with neural networks. url:
http://web.mit.edu/felixsun/www/neural-music.html (visited on 04/02/2016).