AlphaGo uses deep learning algorithms like deep neural networks to master the game of Go. DeepMind's AlphaGo defeated top human Go players by using a technique called reinforcement learning to learn from vast amounts of self-play experience. The document discusses how reinforcement learning works and how AlphaGo applied techniques like Q-learning and deep neural networks to achieve a level of play beyond that of the best human players.

1. หัดเขียน A.I. แบบ
AlphaGo กันชิวๆ
Kan Ouivirach
http://www.tgdaily.com/web/134986-machine-learning-ai-and-digital-intelligences-effect-on-business

2. Kan Ouivirach
Research & Development
Engineer

3. http://www.bigdata-madesimple.com/
Machine Learning
“Given example pairs, induce such
that for given pairs and
generalizes well for unseen ”
–Peter Norvig (2014)
(x, y) f
y = f(x)
x

4. Dog
Cat
Generalization

5. Main Types of Learning
• Supervised Learning
• Unsupervised Learning
• Reinforcement Learning

6. “ไขความลับ อัลฟ่าโกะ การเรียนรู้แบบเชิงลึก
และอนาคตของปัญญาประดิษฐ์”
ดร. สรรพฤทธิ์ มฤคทัต
22 มี.ค. 2559

7. https://deepmind.com/alpha-go.html

8. Google DeepMind:
Ground-breaking AlphaGo masters the game of Go
https://www.youtube.com/watch?v=SUbqykXVx0A

10. We’ll do this today!

11. Minimax Decisions
Reinforcement Learning
and
http://www.123rf.com/photo_8104943_hand-drawn-tic-tac-toe-game-format.html

12. Minimax Decisions

13. MAX(X)
MIN(O)
MAX(X)
• 1 if wins
• -1 if loses

14. MAX(X)
MIN(O)
MAX(X)
+1

15. MAX(X)
MIN(O)
MAX(X)
+1
-1

16. MAX(X)
MIN(O)
MAX(X)
+1
-1
+1

17. MAX(X)
MIN(O)
MAX(X)
+1
-1
+1
-1

18. MAX(X)
MIN(O)
MAX(X)
+1
-1
+1
-1
-1

19. MAX(X)
MIN(O)
MAX(X)
+1
-1
+1
-1
+1
-1

20. MAX(X)
MIN(O)
MAX(X)
+1
-1
+1
-1
-1
+1
-1

21. argmaxa 2 ACTIONS(s)MIN-VALUE(RESULT(state, a))
MINIMAX-DECISION(state)function returns an action
return
MAX-VALUE(state)
TERMINAL-TEST(state) UTILITY(state)
v 1
ACTIONS(state)
v MAX(v, MIN-VALUE(RESULT(s, a)))
v
function
return
returns
then return
a utility value
afor each in do
if
TERMINAL-TEST(state) UTILITY(state)
ACTIONS(state)
v
function
return
returns
then return
a utility value
afor each in do
if
MIN-VALUE(state)
v 1
v MIN(v, MAX-VALUE(RESULT(s, a)))

22. https://github.com/zkan/tictactoe/blob/master/tictactoe_minimax.py

23. Alpha-Beta Pruning

24. [ 1, +1]
↵
max
min

25. 3
max
min
[ 1, +1]
[ 1, +1]
↵

26. [ 1, 3]
3
[ 1, +1]
↵
max
min

27. 3 12
[ 1, 3]
[ 1, +1]
↵
max
min

28. 3 12 8
[3, +1]
[ 1, 3]
↵
max
min

29. 3 12 8 2
[3, 2]
[3, +1]
[ 1, 3]
↵
max
min

30. 3 12 8 2
Prune!
[3, 2]
[3, +1]
[ 1, 3]
↵
max
min

31. http://researchers.lille.inria.fr/~munos/
Reinforcement
Learning

32. Reinforcement Learning (RL)
y = f(x)
z
Given x and z, find a function f that generates y.

33. Agent-Environment Interaction in RL
Agent
Environment
Rt+1
St+1
St Rt At
State Reward Action

34. Policy
• The learning agent's way of behaving at a given
time
• A mapping from perceived states of the
environment to actions to be taken when in
those states
• A simple lookup table

35. Agent
Environment
Rt+1
St+1
St Rt At
State Reward Action
I’m gonna find my optimal policy!

36. y = f(x)
z
Given x and z, find a function f that generates y.
Action
State
Reward
Policy

38. Google DeepMind's Deep Q-learning
playing Atari Breakout
https://www.youtube.com/watch?v=V1eYniJ0Rnk

39. Flappy Bird Hack using
Reinforcement Learning
http://sarvagyavaish.github.io/FlappyBirdRL/

40. http://quotes.lifehack.org/quote/albert-einstein/learning-is-experience-everything-else-is-just/

41. Mystery Game
https://github.com/guiferviz/rl_udacity

42. Exploration & Exploitation
http://mariashriver.com/blog/2012/10/balancing-happiness-and-heartache-in-alzheimers-kerry-lonergan-luksic/

43. Q-Learning
Q(s, a) Q(s, a) + ↵(R(s) + maxa0 Q(s0
, a0
) Q(s, a))
↵ learning rate
discount factor
Q table of action values indexed by state and action
s
a
previous state
previous action
a0
s0
R(s) reward in state s
action taken in state

44. Q-Learning through
Simple Example
1 2
3 4
5
Goal!

45. 1 2
3 4
5
Goal!
1 2
3 4
5 Goal!

46. 1 2
3 4
5 Goal!
0
0
0
00
0
0
0
00
R =
0
B
B
B
B
B
B
@
1 2 3 4 5
1 1 1 0 1 1
2 0 1 1 0 100
3 0 1 1 0 1
4 1 0 0 1 1
5 1 0 1 1 100
1
C
C
C
C
C
C
A
Define a reward function.

47. Q(s, a) Q(s, a) + ↵(R(s) + maxa0 Q(s0
, a0
) Q(s, a))
Suppose:
= 0.8
↵ = 1
1 2
3 4
5 Goal!
0
0
0
00
0
0
0
00
Q =
0
B
B
B
B
B
B
@
1 2 3 4 5
1 0 0 0 0 0
2 0 0 0 0 0
3 0 0 0 0 0
4 0 0 0 0 0
5 0 0 0 0 0
1
C
C
C
C
C
C
A
Let’s start here.
R =
0
B
B
B
B
B
B
@
1 2 3 4 5
1 1 1 0 1 1
2 0 1 1 0 100
3 0 1 1 0 1
4 1 0 0 1 1
5 1 0 1 1 100
1
C
C
C
C
C
C
A

48. 1 2
3 4
5 Goal!
100
0
0
00
0
0
0
00
Q(2, 5) = R(2, 5) + 0.8 ⇥ max(Q(5, 2)) Q(2, 5)
= 100 + 0.8 ⇥ 0 0
= 100
Q =
0
B
B
B
B
B
B
@
1 2 3 4 5
1 0 0 0 0 0
2 0 0 0 0 100
3 0 0 0 0 0
4 0 0 0 0 0
5 0 0 0 0 0
1
C
C
C
C
C
C
A
When choose to go to 5..

49. 1 2
3 4
5 Goal!
100
0
0
800
0
0
0
00
Q(4, 2) = R(4, 2) + 0.8 ⇥ max(Q(2, 1), Q(2, 4), Q(2, 5)) Q(4, 2)
= 0 + 0.8 ⇥ 100 0
= 80
Q =
0
B
B
B
B
B
B
@
1 2 3 4 5
1 0 0 0 0 0
2 0 0 0 0 100
3 0 0 0 0 0
4 0 80 0 0 0
5 0 0 0 0 0
1
C
C
C
C
C
C
A
When choose to go to 2..

50. 1 2
3 4
5 Goal!
100
0
0
800
64
0
0
00
Q(3, 4) = R(3, 4) + 0.8 ⇥ max(Q(4, 2), Q(4, 3)) Q(3, 4)
= 0 + 0.8 ⇥ 80 0
= 64
Q =
0
B
B
B
B
B
B
@
1 2 3 4 5
1 0 0 0 0 0
2 0 0 0 0 100
3 0 0 0 64 0
4 0 80 0 0 0
5 0 0 0 0 0
1
C
C
C
C
C
C
A
When choose to go to 4..

51. http://www.123rf.com/photo_8104943_hand-drawn-tic-tac-toe-game-format.html
State Space
• Board
Actions
• Move
Rewards
• +1 if A.I. wins
• -1 if A.I. loses
• 0.5 if the game’s a draw
Back to our Tic Tac Toe!

52. https://github.com/zkan/tictactoe/blob/master/tictactoe_rl.py

53. Supervised learning
and unsupervised
learning?

54. BetaGo
http://maxpumperla.github.io/betago/