An introduction into Reinforcement Learning, with an outlook on some of the most prominent problems and promising research from the past couple of years.

1. 1
An introduction to
by Xander Steenbrugge
Reinforcement
Learning

2. 2
The difference in mind between man
and the higher animals, great as it is,
certainly is one of degree and not of
kind.
-- Charles Darwin
“

3. 3
Overview
1. A brief history of AI
2. Machine Learning today
3. Introduction to Reinforcement Learning
4. Problems in Reinforcement Learning
5. Promising Research
6. A look into the future

4. 4
A brief history of AI

5. 5
1950: Alan Turing’s “Turing test”

6. 6
1997:
IBM’s
Deep Blue

7. 7
Chess branching factor ~ 35

8. 8
Chessbot basics
MiniMax Search
● Concept:
“Maximize the evaluation of your
move while minimizing opponent's
move evaluation”
● Reviews each possible move sequence
● Has high time cost since every possible
future board position must be evaluated

9. 9
Minimax strategy + brute force heuristics based state
evaluations

10. 10
2011:
Apple’s Siri
IBM wins Jeopardy

11. 11

12. 12
The Deep Learning Revolution: ImageNet 2012
Deep Learning on Google Trends

13. Ultimate Machine Learning with Google Cloud 13
The old, algorithmic approach
“apple”
“orange”
“banana”
IF (round) THEN
IF (orange AND coarse) THEN
“orange”
ELSE IF (green AND smooth) THEN
“apple”
ELSE IF ...
...
ELSE IF …
“banana”

14. Ultimate Machine Learning with Google Cloud 14
Let the machine find the rules
“apple”
“orange”
“banana”
?

15. Confidential & Proprietary
ConvNets

16. Keys to Successful ML
Large Datasets Good Models Lots of Computation

17. 17
Machine Learning Today

18. Confidential + Proprietary

19. Machine Learning is everywhere
already…

20. 20
Rapidly Accelerating Use of Deep Learning at Google
Number of projects using some form of deep learning
2012 2013 2014 2015
1500
1000
500
0
Used across products:

21. 21Confidential & ProprietaryGoogle Cloud Platform 21
Speech recognition
Audio Input
Deep
Recurrent
Neural Network
Text Output
● Reduced word errors by more than 30%
● 20% of Mobile queries are Voice Search
Google Research blog, August 2012, August 2015
“How cold is it
outside?”

22. Confidential & ProprietaryGoogle Cloud Platform 22
Google Translate

23. Confidential + Proprietary

24. Datatonic & you
PLACE IMAGE HERE
24
In popular culture:
+ ‘The next big thing’
+ sentient AI in the next 10 years
+ Will put humans out of a job
+ Foolproof
Machine learning:

25. Datatonic & you
PLACE IMAGE HERE
25
Really:
+ Been around for 60 years now
+ ‘Sentient next year’, every
year, for the last 60 years.
+ AI winters: 1970, 1990, … ?
+ Not foolproof
Machine learning:

26. 26
A person on a beach
flying a kite.
A person skiing down a
snow covered slope.
A group of giraffe standing
next to each other.

27. 27
A woman riding a horse
on a dirt road.
An airplane is parked on
the tarmac at an airport.
A group of people standing
on top of a beach.

28. 28

29. 29
Image classifiers are easily fooled!

30. 30

31. 31
● Supervised: need large amounts of annotated
training data
● Static inference machines
● Bad transfer learning capabilities to new tasks
Practical limitations of current AI systems

32. 32
Introduction to
Reinforcement Learning

33. 33

34. 34
Policy

35. 35
Policy

36. 36

37. 37
Policy networks
Raw Pixels

38. 38

39. 39
Policy Gradients
Run a policy for a while. See what actions led to high rewards. Increase their probability.

40. 40
Practical Applications of RL

41. 41

42. 42

43. Data Centre cooling

44. Confidential + Proprietary

45. Confidential + Proprietary

46. 46
Chess branching factor ~ 35

47. 47
Go branching factor ~ 250

48. 48
Neural nets to the rescue!

49. 49
Deep Learning enhanced MiniMax

50. 50
Advantages that AlphaGo can leverage
1. Fully deterministic: no noise in the game
2. Fully observed: each player has complete information and there are
no hidden variables. (unlike Poker for example)
3. Discrete action space.
4. Each game is relatively short (approximately 200 actions).
5. Target function is clear (win/lose) & fast to evaluate.
6. Huge datasets of human gameplay are available to bootstrap the
learning, so AlphaGo doesn’t have to start from scratch.

51. 51

52. 52
3:53

53. 53
Image Segmentation

54. 54
Deepdrive in GTA V

55. 55
Open AI Universe
Universe Starter Agent

56. 56
Problems in
Reinforcement Learning

57. 57
Bad sample efficiency

58. 58
Cold Start

59. 59
Exploration vs Exploitation

60. 60
Subgoal creation

61. 61
Promising Research

62. 62
Attention

63. 63

64. 64
Memory

65. 65
Domain Transfer

66. 66
PathNet

67. 67
Physical Intuition
Simulation Real
Ground Truth
Prediction

68. 68
Auxiliary Learning Signals
Policy

69. 69
Auxiliary Learning Signals

70. 70
Auxiliary Learning Signals

71. 71
Auxiliary Learning Signals (continued)
Divide observations in 3 classes:
1. Things that the agent can control
2. Things the agent cannot control but affect it
3. Things the agent cannot control but do not affect it
A good feature space for curiosity should model (1) and (2) and
be unaffected by (3).

72. 72
Auxiliary Learning Signals (continued)

73. 73
Auxiliary Learning Signals (continued)

74. 74
Learning to communicate
Observations
Memory State
Message
Policy
Network
Memory
State
Message

75. 75
Third person imitation learning

76. 76
Adversarial Networks

77. 77
Generative Networks

78. 78
Generative Networks

79. 79
Generative Networks

80. 80
Unsupervised Learning’s potential
7
2
1
0
4
FFN < Auto-Encoder < GAN

81. 81

82. Personal thoughts
▪ “Intelligent software will become the main driver of
most technological advances in the next decade”
➢ Self driving cars

83. ▪ “Intelligent software will become the main driver of
most technological advances in the next decade”
➢ Self driving cars
➢ Personal, digital assistants (Siri, Viv, Alexa, ...)
Personal thoughts

84. ▪ “Intelligent software will become the main driver of
most technological advances in the next decade”
➢ Self driving cars
➢ Personal, digital assistants (Siri, Viv, Alexa, ...)
➢ Machine generated/augmented content
Personal thoughts

88. ▪ “Intelligent software will become the main driver of
most technological advances in the next decade”
▪ “Virtual Reality (VR) will become a mainstream
experience sharing platform”
Personal thoughts

89. ▪ “Intelligent software will become the main driver of
most technological advances in the next decade”
▪ “Virtual Reality (VR) will become a mainstream
experience sharing platform”
▪ “Natural language processing will be fundamental
to interacting with all of these new technologies”
Personal thoughts

90. Plenty of problems to solve...

91. Plenty of solutions around...

93. 93
Thanks!

94. 94
We are hiring!
datatonic.com