OpenAI has made progress towards its mission of developing beneficial artificial general intelligence (AGI). This includes developing OpenAI Five, which reached the world's top level at the game Dota 2 through reinforcement learning techniques. OpenAI has also achieved dexterous in-hand manipulation with Dactyl and photorealistic image generation with generative adversarial networks. While challenges remain, deep learning has made rapid progress in areas like computer vision, translation and reinforcement learning. With continued growth in computing power, near-term AGI should be considered a serious possibility and risks should be proactively addressed.

1. Progress towards the OpenAI mission
Ilya Sutskever
Co-founder and Chief Scientist, OpenAI
NOVEMBER 9, 2018

2. OpenAI’s mission
OpenAI’s mission is to ensure that artificial general intelligence
(AGI) — by which we mean highly autonomous systems that
outperform humans at most economically valuable work —
benefits all of humanity.
— The OpenAI Charter

3. Technical progress from OpenAI

4. OpenAI Five

5. Dota

6. Dota is hard
Partial observability
120 heroes (we integrated 18)
20,000 actions per game, massive action space
Pros dedicate their lives to the game, 10K+ hrs of deliberate
practice

7. Dota is popular
Largest professional scene
Annual prize pool of $40M+

8. Dota

9. Our approach
Very large scale reinforcement learning
millennia of practice
LSTM policy = honeybee brain
Self play
Reward shaping

10. Reinforcement learning (RL) actually works!
Nearly all RL experts believed that RL can’t solve tasks as hard
as Dota
Horizon too long

11. Results

12. Dactyl

13. Dexterity

14. Diverse objects

15. Strategy: Sim 2 Real

16. Domain randomization
Train in simulation: randomize perception and physics

17. Domain randomization

18. Curiosity-based exploration

19. Core idea
Novel states = reward
Fix all bugs
Very hard to do

20. Montezuma’s Revenge

21. Montezuma’s Revenge

23. Mario

24. Curiosity early in training

25. Curiosity late in training

26. The OpenAI Mission

27. OpenAI’s mission
OpenAI’s mission is to ensure that artificial general intelligence
(AGI) — by which we mean highly autonomous systems that
outperform humans at most economically valuable work —
benefits all of humanity.
— The OpenAI Charter

28. Impact of AGI
Generate massive wealth
Potential to end poverty, achieve material abundance

29. Impact of AGI
Generate massive wealth
Potential to end poverty, achieve material abundance
Generate science and technology
cure disease, extend life, superhuman healthcare
mitigate global warming, clean the oceans, etc
massively improve education and psychological well being

30. Why is OpenAI’s mission relevant today?
We review progress in the field over the past 6 years
Our conclusion: near term AGI should be taken as a serious
possibility

31. Algorithms

32. Deep Learning at the root of it all
During the past 6 years, deep learning repeatedly and rapidly
broke through “insurmountable” barriers
A multilayered Perceptron
trainable with backpropagation

33. Vision (2012-2016)
HOG feature (2005)—image by Antonio Torrabla

34. Vision (2012-2016)
ImageNet Classification Error (Top 5)
Human 2015
(ResNet)
2016
(GoogleLeNet-v4)
0
5
10
15
20
25
30
3.13.6
5.0
2011
(XRCE)
2012
(AlexNet)
2013
(ZF)
2014
(VGG)
2014
(GoogleLeNet)
26.0
16.4
11.7
7.3 6.7

35. Translation (2014-2018)
BLEU score on EN to FR translation on the WMT dataset

36. Image generation (2014-2018)
GANs over the years: 2014
Goodfellow et al, 2014

37. Image generation (2014-2018)
GANs over the years: 2015
Radford et al, 2015

38. Image generation (2014-2018)
GANs over the years: 2017
Karras et al, 2017

39. Image generation (2014-2018)
GANs over the years: 2018
Brock et al, 2018

40. Reinforcement Learning (2013-2018)
DQN (2013)
Mnih et al, 2013

41. Reinforcement Learning (2013-2018)
TRPO (2015)
Schulman et al, 2015

42. Reinforcement Learning (2013-2018)
AlphaGo (2016)
Silver et al, 2016

43. Reinforcement Learning (2013-2018)
OpenAI Five (2018)
Very large scale: +100,000 CPU cores, +1000 GPUs

44. Reinforcement Learning (2013-2018)
Dactyl (2018)

45. Compute

46. Compute grows rapidly
Neural networks can usefully consume all available compute
Neural networks are extremely parallelizable
300,000x increase in neural net compute used for the largest neural
net experiments over the past 6 years

47. Petaflop/s-days to train
Early production convnet

48. Conclusion

49. Formidable challenges remain
Unsupervised learning
Robust classification
Reasoning
Abstraction
???

50. We’ve been breaking through barriers for 6 years
Will this trend continue, or will it stop?
And if so, when?

51. Means proactively thinking about risks:
Machines pursuing goals misspecified by
their operator
Malicious humans subverting deployed systems
Out-of-control economy that grows without resulting
in improvements to human lives
THIS TALK’S GOAL IS TO PRESENT EVIDENCE THAT:
While highly uncertain,
near-term AGI should be taken
as a serious possibility.

52. Thank you