Many recent advances in deep reinforcement and artificial intelligence learning have stemmed from video games. In this session, we'll explore a brief history of this relationship, looking specifically at how Unity is pushing the boundaries of AI research with the Obstacle Tower Challenge. We'll also show how Unity is leveraging cutting-edge research to solve gaming's biggest challenges with the Unity Machine Learning Agents Toolkit, one of the most popular open-source toolkits for deep learning. Speaker: Matthew Crosby - Imperial College London

2. Driving AI Research
Using Games
Jeffrey Shih
Lead Product Manager, AI
Unity Technologies
@shihzy
Dr. Matt Crosby
Postdoctoral Researcher
Leverhulme Centre for the
Future of Intelligence
Imperial College London
@MaCroPhilosophy
Benjamin Beyret
Researcher
Leverhulme Centre for the
Future of Intelligence
Imperial College London
@BenBeyret

3. Video games have been
used to drive artificial
intelligence research
3

4. Video games have been
used to drive artificial
intelligence research
(for a long time)
4

5. Video games
5
A few examples

6. Video games
6
A few examples
● Atari 2600 Games - DeepMind, OpenAI

7. Video games
7
A few examples
● Atari 2600 Games - DeepMind, OpenAI
● Doom - VizDoom from Poznan University

8. Video games
8
A few examples
● Atari 2600 Games - DeepMind, OpenAI
● Doom - VizDoom from Poznan University
● Quake 3 - DeepMind

9. Video games
9
A few examples
● Atari 2600 Games - DeepMind, OpenAI
● Doom - VizDoom from Poznan University
● Quake 3 - DeepMind
● Minecraft - Microsoft Project Malmo

10. Video games
10
A few examples
● Atari 2600 Games - DeepMind, OpenAI
● Doom - VizDoom from Poznan University
● Quake 3 - DeepMind
● Minecraft - Microsoft Project Malmo
● Starcraft 2 - DeepMind / Blizzard

11. Video games
11
A few examples
● Atari 2600 Games - DeepMind, OpenAI
● Doom - VizDoom from Poznan University
● Quake 3 - DeepMind
● Minecraft - Microsoft Project Malmo
● Starcraft 2 - DeepMind / Blizzard
● Dota 2 - OpenAI Five

12. Video games
12
Some novel approaches in AI

13. Video games
13
Some novel approaches in AI
● Deep Q-Network (DQN) paper published by
DeepMind

14. Video games
14
Some novel approaches in AI
● Deep Q-Network (DQN) paper published by
DeepMind
● Proximal Policy Optimization (PPO) paper
published by OpenAI

15. Video games
15
Some novel approaches in AI
● Deep Q-Network (DQN) paper published by
DeepMind
● Proximal Policy Optimization (PPO) paper
published by OpenAI
● New AI systems and approaches to beating
top players in StarCraft II and DOTA 2

16. 16
Unity for AI
Research

17. 17
Unity for AI
Research

18. 18
Unity for AI
Research

19. Demis Hassabis - co-founder and CEO of
DeepMind
As a former video game designer myself, I couldn’t be more
excited to be collaborating with Unity, creating virtual
environments for developing and testing the kind of smart,
flexible algorithms we need to tackle real-world problems

20. AI-Based Challenges Using Unity
The Animal-AI Olympics

21. History of the Obstacle Tower Challenge
— February 18, 2019: Launched 1st qualifying round on AICrowd with Google
Cloud Platform as a co-sponsor. Over $100K in prizes.
— Qualifying round participation: 2000+ entries from 350+ teams
— May 15, 2019: 2nd round launched
— August 7, 2019: Winners announced and Obstacle Tower open-sourced

22. Obstacle Tower Environment
Research and benchmarks areas
— Vision
– High-fidelity 3D visuals
– Realtime lighting/shadows
— Control
– Platforming puzzles
— Planning
– Complex floor layouts
— Generalization
– Procedural Floors, Rooms, and Visuals
22

23. Different Floors and Themes
23

24. Obstacle Tower Environment
Procedurally generated design
— Each episode a new tower
– Each tower filled with 25 floors
– Each floor filled with rooms
– Each room filled with
obstacles and puzzles!
24

25. Learnings...
Winner: Alex Nicol
— Classifier for object identification
— Imitation Learning
— PPO (Proximal Policy Optimization)
for fine-tuning behavior
2nd: Gianni & Miha
— PPO with modifications
— Sampling algorithm
— 10 Billion steps sampled
3rd: Songbin Choi
— Standard PPO
— Human Play experience added

26. The Animal-AI Olympics
Download it today
github.com/beyretb/AnimalAI-Olympics 26
animalaiolympics.com

27. Animal Cognition Tasks
27

28. Animal Cognition Tasks
28
Can we build AI systems to do this?

29. Animal Cognition Tasks
29
Can we build AI systems to do this?
— No!
— It’s way too hard.

30. Animal Cognition Tasks
30
Can we build AI systems to do this?
— No!
— It’s way too hard.
Can we build a research pathway?

31. Animal Cognition Tasks
31
Can we build AI systems to do this?
— No!
— It’s way too hard.
Can we build a research pathway?
— Yes!
— Using Unity - ML-Agents

32. Robot Corvids
32

33. Robot Corvids
33
Detect food is inside the containers

34. Robot Corvids
34
Detect food is inside the containers
Detect Containers have open tops

35. Robot Corvids
35
Detect food is inside the containers
Detect
objects that
could help
Detect Containers have open tops

36. Robot Corvids
36
Detect food is inside the containers
Successfully grip object
Detect
objects that
could help
Detect Containers have open tops

37. Robot Corvids
37
Detect food is inside the containers
Successfully grip object
Notice different substances
Detect
objects that
could help
Detect Containers have open tops

38. Robot Corvids
38
Not to
mention
anything
required to
actually
solve the
problem!
Detect Containers have open tops
Detect food is inside the containers
Successfully grip object
Notice different substances
Detect
objects that
could help

39. Design Principles
39
— Simple tasks
— Simple affordances
— Abstract from animals

40. The Animal-AI Environment
40

41. The Animal-AI Environment: Possibilities
41
— Goal is to get food
— From simple building blocks
can build complex
structures/experiments.

42. The Animal-AI Environment
42

43. Use example: Maze Navigation
43
Test animals’ navigation skills
Claude Shannon

44. Use example: Maze Navigation
44
Claude Shannon

45. Use example: Maze Navigation
45
Claude Shannon

46. Use example: Maze Navigation
46
Claude Shannon

47. Use example: Maze Navigation
47
Claude Shannon
Experimentation design

48. Use example: Maze Navigation
48
Step 1: design training

49. Use example: Maze Navigation
49
Step 1: design training

50. Use example: Maze Navigation
50
Step 1: design training

51. Use example: Maze Navigation
51
Step 1: design training

52. Use example: Maze Navigation
52
Step 1: design training

53. Use example: Maze Navigation
53
Step 1: design training

54. Use example: Maze Navigation
54
Step 1: design training

55. Use example: Maze Navigation
55
Step 1: design training

56. Use example: Maze Navigation
56
Step 1: design training

57. Use example: Maze Navigation
57
Step 1: design training

58. Use example: Maze Navigation
In practice
AGENT ENVIRONMENT

59. Use example: Maze Navigation
In practice
AGENT ENVIRONMENT
LESSON

60. Use example: Maze Navigation
In practice
AGENT ENVIRONMENT
ACTION
LESSON

61. Use example: Maze Navigation
61
In practice
AGENT ENVIRONMENT
ACTION
OBSERVATIONS
REWARD
LESSON

62. Use example: Maze Navigation
62
In practice
AGENT ENVIRONMENT
ACTION
OBSERVATIONS
REWARD
LESSON

63. Use example: Maze Navigation
63
In practice
AGENT ENVIRONMENT
ACTION
OBSERVATIONS
REWARD
LESSON

64. Use example: Maze Navigation
64
In practice
AGENT ENVIRONMENT
ACTION
OBSERVATIONS
REWARD
LESSON

65. Use example: Maze Navigation
65
In practice
AGENT ENVIRONMENT
ACTION
OBSERVATIONS
REWARD
LESSON

66. Use example: Maze Navigation
66
Step 3: Enjoy

67. 10 Categories 300 tests
67
1. Food
2. Preferences
3. Obstacles
4. Avoidance
5. Spatial Reasoning
6. Generalisation
7. Internal Models
8. Object Permanence
9. Advanced Preferences
10. Causal Reasoning

68. Competition Results
68
— 50+ teams
— 25,000+ correct solutions
— One month left to go!

69. Competition Results
69

70. Competition Results
70

71. Competition Results
71
ME

72. Conclusions
72
— A gaming-style environment allows us to build an arena with
simulated physics and visual inputs.
— Many tests used on animals in the real world can be
translated to our environment.
— Creates a research framework for creating AIs with
animal-like skills.
— A crucial step towards Artificial General Intelligence.

73. Learn more!
Additional Information
(Point Camera)
Chat with us, demos and more!
Visit us at the Games AI Kiosk on
the Expo Floor!
https://tinyurl.com/unite-airesearch