The past few years, some incredible breakthrough was made through games like Alpha Go, to master the games of go; or OpenAI Five to beat the Dota 2 champions. This talk aims to present the machine learning aspects of the AI and games research, which methods are used to create reliable interacting agent and the next frontier of this domain.

1. Why is it important to
solve games with AI?
Anaëlle Laurans
22nd January, 2020 — Meetup WiMLDS Paris
@AnaelleLaurans alaurans.com

2. This talk is about ...
● the research side of the artificial
intelligence for playing games.
○ Doesn’t cover AI in video games that is part
of the game design
● the benefits of solving games for the
research
● some machine learning methods
used in this field
● some challenges to solve

3. AI and Games in 1950’s
1952 - Alexander Douglas - Tic-Tac-Toe
He creates a program that mastered the game.
1959 - Arthur Samuel - Checkers
He creates a self-learning program that inspires
reinforcement learning

4. Reason #1
Games are interesting because
designed to challenge the human
brain.
They are perfect to test our
capacity to develop better AI.

5. Game are milestones
State-space complexity
number of finite state spaces
that a game can offer
Tic-Tac-Toe
● 3 values ( X, O, empty)
● 9 spaces
● Complexity:
legal states

6. Game are milestones
Chess
possible states
solved in 1997
Go
possible states
solved in 2016
Starcraft II
possible states
AlphaStar algorithm presented at
NeurIPS 2019
Universe
protons

7. Reason #2
Games offer rich
human-computer interaction that
involves collaboration or
competition with other players.
Solving games is also learning to
play with humans.

8. Cooperative mode in multi-players games
Our testers reported feeling supported by
their bot teammates, that they learned
from playing alongside these advanced
systems, and that it was generally a fun
experience overall.
Dota 2: Each team has 5 players who start from a corner and must take
the opposite base
Experiment with the OpenAI Five
bots that team alongside human
Quote from OpenAI blog

9. Reason #3
Games use multimodal inputs
and multimodal outputs in a
spatio-temporal context.
Solving games is also challenge
for all AI areas.

10. Multimodality from the agent perspective
Diverse nature of actions:
keyboard combination, pointer,
scrolling, gesture, speech, ...
Diverse nature of the
observation from the agent:
image, text, cue sounds, 3d
space, ...
Challenge to
fusion the
inputs with
temporal
context in
mind

11. What is used to create
engaging and effective
agents?

12. Reinforcement Learning
● The agent learns an optimal policy (aka behavior or strategy) by trial and error
● Needs feedback on its own action
● Its action affects the futur state (aka observation) it receives
● The agent tries to maximize its rewards
Definition: study of learning intelligent behavior
State
AGENT ENVIRONMENT
Reward
Action

13. Deep Learning + Reinforcement Learning
Mnih, Volodymyr, et al. "Human-level control through deep reinforcement learning." Nature
518.7540 (2015): 529.
In 2015, DeepMind
released a paper on
solving Atari games with
deep learning: Deep
Q-Learning.
It takes an image as input
and outputs a value for
each available actions.
Deep Learning is not the
only contribution but it
allows to scale RL to
more complex games.

14. Reinforcement learning landscape

15. Applications of RL
outside games
● Neural architecture search
● Advertising
● Resource managements (energy
control, web delivery, clusters)
● Personalized recommendations
● 3D Animations
● Finance
● ...

16. My work on my spare time
Problem: Today RL is good to resolve one problem.
Research interests: RL Generalization and skills transferability with
procedurally-generated environments
ProcgenenvironmentsbyOpenAI

17. My work on my spare time
Work on interactive visualization
and improving workflow to work
on RL at home.
Visualization using Bokeh Passing to a visualization through GUI

18. Take home message
Games are great if you want to work
on problems:
● with interaction and behaviors
● with multi-modal inputs and
outputs
● with competition / cooperation
between players (bot and human)
Algorithms developed on games can
be used in other domains.

19. Thank you
for you
attention
19
@AnaelleLaurans
alaurans.com