Presented at Deepmind & Blizzard StarCraft2 AI Workshop [Description] Developing advanced StarCraft II RL agent is an extremely challenging project for me. So, I designed the roadmap to hack StarCraft II RL. My first naive idea is to develop Actor-Critic agent and optimal scripted agent. and I'll teach Actor-Critic agent through the trajectories of the optimal scripted agent. This idea will be tested on the mini-game map, "CollectMineralShards". - 2017.11.04 Anaheim Hilton Hotel

1. StarCra& II AI Workshop
[Deepmind&BlizzardStarCraftIIAI Workshop]
Teaching anActor-CriticAgentThrough
Optimal ScriptedAgentTrajectories
Chris Hoyean Song

2. StarCra& II AI Workshop
StarCra& has 4 levels of difficulty.
Easy Level
Intermediate Level
Expert Level
and ..

3. StarCra& II AI Workshop
The ulBmate level
Korean

4. StarCra& II AI Workshop
First, let me introduce
my hoFest girlfriend.

5. StarCra& II AI Workshop
RL

6. StarCra& II AI Workshop
Reinforcement Learning
Tensorflow. "Newbie" Contributor
Microsoft AI MVP
Tensorflow-KR Admin (Korean No1. ML Community)
Ex-game developer (using Unity3D)
sjhshy@gmail.com
Posting StarCraft II Reinforcement Learning Tutorials on
http://chris-chris.ai
Kakao Corp. Data Engineer
- Data Pipeline management, Real-time log processing
- Business Intelligence, Marketing Intelligence
- Develop APIs & Dev-ops

7. StarCra& II AI Workshop
Cool Chris presents
Insert Subtitle Text Here
Insert Subtitle Text Here
Insert Subtitle Text Here
1. Problem DefiniBon
2. Lessons learned from pysc2
3. Actor-CriBc ImitaBon Learning Agent

8. StarCra& II AI Workshop
Problem Definition1

9. StarCra& II AI Workshop
1. Problem Definition
Level 3 :
Level 2 :
Level 4 :
Level 1 :
My topic

10. StarCra& II AI Workshop
1. Problem Definition
1
Develop
Actor-CriBc
agent
2
Get opBmal
trajectories
3
Teach agent
with opBmal
trajectories

11. StarCra& II AI Workshop
1. Problem Definition
Key idea 1 :
SAME padding + 1x1 convoluBon layer
Key idea 2 :
Base acBon masking + autoregressive sub-acBon selecBon

12. StarCra& II AI Workshop
1. Problem Definition
{"step": 1, "done": false, "obs": [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0], [0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0,
0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], "base_acBon": 1, "x": 12, "y": 17, "reward": 0}
hFps://youtu.be/6byY0sXEk6U

13. StarCra& II AI Workshop
1. Problem Definition
Idea 1. Supervised Learning
Source : Mastering the Game of Go with Deep Neural Networks and Tree Search, Deepmind

14. StarCra& II AI Workshop
1. Problem Definition
Idea 2. Hybrid ?!
To be conBnued

15. StarCra& II AI Workshop
Lessons learned from pysc22

16. StarCra& II AI Workshop
Almost 100 million actions you can make at one step
Multi agents : Agents should cooperate for one common goal.
Complexity : Too large action / observation space.
Each agents should solve problems like below
- Strategy
- Economy
- Production
- Tactics
- Recon

17. StarCra& II AI Workshop
Lesson 1
one-hot encoding
1 : marine
3 : mineral
1 : marine 1: mineral
Marine Layer Mineral Layer
3 3 3 3
1 1
1 1 1 1
player_relative

18. StarCra& II AI Workshop
Lesson 2
Make the model simple
Agent can cover all possible
actions in StarCraft II RL
environment.
Agent can select an unit,
and handle control groups
and move them.
Agent can recall control
group, and move the unit.
14 policy network model 7 policy network model 3 policy network model (now)

19. StarCra& II AI Workshop
Lesson 3
tf.clip_by_norm()
This operaBon prevents your model from explosion.
It is very explicit way to handle model stability. Let's tune it!

20. StarCra& II AI Workshop
Lesson 4
32x32 map size
Default map size is 64x64
but, you don't need 4 pixel point
to represent one marine.
Reduce the map size
as much as you can.

21. StarCra& II AI Workshop
First trial
(up, down, left, right)
=> 4 action space
OpenAI baselines – DQN
Started to learn
hFps://youtu.be/xpdQYnnxAko

22. StarCra& II AI Workshop
Second trial
2 DQN Network
x => (32) action space
y => (32) action space
OpenAI baselines – DQN
Failed
hFps://youtu.be/RyFcWKWSGJE

23. StarCra& II AI Workshop
I've tried to implement
the baseline agent architecture on the paper.
Some of my architecture would be incorrect.
Oriol, please give me some feedback if you find any.

24. StarCra& II AI Workshop
Actor-critic model architecture 1
16filters
[?, 32, 32, 16]
32filters
[?, 32, 32, 32]
observaBon
[?, 32,32,3]
5x5 3x3
1 filter
[?, 32, 32, 1]
1x1
convoluBon
1x1

25. StarCra& II AI Workshop
Actor-critic model architecture 2
3x3 1x1
16filters
[?, 32, 32, 16]
32filters
[?, 32, 32, 32]
1 filter
[?, 32, 32, 1]
1x1
convoluBon
We take 1x1 convoluBon layer for the dimension reducBon.
1x1 convoluBon layer squeeze 32 layers into 1 layer.

26. StarCra& II AI Workshop
Actor-critic model architecture 3
1x1
32filters
[?, 32, 32, 32]
1 filter
[?, 32, 32, 1]
1x1
convoluBon
conv_to_fc
[?, 1024, 1]
argmax

27. StarCra& II AI Workshop
Actor-critic model architecture 4
1x1
32filters
[?, 32, 32, 32]
fully_connected
[?, 524, 1]
argmax
conv_to_fc
[?, 32768, 1]
fully_connected
[?, 256, 1]
base acBon : 4/ SELECT_CONTROL_GROUP
masked policy
[?, 524, 1]
masking
unavailable
acBons
env
available
acBons

28. StarCra& II AI Workshop
Actor-critic model architecture 5
pi
fully_connected
[?, 524, 1]
conv_to_fc
[?, 32768, 1]
fully_connected
[?, 256, 1]
sub_acBon 4 / control_group_act
: Recall 0, Set 1, Append 2,
SetAndSteal 3, AppendAndSteal 4
sub_acBon 11 / build_queue_id
if we do not use this result,
do not back-propagate the errors on this layer
argmax
masked pi_sub4
[?, 5, 1]
masked pi_sub11
[?, 10, 1]
argmax
masked pi_sub5
[?, 10, 1]
sub_acBon 5 / control_group_id
: 0 ~ 9

29. StarCra& II AI Workshop
Actor-critic model architecture 6
1x1
32filters
[?, 32, 32, 32]
vf
[?, 1, 1]
conv_to_fc
[?, 32768, 1]
fully_connected
[?, 256, 1]

30. StarCra& II AI Workshop
Third trial
Agent can cover all possible
actions in StarCraft II RL
environment.
14 policy network model
hFps://youtu.be/oMax7wq-KCs

31. StarCra& II AI Workshop
Fourth Trial
Agent can select an unit, and
handle control groups and move
them.
7 policy network model
hFps://youtu.be/gNp3OnPFSRc

32. StarCra& II AI Workshop
Fifth Trial
Agent can recall control group,
and move the unit.
3 policy network model (now)
hFps://youtu.be/V0HqK1-VfLw

33. StarCra& II AI Workshop
Hybrid Actor-Critic Imitation Learning3

34. StarCra& II AI Workshop
Question
I know how to develop
optimal scripted agents.
but how can I make
Actor-Critic agent
learn from it?
Question
Actor-Critic Agent
(Learning Agent)
Optimal Scripted Agent
(Optimal Agent)

35. StarCra& II AI Workshop
Seems naive,
but intuitively make sense.
I propose new way to train Actor-Critic
Agent through optimal scripted agent
trajectories.

36. StarCra& II AI Workshop
Actor-Critic Imitation Learning Agent
Policy
Agent 1 Agent 2 Agent 3 Agent 4
Policy
Actor-CriBc
Policy & Value Network
Policy Policy
trajectories

37. StarCra& II AI Workshop
Actor-Critic Imitation Learning Agent
Policy
Optimal
Script
Agent 1 Agent 2 Agent 3 Agent 4
Optimal
ScriptPolicy
Actor-CriBc
Policy & Value Network
trajectories

38. StarCra& II AI Workshop
baseline : Pure A2C Agent

39. StarCra& II AI Workshop
1. Learn faster
2. Higher max reward
3. Robustness
Learning Rate : Random (e-5 ~ e-3)
X axis : episodes
Y axis : rewards

40. StarCra& II AI Workshop
Detailed description of this idea will be
covered on the paper or the blog post.
The source code is on my Github
https://github.com/chris-chris/pysc2-examples
python train_mineral_shards.py
--algorithm=a2c --num_agents=2
--num_scripts=2

41. StarCra& II AI Workshop
Insert Subtitle Text Here
Insert Subtitle Text Here
Insert Subtitle Text Here
1. Problem DefiniBon
Goal : Make the RL agent learn from expert game plays.
2. Lessons learned from pysc2
Simple model / one-hot encoding / u.clip_by_norm() / Actor-CriBc Architecture
3. Hybrid Actor-CriBc ImitaBon Learning
Hybrid agent learns faster, beFer with robustness.
Wrap up

42. StarCra& II AI Workshop
- Special Thanks to
Seungil You(Google) helped me to understand the papers and the algorithms
and supported me to find bugs and improve my tensorflow RL model.
- Deepmind and Blizzard Team
Thank you for the StarCraft II Learning Environment.
- Thanks for the precious advices
Sungjoon Choi(Disney Research), Nako Sung(Naver),
Woongwon Lee(RLCode), Doyun Lee(NC Soft).

43. StarCra& II AI Workshop
Thank you
sjhshy@gmail.com
http://chris-chris.ai
https://www.facebook.com/ai.chris.chris
https://www.linkedin.com/in/chris-song-0bb03439/