This document discusses using reinforcement learning and meta-level reasoning to develop adaptive high-level strategies in StarCraft. It introduces Q-learning as a reinforcement learning technique to update state-action values. A meta-level reinforcement learning approach is then used to select exploration rates and learning rates over time based on the agent's need to learn. Experiments inject code into StarCraft using BWAPI to implement different Terran strategies and apply meta-level reasoning to select strategies and dynamically adjust hyperparameters. Results showed this approach enabled the development of adaptive high-level strategies in the complex StarCraft environment.

1. ADAPTIVE HIGH-LEVEL
STRATEGY LEARNING IN
STARCRAFT
SBGames 2013
Jiéverson Maissiat and Felipe Meneguzzi

2. Background

3. Reinforcement Learning
A learning technique for agents acting in the
environment, observing the reached states and
the received rewards at each step

4. Reinforcement Learning

5. Q-Learning
Updates the value of a state-action pair after
the action has been taken in the state and an
immediate reward has been received

6. Q-Learning
Q(s,a)
●
●
●
s represents the current state of the world;
a represents the action chosen by the agent;
Q(s,a) represents the value obtained the last time action a was executed at state s. This value is
often called Q-value.

7. Q-Learning
Q(s,a)
●
r
r represents the reward obtained after performing action a in state s;

8. Q-Learning
Q(s,a)
●
fit is responsible to adjust the reward;
fit( r )

9. Q-Learning
Q(s,a) = Q(s,a)
●
+
The Q-value is increased with the adjusted reward;
fit( r )

10. Q-Learning
Q(s,a) = Q(s,a)
●
+
α
*
r
α is the learning-rate, which determines the weight of new information over what the agent
already knows: a factor of 0 prevents the agent from learning anything (by keeping the Q-value
identical to its previous value) whereas a factor of 1 makes the agent consider all newly obtained
information;

11. Q-Learning
Q(s,a) =
●
0
If the Q-value starts in 0...
+
α
*
r

12. Q-Learning
Q(s,a) =
●
0
and the received reward is 2...
+
α
*
2

13. Q-Learning
Q(s,a) =
●
0
with a learning-rate of 0.3...
+
0.3 *
2

14. Q-Learning
0.6
●
=
0
than the Q-value is updated to 0.6;
+
0.3 *
2

15. Q-Learning
-2.4
●
=
0.6
+
Others updates will be done over the first;
0.3 *
-10

16. Q-Learning
Q(s,a) =
●
And so it follows.
-2.4
+
0.3 *
r

17. Parameter Control
α / learning-rate
Influence how the state values are computed,
and ultimately how a policy is generated

18. Learning Rate: α
●
High Learning-rate?
It can prevent the algorithm from converging, or lead to inaccuracies in the
computed value of each state (e.g. a local maximum);
●
Low Learning-rate?
It can hampers the convergence of the Q-value;
●
Varying the Learning-rate?
A typical way to vary the α-value, is to start interactions with a value close
to 1, and then decrease it over time toward 0.However, this approach is not
effective for dynamic environments, since a drastic change in the
environment with a learning-rate close to 0 prevents the agent from
learning the optimal policy in the changed environment.

19. Exploration Policy
ε-greedy
It has the probability ε to select a random action, and a
probability 1 - ε to select the optimal action (which has the
highest Q-value)

20. Meta-Level Reasoning

21. Meta-Level Reasoning

22. StarCraft:
Brood War

23. StarCraft
●
Real-Time Strategy
(RTS)
●
Resource Management
●
Battle!

24. StarCraft

25. StarCraft

26. META-LEVEL
REINFORCEMENT
LEARNING

27. Meta-Level Reasoning

28. Meta-Level Reasoning
Reinforcement
Learning

29. Meta-Level Learning

30. Meta-Level Reasoning
ε-greedy

31. Meta-Level Action Selection
ε-greedy modified:
Instead of keeping the exploitation-rate (ε-value) constant, we apply the
same meta-level reasoning to the ε-value, increasing the exploration rate,
whenever we find that the agent must increase its learning-rate.
The more the agent wants to learn, the more it wants to
explore; if there is nothing to learn, there is nothing to
explore.
ε=α

32. Experiments
in StarCraft

33. Code Injection at StarCraft
● BWAPI
○ Framework in C++ to inject code at
StarCraft;
● BTHAI
○ Bot implemented in BWAPI, with different
high-level strategies;

34. A Reinforcement Learning
Approach for StarCraft
● High-Level Strategy
○ Learning and selection of strategies rather
than micro-actions;
● Learn the end of each match
○ Feedback of victory: +1
○ Feedback of defeat: -1
● Terran vs CPU
○ It was assumed that the agent plays only
as Terran;

35. List of Terran Strategies
of BTHAI
● Marine Rush
● Wraith Harass
● Terran Defensive
● Terran Defensive FB
● Terran Push

36. Results

41. Source Code
BWAPI & StarCraft
https://github.com/jieverson/BTHAIMOD

43. Thank You
contact@jieveron.com
Porto Alegre, 18 de outubro de 2013