Trabajo Ya Publicado presentado en CoSECiVi'14. Resumen: Flocking strategies are sets of behavior rules for the interaction of agents that allow to devise controllers with reduced complexity that generate emerging behavior. In this paper, we present an application of genetic algorithms and flocking strategies to control the Ghost Team in the game Ms. Pac-Man. In particular, we define flocking strategies for the Ghost Team and optimize them for robustness with respect to the stochastic elements of the game and effectiveness against different possible opponents by means of genetic algorithm. The performance of the methodology proposed is tested and compared with that of other standard controllers. The results show that flocking strategies are capable of modeling complex behaviors and produce effective and challenging agents.

1. Evolving Evil: Optimizing Flocking Strategies
through Genetic Algorithms for the Ghost Team
in the Game of Ms. Pac-Man
F. Liberatore (University Rey Juan Carlos)
A. Mora (University of Granada)
P. Castillo (University of Granada)
J.J. Merelo (University of Granada)

2. Outline
- The Game of Ms. Pac-Man.
- State of the Art.
- Flocking.
- Flocking Strategies for the Ghost Team.
- A Genetic Algorithm for Flocking Strategies.
- Experiments.
- Conclusions.

3. What is this about?
In this work we propose:
- A Controller for the Ghost Team in the
game of Ms. Pac-Man …
- … based on Flocking Strategies (more
about this in the following) …
- … optimized by a Genetic Algorithm.

4. The game of Ms. Pac-Man [1/2]
The game of Ms. Pac-Man:
- Released on 1981.
- Chosen for the Ms. Pac-Man vs Ghosts
competition.
- Participants can submit controllers for
Ms.Pac-Man and the Ghosts.
- The objective of Ms. Pac-Man is to
maximize the final score.
- The objective of the Ghosts is to minimize
it.

5. The game of Ms. Pac-Man [2/2]
Movement restrictions for the Ghosts:
- A ghost can never stop.
- A ghost can choose its direction only at a
junction.
- A ghost cannot turn back on itself.
- At every tick of the game all the ghosts
obligatorily reverse their direction
according to a small random probability.

6. State of the Art [1/2]
Most contributions proposed controllers for Ms. Pac-Man:
- Genetic Programming (Alhejali and Lucas 2010, 2011, Brandstetter and
Ahmadi 2012),
- Genetic Algorithms for parameters optimization (Thawonmas 2010),
- Evolved Neural Networks (Burrow and Lucas 2009, Keunhyun and
Sung-Bae 2010),
- Ant Colonies (Martin et al. 2010),
- Monte Carlo Tree Search (Samothrakis et al. 2011, Ikehata and Ito
2011, Alhejali and Lucas 2013),
- Reinforcement Learning (Bom et al. 2013).

7. State of the Art [2/2]
Controllers for the Ghosts:
- Monte Carlo Tree Search (Nguyen and Thawonmas 2011,
2013),
- Influence Maps (Svensson and Johansson 2012),
- Neural Networks (Jia-Yue et al. 2011),
- Genetic Algorithm + Rules (Gagne and Congdon 2012),
- Competitive Co-Evolution of Ms. Pac-Man and Ghosts
controllers (Cardona et al. 2013).

8. Flocking [1/2]
Swarm Intelligence technique:
- Coordinated intelligence that arises from the collective behavior
of decentralized, self-organized systems.
Flocking:
- Developed to mimic lifelike behaviors of groups of beings.
- Flocking systems consist of a population of simple agents (or
boids) interacting locally with one another depending on the
distance between them.

9. Flocking [2/2]
- The agents follow very simple steering behaviors:
· Separation: makes the agent steer away from close flock mates.
· Alignment: makes the agent steer toward the average heading of
the flock.
· Cohesion: makes the agent steer toward the average position of
distant flock mates.
- The interactions between such agents lead to the emergence of
"intelligent" global behavior

10. Flocking Strategies [1/4]
- Generalization of the concept of Flocking.
- The direction taken by a ghost at a junction depends on:
· State:
- Normal,
- Vulnerable,
- Flashing.
· Position WRT that of the other actors:
- Ms. Pac-Man,
- Powerpills,
- Other Ghosts (depending on their state).

11. Flocking Strategies [2/4]
- Each Ghost is surrounded by
concentric rings called
Neighborhoods.
- The neighborhoods can have different
radius and size.
- Each neighborhood defines the
strength of the interaction between
the ghost and the other actors:
· Negative value → Separation
· Positive value → Cohesion

12. Flocking Strategies [3/4]
Formally, a Flocking Strategy is represented by a tuple (N, δ,
Α):
- N, number of neighborhoods, indexed by n=1,…,N.
- δ∈ℝN
>0
, vector of N real positive numbers. Each element δn
is
associated to one neighborhood and represents its maximum
radius.
- Α∈Μ, three-dimensional matrix with real entries. Each element
αs,a,n
is the magnitude of the steering force on the current ghost in
state s, resulting from the interaction with the actor a, falling into
neighborhood n.

13. Flocking Strategies [4/4]
- First, all the interactions with the other
actors are calculated as vectors.
- Then, all the vectors are summed to
find the total interaction vector.
- Finally, the direction taken by the
ghost (UP, DOWN, LEFT, RIGHT)
depends on the component of the
total interaction vector having
maximum absolute value.

14. A Genetic Algorithm for Flocking
Strategies [1/3]
Flocking Strategies could be designed manually by an expert.
We propose a standard Genetic Algorithm

15. A Genetic Algorithm for Flocking
Strategies [2/3]
The Genetic Algorithm has been applied for the automatic
definition of effective Ghosts agents:
- Each individual is a Flocking Strategy, represented as a tuple
(N, δ, Α).
- Random initial population.
- Roulette wheel selection.
- Cross-over: two parents generate one child by random
recombination of their chromosomes.

16. A Genetic Algorithm for Flocking
Strategies [3/3]
- Fitness function:
· The objective is to create effective Ghosts that perform well against
different Ms. Pac-Man strategies.
· Each individual is tested against two Ms. Pac-Man agents:
StarterPacMan (SPM) and NearestPillPacMan (NPPM).
· To ensure robustness in the face of random events, the game is
simulated 30 times for each Ms. Pac-Man agent and 95% confidence
intervals are calculated.
· Fitness is given by:
FITNESS = 1/CI+
SPM
+ 1/CI+
NPPM
Where CI+
represents the upper limit of the 95% confidence interval.

17. Experiments [1/2]
Comparison of the performance of the Ghosts controllers
obtained with different values of the parameter N (number of
Neighborhoods)
N=1 N=2 N=3 N=4 N=5
Best
FITNESS-1
783.38 726.84 815.66 766.96 720.20
Avg.
FITNESS-1
871.30 ±
55.45
861.57 ±
70.13
876.31 ±
65.06
905.86 ±
62.66
863.17 ±
72.15
Avg. CPU
time (s)
1373 ±
150.66
1484.3 ±
122.01
1561 ±
193.94
1562.60 ±
109.90
1473.00 ±
74.02

18. Experiments [2/2]
Performances of the controllers included in the competition
framework.
Controller Aggressive
Ghosts
Legacy Legacy 2
The
Reckoning
Random
Ghosts
Starter
Ghosts
FITNESS-1
1893.13 2210.9 1429.20 4200.70 1603.49

19. Conclusions [1/2]
- Sample controller (link)
- Flocking Strategies: sets of behavior rules that determine the next
move of an agent as a force
resulting from the interaction of the agents in the game.
- Genetic Algorithm presented to design optimized strategies.
- The fitness function evaluates each individual by pitting it against
two Ms. Pac-Man controllers 30 times, so as to avoid noise in the
function.

20. Conclusions [2/2]
- Future lines of research:
· Include in the fitness function the best Ms. Pac-Man controllers
that took part to the competition.
· Compare the Flocking Strategy-based controller with the best
Ghosts controllers that took part to the competition.
· Explore optimization methods other than Genetic Algorithms.
· Change the fitness function structure to reduce the
computation requirements.

21. The End!
Questions?
amorag@geneura.ugr.es
fliberatore@gmail.com