The document discusses the use of pattern recognition and machine learning in computer games. It provides examples of how pattern recognition has been used in games like Black & White, Command & Conquer: Renegade, and Re-Volt. It also discusses limitations of machine learning, including the cost of acquiring, storing, and using knowledge as well as overfitting issues. The document categorizes problems by their level of decision making (strategic, tactical, operational) and examines which machine learning algorithms like genetic algorithms, particle swarm optimization, and hidden Markov models are best suited for different levels.

1. Pattern Recognition
& Limits of Machine
Learning
Game Programming
Kit Ming Chan

2. About Our Authors
• Section 10.3
– Timo Kaukoranta, Jouni Smed, Harri Hakonen
– Department of I.T., University of Turku, Finland
– Smed: computer games, optimization, and scheduling algorithms,
doctoral degree from university of Turku <http://staff.cs.utu.fi/staff/jouni.smed/>
– Hakonen: Algorithms for Computer Games, String Algorithmic,
Software Construction and Object Integrity <
http://staff.cs.utu.fi/staff/harri.hakonen/>
• Section 10.5
– Neil Kirby
• Lucent Tech, Bell Labs, nak@lucent.com, MS in Comp Sci, Ohio
State University.
• currently develops .NET solutions.
• built speech recognition software and teaching at the university
level.
• In his spare time he designs multi-player, tactical combat
computer games. He especially enjoys writing programs for
computer opponents that play well without cheating.

3. • 10.5 Getting around the Limits of Machine
Learning

4. Categories of Limitations
• Four Questions
• Is it cheap to see (recognize) the thing to learn from?
• Is it cheap to store the knowledge?
• Is it cheap to use the knowledge?
• Does learning make the game better?

5. Divisions of Problem Space
• Implicit vs. explicit learning
– Implicit means the game is self motivated to
learn
– Explicit means the game is told to learn
• Learn in the field vs. in development
– Flexibility vs. risky quality assurance

6. Examples of Learning
• Simple question:
– Implicit or explicit learning? Learning in field or development?
– Is it cheap to see/store/use knowledge and does it improve the game?
– How to deal with quality assurance?
• Black & White
– Remember in the last class when the human player teaches his
monster to throw stone at the village?
1. Explicit and In field learning
2. Cheap to see ( just copy the player), Cheap to store (proven by
example), Cheap to use (game runs well on target machines)
3. Improve the game (sure! Increase enjoyment of replaying)
4. To avoid quality assurance problem, creatures are programmed with
innate behaviors that constrain learning

7. Example of Learning
• Command & Conquer Renegade
– A feature that is turned off in the shipped version is a
code that copies player movement and add new paths to
the pathfinder.
1. Implicit and in field learning
2. Cheap to see ( computationally easy)
3. Cheap to store (free! Pathfinder already stores this type of info)
4. Cheap to use (certainly! Just add some more paths to pathfinder)
5. Improve the game (makes AI more intelligent)
6. No quality assurance concern because it makes use of original, safe
component (pathfinder) of the game

8. Example of Learning
Re-volt (racing game)
– A genetic algorithm is used during
development to tune parameters and
reduce lap times
– Implicit & in development learning
– Cheap to see
• ( during game has no cost, but
during development it is costly to run
GA)
– Cheap to store
• ( free! Just change the parameters)
– Cheap to use
• (during game has no cost, but during
development it takes a long time)
– Improve the game
• (makes AI more intelligent)
– No quality assurance concern

9. Limits to Learning?
• What are some limits you can think of now?
Most limitations fall into these categories
1. Knowledge representation
2. Recognizing the knowledge Is the knowledge cheap to see?
3. Storing the knowledge  Is the knowledge cheap to store?
4. Using the knowledge Is the knowledge cheap to use?
The last 3 are simply the first 3 questions we have been
talking about

10. Knowledge Representation
• Consider music coding
– One channel of CD-quality music
• requires 88,200 bytes / sec regardless of content
– MIDI
• a few events per note,
• most music is roughly a few notes per second.
– Printed sheet music
• one event per note
Limit 1: the higher the level of representation, the
easier the knowledge to work with, but the more
expensive to acquire

11. Seeing More Clearly
• Limit 2: AI must store information to deal with the
past
• The present
– Easier to learn than past
• B&W ties feedback to behavior temporally
• C&C Renegade reduces mouse-and-keyboard input into paths,
and pathfinder stores new area connectivity
• Re-Volt uses the lap timer to differentiate good parameters from
suboptimal ones.
• The past
– Racing game, Backgammon are not history dependent
– For history dependent games, it is much more difficult to
understand causal chain in history

12. Seeing More Clearly
• Limit 3: Instance based methods learn
wrong things from data
– Neural nets, Markov models
– Need good training data that resembles testing
– Cannot differentiate similar (but different) data
even if trained with it

13. Seeing More Clearly
• Examples of cases that are hard to learn in instanced-
based learning
1. Neural Networks to train tank recognition
2. Pronunciation of One, Two, and Three in German
http://www.bbc.co.uk/languages/german/lj/language_notes/1_10.shtml

14. Storing new Knowledge
• Limit 4: AI cannot do sophisticated algorithm
on the fly (thus we store knowledge)
• Avoid over-fitting (learning the wrong thing)
– Neural network, Markov
– Improvement: learn from just the new datum
– Overfitting because of small new dataset!
– Neural network can compensate by temporal
differences

15. Using new Knowledge
• Limit 5: If games are not designed to use
new knowledge, new storage and capability
will need to be added
• Games should design to exploit new
knowledge in an integrated fashion
– C&C again, where new data is integrated into
pathfinder
– B&W, new data determines behavior of monster

16. Conclusion
• Machine Learning present in
– In field (player may or may not be aware)
– In development
• Knowledge can be learned
– Explicitly or implicitly
• Limits of Machine Learning
– The cost to see / recognize knowledge
– The cost to store
– The cost to use
• Does it make the game better

17. Limits in Use of Pattern
Recognition
• Limits of Machine Learning
– The cost to see / recognize knowledge
• Expensive!!
– The cost to store
– The cost to use  Could be expensive
– Does it make the game better  hopefully
• Usually in development
– Black and White is in field

18. • “Unfortunately, in the development of AIs for CGs the scope
of pattern recognition has not been widely realized…Our
motivation is to do pattern recognition ourselves to discern
where pattern recognition can be applied in CG” [Hakonen]
• Many classic academic games such as Chess (Deep Blue),
Backgammons, Go made use pattern recognition and
achieved good result.
• Many games do use neural networks, but does not
necessarily use pattern recognition.

19. • 10.3 Understanding Pattern Recognition
Methods

20. Intro to Pattern Recognition 1
• Wikipedia:
• Pattern recognition … can be defined as "the
act of taking in raw data and taking an action
based on the category of the data" [1].
– image analysis, character recognition, speech
analysis, man and machine diagnostics, person
identification and industrial inspection.

21. Intro to Pattern Recognition 2
• Textbook Definition (in the context of CG):
– To abstract relevant information from the game
world and, based on the retrieved information,
construct concepts and deduce patterns for the
use of higher level reasoning and decision-
making systems.

22. Intro to Pattern Recognition 3
Abstract
relevant info,
compare to old
info in KB,
deduce pattern
Choose from a
list of possible
actions
available in the
current state,
balance it to the
requested state

23. Questions
1. Why do we need a decision-making system?
Why don’t we just simply map patterns to some
reaction? (Hint: Do we know everything about
the environment to make the decision?)
The world is not deterministic..
-Built-in randomness
-Human players’ action
2. Where do you think PR can be applied for
computer games?

24. Intro to Pattern Recognition 4
• Some suggested uses of pattern recognition:
– During game:
• Enemy evaluation and prediction
• Coaching
• Group coordination
• Terrain analysis
• Learning
– Black and White
– Command and Conquer Renegade (attempted)
– During development
• Re-Volt

25. • Functional Approach.
• How is the use of PR differ with…
– Levels of decision making ?
– Stance towards the player ?

26. Functional – Level of Decision Making
• Suitability of pattern recognition depends on the level of
decision making:
– Strategic Level
– Tactical Level
– Operational Level

27. Functional– Level of Decision Making
Level of Duration Size of Speed/ Use
decision Plan of Pattern Strategic Game
making Recognition Example
Strategic Long-term All Offline / terrain is analyzed to
players extensive find advantageous
use positions to plan the
maneuvers
Tactical medium A group Delivered decisions made on the
real time / engaging platoons and
some use the battleground
Operational immediate Individual Made in real To shoot, dodge, or
time / no charge
use

28. Functional - Stance
• Enemy
– Provide Challenge
– Demonstrate Intelligent (at least purposeful)
behavior
– PR to aid computer’s decision making
• Prediction and production

29. Functional - Stance
– Ally,
• Augment user interface
– Hints and guide
• Aiding the human
player
• End result should be
visually accessible and
consistent, but not
necessarily complete

30. Functional - Stance
Neutral :
-Context dependent
-Commentory
-highlighting events and
providing background
highlighting events and
background information
- soccer referee
-Judging rule violation

31. Functional– Stance
– Modeled Language?
• A generator labels events and states with symbol
• Modeling recognizes the underlying dependencies
between symbols
• Short term history sufficient

32. Functional– Prediction
– Predict next symbol, calculate probability of the
next action of opponent

33. Functional– Production
– observe opponent and produce next symbol
(memcpy)

34. • PR as these problems:
– Optimization
– Adaptation
– Uncertainty (not going to talk about)
• Algorithm used in each problem area
• How the level of decision making decides
what PR algorithm to use

35. Methodology - Optimization
Optimization
– an objective function (to max or min)
– A set of variables
– A set of constraints
• Pattern recognition as an optimization
problem is to have an objective function to
rank the solution candidates

36. Methodology - Optimization

37. Methodology - Optimization
• Project # 6: tweak AI to beat some opponents
• Marc’s presentation: the tweaking could be
done automatically
• This is related to pattern recognition
– Trying to identify strengths and weaknesses of
opponents

38. Methodology - Optimization
• Age of Empire Example
– Objective: Balance civilizations and units
– A combat comparison simulator is used to test battles
with different troop combinations
– The set of variables: Attributes (armor, hit point,
damage, range)
– The set of constraints: the range of allowed values
– Objective Function: min(difference of the number of
victories of different civilizations in simulator battles)
– Attributes are changed to even out discrepancies

41. These slides are taken from [Street]

42. Methodology - Optimization
• How to find more optimal variable values?
• Heuristic functions
– Fastest and simplest
– Get stuck at local optimum before finding global
– Use multiple search traces instead of one

43. Methodology - Optimization
• How to find more optimal variable values and
avoid local optimum?
• Genetic Algorithm
– Avoid local optimum. A population of candidate
solutions, go thru stages of natural selection,
objective function is to weed out the weak
candidates
– Vulnerable to dependency of variables, CPU
exhaustive

44. Methodology - Optimization
• Another way to find more optimal variable
values and avoid local optimum?
• Swarm Algorithm
– Avoid local optimum (if min speed is set)
– faster than genetic algorithm
– next 3 slides on Particle Swarm Optimization

45. Particle Swarm Optimization 1
– PSO is very similar to genetic algorithm, but does not
have genetic operators like crossover and mutation.
Particles update themselves with the internal velocity.
– The idea is similar to bird flocks searching for food, don’t
know where the food is, but know how far from it
Bird = particle, Food = optimal solution
pbest = the best solution (fitness) a particle has
achieved so far.
gbest = the global best solution of all particles
– pbest and gbest are stored in memory

46. Particle Swarm Optimization 2
• v[] = v0[] + c1 * rand() * (pbest[] - present[]) +
c2 * rand() * (gbest[] - present[]) -----------(a)
• present[] = present0[] + v[] ------------------(b)
v[] = the particle velocity
present[] = the current particle (solution).
rand () = a random number between (0,1).
c1, c2 = learning factors. usually c1 = c2 = 2.
http://uk.geocities.com/markcsinclair/pso.html

47. Optimization & the Level of Decision
Making
• Strategic Level:
– Computationally hard to solve
– With relaxed constraint that the variables are not
interdependent, then genetic algorithm
• Tactical Level:
– Must be responsive
– Single trace, heuristic functions
• Operational Level:
– Real-time,
– simple objective function (switch statement)

48. Methodology - Adaptation
• Adaptation
– The ability to make appropriate responses to
changed or changing circumstances
• Try to model the originator of the modeled
data

49. Methodology - Adaptation
• Adaptation vs.
Optimization
– Adaptation
• looks for a
function behind
a solution,
– Optimization
• looks for a
solution for a
given function

50. Methodology - Adaptation
• When is adaptation useful? Why is it hard to
use?
1. Useful when the affecting factors or mechanisms behind
the phenomena is unknown or dynamic
2. Hard to use because it requires sampling the search
space to cover sufficiently. The more complex the cause
is, the sparser the our sample gets (combinatorial
explosion)

51. Methodology - Adaptation
• Neural Networks
– Can adapt to situation where we do not have background
knowledge of dependencies
– Supervised (predefined categories for results) or
unsupervised learning
– For more information, please read Chp11
• Hidden Markov Model
– System is conditionally independent of the past states
– each state has a probability distribution over the possible
output tokens
– the challenge is to determine the hidden parameters from
the observable parameters
– Can adapt to recurring structure

52. Adaptation and the level of decision making
• Strategic Level
– Supervised or unsupervised learning
– Can afford great computational demand
• Tactical Level
– Hidden Markov models
– More dynamic environment
– Credibility of results can be evaluated
• Operation Level
– Stochastic interpretation for input data, or
– Use a ready-adapted neural network

53. Conclusion
• Pattern Recognition
– PR is not well explored in gaming yet
– Biggest limit is computational complexity
– A lot of time is domain dependent
– Algorithms used depend highly on required
responsiveness
– Functional Approach
• Level of decision making
• Stance
– Methodological Approach
• Optimization
• Adaptation
• Uncertainty ( not discussed )

54. References
• Hu, Xiaohui. “Particle Swarm Optimization Tutorial,”
http://www.swarmintelligence.org/tutorials.php
• Fraser, Neil. “Neural Network Follies.” http://neil.fraser.name/writing/tank/
• German steps, number 1-10.
http://www.bbc.co.uk/languages/german/lj/language_notes/1_10.shtml
• Shutton, R. “Learning to Predict by the Method of Temporal Differences,”
ftp://ftp.cs.umass.edu/pub/anw/pub/sutton/sutton-88.ps.gz
• Kidd, Petersen, Street, “How to Balance a Real-Time Strategy Game: Lessons from the
Age of Empires Series,” http://www.gdconf.com/archives/2001/gstreetprintable3.ppt
• Timo Kaukoranta, Jouni Smed, Harri Hakonen. “Role of Pattern Recognition in Computer
Games.” http://staff.cs.utu.fi/staff/jouni.smed/papers/PRinCG.pdf