Invited talk given at Workshop on Simulations in Virtual BattleSpaces, Spelvetenskapligt Seminarium, Göteborgs Universitet, May 10 2016.

1. BEYOND BELIEVABLE AGENTS
EMPLOYING AI FOR IMPROVING GAME LIKE SIMULATIONS FOR TACTICAL TRAINING
Mirjam Palosaari Eladhari, 10th of May 2016, Spelvetenskapligt Seminarium, Göteborgs Universitet

2. ABOUT ME
➤ Game designer, researcher, and indie developer.
➤ Research associate at Institute of Digital Games in Malta, and
at Dept. of Computer and System Sciences at Stockholm
University
➤ Recently founded Otter Play - one person indie studio
➤ Current interests: AI Based Game Design, Story Making
Games, and co-creation of social intelligent agents.
➤ Past: Game Programmer 2000-02, Liquid Media
Tech Lead, Zero Game Studio, Interactive Institute 2002 - 04
Then: 10+ years of game research & faculty work, mostly in
Game AI & Design.

3. OVERVIEW
➤ Introduction
➤ Believable Agents
➤ approaches
➤ opportunites
➤ AI in military training scenarios, ex.
➤ The gamist problem
➤ What the AI Game programmers
Guild said
➤ AI Based Game Design
➤ Discussion: What are the learning
goals of training simulations, and
how can they be reached? (with or
without AI)

5. KRIEGSSPIEL (1812)
Kriegsspiel, from
the German word for wargame,
was a system used for training
officers in the Prussian army.
The first set of rules was created
in 1812 and named
Instructions for the Representation of
Tactical Maneuvers under the Guise of
a Wargame.
It was originally produced and
developed further by
Lieutenantvon Reiswitz of
the Prussian Army.
http://www.boardgamestudies.info/pdf/issue3/BGS3Hilgers.pdf
Taktischer Kriegsspielapparat, von Domänen- und Kriegsrat Georg Leopold Baron
von
Reiswitz für Friedrich Wihelm III entworfen und 1812 angefertigt. Siftung
Preußischer
Schlösser und Gärten Berlin-Brandenburg (Foto Roman März, Berlin).

7. TURING’S PAPER MACHINE 1962

8. Games and AI -
historically
intertwined.
Many game AI
programmers
considers game
AI to be game
design.

9. ARTIFICIAL INTELLIGENCE (AI)
“The subfield of computer science concerned with the concepts and
methods of symbolic inference by computer and symbolic knowledge
representation for use in making inferences. AI can be seen as an attempt to
model aspects of human thought on computers. It is also sometimes
defined as trying to solve by computer any problem that a human can solve
faster.”
The Free On-line Dictionary of Computing, © 1993-2007
Denis Howe

10. INTELLIGENT AGENTS
In artificial intelligence, an intelligent agent (IA) is an autonomous
entity which observes through sensors and acts upon an
environment using actuators (i.e. it is an agent) and directs its
activity towards achieving goals (i.e. it is "rational", as defined in
economics).
(Russell & Norvig 2003, chpt. 2)

11. AI BASED GAME DESIGN
Diagram is a joint effort of Josh McCoy, Anne Sullivan, Gillian Smith, and me (2011, Santa Cruz)

12. BELIEVABLE AGENTS (BATES 1994)
➤ "one that provide the illusion of life, and thus permits the audiences suspension of
disbelief."
➤ "an interactive analog of the believable characters discussed in the Arts."
➤ "Emotion is one of the primary means to achieve believability, this illusion of life, because
it helps us know that characters really care about what happen in the world, that they truly
have desires".
➤ “Believability places a variety of demands on an interactive character. These include the
appearance of reactivity, goals, emotions and situated social competence, among
others.”
J. Bates. The role of emotions in believable agents. Technical Report CMU-
CS-94-136, School of Computer Science, Carnegie Mellon University, Pittsburgh,
PA, April 1994.

13. COMMON TECHNOLOGIES FOR
BELIEVABLE AGENTS

14. BELIEVABLE CHARACTERS: STATE OF THE ART
• Use of Planning models- reactive & traditional
• Plan-based representations of physical, social and language
actions (domain specific hierarchical planning)
• Emotion modelling with convergence on cognitive appraisal
architectures
• FSMs (used now again for dialogue- moving towards
performance of interactive dialogue)

15. BELIEVABLE CHARACTERS: STATE OF THE ART
• Mood modelled as a function of emotions (sums, averages)
and with decay
• Modelling mood through spreading activation. When a
character is reminded of something that is emotionally
significant, there is an emotional echo
• Episodic memory (temporal query), evaluative labelling,
generalisation.

16. COMMON APPROACHES FOR BELIEVABLE AGENTS
➤ Planning
➤ BDI - Belief, Desire,
Intention
➤ Reactive planning
➤ Hierarchical task networks
(HTNs)
➤ Finite State Machines
(FSMs) and Behaviour Trees
➤ Various architectures for
appraisal and decision making
for Intelligent Virtual Agents
and Synthetic Humans

17. PLANNING
Planning is the process of generating a sequence of actions
that will achieve a goal.
Planning is the process of generating (possibly partial)
representations of future behavior prior to the use of such plans
to constrain or control that behavior. The outcome is usually a
set of actions, with temporal and other constraints on them, for
execution by some agent or agents.
STRIPS (Stanford Research Institute Problem Solver)
First automated planner, 1971.
Language with the same name, used for the inputs to this planner.
This language is the base for most of the languages for expressing automated planning
problem instances in use today.

18. THE BELIEF-DESIRE-INTENTION (BDI) SOFTWARE MODEL
BDI Agents are
➤ Situated - they are embedded in their environment
➤ Goal directed - they have goals that they try to achieve
➤ Reactive - they react to changes in their environment
➤ Social - they can communicate with other agents (including
humans)
Wooldridge, M. (2000). Reasoning About Rational Agents. The MIT Press. ISBN 0-262-23213-8. http://
mitpress.mit.edu/catalog/item/default.asp?ttype=2&tid=3533.

19. BELIEFS - “I BELIEVE”
➤ the informational state of the agent - in other words its
beliefs about the world (including itself and other agents).
Beliefs can also include inference rules, allowing forward
chaining to lead to new beliefs. Typically, this information
will be stored in a database (sometimes called a belief base),
although that is an implementation decision.
➤ Using the term belief - rather than knowledge - recognises
that what an agent believes may not necessarily be true
(and in fact may change in the future).

20. DESIRES - “I WANT”
➤ Desires (or aims) represent the motivational state of the
agent. They represent objectives or situations that the agent
would like to accomplish or bring about.
Examples of desires might be: find the best price, go to the
party, or become rich.

21. INTENTIONS - “I’M GOING TO…”
Intentions represent the deliberative state of the agent: what the
agent has chosen to do. Intentions are desires to which the
agent has to some extent committed (in implemented
systems, this means the agent has begun executing a plan -
towards a goal).
Plans are sequences of actions that an agent can perform to
achieve one or more of its intentions.

22. Mirjam Eladhari, mirjam.eladhari@hgo.se
Högskolan på Gotland Institutioner för
teknik, konst och nya medier
Gotland University, Department of
Technology, Art and New Media
BLACK AND WHITE
OFFICE PLANT #1, Marc
Bohlen and Michael
Mateas, Carnegie Mellon
University, March 1998

23. Mirjam Eladhari, mirjam.eladhari@hgo.se
Högskolan på Gotland Institutioner för
teknik, konst och nya medier
Gotland University, Department of
Technology, Art and New Media
BLACK AND WHITE

24. BLACK AND WHITE AI DEV STRATEGY
To make a plausible agent, there must be an explanation of why
he is in that particular mental state.
In particular, if an agent has a belief about an object, that belief
must be grounded in his perception of that object: creatures
in Black & White do not cheat about their beliefs
– their beliefs are gathered from their perceptions, and there
is no way a creature can have free access to information he
has not gathered from his senses.

25. REACTIVE PLANNING
A group of techniques for action selection by autonomous
agents.
Differ from classical planning in two aspects:
➤ operate in a timely fashion and hence can cope with highly
dynamic and unpredictable environments.
➤ compute just one next action in every instant, based on the
current context.
Reactive planners often (but not always) exploit reactive plans,
which are stored structures describing the agent's priorities and
behaviour.

26. REACTIVE PLAN REPRESENTATION
➤ Finite State Machines (FSMs) and Behavior trees
➤ Fuzzy approaches (Utility Systems)
➤ Connectionists approaches

27. FINITE STATE MACHINES
➤ set of states and transitions between these
states.
➤ Transitions are condition action rules.
➤ In every instant, just one state of the FSM
is active, and its transitions are evaluated.
If a transition is taken it activates another
state. That means, in general transitions
are the rules in the following form: if
condition then activate-new-state.
Example: Isla, D.: Handling complexity in Halo 2.
Gamastura online (2005)
http://www.gamasutra.com/gdc2005/features/20050311/
isla_pfv.htm
Reactive Planning
Traditional - often used

28. BEHAVIOR TREES
➤ Reactive decision making and
control of the virtual creatures
➤ Behavior trees replace the often
intangible growing mess of state
transitions of finite state machines
(FSMs) with a more restrictive but
also more structured traversal
defining approach.
➤ Behavior trees are formed by
hierarchically organizing behavior
sub-trees consisting of nodes.

29. ➤ Fuzzy Logic: conditions, states and actions are no more
boolean or "yes/no"
instead: but are approximate and smooth. Behaviour will
transition smoother, especially in the case of transitions
between two tasks.
➤ Trade-off: It is slower.
Zadeh, L.A. (1965). "Fuzzy sets". Information and Control 8 (3): 338-353
FUZZY APPROACHES
Reactive Planning
Related to ‘utility systems’

30. CONNECTIONISTS APPROACHES
➤ Reactive plans can be expressed by
connectionist networks like artificial
neural networks or free-flow
hierarchies.
➤ units with several input links that
feed the unit with "an abstract
activity" and output links that
propagate the activity to following
units.
➤ Each unit itself works as the activity
transducer. (Typically, the units are
connected in a layered structure.)
Reactive Planning
My ‘mind module’ is an ex of this

31. IVAS AND SYNTHETIC HUMANS
➤ Agents built to interact in real world (as opposed to a fictional
world), so rules and laws are known
➤ Agents mimicking humans, modeled from psychological and
cognition theories, such as the OCC model
➤ Big 5 and affect theory.
➤ Example: psychSim (Stacy Marcella et. al)

32. BELIEVABLE AGENTS -
OPPORTUNITIES
Michael Mateas, Elizabeth Andre, Ruth Aylett, Mirjam
Eladhari, Richard Evans, Ana Paiva, Mike Preuss,
Michael Young
at a Dagstuhl Seminar on AI for games

33. BELIEVABLE CHARACTERS: OPPORTUNITIES
• Most agent learning focuses on easy-to-evaluate criteria
– Learning should be personality-specific
– Online learning must maintain believability while doing
exploration (converge in relatively few exposures)

34. BELIEVABLE CHARACTERS: OPPORTUNITIES
• Combine statistical language model for style with semantic / symbolic
content models
• Dynamically generated character soundtrack communicating character
& social state
• Two directions for more information-rich signals from
– Player: high-bandwidth naturalistic interaction (gestures, gaze); new
communication modes (biometrics, out-of-band interactions like
music selection, player modeling)
– Purposefully set up choices for characters to allow them
• to express personality through choice
• and conversely set up player choices that give information about
player

35. BELIEVABLE CHARACTERS: OPPORTUNITIES
• Take advantage of low-cost Mocap to “correlate” prosodics &
gestures features in a generative models (has to be
parameterized by emotion & personality and social context)
• Using explainable AI to drive interface elements (text
explanations) or thought bubbles

36. EXPRESSION: CHALLENGES
• Rich emotional state currently not expressed
• Language is the elephant in the room
• Multi-modal expression often results in inconsistencies
between modes
• Handcrafted currently works best

37. EXPRESSION: OPPORTUNITIES
• Go beyond naturalistic world simulation affordances to
express character state
– reifying state in characters and objects,
– behaviour explanation (HUD)
– South Park and other stylised comic-inspired forms , music,
abstract visuals

38. AI FOR
MILITARY
TRAINING

39. AI IN VBS

40. IMMERSE PROJECT
An interaction with one of the virtual characters, while an activity continues to take place in the background.

41. IMMERSE
➤ IMMERSE is a projected funded under the DARPA Strategic
Social Interaction Modules program. The goal of the project
is to produce a game-based training environment that
teaches people how to be “good strangers”, supporting the
practice of skills necessary to have successful social
interactions when in novel culture and language contexts.
➤ Virtual dramatic scenarios, realized within the Unity game
engine, in which the player interacts with computer-
controlled autonomous characters in high-consequence social
environments, learning how to quickly recognize and
navigate the social interactions norms in these
environments.

42. IMMERSE
➤ The simulation is embedded in a 3D virtual environment
modeled in the Unity Game Engine. The social simulation is
implemented by integrating a real-time behavior language
with a rule-based artificial intelligence system specifically
designed for turn- based social interactions. The player
interacts with the environment and other embodied virtual
agents through custom automated speech recognition and full
body gestures via input from a Microsoft Kinect.
➤ Pedagogical goals are embedded into each scenario, along with
specialized pedagogical coaching through specialized virtual
agents, to help focus the trainees attention on the most salient
points in the simulation and to work on specific problem areas
identified by the player model.
Expressive Intelligence Studio, UC Santa Cruz

43. FINAL REPORT
http://oai.dtic.mil/oai/oai?
verb=getRecord&metadataPrefix=html&identifier=ADA625663
Report contains example scenarios!

44. BUT…
all these agent technologies,
approaches and applications,
can’t solve all problems

48. THE GAMER MODE PROBLEM
Anders Frank, 2010:
“at different times players may focus on the rules, the fiction or
on both during game play. In military education with games, this
poses a problem when the learner becomes too focused on
the rules, trying to win at any price rather than taking the
representation and what it implies in terms of permissible
behaviour seriously.”
Frank, 2010, Swedish Defence University,
Gamer mode: Identifying and managing unwanted behaviour in military educational wargaming
http://www.diva-portal.org/smash/record.jsf?pid=diva2%3A768474&dswid=4699

49. A Grail of AI research in its infancy:
The Automated Game Master

50. WHAT THE
HIVE MIND OF
THE AI GAME
PROGRAMMERS’
GUILD SAID.
(here only sharing those parts that people
were OK with being shared outside the
workshop)

57. SUMMARY
➤ Manual paper gaming can be an alternative.
➤ Maps are important!
➤ VBS is very expensive.
➤ Little sharing of knowledge because of monetary and national
interests. (Big contracts, national security)
➤ Physical tasks are hard to simulate - focus on strategic
decision making (at least until we have better VR/AR)
➤ Scenario construction: Rules for game play MUST match
training goals

59. AI BASED GAME DESIGN
Diagram is a joint effort of Josh McCoy, Anne Sullivan, Gillian Smith, and me (2011, Santa Cruz)

60. Metaphor: adapt game mechanics to the goals of the game.
The learning goals of the training scenario. Build as true a
representation of the real situation as possible.
AI BASED GAME DESIGN
➤ Diagram is a joint effort of Josh McCoy, Anne Sullivan, Gillian Smith, and me (2011, Santa Cruz)

61. Metaphor: adapt game mechanics to the goals of the game.
The learning goals of the training scenario. Build as true a
representation of the real situation as possible.
AI BASED GAME DESIGN
➤ Diagram is a joint effort of Josh McCoy, Anne Sullivan, Gillian Smith, and me (2011, Santa Cruz)
Be cautions using off-the shelf game play from existing genres
such as FPSes. Players likely to go into gamer mode if controls
and GUI are similar to what they play for entertainment.
Difficulty: if the game-engine used IS an FPS engine. Then it
will be conscious effort to change those parts.

62. Metaphor: adapt game mechanics to the goals of the game.
The learning goals of the training scenario. Build as true a
representation of the real situation as possible.
AI BASED GAME DESIGN
➤ Diagram is a joint effort of Josh McCoy, Anne Sullivan, Gillian Smith, and me (2011, Santa Cruz)
Consider the AI in the same manner: “known” NPC
behaviour creates a feeling of game-ness,
unrealness. Adapt it to the scenario.
Be cautions using off-the shelf game play from existing genres
such as FPSes. Players likely to go into gamer mode if controls
and GUI are similar to what they play for entertainment.
Difficulty: if the game-engine used IS an FPS engine. Then it
will be conscious effort to change those parts.

63. WHAT I DO: CO-CREATION OF SOCIAL INTELLIGENT AGENTS
➤ Agents with personality, moods and emotional memory
➤ Created by players & system in collaboration
➤ In play: agents enacted, allowing for reflection and
perspective change
My past work of this sort:
PataphysicInstitute (2010)
C2Learn suite of games (2014)C2Learn.eu

64. ENDING TALK -> DISCUSSION
➤ What are the current most
important goals for military
training?
➤ How can these challenges be
met - with or without AI?

65. Thank you for listening!
Contact:
info@otter-play.com