Spatio-Temporal Reasoning Over Play-Scripts for Artificially
Intelligent Characters
Christine Talbot
Richard Burton in Hamlet, directed by Sir Gielgud
http://www.youtube.com/watch?v=XRU5yLgs0zw&feature=player_detailpage

Background
2

of 49
Virtual Character Positioning
3
Back-
ground
EMA: A process model of appraisal dynamics (Stacy C. Marsella,
Jonathan Gratch), In Journal of Cognitive Systems Research, volume
10, 2009.
Ada and Grace: Toward Realistic and Engaging Virtual Museum Guides (William
Swartout, David Traum, Ron Artstein, Dan Noren, Paul Debevec, Kerry
Bronnenkant, Josh Williams, Anton Leuski, Shrikanth Narayanan, Diane Piepol, H.
Chad Lane, Jacquelyn Morie, Priti Aggarwal, Matt Liewer, Jen-Yuan Chiang, Jillian
Gerten, Selina Chu, Kyle White), In Proceedings of the 10th International
Conference on Intelligent Virtual Agents (IVA 2010), 2010.

of 49
Mocap Files and Hand-Coding
4
Back-
ground
Discovery News – Avatar: Motion Capture Mirrors Emotions
http://news.discovery.com/videos/avatar-making-the-movie/

of 49
BML and BML Realizers
5
Back-
ground
SmartBody Path Planning
http://smartbody.ict.usc.edu
Hamlet played by robots
Unity using SmartBody
MindMakers Wiki
http://www.mindmakers.org/
projects/bml-1-0/wiki/Wiki

of 49
<act><participant id="GRAVEDIGGER2" role="actor" /><bml><gesture lexeme="POINT" target="GRAVEDIGGER1"
/></bml></act>
<act><participant id="GRAVEDIGGER1" role="actor" /><bml><speech id="sp1" ref="“ type="application/ssml+xml">Give me
leave!</speech></bml></act>
<act><participant id="GRAVEDIGGER1" role="actor" /><bml><gesture lexeme="POINT" target="GRAVEDIGGER2"
/></bml></act>
<act><participant id="GRAVEDIGGER1" role="actor" /><bml><gesture lexeme="POINT" target="GRAVE" /></bml></act>
<act><participant id="GRAVEDIGGER1" role="actor" /><bml><speech id="sp1" ref="" type="application/ssml+xml"> Here lies
the water -- good? </speech></bml></act>
<act><participant id="GRAVEDIGGER1" role="actor" /><bml><gesture lexeme="POINT" target="GRAVE" /></bml></act>
<act><participant id="GRAVEDIGGER1" role="actor" /><bml><speech id="sp1" ref="" type="application/ssml+xml"> Here
stands the man -- good! </speech></bml></act>
<act><participant id="GRAVEDIGGER1" role="actor" /><bml><speech id="sp1" ref="" type="application/ssml+xml"> If the
man go to this water and drown himself, it is willynilly he goes, mark you that! But, </speech></bml></act>
<act><participant id="GRAVEDIGGER1" role="actor" /><bml><gesture lexeme="POINT" target="GRAVE" /></bml></act>
<act><participant id="GRAVEDIGGER1" role="actor" /><bml><speech id="sp1" ref="" type="application/ssml+xml">if the
water come to HIM and drown him, he drowns not him-self; Argal, he that is not guilty of his own death
shortens not his own life!</speech></bml></act>
<act><participant id="GRAVEDIGGER1" role="actor" /><bml><locomotion target="GRAVE" type="basic" manner="walk"
/></bml></act>
<act><participant id="GRAVEDIGGER2" role="actor" /><bml><speech id="sp1" ref="" type="application/ssml+xml">But is
this LAW ?</speech></bml></act>
Still a Lot of Work…
6
Back-
ground
4 hours & 12 minutes for a 10 minute scene!!
Point
Speak
Move

of 49
So How Do We Do It?
7
Back-
ground
A: Excuse me…
B: Can I help you?
A: Yes, where is the post office?
B: Go straight and turn left.
A: Where do I turn left?
B: Turn left at the bus stop -
you can’t miss it.
A: Thank you very much!
B: No problem.

Play-Scripts
8

of 49
GRAVEDIGGER1
Give me leave!
(GRAVEDIGGER2 sits on the side steps)
(Pointing down into the grave)
Here lies the water--good?
(Pointing to the table ledge)
Here stands the man-good!
(Illustrating each point literally with his hands)
If the man go to this water and drown himself, it is willy-nilly he goes,
mark you that! But,
(Pointing first to the grave, then to the ledge)
if the water come to HIM and drown him, he drowns not him-self;
(Greatly pleased with his own logic)
Argal, he that is not guilty of his own death shortens not his own life!
(He goes behind the barricade down into the grave and
prepares to dig)
GRAVEDIGGER2
(Trying to disprove him)
But is this LAW?
Play-Scripts
9
Play-
Scripts
Character Directions
Stage Directions
Stage Directions
Character Directions
Character Directions

of 49
The Baseline
Play-
Scripts
Hamlet
Act 5, Scene 3
(Graveyard Scene)
Richard Burton in Hamlet, directed by Sir Gielgud
http://www.youtube.com/watch?v=XRU5yLgs0zw&feature=pla
yer_detailpage
10
400 BML Commands
4 hours & 12 minutes
10 minute scene

of 49
11
Play-
Scripts
Sentence
Subject NP Actor/Noun
VP
VP Action/Verb
NP Target/Noun
Example Nouns:
GRAVEDIGGER1
GRAVEDIGGER2
HAMLET
HORATIO
Steps
Grave
Audience
Center stage
Stage left
Example Verbs:
Move to
Follow
Look at
Pick up
Put down
Speak
Point to
Example:
(Pointing down into the grave)
Actor = current speaker
Verb = point
Target = grave
Annotation Parsing

of 49
What did it look like?
12
Play-
Scripts

of 49
How Did We Do?
13
Play-
Scripts
HamletGraveDigger1
Ground Truth Simple NLP Method
Character Traces Over Time for Entire Graveyard Scene
C. Talbot and G. M. Youngblood. Spatial Cues in Hamlet. In Proceedings of the 12th International Conference on
Intelligent Virtual Agents, IVA '12, pages 252-259, Berlin, Heidelberg, 2012. Springer-Verlag.

Spatial Rules
14

of 49
What’s Next?
 Applying Spatial Rules
 Conversational Spatial Rules
 Grouping Spatial Rules
 Theatre Rules
 General Rules
15
Spatial
Rules
E. Sundstrom and I. Altman.
Interpersonal Relationships and Personal Space:
Research Review and Theoretical Model. 1976
Counter-Crossing

of 49
Architecture
16
Spatial
Rules
BML

of 49
Architecture
17
Spatial
Rules

of 49
Rules Engine Logic
18
Spatial
Rules

of 49
Position Results
19
Spatial
Rules
C. Talbot and G. M. Youngblood. Shakespearean Spatial Rules. In Proceedings of the 2013 International
Conference on Autonomous Agents and Multi-agent Systems, AAMAS '13, pages 587-594, Richland, SC, 2013.

of 49
Position Results
20
Spatial
Rules
C. Talbot and G. M. Youngblood. Shakespearean Spatial Rules. In Proceedings of the 2013 International
Conference on Autonomous Agents and Multi-agent Systems, AAMAS '13, pages 587-594, Richland, SC, 2013.

Implied Movements
21

of 49
Grave Digger 1 Initiative
22
Implied
Mvmt

of 49
Implied Motion
23
Implied
Mvmt
To be, or not to be—
that is the question:
Whether 'tis nobler in
the mind to suffer ….
I should move
towards
the audience
for my monologue

of 49
24
Implied
Mvmt
Information Captured
 For Each Line of Speech:
 Movement by Speaker or Other
Character
 Number of Lines Spoken Before
 Number of Lines Spoken After
 Annotation Before
 Annotation After
 Number of Lines since Last
Movement
 Number of Repeated Words
 Number of Upper Case Words
 Punctuation Counts
 Parts of Speech Counts
 Type of Movements:
 Fighting
 Jumping
 Gestures
 Object Manipulations
 Locomotion
 Pointing
 Posture
 Gaze

of 49
25
Implied
Mvmt
Machine Learning
 RTextTools in R
 Maximum Entropy
 Random Forests
 Boosting
 SVM
 Specific Movements
 General Movements
 Any Movement By Speaker
 Any Movement at All

of 49
26
Implied
Mvmt
Learning Combinations
 Movement Classifications
 Specific Movements
 Movement High-Level Categories
 Big Movements
 Any Movements
 N-Gram Sizes
 Unigrams
 Bigrams
 Trigrams
 4-grams
 5-grams
 Training Sizes
 Even Split Training / Testing
 Even Split of Positive Examples for
Training / Testing
 Feature Sets
 Text Only
 POS Counts Only
 POS Counts & Text
 POS Counts & Contextual Features
 All Features

of 49
27
Implied
Mvmt
Evaluation Criteria
 Overall Accuracy
 Recall
 Precision
 F1 score
 F0.5 score
 Matthews Correlation Coefficient
 ROC curves

of 49
28
Implied
Mvmt
Evaluation Criteria
 Overall Accuracy
 Recall
 Precision
 F1 score
 F0.5 score
 Matthews Correlation Coefficient
 ROC curves

of 49
29
Implied
Mvmt
Evaluation Criteria
 Overall Accuracy
 Recall
 Precision
 F1 score
 F0.5 score
 Matthews Correlation Coefficient
 ROC curves

of 49
Best Performing
30
Implied
Mvmt
Boosting
SVM
MaxEnt
RandForest
Any Mvmt, POS, Unigrams Any Mvmt, No Text, Unigrams
Gestures, All, 4-grams Any Mvmt, Text, Unigrams
C. Talbot and G. M. Youngblood. Lack of Spatial Indicators in Hamlet. In Florida Artificial Intelligence Research
Society Conference, FLAIRS '13, pages 154-159. Association for the Advancement of Artificial Intelligence, 2013.

of 49
Best Performing
31
Implied
Mvmt
Boosting
SVM
MaxEnt
RandForest
Random
Any Mvmt, POS, Unigrams Any Mvmt, No Text, Unigrams
Gestures, All, 4-grams Any Mvmt, Text, Unigrams
C. Talbot and G. M. Youngblood. Lack of Spatial Indicators in Hamlet. In Florida Artificial Intelligence Research
Society Conference, FLAIRS '13, pages 154-159. Association for the Advancement of Artificial Intelligence, 2013.

Incorporating Human
Characters
32

of 49
So Far…
33
Humans
Sentence
Subject NP Actor/Noun
VP
VP Action/Verb
NP Target/Noun
Speech Movement
Grouping Spatial Rules
Conversational Spatial Rules
Theatre Rules
General Rules
MindMakers Wiki
http://www.mindmakers.org/
projects/bml-1-0/wiki/Wiki
SmartBody Path Planning
http://smartbody.ict.usc.edu

of 49
Adding a Human
 Move correctly, on-time
 Move correctly, wrong time
 Move incorrectly, on-time
 Move incorrectly, wrong time
 Don’t move at all
34
Humans

of 49
35
Humans
 Equilibrium of Forces
 Aesthetically Balanced
 Easy to See Nodes
 Crossings-Free (some)
 Fixed Nodes
 Varying Relationships Based on Data
 Can be Arranged in Pre-defined Shapes (some)
Force-Directed Graphs (FDGs)
T. M. J. Fruchterman, Edward, and E. M. Reingold. Graph Drawing by Force-
Directed Placement. Software: Practice and Experience, 21(11):1129{1164, 1991.

of 49
Force-Directed Graph Structure
 Node Representations:
 Characters
 Human
 Target/Marks/Pawns
 Audience
 Central Grouping Point
36
Humans
A
H
T
A
H
T

of 49
Force-Directed Graph Structure
 Node Representations:
 Characters
 Human
 Target/Marks/Pawns
 Audience
 Central Grouping Point
 Linkages
 Characters – Humans/Characters
 Characters – Targets/Marks/Pawns
 Characters – Audience
 Characters – Central Grouping Point
 Humans – Central Grouping Point
 Central Grouping Point - Audience
 Humans - Audience
37
Humans
A
H
T

of 49
Force-Directed Graph Functions
 Adding Characters
 Characters Leaving
 Moving Characters
 Human Moves
38
Humans
B
H
T
T
A

of 49
A
Force-Directed Graph Functions
 Adding Characters
 Characters Leaving
 Moving Characters
 Human Moves
39
Humans
B
H
T
T

of 49
Force-Directed Graph Functions
 Adding Characters
 Characters Leaving
 Moving Characters
 Human Moves
40
Humans
B
H
T
T
A

of 49
Force-Directed Graph Functions
 Adding Characters
 Characters Leaving
 Moving Characters
 Human Moves
41
Humans
B
H
T

of 49
42
Humans
Forces and Time
δ= distance between nodes
L = length of stage depth
α = constant
C. Talbot and G. M. Youngblood. Positioning Characters Using Forces. In Proceedings of the Cognitive Agents for
Virtual Environments Workshop (CAVE 2013) collocated with AAMAS (W08). IFAMAAS (International Foundation
for Autonomous Agents and Multi-agent Systems), 2013.

of 49
Evaluation Approaches
 Optimal arrangement based on current relationships
 Time-based / sequential arrangement through entire scene
 User evaluation of appropriate positioning
43
Humans

of 49
Humans
Arrangement Based Upon Relationships
44

of 49
Humans
Arrangement Based Upon Relationships
45

of 49
Humans
Arrangement Based Upon Relationships
46

of 49
47
Humans
Evaluation Criteria
Appropriate arrangement based on current relationships
 Even Vertex Distribution
 Measure character distances
 Small Number of Vertices
 Count number of vertices
 Fixed Vertices
 Measure distance from targets/marks
 Centering and Encircling of Groups
 Comparison to semi-circular shape

of 49
48
Humans
Results
100s of Random Relationship Scenarios
 Even Vertex Distribution
 3.14 feet (SD=1.54) between characters
 Small Number of Vertices
 At most 40 vertices in graph, with 12 characters
 Fixed Vertices
 3.30 feet (SD=1.52) from target
 Centering and Encircling of Groups
 Characters formed nice semi-circles
C. Talbot and G. M. Youngblood. Application of Force-Directed Graphs on Character Positioning. In Proceedings
of the Spatial Computing Workshop (SCW 2013) collocated with AAMAS (W09), pages 53-58. IFAMAAS
(International Foundation for Autonomous Agents and Multi-agent Systems), 2013.

of 49
Incorporating Forces for Time-
Based Arrangements 49
Humans

of 49
Evaluation Criteria
 Occlusion
 Clustering
50
Humans

of 49
51
Humans
ResultsCase#
Case
Description
Avg
Occlusion
Average
Clustering
X
Average
Clustering
Y
0Baseline All AI 3.60% 19.50% 14.60%
1Baseline Human 90% 3.60% 19.10% 15.40%
2Baseline Human 50% 2.90% 20.00% 14.70%
3Baseline Human 10% 4.40% 30.90% 28.70%
4Forces All AI 2.40% 16.80% 14.60%
5Forces Human 90% 2.40% 16.80% 14.60%
6Forces Human 50% 1.60% 20.40% 13.80%
7Forces Human 10% 2.40% 20.80% 14.00%
C. Talbot and G. M. Youngblood. Scene Blocking Utilizing Forces. In Florida Artificial Intelligence Research
Society Conference, FLAIRS '14, pages 91-96. Association for the Advancement of Artificial Intelligence, 2014.

of 49
52
Humans
ResultsCase#
Case
Description
Avg
Occlusion
Average
Clustering
X
Average
Clustering
Y
0Baseline All AI 3.60% 19.50% 14.60%
1Baseline Human 90% 3.60% 19.10% 15.40%
2Baseline Human 50% 2.90% 20.00% 14.70%
3Baseline Human 10% 4.40% 30.90% 28.70%
4Forces All AI 2.40% 16.80% 14.60%
5Forces Human 90% 2.40% 16.80% 14.60%
6Forces Human 50% 1.60% 20.40% 13.80%
7Forces Human 10% 2.40% 20.80% 14.00%
C. Talbot and G. M. Youngblood. Scene Blocking Utilizing Forces. In Florida Artificial Intelligence Research
Society Conference, FLAIRS '14, pages 91-96. Association for the Advancement of Artificial Intelligence, 2014.

User Studies
53

of 49
Block World 3D Representation
54
User
Studies

of 49
Survey Questions
1. Characters showed evidence of
engaged listening
2. Characters appeared to perform
suitable movements on cue
3. The pace of the performance was
too fast
4. The pace of the performance was
too slow
5. The use of the space on stage was
appropriate
6. The blocking (positioning and timing
of the characters) was appropriate
7. There was adequate variety in the
staging positions of the characters
8. The characters’ movement onstage
during the performance was believable
in the context of the performance
9. The performance is free from
distracting behavior that does not
contribute to the scene
10. The arrangement of the performers
appropriately conveys the mood of the
scene
11. The character movements provide
appropriate dramatic emphasis
12. There is adequate variety and
balance in the use of the performance
space
13. All visible behaviors appear to be
motivated and coordinated within the
scene
14. The characters were grouped to
give proper emphasis to the right
characters at the right time
15. The characters frequently covered
or blocked each other from your point
of view
16. The movements of the characters
were consistent with the play
17. There was a great deal of random
movement
18. The characters’ reactions to other
characters were believable
19. Characters showed a lack of
engagement when listening
20. The arrangement of the performers
contradicts the mood of the scene
21. The more prominent characters in
the scene were hidden or masked from
your view
22. The characters were too close
together
23. The characters were too far apart
24. The stage space was not utilized to
its full potential
25. All characters were visible from
your point of view throughout the
scene
55
User
Studies

of 49
Results
56
User
Studies
StronglyDisagreeNeutralStronglyAgree
Mean

of 49
Results
57
User
Studies
StronglyDisagreeNeutralStronglyAgree
Mean

Planned Future Work
58

of 49
Planned Future Work
 Additional User Studies (shortened)
 Random
 Baseline
 NLP
 NLP + Rules
 NLP + Rules + FDGs
 Human Interaction User Studies
 Baseline
 NLP + Rules + FDGs
 Generalization
 Identify play-types based on organization
 Apply & evaluate techniques for up to 10 of these
59
Planned

Summary
60

of 49
 Proposed Play-Scripts
 Applied NLP
 Added Rules Engine
 Evaluated Speech for Implied Movement
 Incorporated Human-Controlled Characters
 Added FDGs and Algorithms
 Created Spatial Performance Evaluation
 Initial User Study
Summary
61
Summary

of 49
Christine Talbot
ctalbot1@uncc.edu
Questions?
 C. Talbot. Creating an Artificially Intelligent Director (AID) for Theatre and Virtual Environments. In Proceedings of the
2013 International Conference on Autonomous Agents and Multi-agent Systems, AAMAS '13, pages 1457-1458,
Richland, SC, 2013. International Foundation for Autonomous Agents and Multi-agent Systems.
 C. Talbot and G. M. Youngblood. Spatial Cues in Hamlet. In Proceedings of the 12th International Conference on
Intelligent Virtual Agents, IVA '12, pages 252-259, Berlin, Heidelberg, 2012. Springer-Verlag.
 C. Talbot and G. M. Youngblood. Application of Force-Directed Graphs on Character Positioning. In Proceedings of the
Spatial Computing Workshop (SCW 2013) collocated with AAMAS (W09), pages 53-58. IFAMAAS (International
Foundation for Autonomous Agents and Multi-agent Systems), 2013.
 C. Talbot and G. M. Youngblood. Lack of Spatial Indicators in Hamlet. In Florida Artificial Intelligence Research Society
Conference, FLAIRS '13, pages 154-159. Association for the Advancement of Artificial Intelligence, 2013.
 C. Talbot and G. M. Youngblood. Positioning Characters Using Forces. In Proceedings of the Cognitive Agents for
Virtual Environments Workshop (CAVE 2013) collocated with AAMAS (W08). IFAMAAS (International Foundation for
Autonomous Agents and Multi-agent Systems), 2013.
 C. Talbot and G. M. Youngblood. Shakespearean Spatial Rules. In Proceedings of the 2013 International Conference on
Autonomous Agents and Multi-agent Systems, AAMAS '13, pages 587-594, Richland, SC, 2013. International
Foundation for Autonomous Agents and Multi-agent Systems.
 C. Talbot and G. M. Youngblood. Scene Blocking Utilizing Forces. In Florida Artificial Intelligence Research Society
Conference, FLAIRS '14, pages 91-96. Association for the Advancement of Artificial Intelligence, 2014.
62
Questions
Selected Bibliography Highlighting This Work