This document summarizes research on analyzing temporal micro-level behavioral patterns from multimodal data streams. Key points: - The research collects extensive video, eye tracking, motion and audio data from child-parent interactions, totaling over 12GB per participant. - Analyzing micro-level gaze data aims to discover new patterns and knowledge not predicted by experimental design alone. Interactive data analysis relies on both computational techniques and human domain knowledge. - Preliminary findings show patterns in joint attention states, such as one person being more likely to follow the other's gaze to an attended object or face. Sequential patterns are discovered across data streams using segmentation and alignment. - The goal is to discover more complex patterns from

1. It s About Time Analyzing Temporal Micro
It’s About Time – Analyzing Temporal Micro‐
Level Behavioral Patterns
Chen Yu
Indiana University

2. Temporal Bands of Human Computation
(Ballard, Hayhoe, Pook, & Rao, 1997)
(B ll d H h P k & R 1997)
macro-level
ternal
behavior
micro-level
micro level
ext
behavior
nal
intern

3. Dual Eye Tracking in Child-Parent Interaction
(
(with Linda Smith, Damian Fricker, & Linger Xu)
, , g )
child’s first person view parent’s first person view
from
eye
camera
from head
camera
from head camera
child eye
tracker
adult eye tracker

4. Dual Eye Tracking in Child-Parent
Interaction

5. Multimodal data
• Vision: 720*480, 30 frames/second, 3 cameras, 18,000
frames per dyads.
• Motion tracking: 250 Hz/second, 6 sensors with six
dimensions (x,y,z,h,p,r) on each.
900,000
900 000 data points per dyads
dyads.
• Speech: 44.1Hz
• Eye tracking: 30Hz
• 12GB per participant

6. Dealing with data
• Synchronization of multiple data streams
• Data annotation (automatically or manually)
• Data management
• Data Mining and Knowledge Discovery

7. Multi‐Streaming Multimodal Data
x 10
4
speech
5
2
5
1
5
0
0 500 1000 1500 2000 2500 3000 35
4
x 10
8
6
4
vision
2
0
0 500 1000 1500 2000 2500 3000 3
4
x 10
0 500 1000 1500 2000 2500 3000
……
motion

8. Data Mining
• Our goal of analyzing gaze data is to find new patterns
and gain new knowledge from such data.
• With micro level data, even if we have some predictions
micro-level data
from our experimental designs, we nonetheless lack
precise predictions about the structure and patterns of
data at the micro-level.
• But how can we discover new and meaningful patterns if
we d not know what we are looking for?
do k h l k f
• Discovering new knowledge requires the ability to detect
unknown, surprising, novel, and unexpected patterns.
• A particular challenge is not just from the amount of data but
p g j
from how to extract, select and interpret meaningful patterns
from a sea of complex data.

9. child
gaze
parent
gaze

10. Interactive Data Analysis
top‐down
knowledge
data
data pattern
pattern
visualization extraction
• This solution relies on both computational techniques and human domain
knowledge. Data visualization and pattern extraction techniques provide
candidate patterns through a bottom up way.
bottom-up way
• Compared with “blind” data mining, what we suggest is that researchers with
top down
top-down theoretical knowledge need to be in the loop of data mining and
data analysis.

11. child
gaze
parent
gaze

12. face three objects
red blue red green red blue
child
hild
gaze
parent
gaze
face gaze and mutual gaze joint attention

13. Joint Attention State
Joint Attention State
60%
50%
40%
30%
20%
10%
0%
mutual gaze
t l child‐face/parent‐object parent‐face/child‐object
hild f / t bj t t f / hild bj t same object
bj t different objects
diff t bj t
‐10%

14. child following
child following
face three objects
child
gaze
parent
gaze
parent following
child
g
gaze
parent
gaze

15. What is joint attention made of?
when baby looks mom’s face, mom
when MOM looks baby’s face, baby looks at baby’s face 54% of the time;
when baby looks at an object mom
looks at mom’s face 13% of the
looks at 32% of time
time; when mom looks at an object
baby looks at 14% of time
baby lookshild of time
at 14%
child parent
t
54%(face)
32%(object)
13%(face)
14%(object)
gaze gaze
55% 56%
hand hand
17%
21%
probabilities of following an attended object/face

16. Sequential Patterns From
Multimodal Data Stream
active face-to-
joint attention following face
learner’s
gaze 1 1 1
teacher’s
gaze 2 3 2 3 2 3
teacher’s
hand
action 4 4 4
face three objects

17. Segmentation and Alignment

18. Discovering Statistically Reliable
Sequential Patterns
(Fricker, Zhang, & Yu, 2011)
multiple instances naming utterance face look
describing utterance
sequential prototype
durations and timings within and
across multiple events

19. More complex patterns from
multiple data streams
lti l d t t

21. From Data To Patterns

22. From Patterns To Knowledge
……

23. Interactive Data Mining
top‐down
knowledge
data pattern
visualization extraction
• Human-in-the-Loop data analysis involves active and iterative
examination and exploration of visually displayed patterns to
select useful information and guide further knowledge discovery
discovery.
• “Discovery takes place between the ears” – Ben Shneiderman

24. Visual Data Mining
(Yu, Zhong, Smith, Park, & Huang, 2008, 2009)

25. Acknowledgement
Data collection and pre-processing : Damian Fricker, Amanda
p p g ,
Favata, Char Wozniak, Melissa Elston.
Data Processing: Tian Xu, Damian Fricker, Thomas Smith Henry
Xu Fricker Smith,
Shen.
This research was s ppo ted b NSF BCS 0924248 and NIH T32
esea ch as supported by
HD 07475
Thanks!