Your SlideShare is downloading. ×
0
Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns
Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns
Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns
Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns
Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns
Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns
Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns
Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns
Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns
Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns
Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns
Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns
Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns
Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns
Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns
Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns
Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns
Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns
Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns
Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns
Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns
Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns
Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns
Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns
Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns

532

Published on

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
532
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. It s About Time  Analyzing Temporal MicroIt’s About Time – Analyzing Temporal Micro‐ Level Behavioral Patterns Chen Yu Indiana University
  • 2. Temporal Bands of Human Computation (Ballard, Hayhoe, Pook, & Rao, 1997)  (B ll d H h P k & R 1997)macro-level ternalbehaviormicro-levelmicro level extbehavior nal intern
  • 3. Dual Eye Tracking in Child-Parent Interaction ( (with Linda Smith, Damian Fricker, & Linger Xu) , , g ) child’s first person view parent’s first person view from eye camera from head camera from head camera child eye tracker adult eye tracker
  • 4. Dual Eye Tracking in Child-Parent Interaction
  • 5. Multimodal data• Vision: 720*480, 30 frames/second, 3 cameras, 18,000 frames per dyads.• Motion tracking: 250 Hz/second, 6 sensors with six dimensions (x,y,z,h,p,r) on each. 900,000 900 000 data points per dyads dyads.• Speech: 44.1Hz• Eye tracking: 30Hz• 12GB per participant
  • 6. Dealing with data• Synchronization of multiple data streams• Data annotation (automatically or manually)• Data management• Data Mining and Knowledge Discovery
  • 7. Multi‐Streaming Multimodal Data x 10 4 speech525150 0 500 1000 1500 2000 2500 3000 35 4 x 10864 vision20 0 500 1000 1500 2000 2500 3000 3 4x 100 500 1000 1500 2000 2500 3000 …… motion
  • 8. Data Mining• Our goal of analyzing gaze data is to find new patterns and gain new knowledge from such data.• With micro level data, even if we have some predictions micro-level data from our experimental designs, we nonetheless lack precise predictions about the structure and patterns of data at the micro-level.• But how can we discover new and meaningful patterns if we d not know what we are looking for? do k h l k f• Discovering new knowledge requires the ability to detect unknown, surprising, novel, and unexpected patterns.• A particular challenge is not just from the amount of data but p g j from how to extract, select and interpret meaningful patterns from a sea of complex data.
  • 9. childgazeparentgaze
  • 10. Interactive Data Analysis top‐down  knowledge data  data pattern  pattern visualization extraction• This solution relies on both computational techniques and human domain knowledge. Data visualization and pattern extraction techniques provide candidate patterns through a bottom up way. bottom-up way• Compared with “blind” data mining, what we suggest is that researchers with top down top-down theoretical knowledge need to be in the loop of data mining and data analysis.
  • 11. childgazeparentgaze
  • 12. face three objects red blue red green red bluechild hildgazeparentgaze face gaze and mutual gaze joint attention
  • 13. Joint Attention State Joint Attention State60%50%40%30%20%10% 0% mutual gaze t l child‐face/parent‐object parent‐face/child‐object hild f / t bj t t f / hild bj t same object bj t different objects diff t bj t‐10%
  • 14. child following child following face three objectschildgazeparentgaze parent followingchildggazeparentgaze
  • 15. What is joint attention made of? when baby looks mom’s face, mom  when MOM looks baby’s face, baby  looks at baby’s face 54% of the time;  when baby looks at an object mom  looks at mom’s face 13% of the  looks at 32% of timetime; when mom looks at an object  baby looks at 14% of time baby lookshild of time at 14% child parent t 54%(face) 32%(object) 13%(face) 14%(object) gaze gaze 55% 56% hand hand 17% 21% probabilities of following an attended object/face
  • 16. Sequential Patterns From Multimodal Data Stream active face-to- joint attention following facelearner’sgaze 1 1 1teacher’sgaze 2 3 2 3 2 3teacher’shandaction 4 4 4 face three objects
  • 17. Segmentation and Alignment
  • 18. Discovering Statistically Reliable Sequential Patterns (Fricker, Zhang, & Yu, 2011)multiple instances naming utterance face look describing utterance sequential prototype durations and timings within and across multiple events
  • 19. More complex patterns from multiple data streams lti l d t t
  • 20. From Data To Patterns
  • 21. From Patterns To Knowledge ……
  • 22. Interactive Data Mining top‐down  knowledge data  pattern  visualization extraction• Human-in-the-Loop data analysis involves active and iterative examination and exploration of visually displayed patterns to select useful information and guide further knowledge discovery discovery.• “Discovery takes place between the ears” – Ben Shneiderman
  • 23. Visual Data Mining(Yu, Zhong, Smith, Park, & Huang, 2008, 2009)
  • 24. Acknowledgement Data collection and pre-processing : Damian Fricker, Amanda p p g , Favata, Char Wozniak, Melissa Elston. Data Processing: Tian Xu, Damian Fricker, Thomas Smith Henry Xu Fricker Smith, Shen. This research was s ppo ted b NSF BCS 0924248 and NIH T32 esea ch as supported by HD 07475 Thanks!

×