0
1
Hayley Hung, TUDelft
Socially Intelligent Sensing
Hayley Hung
Pattern Recognition and Bioinformatics Group
Intelligent S...
2
Hayley Hung, TUDelft
What is social behaviour?
3
Hayley Hung, TUDelft
What is Human Face-to-face interaction for ?
Makes our lives easier:
Relationships
Trust
Co-operati...
4
Hayley Hung, TUDelft
Research Mission Statement
To develop algorithms that can model and understand non-
verbal human so...
Hayley Hung, TUDelft
What can you say about this picture?
Hayley Hung, TUDelft
What can you say about this picture?
Relaxed postureGestures
Vocal Behaviour
Mutual Gaze
Interpersona...
7
Hayley Hung, TUDelft
Current Research Frontier
Person
detection
Person
tracking
Gaze
detection
Body pose
estimation
Grou...
8
Hayley Hung, TUDelft
Current Research Frontier
Relationship intimacy
estimation
Conversation
quality estimation
Person
d...
9
Hayley Hung, TUDelft
How to model social behaviour
Sensor Data Feature and Cue Extraction
Data Annotation
Social Behavio...
10
Hayley Hung, TUDelft
Task 1: Estimating Attraction
Source: http://catinbag.blogspot.nl/2010/07/fatal-attraction.html
Ve...
11
Hayley Hung, TUDelft
Speed Dating, Non-verbal cues and Attraction
Can proximity-related video cues be used to automatic...
12
Hayley Hung, TUDelft
Automated Position Extraction
- =
13
Hayley Hung, TUDelft
Automated Position Extraction
14
Hayley Hung, TUDelft
A Sped up Speed Date:
15
Hayley Hung, TUDelft
Speed Dating Results
Predicting attraction
Variance in position is best
feature predictor for
wome...
16
Hayley Hung, TUDelft
Speed Date Experiments : Conclusion
The video channel can indeed be a source of valuable
informati...
17
Hayley Hung, TUDelft
Task 2: Classifiying Social Actions using a
Single Wearable Accelerometer
Hung, Englebienne, Kools...
18
Hayley Hung, TUDelft
Modelling Human Social Behaviour in Dense Crowds
How can we model instantaneous social
behaviour i...
19
Hayley Hung, TUDelft
Our Goal
To develop methods to automatically measure
socially relevant behaviour and moods in dens...
20
Hayley Hung, TUDelft
Our Goal
Use insights from Social
Psychology:
Speakers move more than
listeners (McNeill 2000)
Lau...
21
Hayley Hung, TUDelft
Data: The Scenario
32 volunteers (mostly mutual strangers)
Experiment Stages: Briefing; Meeting an...
22
Hayley Hung, TUDelft
Data: The Scenario
32 volunteers (mostly mutual strangers)
Experiment Stages: Briefing; Meeting an...
23
Hayley Hung, TUDelft
Data: The Scenario
32 volunteers (mostly mutual strangers)
Experiment Stages: Briefing; Meeting an...
24
Hayley Hung, TUDelft
Classifying social actions
Social Actions: speaking, laughing, gesturing, drinking, or
stepping
(%...
25
Hayley Hung, TUDelft
Social Action Conclusion
It is possible to detect socially relevant behaviour.
Can we detect when ...
26
Hayley Hung, TUDelft
Looking to the Future...
The way we behave socially can exhibit strong detectable
patterns, which ...
27
Hayley Hung, TUDelft
Potential Applications: Human Robot Interaction
28
Hayley Hung, TUDelft
Applications: Urban Planning
Behaviour in Public Spaces
How can we measure statistically generalis...
29
Hayley Hung, TUDelft
Potential Applications: Organisational behaviour
30
Hayley Hung, TUDelft
Acknowledgements/Collaborators
Arno Veenstra
Gwenn Englebienne Jeroen Kools
Ben Krose
Maarten
van ...
31
Hayley Hung, TUDelft
Workshop at ACM Multmedia (acmmm13.org/)
Barcelona , October 22
Human Behaviour Understanding for ...
32
Hayley Hung, TUDelft
Data:Sensors
Each participant wore:
indoor positioning device
Proximity and Acceleration sensor
12...
Upcoming SlideShare
Loading in...5
×

Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS

1,881

Published on

Tenure Track Assistant Professor gave this presentation at the first Behavior Design AMS meetup on Thursday, September 19th in Amsterdam.

Published in: Design, Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,881
On Slideshare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS"

  1. 1. 1 Hayley Hung, TUDelft Socially Intelligent Sensing Hayley Hung Pattern Recognition and Bioinformatics Group Intelligent Systems Department
  2. 2. 2 Hayley Hung, TUDelft What is social behaviour?
  3. 3. 3 Hayley Hung, TUDelft What is Human Face-to-face interaction for ? Makes our lives easier: Relationships Trust Co-operation Persuade/influence others Information sharing How could technology help us to understand/interpret/ socially relevant behaviour? How could this help to influence/enhance our experience?
  4. 4. 4 Hayley Hung, TUDelft Research Mission Statement To develop algorithms that can model and understand non- verbal human social behaviour in real life situations. And through this, to understand how to build systems that can enhance people's quality of life by behaving with socially aware intelligence. Develop algorithms that are perceptive to human social behaviour: Social Signal Processing, Machine Learning Enhancing people's quality of life: Human Machine Interaction, Ambient Intelligent Environments, Design, Architecture. Socially aware intelligence: Social and Behavioural Psychology, Ethnography.
  5. 5. Hayley Hung, TUDelft What can you say about this picture?
  6. 6. Hayley Hung, TUDelft What can you say about this picture? Relaxed postureGestures Vocal Behaviour Mutual Gaze Interpersonal Distance
  7. 7. 7 Hayley Hung, TUDelft Current Research Frontier Person detection Person tracking Gaze detection Body pose estimation Group detection Social and Behavioural Pscychology, Ethnography Activity modelling Action recognition Attraction Estimation Rapport Estimation Role Recognition Personality estimation Dominance Estimation
  8. 8. 8 Hayley Hung, TUDelft Current Research Frontier Relationship intimacy estimation Conversation quality estimation Person detection Person tracking Gaze detection Body pose estimation Group detection Conversational event estimation Personality estimation Relationship quality estimation Social and Behavioural Pscychology, Ethnography Activity modelling Action recognition New Problem Definitions
  9. 9. 9 Hayley Hung, TUDelft How to model social behaviour Sensor Data Feature and Cue Extraction Data Annotation Social Behaviour Modelling and Classification Model Performance Evaluation
  10. 10. 10 Hayley Hung, TUDelft Task 1: Estimating Attraction Source: http://catinbag.blogspot.nl/2010/07/fatal-attraction.html Veenstra and Hung, “Do They Like Me? Using Video Cues to Predict Desires during Speed-dates” in ICCV Workshops 2011
  11. 11. 11 Hayley Hung, TUDelft Speed Dating, Non-verbal cues and Attraction Can proximity-related video cues be used to automatically predict attraction in speed-dates?
  12. 12. 12 Hayley Hung, TUDelft Automated Position Extraction - =
  13. 13. 13 Hayley Hung, TUDelft Automated Position Extraction
  14. 14. 14 Hayley Hung, TUDelft A Sped up Speed Date:
  15. 15. 15 Hayley Hung, TUDelft Speed Dating Results Predicting attraction Variance in position is best feature predictor for women (70%). Variance in position of the women and synchrony both perform well (70%) for men. Fusion of all synchrony features Fusion of all movement features
  16. 16. 16 Hayley Hung, TUDelft Speed Date Experiments : Conclusion The video channel can indeed be a source of valuable information in speed-dates Results differ per gender: Movement synchrony information is more important for males than females. For females, information on the movement of their male counterpart gives good results
  17. 17. 17 Hayley Hung, TUDelft Task 2: Classifiying Social Actions using a Single Wearable Accelerometer Hung, Englebienne, Kools, “Estimating Social Actions”, Ubicomp 2013
  18. 18. 18 Hayley Hung, TUDelft Modelling Human Social Behaviour in Dense Crowds How can we model instantaneous social behaviour in extremely large crowds?
  19. 19. 19 Hayley Hung, TUDelft Our Goal To develop methods to automatically measure socially relevant behaviour and moods in dense crowds using just a single tri-axial accelerometer First step: detect socially relevant actions Speaking; Laughing; Gesturing; Stepping; Drinking
  20. 20. 20 Hayley Hung, TUDelft Our Goal Use insights from Social Psychology: Speakers move more than listeners (McNeill 2000) Laughter and joking correlated with sudden bursts of motion (Kendon 1990) Synchronised motion during conversation (Kendon 1990,Chartrand and Bargh 1999)
  21. 21. 21 Hayley Hung, TUDelft Data: The Scenario 32 volunteers (mostly mutual strangers) Experiment Stages: Briefing; Meeting and Mingling; Team formation (groups of 4); Quiz; Award Giving Prizes for top 3 teams 5mx6m recording area
  22. 22. 22 Hayley Hung, TUDelft Data: The Scenario 32 volunteers (mostly mutual strangers) Experiment Stages: Briefing; Meeting and Mingling; Team formation (groups of 4); Quiz; Award Giving Prizes for top 3 teams 5mx6m recording area
  23. 23. 23 Hayley Hung, TUDelft Data: The Scenario 32 volunteers (mostly mutual strangers) Experiment Stages: Briefing; Meeting and Mingling; Team formation (groups of 4); Quiz; Award Giving Prizes for top 3 teams 5mx6m recording area Each participant wore a sensing badge
  24. 24. 24 Hayley Hung, TUDelft Classifying social actions Social Actions: speaking, laughing, gesturing, drinking, or stepping (%) Gesture Step Drink Laugh Speech Precision 59 100 100 100 64 Recall 24 21 21 38 82 F- measure 34 35 35 56 72
  25. 25. 25 Hayley Hung, TUDelft Social Action Conclusion It is possible to detect socially relevant behaviour. Can we detect when people are in the same conversational group? Could we even detect personality traits quality of people's interaction? Quality of people's relationships? ...etc
  26. 26. 26 Hayley Hung, TUDelft Looking to the Future... The way we behave socially can exhibit strong detectable patterns, which are robust to noise. How simple can the extracted features be? How could socially aware systems benefit better design?
  27. 27. 27 Hayley Hung, TUDelft Potential Applications: Human Robot Interaction
  28. 28. 28 Hayley Hung, TUDelft Applications: Urban Planning Behaviour in Public Spaces How can we measure statistically generaliseable changes as a result of interventions?
  29. 29. 29 Hayley Hung, TUDelft Potential Applications: Organisational behaviour
  30. 30. 30 Hayley Hung, TUDelft Acknowledgements/Collaborators Arno Veenstra Gwenn Englebienne Jeroen Kools Ben Krose Maarten van Steen Matt Dobson Claudio Martella Domenic Vossen
  31. 31. 31 Hayley Hung, TUDelft Workshop at ACM Multmedia (acmmm13.org/) Barcelona , October 22 Human Behaviour Understanding for the Interactions in Arts, Creativity, Entertainment and Edutainment Albert Ali Salah (Bogazici University, Turkey) Oya Aran (Idiap Research Institute, Switzerland) Hatice Gunes (Queen Mary University of London, UK)
  32. 32. 32 Hayley Hung, TUDelft Data:Sensors Each participant wore: indoor positioning device Proximity and Acceleration sensor 12 participants wore wireless microphone (for annotation) 3 fish eye cameras (for annotation) 3 accelerometer readings failed (software bug)
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×