Creating Dynamic Social Network Models from Sensor Data Tanzeem Choudhury Intel Research / Affiliate Faculty CSE Dieter Fo...
<ul><li>What  are we doing? </li></ul><ul><li>Why  are we doing it? </li></ul><ul><li>How  are we doing it? </li></ul>
Social Network Analysis <ul><li>Work across the social & physical sciences is increasingly studying the  structure of huma...
Social Networks <ul><li>Social networks are naturally represented and analyzed as graphs </li></ul>
Example Network Properties <ul><li>Degree of a node </li></ul><ul><li>Eigenvector centrality </li></ul><ul><ul><li>global ...
Applications <ul><li>Many practical applications </li></ul><ul><ul><li>Business – discovering organizational bottlenecks <...
The Data Problem <ul><li>Traditionally data comes from manual surveys of people’s recollections </li></ul><ul><ul><li>Very...
Social Network Analysis of Email <ul><li>Science, 6 Jan 2006 </li></ul>
Limits of E-Data <ul><li>Email data is cheap and accurate, but misses </li></ul><ul><ul><li>Face-to-face speech  – the vas...
Research Goal <ul><li>Demonstrate that we can… </li></ul><ul><li>Model social network dynamics by gathering large amounts ...
Questions we want to investigate: <ul><li>Changes in social networks  over time: </li></ul><ul><ul><li>How do interaction ...
Support <ul><li>Human and Social Dynamics – one of five new priority areas for NSF </li></ul><ul><ul><li>$800K award to UW...
Procedure <ul><li>Test group </li></ul><ul><ul><li>32 first-year incoming  CSE graduate students </li></ul></ul><ul><ul><l...
Privacy <ul><li>UW Human Subjects Division approved procedures after 6 months of review and revisions </li></ul><ul><li>Ma...
Data Collection <ul><li>Intel Multi-Modal Sensor Board </li></ul>Real-time audio feature extraction audio features WiFi st...
Data Collection <ul><li>Multi-sensor board sends sensor data stream to iPAQ </li></ul><ul><li>iPAQ computes audio features...
Older Procedure <ul><li>Because the real-time feature extraction software was not finished in time, the Autumn 2005 data c...
Speech Detection <ul><li>From the audio signal, we want to extract features that can be used to determine </li></ul><ul><u...
Speech Production Fundamental frequency  (F0/pitch) and  formant frequencies  (F1, F2 …) are the  most important component...
Speech Production <ul><li>Voiced sounds: Fundamental frequency (i.e. harmonic structure) and energy in lower frequency com...
Goal: Reliably Detect Voiced Chunks in Audio Stream
Speech Features Computed <ul><li>Spectral entropy </li></ul><ul><li>Relative spectral entropy  </li></ul><ul><li>Total ene...
Features used: Autocorrelation Autocorrelation of (a) un-voiced frame and (b)  voiced frame.  Voiced chunks have higher no...
Features used: Spectral Entropy FFT magnitude of (a) un-voiced frame and (b)  voiced frame.  Voiced chunks have lower entr...
Features used: Energy Energy in voiced chunks is concentrated in the lower frequencies  Higher order MEL cepstral coeffici...
Segmenting Speech Regions
Attributes Useful for Inferring Interaction <ul><li>Attributes that can be reliably extracted from sensors: </li></ul><ul>...
Locations <ul><li>Wi-Fi signal strength can be used to determine the approximate location of each speech event </li></ul><...
Topological Location Map <ul><li>Nodes in map are identified by  area types </li></ul><ul><ul><li>Hallway </li></ul></ul><...
Social Network Model <ul><li>Nodes </li></ul><ul><ul><li>Subjects (wearing sensors, have given consent) </li></ul></ul><ul...
Non-instrumented Subjects <ul><li>We may recruit additional subjects who do not wear sensors </li></ul><ul><li>Such subjec...
Disabling Sensor Units <ul><li>As a courtesy, subjects will disable their units in particular classrooms or offices </li><...
Access to the Data <ul><li>Publications about this project will include summary statistics about the social network, e.g.:...
Upcoming SlideShare
Loading in …5
×

hsd-faculty-lunch-jan06

268 views

Published on

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
268
On SlideShare
0
From Embeds
0
Number of Embeds
28
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • hsd-faculty-lunch-jan06

    1. 1. Creating Dynamic Social Network Models from Sensor Data Tanzeem Choudhury Intel Research / Affiliate Faculty CSE Dieter Fox Henry Kautz CSE James Kitts Sociology
    2. 2. <ul><li>What are we doing? </li></ul><ul><li>Why are we doing it? </li></ul><ul><li>How are we doing it? </li></ul>
    3. 3. Social Network Analysis <ul><li>Work across the social & physical sciences is increasingly studying the structure of human interaction </li></ul><ul><ul><li>1967 – Stanley Milgram – 6 degrees of separation </li></ul></ul><ul><ul><li>1973 – Mark Granovetter – strength of weak ties </li></ul></ul><ul><ul><li>1977 –International Network for Social Network Analysis </li></ul></ul><ul><ul><li>1992 – Ronald Burt – structural holes: the social structure of competition </li></ul></ul><ul><ul><li>1998 – Watts & Strogatz – small world graphs </li></ul></ul>
    4. 4. Social Networks <ul><li>Social networks are naturally represented and analyzed as graphs </li></ul>
    5. 5. Example Network Properties <ul><li>Degree of a node </li></ul><ul><li>Eigenvector centrality </li></ul><ul><ul><li>global importance of a node </li></ul></ul><ul><li>Average clustering coefficient </li></ul><ul><ul><li>degree to which graph decomposes into cliques  </li></ul></ul><ul><li>Structural holes </li></ul><ul><ul><li>opportunities for gain by bridging disconnected subgraphs </li></ul></ul>
    6. 6. Applications <ul><li>Many practical applications </li></ul><ul><ul><li>Business – discovering organizational bottlenecks </li></ul></ul><ul><ul><li>Health – modeling spread of communicable diseases </li></ul></ul><ul><ul><li>Architecture & urban planning – designing spaces that support human interaction </li></ul></ul><ul><ul><li>Education – understanding impact of peer group on educational advancement </li></ul></ul><ul><li>Much recent theory on finding random graph models that fit empirical data </li></ul>
    7. 7. The Data Problem <ul><li>Traditionally data comes from manual surveys of people’s recollections </li></ul><ul><ul><li>Very hard to gather </li></ul></ul><ul><ul><li>Questionable accuracy </li></ul></ul><ul><ul><li>Few published data sets </li></ul></ul><ul><ul><li>Almost no longitudinal (dynamic) data </li></ul></ul><ul><li>1990’s – social network studies based on electronic communication </li></ul>
    8. 8. Social Network Analysis of Email <ul><li>Science, 6 Jan 2006 </li></ul>
    9. 9. Limits of E-Data <ul><li>Email data is cheap and accurate, but misses </li></ul><ul><ul><li>Face-to-face speech – the vast majority of human interaction, especially complex communication </li></ul></ul><ul><ul><li>The physical context of communication – useless for studying the relationship between environment and interaction </li></ul></ul><ul><li>Can we gather data on face to face communication automatically? </li></ul>
    10. 10. Research Goal <ul><li>Demonstrate that we can… </li></ul><ul><li>Model social network dynamics by gathering large amounts of rich face-to-face interaction data automatically </li></ul><ul><ul><li>using wearable sensors </li></ul></ul><ul><ul><li>combined with statistical machine learning techniques </li></ul></ul><ul><li>Find simple and robust measures derived from sensor data </li></ul><ul><ul><li>that are indicative of people’s roles and relationships </li></ul></ul><ul><ul><li>that capture the connections between physical environment and network dynamics </li></ul></ul>
    11. 11. Questions we want to investigate: <ul><li>Changes in social networks over time: </li></ul><ul><ul><li>How do interaction patterns dynamically relate to structural position in the network? </li></ul></ul><ul><ul><li>Why do people sharing relationships tend to be similar? </li></ul></ul><ul><ul><li>Can one predict formation or break-up of communities? </li></ul></ul><ul><li>Effect of location on social networks </li></ul><ul><ul><li>What are the spatio-temporal distributions of interactions? </li></ul></ul><ul><ul><li>How do locations serve as hubs and bridges? </li></ul></ul><ul><ul><li>Can we predict the popularity of a particular location? </li></ul></ul>
    12. 12. Support <ul><li>Human and Social Dynamics – one of five new priority areas for NSF </li></ul><ul><ul><li>$800K award to UW / Intel / Georgia Tech team </li></ul></ul><ul><ul><li>Intel at no-cost </li></ul></ul><ul><li>Intel Research donating hardware and internships </li></ul><ul><li>Leveraging work on sensors & localization from other NSF & DARPA projects </li></ul>
    13. 13. Procedure <ul><li>Test group </li></ul><ul><ul><li>32 first-year incoming CSE graduate students </li></ul></ul><ul><ul><li>Units worn 5 working days each month </li></ul></ul><ul><ul><li>Collect data over one year </li></ul></ul><ul><li>Units record </li></ul><ul><ul><li>Wi-Fi signal strength, to determine location </li></ul></ul><ul><ul><li>Audio features adequate to determine when conversation is occurring </li></ul></ul><ul><li>Subjects answer short monthly survey </li></ul><ul><ul><li>Selective ground truth on # of interactions </li></ul></ul><ul><ul><li>Research interests </li></ul></ul><ul><li>All data stored securely </li></ul><ul><ul><li>Indexed by code number assigned to each subject </li></ul></ul>
    14. 14. Privacy <ul><li>UW Human Subjects Division approved procedures after 6 months of review and revisions </li></ul><ul><li>Major concern was privacy, addressed by </li></ul><ul><ul><li>Procedure for recording audio features without recording conversational content </li></ul></ul><ul><ul><li>Procedures for handling data afterwards </li></ul></ul>
    15. 15. Data Collection <ul><li>Intel Multi-Modal Sensor Board </li></ul>Real-time audio feature extraction audio features WiFi strength Coded Database code identifier
    16. 16. Data Collection <ul><li>Multi-sensor board sends sensor data stream to iPAQ </li></ul><ul><li>iPAQ computes audio features and WiFi node identifiers and signal strength </li></ul><ul><li>iPAQ writes audio and WiFi features to SD card </li></ul><ul><li>Each day, subject uploads data using his or her code number to the coded data base </li></ul>
    17. 17. Older Procedure <ul><li>Because the real-time feature extraction software was not finished in time, the Autumn 2005 data collections used a different process (also approved) </li></ul><ul><ul><li>Raw data was encrypted on the SD card </li></ul></ul><ul><ul><li>The upload program simultaneously unencrypted and extracted features </li></ul></ul><ul><ul><li>Only the features were uploaded </li></ul></ul>
    18. 18. Speech Detection <ul><li>From the audio signal, we want to extract features that can be used to determine </li></ul><ul><ul><li>Speech segments </li></ul></ul><ul><ul><li>Number of different participants (but not identity of participants) </li></ul></ul><ul><ul><li>Turn-taking style </li></ul></ul><ul><ul><li>Rate of conversation (fast versus slow speech) </li></ul></ul><ul><li>But the features must not allow the audio to be reconstructed! </li></ul>
    19. 19. Speech Production Fundamental frequency (F0/pitch) and formant frequencies (F1, F2 …) are the most important components for speech synthesis The source-filter Model vocal tract filter
    20. 20. Speech Production <ul><li>Voiced sounds: Fundamental frequency (i.e. harmonic structure) and energy in lower frequency component </li></ul><ul><li>Un-voiced sounds: No fundamental frequency and energy focused in higher frequencies </li></ul><ul><li>Our approach: Detect speech by reliably detecting voiced regions </li></ul><ul><li>We do not extract or store any formant information. At least three formants are required to produce intelligible speech* </li></ul>* 1. Donovan, R. (1996). Trainable Speech Synthesis. PhD Thesis. Cambridge University 2. O’Saughnessy, D. (1987). Speech Communication – Human and Machine, Addison-Wesley.
    21. 21. Goal: Reliably Detect Voiced Chunks in Audio Stream
    22. 22. Speech Features Computed <ul><li>Spectral entropy </li></ul><ul><li>Relative spectral entropy </li></ul><ul><li>Total energy </li></ul><ul><li>Energy below 2kHz (low frequencies) </li></ul><ul><li>Autocorrelation peak values and number of peaks </li></ul><ul><li>High order MEL frequency cepstral coefficients </li></ul>
    23. 23. Features used: Autocorrelation Autocorrelation of (a) un-voiced frame and (b) voiced frame. Voiced chunks have higher non-initial autocorrelation peak and fewer number of peaks (a) (b)
    24. 24. Features used: Spectral Entropy FFT magnitude of (a) un-voiced frame and (b) voiced frame. Voiced chunks have lower entropy than un-voiced chunks, because voiced chunks have more structure Spectral entropy: 3.74 Spectral entropy: 4.21
    25. 25. Features used: Energy Energy in voiced chunks is concentrated in the lower frequencies Higher order MEL cepstral coefficients contain pitch (F0) information. The lower order coefficients are NOT stored
    26. 26. Segmenting Speech Regions
    27. 27. Attributes Useful for Inferring Interaction <ul><li>Attributes that can be reliably extracted from sensors: </li></ul><ul><ul><li>Total number of interactions between people </li></ul></ul><ul><ul><li>Conversation styles – e.g. turn-taking, energy-level </li></ul></ul><ul><ul><li>Location where interactions take place – e.g. office, lobby etc. </li></ul></ul><ul><ul><li>Daily schedule of individuals – e.g. early birds, late nighters </li></ul></ul>
    28. 28. Locations <ul><li>Wi-Fi signal strength can be used to determine the approximate location of each speech event </li></ul><ul><ul><li>5 meter accuracy </li></ul></ul><ul><ul><li>Location computation done off-line </li></ul></ul><ul><li>Raw locations are converted to nodes in a coarse topological map before further analysis </li></ul>
    29. 29. Topological Location Map <ul><li>Nodes in map are identified by area types </li></ul><ul><ul><li>Hallway </li></ul></ul><ul><ul><li>Breakout area </li></ul></ul><ul><ul><li>Meeting room </li></ul></ul><ul><ul><li>Faculty office </li></ul></ul><ul><ul><li>Student office </li></ul></ul><ul><li>Detected conversations are associated with their area type </li></ul>
    30. 30. Social Network Model <ul><li>Nodes </li></ul><ul><ul><li>Subjects (wearing sensors, have given consent) </li></ul></ul><ul><ul><li>Public places (e.g., particular break out area) </li></ul></ul><ul><ul><li>Regions of private locations (e.g., hallway of faculty offices) </li></ul></ul><ul><ul><li>Instances of conversations </li></ul></ul><ul><li>Edges </li></ul><ul><ul><li>Between subjects and conversations </li></ul></ul><ul><ul><li>Between places or regions and conversations </li></ul></ul>
    31. 31. Non-instrumented Subjects <ul><li>We may recruit additional subjects who do not wear sensors </li></ul><ul><li>Such subjects would allow us to infer information about their behavior indirectly, and to appear (coded) as a node in our network model </li></ul><ul><ul><li>E.g., based on their particular office locations </li></ul></ul><ul><li>Only people who have provided written consent appear as entities in our network models </li></ul>
    32. 32. Disabling Sensor Units <ul><li>As a courtesy, subjects will disable their units in particular classrooms or offices </li></ul>
    33. 33. Access to the Data <ul><li>Publications about this project will include summary statistics about the social network, e.g.: </li></ul><ul><ul><li>Clustering coefficient </li></ul></ul><ul><ul><li>Motifs (temporal patterns) </li></ul></ul><ul><li>We will not release the actual graph </li></ul><ul><ul><li>This is prohibited by our HSD approval </li></ul></ul><ul><li>We welcome additional collaborators </li></ul>

    ×