Upcoming SlideShare
Loading in...5

Like this? Share it with your network

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Creating Dynamic Social Network Models from Sensor Data Tanzeem Choudhury Intel Research / Affiliate Faculty CSE Dieter Fox Henry Kautz CSE James Kitts Sociology
  • 2.
    • What are we doing?
    • Why are we doing it?
    • How are we doing it?
  • 3. Social Network Analysis
    • Work across the social & physical sciences is increasingly studying the structure of human interaction
      • 1967 – Stanley Milgram – 6 degrees of separation
      • 1973 – Mark Granovetter – strength of weak ties
      • 1977 –International Network for Social Network Analysis
      • 1992 – Ronald Burt – structural holes: the social structure of competition
      • 1998 – Watts & Strogatz – small world graphs
  • 4. Social Networks
    • Social networks are naturally represented and analyzed as graphs
  • 5. Example Network Properties
    • Degree of a node
    • Eigenvector centrality
      • global importance of a node
    • Average clustering coefficient
      • degree to which graph decomposes into cliques 
    • Structural holes
      • opportunities for gain by bridging disconnected subgraphs
  • 6. Applications
    • Many practical applications
      • Business – discovering organizational bottlenecks
      • Health – modeling spread of communicable diseases
      • Architecture & urban planning – designing spaces that support human interaction
      • Education – understanding impact of peer group on educational advancement
    • Much recent theory on finding random graph models that fit empirical data
  • 7. The Data Problem
    • Traditionally data comes from manual surveys of people’s recollections
      • Very hard to gather
      • Questionable accuracy
      • Few published data sets
      • Almost no longitudinal (dynamic) data
    • 1990’s – social network studies based on electronic communication
  • 8. Social Network Analysis of Email
    • Science, 6 Jan 2006
  • 9. Limits of E-Data
    • Email data is cheap and accurate, but misses
      • Face-to-face speech – the vast majority of human interaction, especially complex communication
      • The physical context of communication – useless for studying the relationship between environment and interaction
    • Can we gather data on face to face communication automatically?
  • 10. Research Goal
    • Demonstrate that we can…
    • Model social network dynamics by gathering large amounts of rich face-to-face interaction data automatically
      • using wearable sensors
      • combined with statistical machine learning techniques
    • Find simple and robust measures derived from sensor data
      • that are indicative of people’s roles and relationships
      • that capture the connections between physical environment and network dynamics
  • 11. Questions we want to investigate:
    • Changes in social networks over time:
      • How do interaction patterns dynamically relate to structural position in the network?
      • Why do people sharing relationships tend to be similar?
      • Can one predict formation or break-up of communities?
    • Effect of location on social networks
      • What are the spatio-temporal distributions of interactions?
      • How do locations serve as hubs and bridges?
      • Can we predict the popularity of a particular location?
  • 12. Support
    • Human and Social Dynamics – one of five new priority areas for NSF
      • $800K award to UW / Intel / Georgia Tech team
      • Intel at no-cost
    • Intel Research donating hardware and internships
    • Leveraging work on sensors & localization from other NSF & DARPA projects
  • 13. Procedure
    • Test group
      • 32 first-year incoming CSE graduate students
      • Units worn 5 working days each month
      • Collect data over one year
    • Units record
      • Wi-Fi signal strength, to determine location
      • Audio features adequate to determine when conversation is occurring
    • Subjects answer short monthly survey
      • Selective ground truth on # of interactions
      • Research interests
    • All data stored securely
      • Indexed by code number assigned to each subject
  • 14. Privacy
    • UW Human Subjects Division approved procedures after 6 months of review and revisions
    • Major concern was privacy, addressed by
      • Procedure for recording audio features without recording conversational content
      • Procedures for handling data afterwards
  • 15. Data Collection
    • Intel Multi-Modal Sensor Board
    Real-time audio feature extraction audio features WiFi strength Coded Database code identifier
  • 16. Data Collection
    • Multi-sensor board sends sensor data stream to iPAQ
    • iPAQ computes audio features and WiFi node identifiers and signal strength
    • iPAQ writes audio and WiFi features to SD card
    • Each day, subject uploads data using his or her code number to the coded data base
  • 17. Older Procedure
    • Because the real-time feature extraction software was not finished in time, the Autumn 2005 data collections used a different process (also approved)
      • Raw data was encrypted on the SD card
      • The upload program simultaneously unencrypted and extracted features
      • Only the features were uploaded
  • 18. Speech Detection
    • From the audio signal, we want to extract features that can be used to determine
      • Speech segments
      • Number of different participants (but not identity of participants)
      • Turn-taking style
      • Rate of conversation (fast versus slow speech)
    • But the features must not allow the audio to be reconstructed!
  • 19. Speech Production Fundamental frequency (F0/pitch) and formant frequencies (F1, F2 …) are the most important components for speech synthesis The source-filter Model vocal tract filter
  • 20. Speech Production
    • Voiced sounds: Fundamental frequency (i.e. harmonic structure) and energy in lower frequency component
    • Un-voiced sounds: No fundamental frequency and energy focused in higher frequencies
    • Our approach: Detect speech by reliably detecting voiced regions
    • We do not extract or store any formant information. At least three formants are required to produce intelligible speech*
    * 1. Donovan, R. (1996). Trainable Speech Synthesis. PhD Thesis. Cambridge University 2. O’Saughnessy, D. (1987). Speech Communication – Human and Machine, Addison-Wesley.
  • 21. Goal: Reliably Detect Voiced Chunks in Audio Stream
  • 22. Speech Features Computed
    • Spectral entropy
    • Relative spectral entropy
    • Total energy
    • Energy below 2kHz (low frequencies)
    • Autocorrelation peak values and number of peaks
    • High order MEL frequency cepstral coefficients
  • 23. Features used: Autocorrelation Autocorrelation of (a) un-voiced frame and (b) voiced frame. Voiced chunks have higher non-initial autocorrelation peak and fewer number of peaks (a) (b)
  • 24. Features used: Spectral Entropy FFT magnitude of (a) un-voiced frame and (b) voiced frame. Voiced chunks have lower entropy than un-voiced chunks, because voiced chunks have more structure Spectral entropy: 3.74 Spectral entropy: 4.21
  • 25. Features used: Energy Energy in voiced chunks is concentrated in the lower frequencies Higher order MEL cepstral coefficients contain pitch (F0) information. The lower order coefficients are NOT stored
  • 26. Segmenting Speech Regions
  • 27. Attributes Useful for Inferring Interaction
    • Attributes that can be reliably extracted from sensors:
      • Total number of interactions between people
      • Conversation styles – e.g. turn-taking, energy-level
      • Location where interactions take place – e.g. office, lobby etc.
      • Daily schedule of individuals – e.g. early birds, late nighters
  • 28. Locations
    • Wi-Fi signal strength can be used to determine the approximate location of each speech event
      • 5 meter accuracy
      • Location computation done off-line
    • Raw locations are converted to nodes in a coarse topological map before further analysis
  • 29. Topological Location Map
    • Nodes in map are identified by area types
      • Hallway
      • Breakout area
      • Meeting room
      • Faculty office
      • Student office
    • Detected conversations are associated with their area type
  • 30. Social Network Model
    • Nodes
      • Subjects (wearing sensors, have given consent)
      • Public places (e.g., particular break out area)
      • Regions of private locations (e.g., hallway of faculty offices)
      • Instances of conversations
    • Edges
      • Between subjects and conversations
      • Between places or regions and conversations
  • 31. Non-instrumented Subjects
    • We may recruit additional subjects who do not wear sensors
    • Such subjects would allow us to infer information about their behavior indirectly, and to appear (coded) as a node in our network model
      • E.g., based on their particular office locations
    • Only people who have provided written consent appear as entities in our network models
  • 32. Disabling Sensor Units
    • As a courtesy, subjects will disable their units in particular classrooms or offices
  • 33. Access to the Data
    • Publications about this project will include summary statistics about the social network, e.g.:
      • Clustering coefficient
      • Motifs (temporal patterns)
    • We will not release the actual graph
      • This is prohibited by our HSD approval
    • We welcome additional collaborators