HCI and Smartphone Data at Scale
Upcoming SlideShare
Loading in...5
×
 

HCI and Smartphone Data at Scale

on

  • 265 views

 

Statistics

Views

Total Views
265
Views on SlideShare
265
Embed Views
0

Actions

Likes
0
Downloads
4
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Start out with a statement that probably won’t be controversial, which is that smartphones are pervasiveAbout 40% of all mobile phones sold today are smartphones, and the number is rapidly growingWhat’s also interestingare trends in how people use these smartphoneshttp://blog.sciencecreative.com/2011/03/16/the-authentic-online-marketer/http://www.generationalinsights.com/millennials-addicted-to-their-smartphones-some-suffer-nomophobia/In fact, Millennials don’t just sleep with their smartphones. 75% use them in bed before going to sleep and 90% check them again first thing in the morning.  Half use them while eating and third use them in the bathroom. A third check them every half hour. Another fifth check them every ten minutes. A quarter of them check them so frequently that they lose count.http://www.androidtapp.com/how-simple-is-your-smartphone-to-use-funny-videos/Pew Research CenterAround 83 percent of those 18- to 29-year-olds sleep with their cell phones within reach. http://persquaremile.com/category/suburbia/
  • Smartphones intimate part of our livesLocation,call logs,SMS,pics, moreCan capture human behavior atunprecedented fidelity and scale
  • http://www.flickr.com/photos/robby_van_moor/478725670/
  • We know these relationships, but computers have an overly simplified model of our relationships, usually just “friend”Can we do better?
  • Image adapted from Real Life Social Network, by Paul Adams
  • Picture of “Robin Sage”
  • to understand the dynamics, structure, and character of a city
  • If you just looked at the geography only, you might break things down as follows…
  • http://www-958.ibm.com/software/data/cognos/manyeyes/visualizations/b2794c5a60c611e18bfd000255111976/comments/b27c2c4060c611e18bfd000255111976
  • DARPAGoogleCMU CyLab
  • Intuitively, if we are co-located in a highly public place, it’s not a very strong signal
  • 2.8m location sightings of 489 volunteers in Pittsburgh
  • 2.8m location sightings of 489 volunteers in Pittsburgh
  • Livehoods useful for recommender systems, e.g. not recommending things across boundaries

HCI and Smartphone Data at Scale HCI and Smartphone Data at Scale Presentation Transcript

  • ©2009CarnegieMellonUniversity:1 HCI and Smartphone Data at Scale IBM Research July 29, 2013 Shah Amini Justin Cranshaw Afsaneh Doryab Jialiu Lin Jun-Ki Min Jason Wiese Jason Hong Norman Sadeh Joy Zhang John Zimmerman Computer Human Interaction: Mobility Privacy Security
  • ©2013CarnegieMellonUniversity:2 Smartphones are Pervasive • 50% penetration in the US mid-2012 • In 2013Q1, majority of phones sold worldwide • 50 billion apps downloaded on each of Apple and Android
  • ©2013CarnegieMellonUniversity:3 Smartphones are Intimate Mobile phones and millennials (Pew 2012): • 75% use in bed before going to sleep • 83% sleep with their mobile phones • 90% check first thing in the morning • Half use them while eating • A third use them in the bathroom (!) • A fifth check them every ten minutes
  • ©2013CarnegieMellonUniversity:4 Smartphone Data is Intimate Who we know (contact list) Who we call (call log) Who we text (sms log)
  • ©2013CarnegieMellonUniversity:5 Smartphone Data is Intimate Where we go (gps, foursquare) Photos (some geotagged) Sensors (accel, sound, light)
  • ©2013CarnegieMellonUniversity:6 The Opportunity • We are creating a worldwide sensor network with these smartphones • We can now capture and analyze human behavior at unprecedented fidelity and scale
  • ©2013CarnegieMellonUniversity:7 Three Threads of Research • Augmented Social Graph – Create richer computational models of our social relationships with others • Urban Analytics – Create viz and models of cities based on geotagged social media • CrowdScanning Apps – Crowdsourcing and other techniques to analyze privacy behaviors of apps
  • ©2013CarnegieMellonUniversity:8 Modeling Social Relationships • If you were in a jail in Mexico, which of the 500+ “friends” in your phone contact list and on Facebook would come and get you out?
  • ©2013CarnegieMellonUniversity:9 Modeling Social Relationships • Can we use smartphone data to build a richer augmented social graph? – models tie strength, group, role
  • ©2013CarnegieMellonUniversity:10 Why Better Models? • Secure invitations – Who is this person friending me? • Communication triage • Better info finding (weak ties) • Configuration of privacy policies – Tie strength strongly correlated with what personal info people willing to share (Wiese et al, Ubicomp 2011) • Early detection of depression – Less communication with strong ties, less mobility, lots of fast food, insomnia
  • ©2013CarnegieMellonUniversity:11 Ongoing Work: Sleep Data Sleep data (self-reported ground truth) Sensor data
  • ©2013CarnegieMellonUniversity:12 Using Call Log, SMS, Contacts
  • ©2013CarnegieMellonUniversity:13 Using Call Log, SMS, Contacts
  • ©2013CarnegieMellonUniversity:14 Using Call Log, SMS, Contacts
  • ©2013CarnegieMellonUniversity:15 User Study on Relationships • 40 Participants – 13 male and 27 female (age 19-50) – 55% student, 35% employed, 10% unemp • Data collection – Phone: Contact list, call & SMS logs – Facebook: Friend list from Facebook backup – Self-report for 70 contacts: Demographics, group, closeness (1 – 5 = feel very close)
  • ©2013CarnegieMellonUniversity:16 Life Facets • Can classify life facets {work, social, home} at 90.1% – If at least one comm. – Just contact list, call log, SMS log • Correlations Min et al, Mining Smartphone Data to Classify Life-Facets of Social Relationships, CSCW 2013
  • ©2013CarnegieMellonUniversity:17 Ongoing Work: Tie Strength • However, tie strength much harder to predict, 74.6% for {low, med, high} – We thought this would be easy… – Other modes of communication • Skype, IM, email, face-to-face – Stage of relationship / maintenance comm.
  • ©2013CarnegieMellonUniversity:18 Three Threads of Research • Augmented Social Graph – Create richer computational models of our social relationships with others • Urban Analytics – Create viz and models of cities based on geotagged social media • CrowdScanning Apps – Crowdsourcing and other techniques to analyze privacy behaviors of apps
  • ©2013CarnegieMellonUniversity:19 The Problem • Today’s methods for gathering data about cities are slow, expensive, and limited – Ex. Travel Behavioral Inventory for traffic flows every 10-20 years and 100s of people – US Census 2010 cost $13 billion – Quality of life surveys (sociology, city govts) go door-to-door and interview people • Some approaches today: – Call Data Records, but granularity – Deploy a custom app, but scale and utility
  • ©2013CarnegieMellonUniversity:20 The Vision: Urban Analytics • Goal: Use smartphones + social media + machine learning to offer new and useful insights about a city in a manner that is cheap, fast, and highly scalable
  • ©2013CarnegieMellonUniversity:21 Livehoods, Our First Urban Analytics Tool • The character of an urban area is defined not just by the types of places found there, but also by the people that make it part of their daily life Cranshaw et al, The Livehoods Project: Utilizing Social Media to Understand the Dynamics of a City, ICWSM 2012.
  • ©2013CarnegieMellonUniversity:22 What comes to mind when you picture your neighborhood?
  • ©2013CarnegieMellonUniversity:23 You’re probably not imagining this. The Image of a Neighborhood
  • ©2013CarnegieMellonUniversity:24 The Image of a Neighborhood What you’re imagining probably looks a lot more like this. Every citizen has had long associations with some part of his city, and his image is soaked in memories and meanings. ---Kevin Lynch, The Image of a City
  • ©2013CarnegieMellonUniversity:25 Kevin Lynch, 1960 Stanley Milgram, 1977 Studying Perceptions: Cognitive Maps
  • ©2013CarnegieMellonUniversity:26 Two Perspectives “Politically constructed” “Socially constructed” Neighborhoods have fixed borders defined by the city government. Neighborhoods are organic, cultural artifacts. Borders are blurry, imprecise, and may be different to different people.
  • ©2013CarnegieMellonUniversity:27 Two Perspectives “Socially constructed” Neighborhoods are organic, cultural artifacts. Borders are blurry, imprecise, and may be different to different people. Can we discover automated ways of identifying the “organic” boundaries of the city? Can we extract local cultural knowledge from social media? Can we build a collective cognitive map from data?
  • ©2013CarnegieMellonUniversity:28 Livehoods Data Source • Crawled 18m check-ins from foursquare – Claims 20m users – People who linked their foursquare accts to Twitter • Spectral clustering based on geographic and social proximity
  • ©2013CarnegieMellonUniversity:29 ☺ ☺ ☺ ☺ If you watch check-ins over time, you’ll notice that groups of like-minded people tend to stay in the same areas.
  • ©2013CarnegieMellonUniversity:30 We can aggregate these patterns to compute relationships between check- in venues.
  • ©2013CarnegieMellonUniversity:31 These relationships can then be used to identify natural borders in the urban landscape.
  • ©2013CarnegieMellonUniversity:32 Livehood 2 Livehood 1 We call the discovered clusters “Livehoods” reflecting their dynamic character.
  • ©2013CarnegieMellonUniversity:33 Try it out at livehoods.org
  • ©2013CarnegieMellonUniversity:34 Evaluation • Interviewed 27 locals – Residents, urban planners, businesses – Asked them to draw their mental maps of areas first – Then showed them our maps and solicited feedback
  • ©2013CarnegieMellonUniversity:35 South Side Pittsburgh
  • ©2013CarnegieMellonUniversity:36 South Side Pittsburgh
  • ©2013CarnegieMellonUniversity:37 South Side Pittsburgh Carson Street runs along the length of South Side, and is densely packed with bars, restaurants, tattoo parlors, and clothing and furniture shops. It is the most popular destination for nightlife.
  • ©2013CarnegieMellonUniversity:38 South Side Pittsburgh South Side Works is a recently built, mixed-use outdoor shopping mall, containing nationally branded apparel stores and restaurants, upscale condominiums, and corporate offices.
  • ©2013CarnegieMellonUniversity:39 South Side Pittsburgh There is an small, somewhat older strip- mall that contains the only super market (grocery) in South Side. It also has a liquor store, an auto-parts store, a furniture rental store and other small chain stores.
  • ©2013CarnegieMellonUniversity:40 South Side Pittsburgh The rest of South Side is predominantly residential, consisting of mostly smaller row houses.
  • ©2013CarnegieMellonUniversity:41 South Side Pittsburgh
  • ©2013CarnegieMellonUniversity:42 South Side Pittsburgh Livehoods Found in South Side LH8 LH9LH7 LH6 I’ll show evidence in support of the Livehoods clusters in South Side, and will describe the forces that people highlighted.
  • ©2013CarnegieMellonUniversity:43 South Side Pittsburgh Demographic Differences LH8 LH9LH7 LH6 LH8 vs LH9 “Ha! Yes! See, here is my division! Yay! Thank you algorithm! ... I definitely feel where the South Side Works, and all of that is, is a very different feel.”
  • ©2013CarnegieMellonUniversity:44 South Side Pittsburgh Architecture & Urban Design LH8 LH9LH7 LH6 LH7 vs LH8 “from an urban standpoint it is a lot tighter on the western part once you get west of 17th or 18th [LH7].”
  • ©2013CarnegieMellonUniversity:45 South Side Pittsburgh Safety LH8 LH9LH7 LH6 LH7 vs LH8 “Whenever I was living down on 15th Street [LH7] I had to worry about drunk people following me home, but on 23rd [LH8] I need to worry about people trying to mug you... so it’s different. It’s not something I had anticipated, but there is a distinct difference between the two areas of the South Side.”
  • ©2013CarnegieMellonUniversity:46 South Side Pittsburgh Demographic Differences LH8 LH9LH7 LH6 LH6 “There is this interesting mix of people there I don’t see walking around the neighborhood. I think they are coming to the Giant Eagle [grocery store] from lower income neighborhoods... I always assumed they came from up the hill.”
  • ©2013CarnegieMellonUniversity:47 South Side Pittsburgh “I always assumed they came from up the hill.”
  • ©2013CarnegieMellonUniversity:48 Bezerkeley, CA
  • ©2013CarnegieMellonUniversity:49 Other Potential Urban Analytics
  • ©2013CarnegieMellonUniversity:50 Three Threads of Research • Augmented Social Graph – Create richer computational models of our social relationships with others • Urban Analytics – Create viz and models of cities based on geotagged social media • CrowdScanning Apps – Crowdsourcing and other techniques to analyze privacy behaviors of apps
  • ©2013CarnegieMellonUniversity:51 Shares your location, gender, unique phone ID, phone# with advertisers Uploads your entire contact list to their server (including phone #s) What are your apps really doing?
  • ©2013CarnegieMellonUniversity:52 Many Smartphone Apps Have “Unusual” Permissions App Permissions Used Tiny Flashlight + LED Internet Access, phone# Backgrounds Contact List Dictionary Location Bible Quotes Location
  • ©2013CarnegieMellonUniversity:53 Android • What do these permissions mean? • Why does app need this permission? • When does it use these permissions?
  • ©2013CarnegieMellonUniversity:54 CrowdScanning Core Ideas • Idea 1: find the gap between what people expect an app to do and what it actually does Lin et al, Expectation and Purpose: Understanding User’s Mental Models of Mobile App Privacy thru Crowdsourcing. Ubicomp 2012.
  • ©2013CarnegieMellonUniversity:55 Nissan Maxima Gear Shift
  • ©2013CarnegieMellonUniversity:56 Privacy as Expectations • Apply this same idea of mental models for privacy – Compare what people expect an app to do vs what an app actually does – Emphasize the biggest gaps, misconceptions that many people had App Behavior (What an app actually does) User Expectations (What people think the app does)
  • ©2013CarnegieMellonUniversity:57 Crowdsourcing Privacy • Idea 2: use crowdsourcing to do this (crowdsource privacy) • Few people read privacy policies – We want to install the app – Reading policies not part of main task – Complexity of these policies (the pain!!!) – Clear cost (time) for unclear benefit • Crowdsourcing can mitigate these problems
  • ©2013CarnegieMellonUniversity:58 10% users were surprised this app wrote contents to their SD card. 25% users were surprised this app sent their approximate location to dictionary.com for searching nearby words. 85% users were surprised this app sent their phone’s unique ID to mobile ads providers. 0% users were surprised this app could control their audio settings. See all 90% users were surprised this app sent their precise location to mobile ads providers. 95% users were surprised this app sent their approximate location to mobile ads providers. 95% users were surprised this app sent their phone’s unique ID to mobile ads providers. 0% users were surprised this app can control camera flashlight.
  • ©2013CarnegieMellonUniversity:59 Our Study on App Privacy • Showed crowd workers screenshots and description of app (from Google Play) – 56 of top 100 Android Apps • Showed permissions one at a time – Only those related to privacy • Expectation Condition – Why they think the app uses permission – How comfortable they were with it • Purpose Condition – We gave an explanation (based on our analysis) – Asked how comfortable they were with it
  • ©2013CarnegieMellonUniversity:60 Results for Location Data (N=20 per app, Expectations Condition) App Comfort Level (-2 – 2) Maps 1.52 GasBuddy 1.47 Weather Channel 1.45 Foursquare 0.95 TuneIn Radio 0.60 Evernote 0.15 Angry Birds -0.70 Brightest Flashlight Free -1.15 Toss It -1.2
  • ©2013CarnegieMellonUniversity:61 Showing Purpose Lowers Concerns • All differences statistically significant • Big increases for dictionary, Shazam, Air Control Lite, and others (> 1.0) App Comfort w/ Purpose Comfort w/o Purpose Device ID 0.47 ( =0.30) -0.10 ( =0.41) Contact List 0.66 ( =0.22) 0.16 ( =0.54) Network Location 0.90 ( =0.53) 0.65 ( =0.55) GPS Location 0.72 ( =0.62) 0.35 ( =0.73)
  • ©2013CarnegieMellonUniversity:62 Ongoing Work • Scaling up analysis – 600k+ apps on Android market – Static & dynamic analysis + clustering to build models of apps • Ex. “Games that use location data” -1.3 • Gort tool
  • ©2013CarnegieMellonUniversity:63 Summary • Smartphones offer big opportunity to understand human behavior at unprecedented fidelity and scale • Augmented Social Graph • Urban Analytics • CrowdScanning
  • ©2013CarnegieMellonUniversity:64 Thanks! More info at cmuchimps.org or email jasonh@cs.cmu.edu Special thanks to: • Army Research Office • National Science Foundation • Alfred P. Sloan Foundation • DARPA • Google • CMU Cylab Join our community for researchers at: www.reddit.com/r/pervasivecomputing
  • ©2013CarnegieMellonUniversity:65
  • ©2013CarnegieMellonUniversity:66 66 Using features such as location entropy significantly improves performance over shallow features such as number of co-locations
  • ©2013CarnegieMellonUniversity:67 67
  • ©2013CarnegieMellonUniversity:68 Using Location Data to Infer Friendships • 2.8m location sightings of 489 users of Locaccino friend finder in Pittsburgh • Place entropy for inferring social quality of a place – #unique people seen in a place – 0.0002 x 0.0002 lat/lon grid, ~30m x 30m Cranshaw et al, Bridging the Gap Between Physical Location and Online Social Networks, Ubicomp 2010
  • ©2013CarnegieMellonUniversity:69 • Insert graph here • Describe entropy
  • ©2013CarnegieMellonUniversity:70 Inferring Friendships • 67 different machine learning features – Location diversity (and entropy) – Intensity and Duration – Specificity (TF-IDF) – Graph structure (overlap in friends) • 92% accuracy in predicting friend/not – Location entropy improves performance over shallow features like #co-locations
  • ©2013CarnegieMellonUniversity:71 Most Unexpected Uses (N=20 per app, Expectations Condition) • Found strong correlation between expectations & comfort level (r=0.91) Apps using Contact List Comfort Level (-2 – 2) Backgrounds HD Wallpaper -1.35 Pandora -0.70 GO Launcher EX -0.75
  • ©2013CarnegieMellonUniversity:72 • Insert graph here • Describe entropy Co-location data to infer friendship Using place entropy, accuracy of 92% Can also infer number of friends
  • ©2013CarnegieMellonUniversity:73 Topic Modeling (LDA)
  • ©2013CarnegieMellonUniversity:74 Brooklyn Queens Expressway