What’s in a place? Adventures with Location-Aware Media Rahul Nair Yahoo! Research Berkeley

Flickr “geotagged” 20+ million images Can we do better?

20+ million images

Talk Outline Extracting information from geo-tagged photos World Explorer: Visualization ZoneTag: Creation Zurfer: Consumption

Extracting information from geo-tagged photos

World Explorer: Visualization

ZoneTag: Creation

Zurfer: Consumption

Attraction Map of Paris Stanley Milgram, 1976. Psychological Maps of Paris

Stanley Milgram, 1976.

Psychological Maps of Paris

Attraction Map of Paris Y!RB, 2007.

Y!RB, 2007.

Location-driven Modeling Derive meaningful data about map regions E.g., representative tags, photos

Derive meaningful data about map regions

E.g., representative tags, photos

Data Description

Issues Sparse data set Photographer bias In location In tags Incorrect data

Sparse data set

Photographer bias

In location

In tags

Incorrect data

Heuristics Number of photographs denotes the “importance” of a location Users will use a common subset of tags to describe objects/locations Concentrated tag usage indicates descriptiveness

Number of photographs denotes the “importance” of a location

Users will use a common subset of tags to describe objects/locations

Concentrated tag usage indicates descriptiveness

Algorithm Clustering: k-Means, get set of k clusters “ Document” C is bag of all tags in cluster For each tag in C calculate: TF = |P(C,t)| IDF = |P(R)| / |P(R, t)| UF = |U(C,t)|/|U(C)|

Clustering: k-Means, get set of k clusters

“ Document” C is bag of all tags in cluster

For each tag in C calculate:

TF = |P(C,t)|

IDF = |P(R)| / |P(R, t)|

UF = |U(C,t)|/|U(C)|

Scoring Score (t) = TF * IDF * UF Threshold values 30+ photographs Minimum 3 users Score > 1 Final dataset: (tag, score, latitude, longitude)

Score (t) = TF * IDF * UF

Threshold values

30+ photographs

Minimum 3 users

Score > 1

Final dataset: (tag, score, latitude, longitude)

Talk Outline Mining information from geo-tagged photos World Explorer: Visualization ZoneTag: Creation Zurfer: Consumption

Mining information from geo-tagged photos

World Explorer: Visualization

ZoneTag: Creation

Zurfer: Consumption

DEMO

Precomputation Divide the world into equal sized non-overlapping tiles Compute and store the tags for each tile Repeat for different zoom levels

Divide the world into equal sized non-overlapping tiles

Compute and store the tags for each tile

Repeat for different zoom

levels

Retrieval Find the tile level closest in size to the request area Select the tiles that fully cover the request area Return the tags that fall within the request area

Find the tile level closest in size to the request area

Select the tiles that fully cover the request area

Return the tags that fall within the request area

User Study 10 subjects 6 female, 4 male Ages 20-60 Varying technical knowledge No geotagged photos of their own

10 subjects

6 female, 4 male

Ages 20-60

Varying technical knowledge

No geotagged photos of their own

Experiment tasks Vacation recap San Francisco tour Explore a new city

Vacation recap

San Francisco tour

Explore a new city

Recall Reminded the subject about locations “It brings out memories” “Oh my God! This place has the best restaurants” “We wanted to see the Polynesian Cultural Center"

Reminded the subject about locations

“It brings out memories”

“Oh my God! This place has the best restaurants”

“We wanted to see the Polynesian Cultural Center"

Discovery Participants discovered previously unknown locations and events “I’ve never heard of this festival” “There is car racing which I'd probably go see”

Participants discovered previously unknown locations and events

“I’ve never heard of this festival”

“There is car racing which I'd probably go see”

Needle & Haystack Excellent visualization of the Haystack Hard to find specific information “Where was Culver City again?” No way to search “I guess what I’m looking for are bull fighting pictures”

Excellent visualization of the Haystack

Hard to find specific information

“Where was Culver City again?”

No way to search

“I guess what I’m looking for are bull fighting pictures”

Other Responses Gets the “vibe” of a place Share with other people Tags did not always match the mental model of a location Wanted more tags Want more info about tags

Gets the “vibe” of a place

Share with other people

Tags did not always match the mental model of a location

Wanted more tags

Want more info about tags

Conclusions Extract meaningful aggregate information from georeferenced data Allows users to explore locations in a new way Users like using the overview but also want the ability to search

Extract meaningful aggregate information from georeferenced data

Allows users to explore locations in a new way

Users like using the overview but also want the ability to search

Future work Adding search capability Show photos in places with no tags Differentiate locations and events Apply to other types of georeferenced data

Adding search capability

Show photos in places with no tags

Differentiate locations and events

Apply to other types of georeferenced data

Talk Outline Mining information from geo-tagged photos World Explorer: Visualization ZoneTag: Creation Zurfer: Consumption

Mining information from geo-tagged photos

World Explorer: Visualization

ZoneTag: Creation

Zurfer: Consumption

Current Mobile Experience Difficult to share (or even save!) Hard to find No context No semantic information Current mobile experience?

Difficult to share (or even save!)

Hard to find

No context

No semantic information

Current Mobile Experience Difficult to share (or even save!) Hard to find No context No semantic information

Difficult to share (or even save!)

Hard to find

No context

No semantic information

ZoneTag Experience 2-click upload (same key!) Photo uploaded with location and time metadata

2-click upload (same key!)

Photo uploaded with location and time metadata

Where does location come from? Bluetooth GPS (when available) User-contributed cell tower mapping

Bluetooth GPS (when available)

User-contributed cell tower mapping

ZoneTag Experience Tagging made easy Tag/annotate your photos from the phone

Tagging made easy

Tag/annotate your photos from the phone

Where do tags come from? Tags I used in this context (`home’) Tags my friends used in this context Tags other people used in this context (‘Ricoh’, ‘California Research Center’) E.g., TagMaps data

Tags I used in this context (`home’)

Tags my friends used in this context

Tags other people used in this context (‘Ricoh’, ‘California Research Center’)

E.g., TagMaps data

Where do tags come from? Stuff around you: Yahoo! Local Upcoming.org Stuff from you (any RSS 2.0 feed): Calendar Favorite hangouts (Wayfaring, Plazes, Socialight)

Stuff around you:

Yahoo! Local

Upcoming.org

Stuff from you (any RSS 2.0 feed):

Calendar

Favorite hangouts (Wayfaring, Plazes, Socialight)

Example of Tagging

I’m Too Lazy (you’re not alone) Tagging is the means, not the goal Benefits even if you never tagged a single image (Bradley Horowitz, elatable.com) (?) (?) (?)

Tagging is the means, not the goal

Benefits even if you never tagged a single image

Talk Outline Mining information from geo-tagged photos World Explorer: Visualization ZoneTag: Creation Zurfer: Consumption

Mining information from geo-tagged photos

World Explorer: Visualization

ZoneTag: Creation

Zurfer: Consumption

Flickr “geotagged” 20+ million images Can we do mobile?

20+ million images

Strengths of Mobile Personal Easy to access Networked Context aware The ultimate photo wallet

Personal

Easy to access

Networked

Context aware

The ultimate photo wallet

Design Goals Engagement and Discovery Spatial Social Allow customization Complete access to your photos Search and Filtering capability

Engagement and Discovery

Spatial

Social

Allow customization

Complete access to your photos

Search and Filtering capability

Channel Metaphor Each row is a single channel Navigate using 4 way Left & right to browse channel (infinite scroll) Up & down to change channels

Each row is a single channel

Navigate using 4 way

Left & right to browse channel (infinite scroll)

Up & down to change channels

Detail view Enlarge photo Scroll through channel Photo details Add comments Add to favourites

Enlarge photo

Scroll through channel

Photo details

Add comments

Add to favourites

Nearby photos channel Photos from the users current location Local Highlights My Nearby

Photos from the users current location

Local Highlights

My Nearby

Social Channels Contacts photos Expanded view Recent Activity

Contacts photos

Expanded view

Recent Activity

My Stuff Channel My Photos By Location By Tag My Photo Wallet My Favourites

My Photos

By Location

By Tag

My Photo Wallet

My Favourites

Custom Channels Users can create channels to match their interests Tags Groups Location aware (dogs near me)

Users can create channels to match their interests

Tags

Groups

Location aware (dogs near me)

Search and Filter

Try it out http://zurfer.research.yahoo.com

http://zurfer.research.yahoo.com

Conclusions It is possible to extract information from georeferenced media Users like browsing the extracted data It can help users tag new media We hope it helps them browse on a mobile device

It is possible to extract information from georeferenced media

Users like browsing the extracted data

It can help users tag new media

We hope it helps them browse on a mobile device

Questions? Rahul Nair [email_address] http://tagmaps.research.yahoo.com http://zonetag.research.yahoo.com http://zurfer.research.yahoo.com