Generating Summaries and Visualization for Large Collections of Geo-referenced Photographs Alexander Jaffe*, Mor Naaman *, Tamir Tassa † , Marc Davis $ *Yahoo! Research Berkeley † Open University of Israel $ Yahoo! Research

Attraction Map of Paris Stanley Milgram, 1976. Psychological Maps of Paris

Stanley Milgram, 1976.

Psychological Maps of Paris

Attraction Map of London Jaffe et al, 2006.

Jaffe et al, 2006.

Information Overload? Flickr “geotagged”

Flickr “geotagged”

Overview Problem definition Intuition for solution Algorithm for summarization Visualizing the dataset Evaluation Demo?

Problem definition

Intuition for solution

Algorithm for summarization

Visualizing the dataset

Evaluation

Demo?

Problem Definition Dataset: ( photo_id , user_id, latitude, longitude) ( photo_id , tag ) Result: (photo_id, rank) Given all photos from a geographic region, find a “representative” summary set

Dataset:

( photo_id , user_id, latitude, longitude)

( photo_id , tag )

Result:

(photo_id, rank)

Given all photos from a geographic region, find a “representative” summary set

Issues to Tackle Noisy data Whatever, color, city, spectrum, santa barbara, california, usa, Lookatme, Herbert Bayer Chromatic Gate Photographer biases In locations In Tags Wrong data

Noisy data

Photographer biases

In locations

In Tags

Wrong data

Intuition More “activity” in a certain location indicates importance of that location Tag that are unique to a certain location can suggest importance of that location

More “activity” in a certain location indicates importance of that location

Tag that are unique to a certain location can suggest importance of that location

(Very) Simple Example

Algorithm Overview Hierarchical Clustering of the location data For each cluster, generate cluster score Recursively generate ordering of all photos in each cluster, based on subcluster score and ordering

Hierarchical Clustering of the location data

For each cluster, generate cluster score

Recursively generate ordering of all photos in each cluster, based on subcluster score and ordering

The Clustered Return of the (Very) Simple Example! 4, 6, 5 8,7 4,8,6,5,7 20 10

Generating a Summary A complete ranking is produced for all photos in the dataset An n -photo summary is simply the first n photos in this ranking.

A complete ranking is produced for all photos in the dataset

An n -photo summary is simply the first n photos in this ranking.

Generating Cluster Scores Main Factors: Number of photos Relevance (bias) factors “ Tag Distinguishability” “ Photographer Distinguishability”

Main Factors:

Number of photos

Relevance (bias) factors

“ Tag Distinguishability”

“ Photographer Distinguishability”

Tag Distinguishability A measure of uniqueness of concepts represented in the cluster (“document”) TF/IDF based Compute frequency of each tag (TF) Compute (inverse) frequency of tag in the rest of the dataset (IDF) Aggregate TF/IDF over all tags in cluster using L2 norm Or, if you like formulas: Read the damn paper!

A measure of uniqueness of concepts represented in the cluster (“document”)

TF/IDF based

Compute frequency of each tag (TF)

Compute (inverse) frequency of tag in the rest of the dataset (IDF)

Aggregate TF/IDF over all tags in cluster using L2 norm

Or, if you like formulas:

Summary of San Francisco Golden Gate Bridge TransAmerica AT&T Baseball Park Golden Gate Twin Peaks Golden Gate Bay Bridge Ocean Beach Chinatown

Progress Bar (almost done) Problem definition Intuition for solution Algorithm for summarization Visualizing the dataset Evaluation Demo?

Problem definition

Intuition for solution

Algorithm for summarization

Visualizing the dataset

Evaluation

Demo?

Tag Maps Observation: The algorithm identifies “representative” locations The algorithm identifies unique, important tags Can be used to visualize the dataset!

Observation:

The algorithm identifies “representative” locations

The algorithm identifies unique, important tags

Tag Maps

Tag Maps

Ok, how do we evaluate this? Direct human-evaluation of algorithmic results Evaluated Tag Maps with various weighting options Compared summaries to 3 base conditions Compared chosen locations to top 15 locations selected by humans (Milgram-style)

Direct human-evaluation of algorithmic results

Evaluated Tag Maps with various weighting options

Compared summaries to 3 base conditions

Compared chosen locations to top 15 locations selected by humans (Milgram-style)

Maybe we have time for a demo

Maybe we have time for Q’s http://zonetag.research.yahoo.com (applied in prototype cameraphone app) http://blog.yahooresearchberkeley.com (more on this and other topics) Become an intern, get involved: Email me. Mor Naaman [email_address]

http://zonetag.research.yahoo.com

(applied in prototype cameraphone app)

http://blog.yahooresearchberkeley.com

(more on this and other topics)

Become an intern, get involved:

Email me.

Mor Naaman

[email_address]