How Flickr Helps us Make Sense of the World: Context and Content in Community-Contributed Media Collections Lyndon Kennedy Mor Naaman* Share Ahern Rahul Nair Tye Rattenbury Yahoo! Research Berkeley Yahoo! Advanced Development Research *me

Community-contributed? Media Descriptive text (title, caption, tag) Discussions and comments Views and view patterns Item use and feedback Reuse and remix Micro- and explicit recommendations “ Context Metadata” …

Media

Descriptive text (title, caption, tag)

Discussions and comments

Views and view patterns

Item use and feedback

Reuse and remix

Micro- and explicit recommendations

“ Context Metadata”

…

Research Challenges Content is still hard… Unstructured data (no semantics) Noise Scale Computation Long tail implies no supervised learning Bias/feedback/Spam

Content is still hard…

Unstructured data (no semantics)

Noise

Scale

Computation

Long tail implies no supervised learning

Bias/feedback/Spam

Foremost Challenge: What’s the user problem? Navigation/exploration Recommendation New application Other? Grounded in real needs What impact on the community? “ Social Media Cycle”

What’s the user problem?

Navigation/exploration

Recommendation

New application

Other?

Grounded in real needs

What impact on the

community?

In Particular… No tigers, beaches and sunsets. Please.

No tigers, beaches and sunsets. Please.

Flickr Tigers

Good news! Patterns That Make Sense: Semantic space Activity and viewing data User/personal data Social network And, location/time:

Semantic space

Activity and viewing data

User/personal data

Social network

And, location/time:

Data Description

That Noise…. Noisy data Photographer biases Wrong data 6 kms 5 kms

Noisy data

Photographer biases

Wrong data

Tag Patterns

Tag Patterns

Tag Patterns

Experiments byobw We can derive tag semantics using location and time metadata. Also see [Rattenbury et al, SIGIR 2007]

Can We Create Useful Applications? Flickr “geotagged” in San Francisco

Flickr “geotagged” in San Francisco

Intuition More “activity” in a certain location indicates importance of that location Tag that are unique to a certain location can represent the location better

More “activity” in a certain location indicates importance of that location

Tag that are unique to a certain location can represent the location better

Tag Maps - SF

Make a World Explorer http://tagmaps.research.yahoo.com Also see [Ahern et al., JCDL 2007]

Rolling in Content So far, we leveraged metadata patterns to find What are the geo-driven features Where people take photos of these features Can we utilized content analysis? Hmmm….

So far, we leveraged metadata patterns to find

What are the geo-driven features

Where people take photos of these features

Can we utilized content analysis?

Hmmm….

Handling scale Reduce computation requirements Filter using metadata Unsupervised methods Effective for long tail without training

Reduce computation requirements

Filter using metadata

Unsupervised methods

Effective for long tail without training

Problem: Better Visual Summaries Locations and Landmarks Raw Data Visual Summary?

The Problem, in Short and more of this… … without explicitly knowing the difference. Find less of this…

Location can help Enough visual similarity for learning?

Finding Representative Photos

Visual Features Color : moments over a 5x5 grid Texture : Gabor over global image Interest points : SIFT

Color : moments over a 5x5 grid

Texture : Gabor over global image

Interest points : SIFT

Ranking images: point-wise links Form links between images via matching SIFT points. Rank by degree of connectivity.

Landmark Graph Structure More connected Less connected

Results: Palace of Fine Arts Tags-only Tags+Location Tags+Location+Visual X X X X X X X

Initial Evaluation Select 10 landmarks to evaluate Identify landmarks region(s) of relevance Apply visual approach to discover representative images Evaluate using Precision @ 10

Select 10 landmarks to evaluate

Identify landmarks region(s) of relevance

Apply visual approach to discover representative images

Evaluate using Precision @ 10

Performance Average +45% from visual +30% from location

Evaluation Issues Degrees of “Representativeness”

Degrees of “Representativeness”

Evaluation Issues Diversity of Results

Diversity of Results

Conclusions Noise can be handled (sometimes) Can generate some structure from the unstructured Content can help with the right tasks Bias and Spam?

Noise can be handled (sometimes)

Can generate some structure from the unstructured

Content can help with the right tasks

Bias and Spam?

Thanks With: Lyndon Kennedy, Shane Ahern, Rahul Nair, Tye Rattenbury Jeannie Yang, Nathan Good, Simon King In the papers: MIR06, JCDL07, SIGIR07 Have a Nokia phone? Check out ZoneTag and Zurfer Read more, follow: http://www.whyrb.com Slides: http://slideshare.net/mor Mor Naaman: mor@yahoo-inc.com

With: Lyndon Kennedy, Shane Ahern, Rahul Nair, Tye Rattenbury

Jeannie Yang, Nathan Good, Simon King

In the papers: MIR06, JCDL07, SIGIR07

Have a Nokia phone? Check out ZoneTag and Zurfer

Read more, follow: http://www.whyrb.com

Slides: http://slideshare.net/mor

Mor Naaman: mor@yahoo-inc.com

APIs for all! Everything we can do, you can do (better). APIs include: Cell Tower ID database Suggested Tags TagMaps data TagMaps Widget ZoneTag RSS feeds, Action Tags http://developer.yahoo.com/yrb/

Everything we can do, you can do (better). APIs include:

Cell Tower ID database

Suggested Tags

TagMaps data

TagMaps Widget

ZoneTag RSS feeds, Action Tags

Tag Maps - Paris