How Flickr Helps us Make Sense of the World: Context and Content in Community-Contributed Media Collections Lyndon Kennedy...
Community-contributed? <ul><li>Media  </li></ul><ul><li>Descriptive text (title, caption, tag) </li></ul><ul><li>Discussio...
Research Challenges <ul><li>Content is still hard… </li></ul><ul><li>Unstructured data (no semantics) </li></ul><ul><li>No...
Foremost Challenge: <ul><li>What’s the user problem? </li></ul><ul><ul><li>Navigation/exploration </li></ul></ul><ul><ul><...
In Particular… <ul><li>No tigers, beaches and sunsets. Please. </li></ul>
Flickr Tigers
Good news! Patterns That Make Sense: <ul><li>Semantic space </li></ul><ul><li>Activity and viewing data </li></ul><ul><li>...
Data Description
That Noise…. <ul><li>Noisy data </li></ul><ul><li>Photographer biases </li></ul><ul><li>Wrong data </li></ul>6 kms 5 kms
Tag Patterns
Tag Patterns
Tag Patterns
Experiments byobw We can derive tag semantics using location and time metadata. Also see [Rattenbury et al, SIGIR 2007]
Can We Create Useful Applications? <ul><ul><li>Flickr “geotagged” in San Francisco </li></ul></ul>
Intuition <ul><ul><li>More “activity” in a certain location indicates importance of that location </li></ul></ul><ul><ul><...
Tag Maps - SF
Make a World Explorer http://tagmaps.research.yahoo.com Also see [Ahern et al., JCDL 2007]
Rolling in Content <ul><li>So far, we leveraged metadata patterns to find </li></ul><ul><ul><li>What are the geo-driven fe...
Handling scale <ul><li>Reduce computation requirements </li></ul><ul><ul><li>Filter using metadata </li></ul></ul><ul><li>...
Problem: Better Visual Summaries Locations and Landmarks Raw Data Visual Summary?
The Problem, in Short and more of this… … without explicitly knowing the difference. Find less of this…
Location can help Enough visual similarity for learning?
Finding Representative Photos
Visual Features <ul><li>Color : moments over a 5x5 grid </li></ul><ul><li>Texture : Gabor over global image </li></ul><ul>...
Ranking images: point-wise links Form links between images via matching SIFT points. Rank by degree of connectivity.
Landmark Graph Structure More connected Less connected
Results: Palace of Fine Arts Tags-only Tags+Location Tags+Location+Visual X X X X X X X
Initial Evaluation <ul><li>Select 10 landmarks to evaluate </li></ul><ul><li>Identify landmarks region(s) of relevance </l...
Performance Average +45% from visual +30% from location
Evaluation Issues <ul><li>Degrees of “Representativeness” </li></ul>
Evaluation Issues <ul><li>Diversity of Results </li></ul>
Conclusions <ul><li>Noise can be handled (sometimes) </li></ul><ul><li>Can generate some structure from the unstructured <...
Thanks <ul><li>With:  Lyndon Kennedy, Shane Ahern, Rahul Nair, Tye Rattenbury </li></ul><ul><li>Jeannie Yang, Nathan Good,...
APIs for all! <ul><li>Everything we can do, you can do (better). APIs include: </li></ul><ul><ul><li>Cell Tower ID databas...
Tag Maps - Paris
Upcoming SlideShare
Loading in...5
×

How Flickr Helps us Make Sense of the World

3,606

Published on

ACM Multimedia 2007 presentation: How Flickr Helps us Make Sense of the World: Context and Content in Community-Contributed Media Collections.

Published in: Sports, Technology
0 Comments
14 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,606
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
14
Embeds 0
No embeds

No notes for slide
  • Lyndon is phd intern from Columbia Tye phd intern from Berkeley But really, I will talk about some research challenges and potential in community-generated collections of media, and how we begin to address those.
  • Transcript of "How Flickr Helps us Make Sense of the World"

    1. 1. How Flickr Helps us Make Sense of the World: Context and Content in Community-Contributed Media Collections Lyndon Kennedy Mor Naaman* Share Ahern Rahul Nair Tye Rattenbury Yahoo! Research Berkeley Yahoo! Advanced Development Research *me
    2. 2. Community-contributed? <ul><li>Media </li></ul><ul><li>Descriptive text (title, caption, tag) </li></ul><ul><li>Discussions and comments </li></ul><ul><li>Views and view patterns </li></ul><ul><li>Item use and feedback </li></ul><ul><li>Reuse and remix </li></ul><ul><li>Micro- and explicit recommendations </li></ul><ul><li>“ Context Metadata” </li></ul><ul><li>… </li></ul>
    3. 3. Research Challenges <ul><li>Content is still hard… </li></ul><ul><li>Unstructured data (no semantics) </li></ul><ul><li>Noise </li></ul><ul><li>Scale </li></ul><ul><ul><li>Computation </li></ul></ul><ul><ul><li>Long tail implies no supervised learning </li></ul></ul><ul><li>Bias/feedback/Spam </li></ul>
    4. 4. Foremost Challenge: <ul><li>What’s the user problem? </li></ul><ul><ul><li>Navigation/exploration </li></ul></ul><ul><ul><li>Recommendation </li></ul></ul><ul><ul><li>New application </li></ul></ul><ul><ul><li>Other? </li></ul></ul><ul><li>Grounded in real needs </li></ul><ul><li>What impact on the </li></ul><ul><li>community? </li></ul>“ Social Media Cycle”
    5. 5. In Particular… <ul><li>No tigers, beaches and sunsets. Please. </li></ul>
    6. 6. Flickr Tigers
    7. 7. Good news! Patterns That Make Sense: <ul><li>Semantic space </li></ul><ul><li>Activity and viewing data </li></ul><ul><li>User/personal data </li></ul><ul><li>Social network </li></ul><ul><li>And, location/time: </li></ul>
    8. 8. Data Description
    9. 9. That Noise…. <ul><li>Noisy data </li></ul><ul><li>Photographer biases </li></ul><ul><li>Wrong data </li></ul>6 kms 5 kms
    10. 10. Tag Patterns
    11. 11. Tag Patterns
    12. 12. Tag Patterns
    13. 13. Experiments byobw We can derive tag semantics using location and time metadata. Also see [Rattenbury et al, SIGIR 2007]
    14. 14. Can We Create Useful Applications? <ul><ul><li>Flickr “geotagged” in San Francisco </li></ul></ul>
    15. 15. Intuition <ul><ul><li>More “activity” in a certain location indicates importance of that location </li></ul></ul><ul><ul><li>Tag that are unique to a certain location can represent the location better </li></ul></ul>
    16. 16. Tag Maps - SF
    17. 17. Make a World Explorer http://tagmaps.research.yahoo.com Also see [Ahern et al., JCDL 2007]
    18. 18. Rolling in Content <ul><li>So far, we leveraged metadata patterns to find </li></ul><ul><ul><li>What are the geo-driven features </li></ul></ul><ul><ul><li>Where people take photos of these features </li></ul></ul><ul><li>Can we utilized content analysis? </li></ul><ul><li>Hmmm…. </li></ul>
    19. 19. Handling scale <ul><li>Reduce computation requirements </li></ul><ul><ul><li>Filter using metadata </li></ul></ul><ul><li>Unsupervised methods </li></ul><ul><ul><li>Effective for long tail without training </li></ul></ul>
    20. 20. Problem: Better Visual Summaries Locations and Landmarks Raw Data Visual Summary?
    21. 21. The Problem, in Short and more of this… … without explicitly knowing the difference. Find less of this…
    22. 22. Location can help Enough visual similarity for learning?
    23. 23. Finding Representative Photos
    24. 24. Visual Features <ul><li>Color : moments over a 5x5 grid </li></ul><ul><li>Texture : Gabor over global image </li></ul><ul><li>Interest points : SIFT </li></ul>
    25. 25. Ranking images: point-wise links Form links between images via matching SIFT points. Rank by degree of connectivity.
    26. 26. Landmark Graph Structure More connected Less connected
    27. 27. Results: Palace of Fine Arts Tags-only Tags+Location Tags+Location+Visual X X X X X X X
    28. 28. Initial Evaluation <ul><li>Select 10 landmarks to evaluate </li></ul><ul><li>Identify landmarks region(s) of relevance </li></ul><ul><li>Apply visual approach to discover representative images </li></ul><ul><li>Evaluate using Precision @ 10 </li></ul>
    29. 29. Performance Average +45% from visual +30% from location
    30. 30. Evaluation Issues <ul><li>Degrees of “Representativeness” </li></ul>
    31. 31. Evaluation Issues <ul><li>Diversity of Results </li></ul>
    32. 32. Conclusions <ul><li>Noise can be handled (sometimes) </li></ul><ul><li>Can generate some structure from the unstructured </li></ul><ul><li>Content can help with the right tasks </li></ul><ul><li>Bias and Spam? </li></ul>
    33. 33. Thanks <ul><li>With: Lyndon Kennedy, Shane Ahern, Rahul Nair, Tye Rattenbury </li></ul><ul><li>Jeannie Yang, Nathan Good, Simon King </li></ul><ul><li>In the papers: MIR06, JCDL07, SIGIR07 </li></ul><ul><li>Have a Nokia phone? Check out ZoneTag and Zurfer </li></ul><ul><li>Read more, follow: http://www.whyrb.com </li></ul><ul><li>Slides: http://slideshare.net/mor </li></ul><ul><li>Mor Naaman: mor@yahoo-inc.com </li></ul>
    34. 34. APIs for all! <ul><li>Everything we can do, you can do (better). APIs include: </li></ul><ul><ul><li>Cell Tower ID database </li></ul></ul><ul><ul><li>Suggested Tags </li></ul></ul><ul><ul><li>TagMaps data </li></ul></ul><ul><ul><li>TagMaps Widget </li></ul></ul><ul><ul><li>ZoneTag RSS feeds, Action Tags </li></ul></ul>http://developer.yahoo.com/yrb/
    35. 35. Tag Maps - Paris

    ×