How Flickr Helps us Make Sense of the World

Slide 1: Ho w Flickr He lps us Make Se nse o f the Wo rld: Co nte xt and Co nte nt in Co mmunity- Co ntribute d Me dia Co lle ctio ns Lyndon Kennedy Mor Naaman* Share Ahern Rahul Nair Tye Rattenbury Yahoo! Research Berkeley Yahoo! Advanced Development Research *me

Slide 2: Co m m unity-co ntribute d? • Media • Descriptive text (title, caption, tag) • Discussions and comments • Views and view patterns • Item use and feedback • Reuse and remix • Micro- and explicit recommendations • “Context Metadata” •… Mo r Naam an - Ho w Flickr He lps us... 2 | Y!RB, Y!ADD 2007

Slide 3: Re se arch Challe nge s • Content is still hard… • Unstructured data (no semantics) • Noise • Scale – Computation – Long tail implies no supervised learning • Bias/feedback/Spam Mo r Naam an - Ho w Flickr He lps us... 3 | Y!RB, Y!ADD 2007

Slide 4: Fo re m o st Challe nge : • What’s the user problem? – Navigation/exploration – Recommendation – New application – Other? • Grounded in real needs • What impact on the community? “So cial Me dia Cycle ” Mo r Naam an - Ho w Flickr He lps us... 4 | Y!RB, Y!ADD 2007

Slide 5: In Particular… • No tigers, beaches and sunsets. Please. Mo r Naam an - Ho w Flickr He lps us... 5 | Y!RB, Y!ADD 2007

Slide 6: Flickr Tige rs Mo r Naam an - Ho w Flickr He lps us... 6 | Y!RB, Y!ADD 2007

Slide 7: Go o d ne w s! Patte rns That Make Se nse : • Semantic space • Activity and viewing data • User/personal data • Social network • And, location/time: Mo r Naam an - Ho w Flickr He lps us... 7 | Y!RB, Y!ADD 2007

Slide 8: Data De scriptio n Mo r Naam an - Ho w Flickr He lps us... 8 | Y!RB, Y!ADD 2007

Slide 9: That No ise …. • Noisy data • Photographer biases • Wrong data 5 kms 6 km s Mo r Naam an - Ho w Flickr He lps us... 9 | Y!RB, Y!ADD 2007

Slide 10: Tag Patte rns Mo r Naam an - Ho w Flickr He lps us... 10 | Y!RB, Y!ADD 2007

Slide 13: Expe rim e nts We can derive tag semantics using location and time metadata. Also see [Rattenbury et al, SIGIR 2007] byobw Mo r Naam an - Ho w Flickr He lps us... 13 | Y!RB, Y!ADD 2007

Slide 14: Can We Cre ate Use ful Applicatio ns? Flickr “ge o tagge d” in San Francisco Mo r Naam an - Ho w Flickr He lps us... 14 | Y!RB, Y!ADD 2007

Slide 15: Intuitio n Mo re “activity” in a ce rtain lo catio n indicate s im po rtance o f that lo catio n Tag that are unique to a ce rtain lo catio n can re pre se nt the lo catio n be tte r Mo r Naam an - Ho w Flickr He lps us... 15 | Y!RB, Y!ADD 2007

Slide 16: Tag Maps - SF Mo r Naam an - Ho w Flickr He lps us... 16 | Y!RB, Y!ADD 2007

Slide 17: Make a Wo rld Explo re r http:/ / tagm aps.re se arch.yaho o .co m Also see [Ahern et al., JCDL 2007] Mo r Naam an - Ho w Flickr He lps us... 17 | Y!RB, Y!ADD 2007

Slide 18: Ro lling in Co nte nt • So far, we leveraged metadata patterns to find – What are the geo-driven features – Where people take photos of these features • Can we utilized content analysis? • Hmmm…. Mo r Naam an - Ho w Flickr He lps us... 18 | Y!RB, Y!ADD 2007

Slide 19: Handling scale • Reduce computation requirements – Filter using metadata • Unsupervised methods – Effective for long tail without training Mo r Naam an - Ho w Flickr He lps us... 19 | Y!RB, Y!ADD 2007

Slide 20: Pro ble m : Be tte r Visual Sum m arie s Locations and Raw Data Visual Summary? Landmarks Mo r Naam an - Ho w Flickr He lps us... 20 | Y!RB, Y!ADD 2007

Slide 21: The Pro ble m , in Sho rt Find le ss o f and m o re o f this… this… …w itho ut e xplicitly kno w ing the diffe re nce . Mo r Naam an - Ho w Flickr He lps us... 21 | Y!RB, Y!ADD 2007

Slide 22: Lo catio n can he lp Enough visual similarity for learning? Mo r Naam an - Ho w Flickr He lps us... 22 | Y!RB, Y!ADD 2007

Slide 23: Finding Re pre se ntative Pho to s Mo r Naam an - Ho w Flickr He lps us... 23 | Y!RB, Y!ADD 2007

Slide 24: Visual Fe ature s • Co lo r: moments over a 5x5 grid • Te xture : Gabor over global image • Inte re st po ints: SIFT Mo r Naam an - Ho w Flickr He lps us... 24 | Y!RB, Y!ADD 2007

Slide 25: Ranking im age s: po int-w ise links Form links between images via matching SIFT points. Rank by degree of connectivity. Mo r Naam an - Ho w Flickr He lps us... 25 | Y!RB, Y!ADD 2007

Slide 26: Landm ark Graph Structure Less connected More connected Mo r Naam an - Ho w Flickr He lps us... 26 | Y!RB, Y!ADD 2007

Slide 27: Re sults: Palace o f Fine Arts X X X XX X X Tags-only Tags+Location Tags+Location+Visual Mo r Naam an - Ho w Flickr He lps us... 27 | Y!RB, Y!ADD 2007

Slide 28: Initial Evaluatio n • Select 10 landmarks to evaluate • Identify landmarks region(s) of relevance • Apply visual approach to discover representative images • Evaluate using Precision @ 10 Mo r Naam an - Ho w Flickr He lps us... 28 | Y!RB, Y!ADD 2007

Slide 29: Pe rfo rm ance +45% from visual +30% from location Average Mo r Naam an - Ho w Flickr He lps us... 29 | Y!RB, Y!ADD 2007

Slide 30: Evaluatio n Issue s • Degrees of “Representativeness” Mo r Naam an - Ho w Flickr He lps us... 30 | Y!RB, Y!ADD 2007

Slide 31: Evaluatio n Issue s • Diversity of Results Mo r Naam an - Ho w Flickr He lps us... 31 | Y!RB, Y!ADD 2007

Slide 32: Co nclusio ns • Noise can be handled (sometimes) • Can generate some structure from the unstructured • Content can help with the right tasks • Bias and Spam? Mo r Naam an - Ho w Flickr He lps us... 32 | Y!RB, Y!ADD 2007

Slide 33: Thanks With: Lyndon Kennedy, Shane Ahern, Rahul Nair, Tye Rattenbury Jeannie Yang, Nathan Good, Simon King In the papers: MIR06, JCDL07, SIGIR07 Have a Nokia phone? Check out ZoneT and Zurfer ag Read more, follow: http:/www.whyrb.com / Slides: http:/slideshare.net/ / mor MN or aaman: mor@ yahoo-inc.com Mo r Naam an - Ho w Flickr He lps us... 33 | Y!RB, Y!ADD 2007

Slide 34: APIs fo r all! • Everything we can do, you can do (better). APIs include: – Cell Tower ID database – Suggested Tags – TagMaps data – TagMaps Widget – ZoneTag RSS feeds, Action Tags http://developer.yahoo.com/yrb/ Mo r Naam an - Ho w Flickr He lps us... 34 | Y!RB, Y!ADD 2007

Slide 35: Tag Maps - Paris Mo r Naam an - Ho w Flickr He lps us... 35 | Y!RB, Y!ADD 2007

Slideshare.net (beta)