Columbia Talk: Landmark Search and Community-Contributed Multimedia

The Landmark Revolut ion: I mproving I mage Search and Explorat ion f or Locat ion- Driven Queries M or N aam an Y ahoo! R esearch B erkeley Y ahoo! A dvanced D evelopm ent D i si vi on

How Flickr Helps us Make Sense of t he World: Cont ext and Cont ent in Communit y- Cont ribut ed Media Collect ions M or N aam an Y ahoo! R esearch B erkeley Y ahoo! A dvanced D evelopm ent D i si vi on

Dat a Descript ion Lyndon Kennedy, Mor Naaman 3 | Y!ADD, 2007

Tag Pat t erns Lyndon Kennedy, Mor Naaman 4 | Y!ADD, 2007

Communit y- cont ribut ed: Bet t er Dat a? • M edi a • D escri ve text (ti e , capti , tag) pti tl on • Di scussions and com m ents • V i s and vi patterns ew ew • Item use and feedback • R euse and rem ix • M i - and expl ci recom m endati cro it ons • “ontext M etadata” C •… Lyndon Kennedy, Mor Naaman 10 | Y!ADD, 2007

Pat t erns That Make Sense • S em anti space c • A cti ty and vi i data vi ew ng • U ser/ personaldata • S ocialnetw ork • Locat ion/ t ime Lyndon Kennedy, Mor Naaman 11 | Y!ADD, 2007

Tag Pat t erns: Beyond Geo Lyndon Kennedy, Mor Naaman 12 | Y!ADD, 2007

Flickr Tigers Lyndon Kennedy, Mor Naaman 13 | Y!ADD, 2007

Older Tigers? • N o tigers, beaches and sunsets. ease . Pl Lyndon Kennedy, Mor Naaman 14 | Y!ADD, 2007

Research Challenges • C ontent i sti lhard … s l • U nstructured data (no sem anti ) cs • T ags, not ground truth labels – F al negati and posi ves se ve ti – If that even m eans anything • N oise • S cale – Com putation – Long tai m pl es no supervi li i sed learning • B i / feedback / S pam as Lyndon Kennedy, Mor Naaman 15 | Y!ADD, 2007

That Noise…. • N oi data sy • Photographer biases • W rong data 5 k ms 6 km s Lyndon Kennedy, Mor Naaman 16 | Y!ADD, 2007

Foremost Challenge: • W hat’s the user probl ? em – N avigati / expl on oration – R ecom m endation – N ew appl cati i on – O ther? • G rounded i realneeds n • W hat i pact on the m com m uni ? ty “Social Media Cycle” Lyndon Kennedy, Mor Naaman 17 | Y!ADD, 2007

Talk Out line • Visual ze i – Creati a W orl E xpl ng d orer • G enerate know ledge – E xtracti T ag S em anti ng cs • S earch – Landm ark search Lyndon Kennedy, Mor Naaman 18 | Y!ADD, 2007

Surely, we can do bet t er t han t his Flickr “geot agged” in San Francisco Lyndon Kennedy, Mor Naaman 19 | Y!ADD, 2007

Simple Model (phot o_ id, user_ id, t ime, lat it ude, longit ude) (phot o_ id, t ag) Lyndon Kennedy, Mor Naaman 20 | Y!ADD, 2007

I nt uit ion More “act ivit y” in a cert ain locat ion indicat es import ance of t hat locat ion Tag t hat are unique t o a cert ain locat ion can represent t he locat ion bet t er Lyndon Kennedy, Mor Naaman 21 | Y!ADD, 2007

Translat ion int o simple algorit hm • Clusteri of photos ng • S cori of tags ng – T F / ID F / U F Lyndon Kennedy, Mor Naaman 22 | Y!ADD, 2007

Tag Maps - SF Lyndon Kennedy, Mor Naaman 23 | Y!ADD, 2007

At t ract ion Maps of Paris S tanley M i gram , l 1976. ”Psychological Maps of Paris” Lyndon Kennedy, Mor Naaman 24 | Y!ADD, 2007

At t ract ion Maps of Paris Y !R B , 2006. ”Tag Maps: World Explorer” Lyndon Kennedy, Mor Naaman 25 | Y!ADD, 2007

Make a World Explorer ht t p: / / t agmaps. research. yahoo. com A l see [A hern et al J CD L 2007] ., so Lyndon Kennedy, Mor Naaman 26 | Y!ADD, 2007

Summary of San Francisco Golden Gat e Bridge TransAmerica AT&T Baseball Park Golden Gat e Twin Peaks Golden Gat e Ocean Beach Bay Bridge Chinat own Lyndon Kennedy, Mor Naaman 27 | Y!ADD, 2007

Tag Maps - Paris - Les Blogs? Lyndon Kennedy, Mor Naaman 28 | Y!ADD, 2007

Tag- based Modeling • D eri m eani ve ngfuldata about i vi ndi dualtags • B ased on the tag ’s m etadata patterns • E .g., Yahoo! Mission College, SIGIR 2007. Lyndon Kennedy, Mor Naaman 30 | Y!ADD, 2007

Ext ended Model (phot o_ id, user_ id, t ime, lat it ude, longit ude) (phot o_ id, t ag) (t ag, locat ion) (t ag, t ime) Lyndon Kennedy, Mor Naaman 31 | Y!ADD, 2007

Tag Semant ics • Im proved i age search through query sem anti m cs • A utom ati pl - and event-gazetteers c ace • A ssoci on of m i ng ti e / pl ati ssi m ace data based on tags •… Lyndon Kennedy, Mor Naaman 33 | Y!ADD, 2007

San Francisco Experiment s ~43 k photos ~800 tags San Francisco Dat aset : 42, 000 Phot os 800+ popular t ags Lyndon Kennedy, Mor Naaman 34 | Y!ADD, 2007

Experiment s Result s: BYOBW! We can derive t ag semant ics using locat ion and t ime met adat a. [Rat t enbury et al, SI GI R 2007] byobw Lyndon Kennedy, Mor Naaman 35 | Y!ADD, 2007

Rolling in Cont ent • S o far, w e leveraged m etadata patterns to find – W hat are the geo-driven features – W here peopl take photos of these features e • C an w e uti i l zed content anal s? ysi • Hmmm…. Lyndon Kennedy, Mor Naaman 37 | Y!ADD, 2007

Handling scale • R educe com putati requi on rem ents – F i ter usi m etadata l ng • U nsupervised m ethods – E ffecti for l ve ong tai i lw thout trai ng ni Lyndon Kennedy, Mor Naaman 38 | Y!ADD, 2007

Building Visual Summaries Raw Data Locations and Names Visual Summary? Lyndon Kennedy, Mor Naaman 39 | Y!ADD, 2007

The Problem, in Short Find less of and more of t his… t his… … hout explicit ly wit knowing t he dif f erence. Lyndon Kennedy, Mor Naaman 40 | Y!ADD, 2007

Locat ion can help E nough visual si i ari for m l ty earni ? l ng Lyndon Kennedy, Mor Naaman 41 | Y!ADD, 2007

Finding Represent at ive Phot os Lyndon Kennedy, Mor Naaman 42 | Y!ADD, 2007

Visual Feat ures • Color: m om ents over a 5 x 5 grid • Text ure: G abor over globali age m • I nt erest point s: S IF T Lyndon Kennedy, Mor Naaman 43 | Y!ADD, 2007

Learning f rom noisy labels Lyndon Kennedy, Mor Naaman 44 | Y!ADD, 2007

Clust ering • K -m eans over l -l ow evelfeatures (texture and col ) or • V ary val of K w i totalnum ber of photographs ue th (avg. cluster si ~ 20) ze Lyndon Kennedy, Mor Naaman 45 | Y!ADD, 2007

Ranking clust ers • N um ber of users – M ore users -> m ore shared interest • T em poralspread – Persistent over ti e -> m ore l kel to be locati , not event m iy on – Alternatel use m ethod descri y bed earl er i • Visualcoherence – M easure of diversi of vi ty sualcluster • Visualconnecti ty vi – M ore on thi l s ater… Lyndon Kennedy, Mor Naaman 46 | Y!ADD, 2007

Finding Represent at ive Phot os Lyndon Kennedy, Mor Naaman 47 | Y!ADD, 2007

Ranking images: low- level similarit y E ucl dean di i stance from cluster centroi i col dn or and texture space . Lyndon Kennedy, Mor Naaman 48 | Y!ADD, 2007

Ranking images: discriminat ive model S am pl pseudo- e negati ves from outside uster. of cl Learn S V M m odelover col / texture space . or R ank by distance from S V M m argi . n Lyndon Kennedy, Mor Naaman 49 | Y!ADD, 2007

Point - wise Linking Lyndon Kennedy, Mor Naaman 50 | Y!ADD, 2007

Ranking images: point - wise links F orm l nks betw een i i ages vi m atchi m a ng S IF T poi . nts R ank by degree of connecti ty. vi Lyndon Kennedy, Mor Naaman 51 | Y!ADD, 2007

Landmark Graph St ruct ure Less connected More connected Lyndon Kennedy, Mor Naaman 52 | Y!ADD, 2007

Coit Tower: Two Main Views Shots from Coit Tower Far or occluded shots Shots of Coit Tower Lyndon Kennedy, Mor Naaman 53 | Y!ADD, 2007

Ranking images: f usion • S el -si i ari : E ucl dean di f m l ty i stance from centroi i dn l -l ow evelfeature space . • Di m nati : di scri i ve stance from S V M deci on si boundary. • Poi -w i : degree of the photo nt se • Fusion: sum of scores, norm al zed vi si oi i a gm d function Lyndon Kennedy, Mor Naaman 54 | Y!ADD, 2007

Result s: Palace of Fine Art s X X X XX X X Tags-only Tags+Location Tags+Location+Visual Lyndon Kennedy, Mor Naaman 55 | Y!ADD, 2007

Evaluat ion • D ataset: geo-tagged B ay A rea photos from F l ckr i • S elect 10 landm arks to evaluate • A ppl al thm (and basel ne ) to di y gori i scover representati i ages ve m Lyndon Kennedy, Mor Naaman 56 | Y!ADD, 2007

Perf ormance: Precision +45% w/visual +30% w/location Lyndon Kennedy, Mor Naaman 57 | Y!ADD, 2007

More Result s: Golden Gat e Bridge X X X X XX XX X T ags-onl T ags+Locati T ags+Locati +V i y on on sual Lyndon Kennedy, Mor Naaman 58 | Y!ADD, 2007

Evaluat ion I ssues • Preci <> R epresentati se ve Lyndon Kennedy, Mor Naaman 59 | Y!ADD, 2007

Evaluat ion I ssues • Preci <> D i se verse Lyndon Kennedy, Mor Naaman 60 | Y!ADD, 2007

Perf ormance: Represent at ive Lyndon Kennedy, Mor Naaman 61 | Y!ADD, 2007

Image Search: Proposed interface Lyndon Kennedy, Mor Naaman 62 | Y!ADD, 2007

Conclusions • Locati i strong predi on s ctor of content • Landm arks and geo-rel ated queri can be i es denti ed fi • C om puter vi on can w ork . S om eti es. si m Lyndon Kennedy, Mor Naaman 63 | Y!ADD, 2007

API s f or all! • E verythi w e can do, you can do (better). A PIs ng i ude : ncl – Cel ow er ID database lT – S uggested T ags based on context – T agM aps data – T agM aps W idget http://developer.yahoo.com/yrb/ Lyndon Kennedy, Mor Naaman 64 | Y!ADD, 2007

Thanks With: L yndo K ennedy, S haneA hern, R ahul N air, T yeR attenbury, J eannieYang, N athan Good, S imon K ing. n In the papers: M IR 06, J CD L 07, S IG IR 07, M M 07 A l ask m e about: Z oneT ag , Z urfer, F i E agl so re e R ead more, follow: http://www.whyrb.com P ast talks slides: http://slideshare.net/mor M or N aaman Lyndon Kennedy, Mor Naaman 65 | Y!ADD, 2007

Columbia Talk: Landmark Search and Community-Contributed Multimedia

2 comments

Notes on slide 1

Favorites, Groups & Events