Your SlideShare is downloading. ×
The Landmark Revolut ion:
I mproving I mage Search and
                     Explorat ion
 f or Locat ion- Driven Queries

...
How Flickr Helps us Make Sense of t he
                                 World:
   Cont ext and Cont ent in Communit y-
   ...
Dat a Descript ion




                       Lyndon Kennedy, Mor Naaman
3 | Y!ADD, 2007
Tag Pat t erns




                   Lyndon Kennedy, Mor Naaman
4 | Y!ADD, 2007
Tag Pat t erns




                   Lyndon Kennedy, Mor Naaman
5 | Y!ADD, 2007
Tag Pat t erns




                   Lyndon Kennedy, Mor Naaman
6 | Y!ADD, 2007
Tag Pat t erns




                   Lyndon Kennedy, Mor Naaman
7 | Y!ADD, 2007
Tag Pat t erns




                   Lyndon Kennedy, Mor Naaman
8 | Y!ADD, 2007
Tag Pat t erns




                   Lyndon Kennedy, Mor Naaman
9 | Y!ADD, 2007
Communit y- cont ribut ed: Bet t er Dat a?

        • M edi a
        • D escri ve text (ti e , capti , tag)
             ...
Pat t erns That Make Sense

        • S em anti space
                   c
        • A cti ty and vi i data
              ...
Tag Pat t erns: Beyond Geo




                               Lyndon Kennedy, Mor Naaman
12 | Y!ADD, 2007
Flickr Tigers




                   Lyndon Kennedy, Mor Naaman
13 | Y!ADD, 2007
Older Tigers?

        • N o tigers, beaches
          and sunsets.
                ease .
              Pl




          ...
Research Challenges

        • C ontent i sti lhard …
                    s   l
        • U nstructured data (no sem anti ...
That Noise….

        • N oi data
              sy
        • Photographer biases
        • W rong data




               ...
Foremost Challenge:

        • W hat’s the user probl ?
                                em
                   – N avigati ...
Talk Out line

        • Visual ze
                i
                   – Creati a W orl E xpl
                           ...
Surely, we can do bet t er t han t his

                               Flickr
                               “geot agged” ...
Simple Model


                   (phot o_ id, user_ id, t ime,
                          lat it ude, longit ude)
        ...
I nt uit ion

           More “act ivit y” in a cert ain locat ion
           indicat es import ance of t hat locat ion

 ...
Translat ion int o simple algorit hm

        • Clusteri of photos
                   ng
        • S cori of tags
        ...
Tag Maps - SF




                   Lyndon Kennedy, Mor Naaman
23 | Y!ADD, 2007
At t ract ion Maps of Paris

                                  S tanley
                                  M i gram ,
     ...
At t ract ion Maps of Paris

                                  Y !R B , 2006.
                                  ”Tag Maps:...
Make a World Explorer




 ht t p: / / t agmaps. research. yahoo. com
 A l see [A hern et al J CD L 2007]
                ...
Summary of San Francisco

                   Golden Gat e Bridge   TransAmerica



                                       ...
Tag Maps - Paris - Les Blogs?




                                  Lyndon Kennedy, Mor Naaman
28 | Y!ADD, 2007
Talk Out line

        • Visual ze
                i
                   – Creati a W orl E xpl
                           ...
Tag- based Modeling

        • D eri m eani
               ve      ngfuldata about i vi
                                  ...
Ext ended Model


                   (phot o_ id, user_ id, t ime,
                          lat it ude, longit ude)
     ...
Tag Pat t erns




                   Lyndon Kennedy, Mor Naaman
32 | Y!ADD, 2007
Tag Semant ics

        • Im proved i age search through query sem anti
                     m                            ...
San Francisco Experiment s

  ~43 k photos
  ~800 tags




San Francisco Dat aset :
42, 000 Phot os
800+ popular t ags
   ...
Experiment s




     Result s: BYOBW!
     We can derive t ag semant ics using locat ion and t ime
     met adat a.
     ...
Talk Out line

        • Visual ze
                i
                   – Creati a W orl E xpl
                           ...
Rolling in Cont ent

        • S o far, w e leveraged m etadata patterns to find
                   – W hat are the geo-dr...
Handling scale

        • R educe com putati requi
                            on    rem ents
                   – F i ter...
Building Visual Summaries




                   Raw Data   Locations and Names
                                          ...
The Problem, in Short
  Find less of               and more of t his…
t his…




 … hout explicit ly
  wit
knowing t he di...
Locat ion can help




                       E nough visual
                       si i ari for
                         ...
Finding Represent at ive Phot os




                                     Lyndon Kennedy, Mor Naaman
42 | Y!ADD, 2007
Visual Feat ures

        • Color: m om ents over a 5 x 5 grid
        • Text ure: G abor over globali age
               ...
Learning f rom noisy labels




                                Lyndon Kennedy, Mor Naaman
44 | Y!ADD, 2007
Clust ering

        • K -m eans over l -l
                          ow evelfeatures
          (texture and col )
        ...
Ranking clust ers

        • N um ber of users
                   – M ore users -> m ore shared interest
        • T em po...
Finding Represent at ive Phot os




                                     Lyndon Kennedy, Mor Naaman
47 | Y!ADD, 2007
Ranking images: low- level similarit y




                              E ucl dean di
                                   ...
Ranking images: discriminat ive model

                             S am pl pseudo-
                                    e
...
Point - wise Linking




                         Lyndon Kennedy, Mor Naaman
50 | Y!ADD, 2007
Ranking images: point - wise links

                              F orm l nks betw een
                                   ...
Landmark Graph St ruct ure




                   Less
                   connected



              More
              co...
Coit Tower: Two Main Views

                       Shots from

                       Coit Tower

                    Far ...
Ranking images: f usion

        • S el -si i ari : E ucl dean di
              f m l ty          i       stance from cent...
Result s: Palace of Fine Art s




                     X   X
                   X
                   XX X
               ...
Evaluat ion

        • D ataset: geo-tagged B ay A rea photos from F l ckr
                                               ...
Perf ormance: Precision




                                         +45%
                                         w/visua...
More Result s: Golden Gat e Bridge


                   X
                                       X
                   X
  ...
Evaluat ion I ssues

        • Preci <> R epresentati
               se               ve




                             ...
Evaluat ion I ssues

        • Preci <> D i
               se     verse




                              Lyndon Kennedy, ...
Perf ormance: Represent at ive




                                   Lyndon Kennedy, Mor Naaman
61 | Y!ADD, 2007
Image Search:
 Proposed
 interface




                   Lyndon Kennedy, Mor Naaman
62 | Y!ADD, 2007
Conclusions

        • Locati i strong predi
                 on s              ctor of content
        • Landm arks and g...
API s f or all!

        • E verythi w e can do, you can do (better). A PIs
                   ng
          i ude :
      ...
Thanks

         With: L yndo K ennedy, S haneA hern, R ahul N air, T yeR attenbury, J eannieYang, N athan Good, S imon K ...
Upcoming SlideShare
Loading in...5
×

Columbia Talk: Landmark Search and Community-Contributed Multimedia

1,617

Published on

A talk at Columbia University, Nov 13th 2007.

Published in: Education, Technology
2 Comments
0 Likes
Statistics
Notes
  • Be the first to like this

No Downloads
Views
Total Views
1,617
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
2
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Columbia Talk: Landmark Search and Community-Contributed Multimedia"

  1. 1. The Landmark Revolut ion: I mproving I mage Search and Explorat ion f or Locat ion- Driven Queries M or N aam an Y ahoo! R esearch B erkeley Y ahoo! A dvanced D evelopm ent D i si vi on
  2. 2. How Flickr Helps us Make Sense of t he World: Cont ext and Cont ent in Communit y- Cont ribut ed Media Collect ions M or N aam an Y ahoo! R esearch B erkeley Y ahoo! A dvanced D evelopm ent D i si vi on
  3. 3. Dat a Descript ion Lyndon Kennedy, Mor Naaman 3 | Y!ADD, 2007
  4. 4. Tag Pat t erns Lyndon Kennedy, Mor Naaman 4 | Y!ADD, 2007
  5. 5. Tag Pat t erns Lyndon Kennedy, Mor Naaman 5 | Y!ADD, 2007
  6. 6. Tag Pat t erns Lyndon Kennedy, Mor Naaman 6 | Y!ADD, 2007
  7. 7. Tag Pat t erns Lyndon Kennedy, Mor Naaman 7 | Y!ADD, 2007
  8. 8. Tag Pat t erns Lyndon Kennedy, Mor Naaman 8 | Y!ADD, 2007
  9. 9. Tag Pat t erns Lyndon Kennedy, Mor Naaman 9 | Y!ADD, 2007
  10. 10. Communit y- cont ribut ed: Bet t er Dat a? • M edi a • D escri ve text (ti e , capti , tag) pti tl on • Di scussions and com m ents • V i s and vi patterns ew ew • Item use and feedback • R euse and rem ix • M i - and expl ci recom m endati cro it ons • “ontext M etadata” C •… Lyndon Kennedy, Mor Naaman 10 | Y!ADD, 2007
  11. 11. Pat t erns That Make Sense • S em anti space c • A cti ty and vi i data vi ew ng • U ser/ personaldata • S ocialnetw ork • Locat ion/ t ime Lyndon Kennedy, Mor Naaman 11 | Y!ADD, 2007
  12. 12. Tag Pat t erns: Beyond Geo Lyndon Kennedy, Mor Naaman 12 | Y!ADD, 2007
  13. 13. Flickr Tigers Lyndon Kennedy, Mor Naaman 13 | Y!ADD, 2007
  14. 14. Older Tigers? • N o tigers, beaches and sunsets. ease . Pl Lyndon Kennedy, Mor Naaman 14 | Y!ADD, 2007
  15. 15. Research Challenges • C ontent i sti lhard … s l • U nstructured data (no sem anti ) cs • T ags, not ground truth labels – F al negati and posi ves se ve ti – If that even m eans anything • N oise • S cale – Com putation – Long tai m pl es no supervi li i sed learning • B i / feedback / S pam as Lyndon Kennedy, Mor Naaman 15 | Y!ADD, 2007
  16. 16. That Noise…. • N oi data sy • Photographer biases • W rong data 5 k ms 6 km s Lyndon Kennedy, Mor Naaman 16 | Y!ADD, 2007
  17. 17. Foremost Challenge: • W hat’s the user probl ? em – N avigati / expl on oration – R ecom m endation – N ew appl cati i on – O ther? • G rounded i realneeds n • W hat i pact on the m com m uni ? ty “Social Media Cycle” Lyndon Kennedy, Mor Naaman 17 | Y!ADD, 2007
  18. 18. Talk Out line • Visual ze i – Creati a W orl E xpl ng d orer • G enerate know ledge – E xtracti T ag S em anti ng cs • S earch – Landm ark search Lyndon Kennedy, Mor Naaman 18 | Y!ADD, 2007
  19. 19. Surely, we can do bet t er t han t his Flickr “geot agged” in San Francisco Lyndon Kennedy, Mor Naaman 19 | Y!ADD, 2007
  20. 20. Simple Model (phot o_ id, user_ id, t ime, lat it ude, longit ude) (phot o_ id, t ag) Lyndon Kennedy, Mor Naaman 20 | Y!ADD, 2007
  21. 21. I nt uit ion More “act ivit y” in a cert ain locat ion indicat es import ance of t hat locat ion Tag t hat are unique t o a cert ain locat ion can represent t he locat ion bet t er Lyndon Kennedy, Mor Naaman 21 | Y!ADD, 2007
  22. 22. Translat ion int o simple algorit hm • Clusteri of photos ng • S cori of tags ng – T F / ID F / U F Lyndon Kennedy, Mor Naaman 22 | Y!ADD, 2007
  23. 23. Tag Maps - SF Lyndon Kennedy, Mor Naaman 23 | Y!ADD, 2007
  24. 24. At t ract ion Maps of Paris S tanley M i gram , l 1976. ”Psychological Maps of Paris” Lyndon Kennedy, Mor Naaman 24 | Y!ADD, 2007
  25. 25. At t ract ion Maps of Paris Y !R B , 2006. ”Tag Maps: World Explorer” Lyndon Kennedy, Mor Naaman 25 | Y!ADD, 2007
  26. 26. Make a World Explorer ht t p: / / t agmaps. research. yahoo. com A l see [A hern et al J CD L 2007] ., so Lyndon Kennedy, Mor Naaman 26 | Y!ADD, 2007
  27. 27. Summary of San Francisco Golden Gat e Bridge TransAmerica AT&T Baseball Park Golden Gat e Twin Peaks Golden Gat e Ocean Beach Bay Bridge Chinat own Lyndon Kennedy, Mor Naaman 27 | Y!ADD, 2007
  28. 28. Tag Maps - Paris - Les Blogs? Lyndon Kennedy, Mor Naaman 28 | Y!ADD, 2007
  29. 29. Talk Out line • Visual ze i – Creati a W orl E xpl ng d orer • G enerate know ledge – E xtracti T ag S em anti ng cs • S earch – Landm ark search Lyndon Kennedy, Mor Naaman 29 | Y!ADD, 2007
  30. 30. Tag- based Modeling • D eri m eani ve ngfuldata about i vi ndi dualtags • B ased on the tag ’s m etadata patterns • E .g., Yahoo! Mission College, SIGIR 2007. Lyndon Kennedy, Mor Naaman 30 | Y!ADD, 2007
  31. 31. Ext ended Model (phot o_ id, user_ id, t ime, lat it ude, longit ude) (phot o_ id, t ag) (t ag, locat ion) (t ag, t ime) Lyndon Kennedy, Mor Naaman 31 | Y!ADD, 2007
  32. 32. Tag Pat t erns Lyndon Kennedy, Mor Naaman 32 | Y!ADD, 2007
  33. 33. Tag Semant ics • Im proved i age search through query sem anti m cs • A utom ati pl - and event-gazetteers c ace • A ssoci on of m i ng ti e / pl ati ssi m ace data based on tags •… Lyndon Kennedy, Mor Naaman 33 | Y!ADD, 2007
  34. 34. San Francisco Experiment s ~43 k photos ~800 tags San Francisco Dat aset : 42, 000 Phot os 800+ popular t ags Lyndon Kennedy, Mor Naaman 34 | Y!ADD, 2007
  35. 35. Experiment s Result s: BYOBW! We can derive t ag semant ics using locat ion and t ime met adat a. [Rat t enbury et al, SI GI R 2007] byobw Lyndon Kennedy, Mor Naaman 35 | Y!ADD, 2007
  36. 36. Talk Out line • Visual ze i – Creati a W orl E xpl ng d orer • G enerate know ledge – E xtracti T ag S em anti ng cs • S earch – Landm ark search Lyndon Kennedy, Mor Naaman 36 | Y!ADD, 2007
  37. 37. Rolling in Cont ent • S o far, w e leveraged m etadata patterns to find – W hat are the geo-driven features – W here peopl take photos of these features e • C an w e uti i l zed content anal s? ysi • Hmmm…. Lyndon Kennedy, Mor Naaman 37 | Y!ADD, 2007
  38. 38. Handling scale • R educe com putati requi on rem ents – F i ter usi m etadata l ng • U nsupervised m ethods – E ffecti for l ve ong tai i lw thout trai ng ni Lyndon Kennedy, Mor Naaman 38 | Y!ADD, 2007
  39. 39. Building Visual Summaries Raw Data Locations and Names Visual Summary? Lyndon Kennedy, Mor Naaman 39 | Y!ADD, 2007
  40. 40. The Problem, in Short Find less of and more of t his… t his… … hout explicit ly wit knowing t he dif f erence. Lyndon Kennedy, Mor Naaman 40 | Y!ADD, 2007
  41. 41. Locat ion can help E nough visual si i ari for m l ty earni ? l ng Lyndon Kennedy, Mor Naaman 41 | Y!ADD, 2007
  42. 42. Finding Represent at ive Phot os Lyndon Kennedy, Mor Naaman 42 | Y!ADD, 2007
  43. 43. Visual Feat ures • Color: m om ents over a 5 x 5 grid • Text ure: G abor over globali age m • I nt erest point s: S IF T Lyndon Kennedy, Mor Naaman 43 | Y!ADD, 2007
  44. 44. Learning f rom noisy labels Lyndon Kennedy, Mor Naaman 44 | Y!ADD, 2007
  45. 45. Clust ering • K -m eans over l -l ow evelfeatures (texture and col ) or • V ary val of K w i totalnum ber of photographs ue th (avg. cluster si ~ 20) ze Lyndon Kennedy, Mor Naaman 45 | Y!ADD, 2007
  46. 46. Ranking clust ers • N um ber of users – M ore users -> m ore shared interest • T em poralspread – Persistent over ti e -> m ore l kel to be locati , not event m iy on – Alternatel use m ethod descri y bed earl er i • Visualcoherence – M easure of diversi of vi ty sualcluster • Visualconnecti ty vi – M ore on thi l s ater… Lyndon Kennedy, Mor Naaman 46 | Y!ADD, 2007
  47. 47. Finding Represent at ive Phot os Lyndon Kennedy, Mor Naaman 47 | Y!ADD, 2007
  48. 48. Ranking images: low- level similarit y E ucl dean di i stance from cluster centroi i col dn or and texture space . Lyndon Kennedy, Mor Naaman 48 | Y!ADD, 2007
  49. 49. Ranking images: discriminat ive model S am pl pseudo- e negati ves from outside uster. of cl Learn S V M m odelover col / texture space . or R ank by distance from S V M m argi . n Lyndon Kennedy, Mor Naaman 49 | Y!ADD, 2007
  50. 50. Point - wise Linking Lyndon Kennedy, Mor Naaman 50 | Y!ADD, 2007
  51. 51. Ranking images: point - wise links F orm l nks betw een i i ages vi m atchi m a ng S IF T poi . nts R ank by degree of connecti ty. vi Lyndon Kennedy, Mor Naaman 51 | Y!ADD, 2007
  52. 52. Landmark Graph St ruct ure Less connected More connected Lyndon Kennedy, Mor Naaman 52 | Y!ADD, 2007
  53. 53. Coit Tower: Two Main Views Shots from Coit Tower Far or occluded shots Shots of Coit Tower Lyndon Kennedy, Mor Naaman 53 | Y!ADD, 2007
  54. 54. Ranking images: f usion • S el -si i ari : E ucl dean di f m l ty i stance from centroi i dn l -l ow evelfeature space . • Di m nati : di scri i ve stance from S V M deci on si boundary. • Poi -w i : degree of the photo nt se • Fusion: sum of scores, norm al zed vi si oi i a gm d function Lyndon Kennedy, Mor Naaman 54 | Y!ADD, 2007
  55. 55. Result s: Palace of Fine Art s X X X XX X X Tags-only Tags+Location Tags+Location+Visual Lyndon Kennedy, Mor Naaman 55 | Y!ADD, 2007
  56. 56. Evaluat ion • D ataset: geo-tagged B ay A rea photos from F l ckr i • S elect 10 landm arks to evaluate • A ppl al thm (and basel ne ) to di y gori i scover representati i ages ve m Lyndon Kennedy, Mor Naaman 56 | Y!ADD, 2007
  57. 57. Perf ormance: Precision +45% w/visual +30% w/location Lyndon Kennedy, Mor Naaman 57 | Y!ADD, 2007
  58. 58. More Result s: Golden Gat e Bridge X X X X XX XX X T ags-onl T ags+Locati T ags+Locati +V i y on on sual Lyndon Kennedy, Mor Naaman 58 | Y!ADD, 2007
  59. 59. Evaluat ion I ssues • Preci <> R epresentati se ve Lyndon Kennedy, Mor Naaman 59 | Y!ADD, 2007
  60. 60. Evaluat ion I ssues • Preci <> D i se verse Lyndon Kennedy, Mor Naaman 60 | Y!ADD, 2007
  61. 61. Perf ormance: Represent at ive Lyndon Kennedy, Mor Naaman 61 | Y!ADD, 2007
  62. 62. Image Search: Proposed interface Lyndon Kennedy, Mor Naaman 62 | Y!ADD, 2007
  63. 63. Conclusions • Locati i strong predi on s ctor of content • Landm arks and geo-rel ated queri can be i es denti ed fi • C om puter vi on can w ork . S om eti es. si m Lyndon Kennedy, Mor Naaman 63 | Y!ADD, 2007
  64. 64. API s f or all! • E verythi w e can do, you can do (better). A PIs ng i ude : ncl – Cel ow er ID database lT – S uggested T ags based on context – T agM aps data – T agM aps W idget http://developer.yahoo.com/yrb/ Lyndon Kennedy, Mor Naaman 64 | Y!ADD, 2007
  65. 65. Thanks With: L yndo K ennedy, S haneA hern, R ahul N air, T yeR attenbury, J eannieYang, N athan Good, S imon K ing. n In the papers: M IR 06, J CD L 07, S IG IR 07, M M 07 A l ask m e about: Z oneT ag , Z urfer, F i E agl so re e R ead more, follow: http://www.whyrb.com P ast talks slides: http://slideshare.net/mor M or N aaman Lyndon Kennedy, Mor Naaman 65 | Y!ADD, 2007

×