Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

"Video Killed the Radio Star": From MTV to Snapchat

423 views

Published on

guest lecture at Knowledge & Media course 2016
VU University Amsterdam

Published in: Technology
  • Who Else Wants To Cure Their Acne, Regain Their Natural Inner Balance and Achieve LASTING Clear Skin? Click Here ▲▲▲ http://t.cn/AiWGkfAm
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

"Video Killed the Radio Star": From MTV to Snapchat

  1. 1. “Video Killed the Radio Star” the path from MTV to Snapchat Lora Aroyo http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  2. 2. The  CNN/YouTube  Republican  Debate  on  2007-­‐11-­‐28  
  3. 3. The  CNN/YouTube  Republican  Debate  on  2007-­‐11-­‐28  
  4. 4. The  CNN/YouTube  Republican  Debate  on  2007-­‐11-­‐28  
  5. 5. The  CNN/YouTube  Republican  Debate  on  2007-­‐11-­‐28  
  6. 6. The  CNN/YouTube  Republican  Debate  on  2007-­‐11-­‐28  
  7. 7. The  CNN/YouTube  Republican  Debate  on  2007-­‐11-­‐28  
  8. 8. h;p://www.blogherald.com/2010/10/27/history-­‐of-­‐online-­‐video/    
  9. 9. massive  amount  of  digital  content  to  explore  …   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  10. 10. but  at  some  point  it  all  looks  the  same  …   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  11. 11. Massive Scale: A lifetime of video content is uploaded to YouTube everyday. Granularity Mismatch: Searching for the relevant video fragments is still not possible. Passive Engagement: Video is still primarily a linear net-time viewing activity
  12. 12. … people search & browse with some implicit relevance in mind http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  13. 13. snapchat  genera8on  …   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  14. 14. audiences  feel  disconnected  &  lost  …   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  15. 15. there  is  huge  seman8c  &  cultural  GAP   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  16. 16. so=ware  systems  are  ever  more  intelligent   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo but  they  don’t  actually  understand  people  
  17. 17. focus  on  human  knowledge  in  machine-­‐readable  form   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo but  there  are  types  of  human  knowledge                           that  can’t  be  captured  by  machines  
  18. 18. classical  AI  involves  human  experts  to  manually   provide  training  knowledge  for  machines   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo human  expert-­‐based  ground  truth  does  not  scale     for  current  demand  for  machines  to  deal  with  wide   ranges  of  real-­‐world  tasks  and  contexts    
  19. 19.              we  need  to  be  able  to  ….                support  of  mulGple  perspecGves   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  20. 20. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo to  provide  an  approach  to  capturing  human  knowledge   in  a  way  that  is  scalable  &  adequate  to  real-­‐world  needs   the  key  scien8fic  challenge  is  
  21. 21. Goodbye Single Truth Hello Multiple Perspectives
  22. 22. humans  accurately  perform  interpreta8on  tasks   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  23. 23. humans  accurately  perform  interpreta8on  tasks   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo can  their  effort  be  adequately  harnessed  in  a   scien8fically  reliable  manner  that  scales  across  tasks,   contexts  &  data  modali8es?  
  24. 24. Quan8ty  is  the  new  Quality   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo Human  Computa8on  adopts  human  intelligence  at   scale  to  improve  purely  machine-­‐based  systems  
  25. 25. diversity  of  opinion   Independent   decentralized   aggregated     James  Surowiecki   “the  wise  crowd”   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  26. 26. a  novel  approach  to  gather  diversity  of  perspec8ves  &   opinions  from  the  crowd,  expand  expert  vocabularies  with   these  and  gather  new  type  of  gold  standard  for  machines     L.  Aroyo,  C.  Welty:  Crowd  Truth:  Harnessing  disagreement  in  crowdsourcing  a  rela?on  extrac?on  gold  standard.  ACM  WebSci  2013.   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo L.  Aroyo,  C.  Welty.  The  Three  Sides  of  CrowdTruth,  Journal  of  Human  Computa?on,  2014   http://CrowdTruth.org http://data.CrowdTruth.org/ http://game.crowdtruth.org
  27. 27. Visual  Content  Domina8on   •  90%  of  informa8on  transmiSed  to  the  brain  is  visual  (processed  60,000X  faster  in   the  brain  than  text)   •  Videos  increase  average  page  conversion  rates  by  86%   •  Visuals  are  social-­‐media-­‐ready/friendly  -­‐  easily  sharable     •  Posts  with  visuals  receive  94%  more  page  visits   •  Visuals  are  becoming  easier  and  easier  to  create  as  photo  /  video  ediGng  tools   become  more  accessible  
  28. 28. any piece of media can be the starting point to a world of compelling visual experiences. turning “mute” images into content-aware images.
  29. 29. NEW JERSEY HUDSON RIVER CENTRAL PARK URBANIZATION VERIZON METLIFE BUILDING SUNSET EAST RIVER NEW YORK CITY SKYSCRAPER UPPER EAST SIDE turning “mute” images into content-aware images. any piece of media can be the starting point to a world of compelling visual experiences.
  30. 30. combining machine processing with crowdsourcing for enriching, curating & gathering metadata quickly & cheaply — at scale. NEW JERSEY HUDSON RIVER CENTRAL PARK URBANIZATION VERIZON METLIFE BUILDING SUNSET EAST RIVER NEW YORK CITY SKYSCRAPER UPPER EAST SIDE
  31. 31. NEW JERSEY HUDSON RIVER CENTRAL PARK URBANIZATION VERIZON NEW YORK CITY SKYSCRAPER METLIFE BUILDING UPPER EAST SIDE EAST RIVER MIDTOWN MANHATTAN PAN-AM BUILDING PAN-AM AIRLINES HELICOPTER CRASH AIR TRAVEL ARCHITECTURE turning “context-free” images in relationship-aware images
  32. 32. NEW JERSEY HUDSON RIVER CENTRAL PARK URBANIZATION VERIZON NEW YORK CITY SKYSCRAPER METLIFE BUILDING UPPER EAST SIDE EAST RIVER MIDTOWN MANHATTAN PAN-AM BUILDING PAN-AM AIRLINES HELICOPTER CRASH AIR TRAVEL ARCHITECTURE … not only images, but also for videos YOUTUBE: NYC FROM THE EMPIRE STATE BUILDING allowing viewers to explore relationships across themes, locations, characters, etc. — within a video.
  33. 33. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo h;p://www.adweek.com/socialGmes/millennials-­‐love-­‐video-­‐on-­‐mobile-­‐social-­‐channels-­‐infographic/622313    
  34. 34. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  35. 35. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  36. 36. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  37. 37. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  38. 38. BRIDGING THE GAP BETWEEN PEOPLE & THE OVERWHELMING AMOUNT OF ONLINE MULTIMEDIA CONTENT
  39. 39. HyperVideos: Link video fragments in non-linear paths Binging Engagement: Construct continuous and interactive experiences Video Snacks: Break video down into snackable moments SOLUTIONS
  40. 40. •  Decomposing & granular description of images & videos. •  Constructing mediaGraph with rich media semantics. •  Continuously enriching & consolidating machine, expert, & user content descriptions. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  41. 41. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  42. 42. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  43. 43. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  44. 44. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  45. 45. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  46. 46. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  47. 47. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  48. 48. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  49. 49. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  50. 50. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  51. 51. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo Machines  &  Crowds  
  52. 52. http://waisda.nl Crowdsourcing  Video  Tags     @Sound  and  Vision  
  53. 53. @waisdahSp://waisda.nl  
  54. 54. Two  Pilots  
  55. 55. Results  of  First  Pilot  
  56. 56. – The  first  6  months:   •  44.362  pageviews   •  12.279  visits  (3+  min  online)   •  555  registered  players  (thousands  anonymous  players!)   – 340.551  tags  added  to  602  items   – 137.421  matches   Results  of  First  Pilot  
  57. 57. 11    PartcipaGng  Museums   1,782    Works  of  Art  in  the  Research     36,981  Tags  collected     2,017    Users  who  tagged     First  two  years  (2006-­‐2008)   Q: Why did you tag? 0% 20% 40% 60% 80% 100% don't remember to connect with others so that I could find works again later other (please specify) to learn about art to improve search for other users for fun to help museums document art work Public MMA
  58. 58. Tags  by  Documentalists   •  Tags  describe  mainly  short  segments   •  Tags  are  oaen  not  very  specific   •  Tags  not  describe  programmes  as  a  whole   •  User  tags  were  useful  &  specific  -­‐-­‐>  domain  dependent  
  59. 59. user vocabulary 8% in professional vocabulary 23% in Dutch lexicon 89% found on Google locations (7%) engeland persons (31%) objects (57%) On  the  Role  of  User-­‐Generated  Metadata  in  A/V  Collec?ons   Riste  Gligorov  et  al.  KCAP  Int.  Conference  on  Knowledge  Capture  2011   Crowd  vs.  Professionals  
  60. 60. System MAP All user tags 0.219 Consensus user tags only 0.143 NCRV tags 0.138 NCRV catalog 0.077 Captions 0.157 Captions + User tags 0.247 Captions + NCRV catalog 0.183 Captions + NCRV tags 0.201 NCRV tags + User tags 0.263 NCRV tags + NCRV catalog 0.150 All – User tags 0.208 All 0.276 All tags better than consensus only • Improvement of 53% • Consensus tags have • higher precision: 0.59 vs. 0.49 • but lower recall: 0.28 vs. 0.42 WAISDA?  Tags  vs.  Rest  
  61. 61. System MAP All user tags 0.219 Consensus user tags only 0.143 NCRV tags 0.138 NCRV catalog 0.077 Captions 0.157 Captions + User tags 0.247 Captions + NCRV catalog 0.183 Captions + NCRV tags 0.201 NCRV tags + User tags 0.263 NCRV tags + NCRV catalog 0.150 All – User tags 0.208 All 0.276 All tags better than rest • Individually • beat NCRV tags by 69% • beat captions by 39% WAISDA?  Tags  vs.  Rest  
  62. 62. System MAP All user tags 0.219 Consensus user tags only 0.143 NCRV tags 0.138 NCRV catalog 0.077 Captions 0.157 Captions + User tags 0.247 Captions + NCRV catalog 0.183 Captions + NCRV tags 0.201 NCRV tags + User tags 0.263 NCRV tags + NCRV catalog 0.150 All – User tags 0.208 All 0.276 All tags better than rest • Individually • beat NCRV tags by 69% • beat captions by 39% • Combined • Improvement of 5% WAISDA?  Tags  vs.  Rest  
  63. 63. System MAP All user tags 0.219 Consensus user tags only 0.143 NCRV tags 0.138 NCRV catalog 0.077 Captions 0.157 Captions + User tags 0.247 Captions + NCRV catalog 0.183 Captions + NCRV tags 0.201 NCRV tags + User tags 0.263 NCRV tags + NCRV catalog 0.150 All – User tags 0.208 All 0.276 All data performs best • largely due to contribution of user tags – 33% WAISDA?  Tags  vs.  Rest  
  64. 64. System MAP All user tags 0.219 Consensus user tags only 0.143 NCRV tags 0.138 NCRV catalog 0.077 Captions 0.157 Captions + User tags 0.247 Captions + NCRV catalog 0.183 Captions + NCRV tags 0.201 NCRV tags + User tags 0.263 NCRV tags + NCRV catalog 0.150 All – User tags 0.208 All 0.276 All tags better than consensus only • Improvement of 53% • Consensus tags have • higher precision: 0.59 vs. 0.49 • but lower recall: 0.28 vs. 0.42 All tags better than rest • Individually • beat NCRV tags by 69% • beat captions by 39% All data performs best • largely due to contribution of user tags – 33% • Combined • Improvement of 5% WAISDA?  Tags  vs.  Rest  
  65. 65. Current  Pilot   h;p://spotvogel.vroegevogels.vara.nl/  
  66. 66. Accurator ask the right crowd, enrich your collection hSp://annotate.accurator.nl     Crowdsourcing  &  Nichesourcing   @Rijksmuseum  
  67. 67. Rijksmuseum Amsterdam collection over 1 million artworks
  68. 68. only a small fraction of about 8000 items are currently on display
  69. 69. … online collection grows 125.000 artworks already available another 40.000 are added every year
  70. 70. expertise of museum professionals is in describing & annotating collection with art- historical information, e.g. when they were created, by whom, etc.
  71. 71. detailed information about depicted objects, e.g. which species the animal or plant belongs to, is in most cases not available
  72. 72. annotated only with “bird with blue head near branch with red leaf” species of the bird and the plant are missing
  73. 73. use crowdsourcing to get more annotations use nichesourcing, i.e. niches of people with the right expertise, to add more specific information
  74. 74. use sources like Twitter to find experts or groups of experts on certain areas, e.g. bird lovers, ornithologists or people who enjoy bird- watching in their spare time
  75. 75. platform where users enter tags: (1) structured vocabulary terms or (2) free text hSp://annotate.accurator.nl  
  76. 76. for tasks that are too difficult: game in which players can carry out an expert annotation task with some assistance
  77. 77. BIRDWATCHING RIJKSMUSEUM Sunday October 4, 10.00 am - 14.00 pm Cuypers Library Rijksmuseum On World Animal Day, the Rijksmuseum will host a birdwatching day in collaboration with Naturalis Biodiversity Center, Wikimedia Netherlands and the COMMIT/ SEALINCMedia project. We are looking for bird watchers to join an expedi- tion through the digital collections and help the museums identify bird species in works of art.
  78. 78. dive.beeldengeluid.nl   In  Digital   Hermeneu8cs   Event-­‐centric  Explora8on     @Sound  &  Vision  and  Royal  Library   3rd  Price  at  the  SemanGc  Web  Challenge  2014  
  79. 79. OPENIMAGES.EU   •  3000  videos     •  NL  InsGtute  for  Sound  &  Vision   •  mostly  news  broadcasts   DELPHER.NL   •  1.5  Million  Scans  of   •  Radio  bulleGns     •  (hand  annotated)   •  1937  –  1984                              
  80. 80. Simple  Event  Model  (SEM)   OpenAnnota8on  (OA)  &  SKOS   DIVE:MEDIA OBJECT   SEM:EVENT   SEM:PLACE   SEM:TIME   SEM:ACTOR   SKOS:CONCEPT   OA:ANNOTATION   •  LINKS  TO  EUROPEANA  (MULTILINGUAL)   •  LINKS  TO  DBPEDIA    
  81. 81. Digital  Submarine  UI   Infinity  of  Explora8on   Events  Linking  Objects   Crowd  Bringing     the  Human  Perspec8ves   Linked  (Open)  Data  
  82. 82. En8ty  &  Event  Extra8on  with  CrowdTruth.org   ENTITY EXTRACTION EVENTS CROWDSOURCING AND LINKING TO CONCEPTS THROUGH CROWDTRUTH.ORG SEGMENTATION & KEYFRAMES LINKING EVENTS AND CONCEPTS TO KEYFRAMES http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  83. 83. Erp,  M.  van;  Oomen,  J.;  Segers,  R.;  Akker,  C.  van  de;  Aroyo,  L.;  Jacobs,  G.;  Legêne,  S;  Meij,  L.  van  der;O  ssenbruggen,  J.R.  van;  Schreiber,  G.   AutomaGc  Heritage  Metadata  Enrichment  with  Historic  Events  Museums  and  the  Web  2011  h;p://www.museumsandtheweb.com/mw2011/ papers/automaGc_heritage_metadata_enrichment_with_hi   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  84. 84. engaging users through event narratives http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  85. 85. “Digital  HermeneuGcs:  Agora  and  the  online  understanding  of  cultural   heritage”  In  proc.  of  Web  Science  Conference,  (ACM:  New  York,  2011)   Interpreta8on  Support  for  Online  CollecGons  
  86. 86. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo Explora8ve  Search  
  87. 87. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo Engagement  with  Games  
  88. 88. Links  from  the  slides   On  the  Web •  http://waida.nl •  http://prestoprime.org •  http://agora.cs.vu.nl •  http://sealincmedia.wordpress.com •  http://dive.beeldengeluid.nl •  http://diveplu.beeldengeluid.nl •  http://annotate.accurator.nl •  http://accurator.nl •  http://crowdtruth.org •  http://data.crowdtruth.org •  http://game.crowdtruth.org •  http://www.adweek.com/socialtimes/ millennials-love-video-on-mobile-social- channels-infographic/622313 •  http://www.blogherald.com/2010/10/27/ history-of-online-video/ •  http://wm.cs.vu.nl   On  TwiSer   @waisda   @agora-­‐project   @sealincmedia   @prestocenter   @vistatv   #CrowdTruth   #Accurator     http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  89. 89. Lecture  Reading  Material   h;p://www.aaai.org/ojs/index.php/aimagazine/arGcle/view/2564     Truth  Is  a  Lie:  Crowd  Truth  and  the  Seven  Myths  of  Human  AnnotaGon   h;ps://www.wired.com/2006/06/crowds/     THE  RISE  OF  CROWDSOURCING     h;ps://www.microsoa.com/en-­‐us/research/project/algorithmic-­‐crowdsourcing/     h;p://cci.mit.edu/publicaGons/CCIwp2011-­‐04.pdf     Programming  the  Global  Brain   h;p://www.orchid.ac.uk/eprints/248/1/main.pdf     The  ACTIVECROWDTOOLKIT:  An  Open-­‐Source  Tool  for  Benchmarking  AcGve   Learning  Algorithms  for  Crowdsourcing  Research   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo

×