Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Crowdsourcing & Nichesourcing: Enriching Cultural Heritage with Experts & Crowds

841 views

Published on

Presentation at the "Past, Present and Future of Digital Humanities & Social Sciences in the Netherlands" event, http://www.ehumanities.nl/past-present-and-future-of-digital-humanities-social-sciences-in-the-netherlands-programme-and-abstracts-2/

Published in: Technology
  • Be the first to comment

Crowdsourcing & Nichesourcing: Enriching Cultural Heritage with Experts & Crowds

  1. 1.  Crowdsourcing  &  Nichesourcing:     Enriching  Cultural  Heritage   with  Experts  &  Crowds   Lora  Aroyo   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  2. 2. so9ware  systems  are  ever  more  intelligent   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo but  they  don’t  actually  understand  people  
  3. 3. focus  on  human  knowledge  in  machine-­‐readable  form   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo but  there  are  types  of  human  knowledge                           that  can’t  be  captured  by  machines  
  4. 4. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo in  a  form  that  machines  can  acquire  &  process   urge  computer  science  research     to  push  fronCers  of  human  knowledge  representaCon    
  5. 5. classical  AI  involves  human  experts  to  manually   provide  training  knowledge  for  machines   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo human  expert-­‐based  ground  truth  does  not  scale     for  current  demand  for  machines  to  deal  with  wide   ranges  of  real-­‐world  tasks  and  contexts    
  6. 6. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo to  provide  an  approach  to  capturing  human  knowledge   in  a  way  that  is  scalable  &  adequate  to  real-­‐world  needs   the  key  scienCfic  challenge  is  
  7. 7. QuanCty  is  the  new  Quality   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo Human  ComputaCon  adopts  human  intelligence  at   scale  to  improve  purely  machine-­‐based  systems  
  8. 8. humans  accurately  perform  interpretaCon  tasks   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  9. 9. humans  accurately  perform  interpretaCon  tasks   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo can  their  effort  be  adequately  harnessed  in  a   scienCfically  reliable  manner  that  scales  across  tasks,   contexts  &  data  modaliCes?  
  10. 10. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo Research  Projects  
  11. 11. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo 1.  extracCng  knowledge  from  social  annotaCon   Research  Projects  
  12. 12. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo 1.  extracCng  knowledge  from  social  annotaCon   2.  harnessing  of  non-­‐expert  human  knowledge   Research  Projects  
  13. 13. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo 1.  extracCng  knowledge  from  social  annotaCon   2.  harnessing  of  non-­‐expert  human  knowledge   3.  nichesourcing  of  expert  knowledge     Research  Projects  
  14. 14. 4.  capturing  event  knowledge   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo 1.  extracCng  knowledge  from  social  annotaCon   2.  harnessing  of  non-­‐expert  human  knowledge   3.  nichesourcing  of  expert  knowledge     Research  Projects  
  15. 15. 1.  Human  Knowledge  from  Social  Tagging   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  16. 16.  Waisda?  @  Beeld  en  Geluid   – an  online  video  tagging  game,  e.g.  NCRV  and  VARA     – power  of  the  crowd  for  Web-­‐scale  enrichment  of  A/V   collecAons   – significant  accuracy  for  improving  IR  beyond  the  state-­‐ of-­‐the-­‐art  of  expert-­‐based  annotaAons   – sustainable  means  for  engaging  large  volunteer  crowds     http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo 1.  Human  Knowledge  from  Social  Tagging  
  17. 17. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo hZp://waisda.nl/   hZp://spotvogel.vroegevogels.vara.nl/   Engage  with  Games  in  Entertainment  
  18. 18. Following the grandeur of Baroque, Rococo art is often dismissed as frivolous and unserious, but Waldemar Januszczak disagrees. […] The first episode is about travel in the 18th century and how it impacted greatly on some of the finest art ever made. The world was getting smaller and took on new influences shown in the glorious Bavarian pilgrimage architecture, Canaletto's romantic Venice and the blossoming of exotic designs and tastes all over Europe. Rococo:  Travel,  pleasure,  madness   A  boarding  school  where  boys  from  the  Dutch  East  Indies   receive  a  vocaAonal  educaAon  has  been  set  up  by  the   ministry  of  Social  Affairs  near  Batavia.  Trumpeter  sounds   the  reveille;  -­‐  the  boys  get  out  of  bed;  -­‐  the  muster  is  held  in   front  of  the  building;  aIerwards  the  boys  stand  in  line  with  a   food  bowl;  they  get  rice  pudding  and  eat  this  around  long   tables  in  the  open  air;  -­‐  the  boys  get  instrucAons  and   pracAce  a.o.  drawing  shapes,  forging,  metalworking,  filing   and  woodworking;   News  from  Indonesia:  youth  care   hZp://vista-­‐tv.eu   hZp://dive.beeldengeluid.nl   Crowdsourcing  for  Video  Analysis  
  19. 19. 2.  Harnessing  Non-­‐expert  Human   Knowledge  in  Medical  Domain   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  20. 20. 2.  Harnessing  Non-­‐expert  Human   Knowledge  in  Medical  Domain    Training  IBM  Watson   – gathering  higher  quality  ground  truth  data  from  the   crowd  than  from  experts   – open  source  human-­‐machine  computaAon  framework   for  accurately  measuring  quality  of  resulAng  data   – CrowdTruth.org     http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  21. 21. http://CrowdTruth.org http://data.CrowdTruth.org/ http://game.crowdtruth.org Achieving Expert-Level Annotation Quality with the Crowd The Case of Medical Relation Extraction http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  22. 22. Crowdsourcing  for  Text  Analysis   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  23. 23. hNp://game.crowdtruth.org     Engage  with  Games  in  Sciences  
  24. 24. 3.  Nichesourcing  Expert  Knowledge     http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  25. 25. 3.  Nichesourcing  Expert  Knowledge     Cultural  Heritage   – Different  layers  and  domains  of  experAse     – Intrinsic  moAvaAon  of  knowledgeable  crowds   – Accurator.nl  @  Rijksmuseum   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo seman'c  enrichment  with  LOD  can  be  complemented   with  domain  knowledge  from  niches  of  experts  
  26. 26. there  are  massive  niches  of  experts  online  …   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  27. 27. hNp://annotate.accurator.nl/     Engage  the  Niche  in  Cultural  Heritage   hNp://accurator.nl/    
  28. 28. Engage  with  Games  in  Cultural  Heritage  
  29. 29. 4.  Capturing  Event  Knowledge   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  30. 30. 4.  Capturing  Event  Knowledge   interpretaCon  support  framework  Digital  HermeneuCcs     –  text,  images  and  videos,  e.g.  newspapers,  tweets,  TV  and   radio  programs,  and  images     –  events  carry  ambiguity,  bias  &  wide  range  of  perspecAves   –  knowledge  representaAon  challenges,  e.g  informaAon   granularity,  temporal  and  provenance  modeling   –  Agora,  DIVE+  @  KB,  Beeld  en  Geluid,  Amsterdam  Museum   –  Crowddriven.nl  for  TV  show  Game  of  Thrones   http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
  31. 31. engaging users through event narratives
  32. 32. “Digital  Hermeneu/cs:  Agora  and  the  online  understanding  of  cultural   heritage”  In  proceedings  of  Web  Science  Conference,  (ACM:  New  York,   2011)   InterpretaCon  Support  for  Scholars  
  33. 33. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo Browsing  support  for  wide  audience  
  34. 34. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo Engage  with  Games  in  Entertainment  
  35. 35. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo understanding  perspecCves   – diversity  of  opinions  from  the  crowd   – mulAtude  of  contexts   – ambiguity  in  language,  images  or  videos   – independent  interpretaCons   – aggregated  view  –  the  big  picture   – a  new  approach  to  understanding  semanCcs  by   harnessing  the  power  &  diversity  (disagreement  on  the   correct  interpretaAon)  of  the  crowd   – human  disagreement  is  essenCal  in  helping  machines   with  semanCc  interpretaCon    
  36. 36. Goodbye to Truth Hello Perspectives
  37. 37. Goodbye to Truth Human Computation making sense of human semantics
  38. 38. Goodbye to Truth Human Computation making sense of human semantics http://lora-aroyo.org http://slideshare.net/laroyo @laroyo

×