ZenCrowd	
  
Leveraging	
  Probabilis3c	
  Reasoning	
  and	
  
Crowdsourcing	
  Techniques	
  for	
  Large-­‐Scale	
  
En...
MoFvaFon	
  
•  Linked	
  Open	
  Data	
  (LOD)	
  
•  Linking	
  enFty	
  from	
  text	
  to	
  LOD	
  
– Rich	
  snippet...
Gianluca	
  DemarFni,	
  eXascale	
  Infolab	
   3	
  
hVp://dbpedia.org/resource/Facebook	
  
hVp://dbpedia.org/resource/...
Crowdsourcing	
  
•  Exploit	
  human	
  intelligence	
  to	
  solve	
  
– Tasks	
  simple	
  for	
  humans,	
  complex	
 ...
ZenCrowd	
  
•  Combine	
  both	
  algorithmic	
  and	
  manual	
  linking	
  
•  Automate	
  manual	
  linking	
  via	
  ...
Outline	
  
•  Related	
  approaches	
  
•  ZenCrowd:	
  System	
  architecture	
  
•  ProbabilisFc	
  model	
  to	
  comb...
Related	
  Approaches	
  
•  EnFty	
  Linking	
  
•  Ad-­‐hoc	
  Object	
  Retrieval	
  
– Keyword	
  queries	
  
– Lookin...
ZenCrowd	
  Architecture	
  
Micro
Matching
Tasks
HTML
Pages
HTML+ RDFa
Pages
LOD Open Data Cloud
Crowdsourcing
Platform
Z...
Algorithmic	
  Matching	
  
•  Inverted	
  index	
  over	
  LOD	
  enFFes	
  
– DBPedia,	
  Freebase,	
  Geonames,	
  NYT	...
EnFty	
  Factor	
  Graphs	
  
•  Graph	
  components	
  
– Workers,	
  links,	
  clicks	
  
– Prior	
  probabiliFes	
  
– ...
EnFty	
  Factor	
  Graphs	
  
•  Training	
  phase	
  
– IniFalize	
  worker	
  priors	
  
– with	
  k	
  matches	
  on	
 ...
Experimental	
  EvaluaFon	
  
•  Datasets	
  
–  25	
  news	
  arFcles	
  from	
  
•  CNN.com	
  (Global	
  news)	
  
•  N...
The	
  micro-­‐task	
  
22-­‐Apr-­‐12	
   Gianluca	
  DemarFni,	
  eXascale	
  Infolab	
   13	
  
Experimental	
  EvaluaFon	
  
•  AutomaFc	
  Approach:	
  P/R	
  trade-­‐off	
  
0"
0.1"
0.2"
0.3"
0.4"
0.5"
0.6"
0.7"
0.8"...
Experimental	
  EvaluaFon	
  
•  EnFty	
  Linking	
  with	
  Crowdsourcing	
  and	
  
agreement	
  vote	
  (at	
  least	
 ...
Experimental	
  EvaluaFon	
  
•  Worker	
  community	
  and	
  textual	
  context	
  
0"
0.1"
0.2"
0.3"
0.4"
0.5"
0.6"
0.7...
Experimental	
  EvaluaFon	
  
•  Worker	
  SelecFon	
  
Top$US$
Worker$
0$
0.5$
1$
0$ 250$ 500$
Worker&Precision&
Number&o...
Experimental	
  EvaluaFon	
  
•  EnFty	
  Linking	
  with	
  ZenCrowd	
  
– Training	
  with	
  first	
  5	
  enFFes	
  +	
...
Comparing	
  3	
  matching	
  techniques	
  
•  ZenCrowd	
  best	
  for	
  75%	
  of	
  documents	
  
22-­‐Apr-­‐12	
   Gi...
Lessons	
  Learnt	
  
•  Crowdsourcing	
  +	
  Prob	
  reasoning	
  works!	
  
•  But	
  
– Different	
  worker	
  communiF...
Conclusions	
  
•  ZenCrowd:	
  ProbabilisFc	
  reasoning	
  over	
  automaFc	
  and	
  
crowdsourcing	
  methods	
  for	
...
Upcoming SlideShare
Loading in …5
×

ZenCrowd: Leveraging Probabilistic Reasoning and Crowdsourcing Techniques for Large-Scale Entity Linking

604 views

Published on

Conference talk at WWW 2012, , April 2012, Lyon, France

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
604
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

ZenCrowd: Leveraging Probabilistic Reasoning and Crowdsourcing Techniques for Large-Scale Entity Linking

  1. 1. ZenCrowd   Leveraging  Probabilis3c  Reasoning  and   Crowdsourcing  Techniques  for  Large-­‐Scale   En3ty  Linking       Gianluca  Demar3ni,  Djellel  Eddine   Difallah,  and  Philippe  Cudré-­‐Mauroux   eXascale  Infolab,  University  of  Fribourg   Switzerland  
  2. 2. MoFvaFon   •  Linked  Open  Data  (LOD)   •  Linking  enFty  from  text  to  LOD   – Rich  snippets   – EnFty-­‐centric  search   •  Linking   – Algorithmic   – Manual  (NYT  arFcles)   HTML+ RDFa Pages 22-­‐Apr-­‐12   Gianluca  DemarFni,  eXascale  Infolab   2  
  3. 3. Gianluca  DemarFni,  eXascale  Infolab   3   hVp://dbpedia.org/resource/Facebook   hVp://dbpedia.org/resource/Instagram   Zase:Instagram   owl:sameAs   Google   Android   <p>Facebook  is  not  waiFng  for  its  iniFal   public  offering  to  make  its  first  big   purchase.</p><p>In  its  largest   acquisiFon  to  date,  the  social  network   has  purchased  Instagram,  the  popular   photo-­‐sharing  applicaFon,  for  about  $1   billion  in  cash  and  stock,  the  company   said  Monday.</p>   <p><span  about="hVp://dbpedia.org/resource/ Facebook"><cite  property=”rdfs:label">Facebook</ cite>  is  not  waiFng  for  its  iniFal  public  offering  to   make  its  first  big  purchase.</span></p><p><span   about="hVp://dbpedia.org/resource/Instagram">In   its  largest  acquisiFon  to  date,  the  social  network  has   purchased  <cite  property=”rdfs:label">Instagram</ cite>  ,  the  popular  photo-­‐sharing  applicaFon,  for   about  $1  billion  in  cash  and  stock,  the  company  said   Monday.</span></p>   RDFa   enrichment   HTML:  
  4. 4. Crowdsourcing   •  Exploit  human  intelligence  to  solve   – Tasks  simple  for  humans,  complex  for  machines   – With  a  large  number  of  humans  (the  Crowd)   – Small  problems:  micro-­‐tasks  (Amazon  MTurk)   •  Examples   – Wikipedia,  Image  tagging   •  IncenFves   – Financial,  fun,  visibility   22-­‐Apr-­‐12   Gianluca  DemarFni,  eXascale  Infolab   4  
  5. 5. ZenCrowd   •  Combine  both  algorithmic  and  manual  linking   •  Automate  manual  linking  via  crowdsourcing   •  Dynamically  assess  human  workers  with  a   probabilisFc  reasoning  framework   22-­‐Apr-­‐12   5   Crowd   Algorithms  Machines  
  6. 6. Outline   •  Related  approaches   •  ZenCrowd:  System  architecture   •  ProbabilisFc  model  to  combine  automaFc   matching  and  crowdsourcing  results   •  Experimental  evaluaFon   22-­‐Apr-­‐12   Gianluca  DemarFni,  eXascale  Infolab   6  
  7. 7. Related  Approaches   •  EnFty  Linking   •  Ad-­‐hoc  Object  Retrieval   – Keyword  queries   – Looking  for  a  specific  enFty   – IR  indexing  and  ranking  over  RDF  data   •  Crowdsourcing   – Training  set,  tagging,  annotaFon   – IR  evaluaFon   – CrowdDB   22-­‐Apr-­‐12   Gianluca  DemarFni,  eXascale  Infolab   7  
  8. 8. ZenCrowd  Architecture   Micro Matching Tasks HTML Pages HTML+ RDFa Pages LOD Open Data Cloud Crowdsourcing Platform ZenCrowd Entity Extractors LOD Index Get Entity Input Output Probabilistic Network Decision Engine Micro- TaskManager Workers Decisions Algorithmic Matchers 22-­‐Apr-­‐12   Gianluca  DemarFni,  eXascale  Infolab   8  
  9. 9. Algorithmic  Matching   •  Inverted  index  over  LOD  enFFes   – DBPedia,  Freebase,  Geonames,  NYT   •  TF-­‐IDF  (IR  ranking  funcFon)   •  Top  ranked  URIs  linked  to  enFFes  in  docs   •  Threshold  on  the  ranking  funcFon  or  top  N   22-­‐Apr-­‐12   Gianluca  DemarFni,  eXascale  Infolab   9  
  10. 10. EnFty  Factor  Graphs   •  Graph  components   – Workers,  links,  clicks   – Prior  probabiliFes   – Link  Factors   – Constraints   •  ProbabilisFc   Inference   – Select  all  links  with   posterior  prob  >τ   w1 w2 l1 l2 pw1( ) pw2( ) lf1( ) lf2( ) pl1( ) pl2( ) l3 lf3( ) pl3( ) c11 c22 c12 c21 c13 c23 u2-3( )sa1-2( ) 2  workers,  6  clicks,  3  candidate  links   Link  priors   Worker   priors   Observed   variables   Link   factors   SameAs   constraints   Dataset   Unicity   constraints 22-­‐Apr-­‐12   Gianluca  DemarFni,  eXascale  Infolab   10  
  11. 11. EnFty  Factor  Graphs   •  Training  phase   – IniFalize  worker  priors   – with  k  matches  on  known  answers   •  UpdaFng  worker  Priors   – Use  link  decision  as  new  observaFons   – Compute  new  worker  probabiliFes   •  IdenFfy  (and  discard)  unreliable  workers   22-­‐Apr-­‐12   Gianluca  DemarFni,  eXascale  Infolab   11  
  12. 12. Experimental  EvaluaFon   •  Datasets   –  25  news  arFcles  from   •  CNN.com  (Global  news)   •  NYTimes.com  (Global  news)   •  Washington-­‐post.com  (US  local  news)   •  Timesofindia.indiaFmes.com  (India  local  news)   •  Swissinfo.com  (Switzerland  local  news)   –  40M  enFFes  (Freebase,  DBPedia,  Geonames,  NYT)   •  80  workers  on  Amazon  MTurk,  $0.01  per  enFty   •  Standard  evaluaFon  measures:  Prec,  Recall,  Acc   •  Gold  Standard:  editorial  selecFon   22-­‐Apr-­‐12   Gianluca  DemarFni,  eXascale  Infolab   12  
  13. 13. The  micro-­‐task   22-­‐Apr-­‐12   Gianluca  DemarFni,  eXascale  Infolab   13  
  14. 14. Experimental  EvaluaFon   •  AutomaFc  Approach:  P/R  trade-­‐off   0" 0.1" 0.2" 0.3" 0.4" 0.5" 0.6" 0.7" 0.8" 0.9" 1" 1" 2" 3" 4" 5" Precision)/)Recall) Top)N)Results) P" R" 22-­‐Apr-­‐12   Gianluca  DemarFni,  eXascale  Infolab   14  
  15. 15. Experimental  EvaluaFon   •  EnFty  Linking  with  Crowdsourcing  and   agreement  vote  (at  least  2  out  of  5  workers   select  the  same  URI)           Top-­‐1  precision:  0.70   the URIs with at least 2 votes are selected as valid links (we tried various thresholds and manually picked 2 in the end since it leads to the highest precision scores while keep- ing good recall values for our experiments). We report on the performance of this crowdsourcing technique in Table 2. The values are averaged over all linkable entities for di↵erent document types and worker communities. Table 2: Performance results for crowdsourcing with agreement vote over linkable entities. US Workers Indian Workers P R A P R A GL News 0.79 0.85 0.77 0.60 0.80 0.60 US News 0.52 0.61 0.54 0.50 0.74 0.47 IN News 0.62 0.76 0.65 0.64 0.86 0.63 SW News 0.69 0.82 0.69 0.50 0.69 0.56 All News 0.74 0.82 0.73 0.57 0.78 0.59 The first question we examine is whether there is a di↵er- ence in reliability between the various populations of work- Figure textua Entity L We n ference method consisti phase, c the wor is know In ord ence in of bad ered as clicks o In our e swers in 22-­‐Apr-­‐12   Gianluca  DemarFni,  eXascale  Infolab   15  
  16. 16. Experimental  EvaluaFon   •  Worker  community  and  textual  context   0" 0.1" 0.2" 0.3" 0.4" 0.5" 0.6" 0.7" 0.8" 0.9" 1" 0" 0.2" 0.4" 0.6" 0.8" 1" Recall& Precision& US" India" 0" 0.1" 0.2" 0.3" 0.4" 0.5" 0.6" 0.7" 0.8" 0.9" 1" 1" 2" 3" 4" 5" 6" 7" 8" 9" 10" Precision) Document) Simple" Snippet" 22-­‐Apr-­‐12   Gianluca  DemarFni,  eXascale  Infolab   16  
  17. 17. Experimental  EvaluaFon   •  Worker  SelecFon   Top$US$ Worker$ 0$ 0.5$ 1$ 0$ 250$ 500$ Worker&Precision& Number&of&Tasks& US$Workers$ IN$Workers$ 0.6$ 0.62$ 0.64$ 0.66$ 0.68$ 0.7$ 0.72$ 0.74$ 0.76$ 0.78$ 0.8$ 1$ 2$ 3$ 4$ 5$ 6$ 7$ 8$ 9$ Precision) Top)K)workers) 22-­‐Apr-­‐12   Gianluca  DemarFni,  eXascale  Infolab   17  
  18. 18. Experimental  EvaluaFon   •  EnFty  Linking  with  ZenCrowd   – Training  with  first  5  enFFes  +  5%  aserwards   – 3  consecuFve  bad  answers  lead  to  blacklisFng   di↵er- work- s per- point all en- tasks higher orkers ws as rms of ed on clicks on the links, hence generating noise in our system. In our experiments, we consider that 3 consecutive bad an- swers in the training phase is enough to identify the worker as a spammer and to blacklist him/her. We report the aver- age results of ZenCrowd when exploiting the training phase, constraints, and blacklisting in Table 3. As we can observe, precision and accuracy values are higher in all cases when compared to the agreement vote approach. Table 3: Performance results for crowdsourcing with ZenCrowd over linkable entities. US Workers Indian Workers P R A P R A GL News 0.84 0.87 0.90 0.67 0.64 0.78 US News 0.64 0.68 0.78 0.55 0.63 0.71 IN News 0.84 0.82 0.89 0.75 0.77 0.80 SW News 0.72 0.80 0.85 0.61 0.62 0.73 All News 0.80 0.81 0.88 0.64 0.62 0.76 Finally, we compare ZenCrowd to the state of the art22-­‐Apr-­‐12   Gianluca  DemarFni,  eXascale  Infolab   18  
  19. 19. Comparing  3  matching  techniques   •  ZenCrowd  best  for  75%  of  documents   22-­‐Apr-­‐12   Gianluca  DemarFni,  eXascale  Infolab   19   0" 0.1" 0.2" 0.3" 0.4" 0.5" 0.6" 0.7" 0.8" 0.9" 1" 1" 2" 3" 4" 5" 6" 7" 8" 9" 10"11"12"13"14"15"16"17"18"19"20"21"22"23"24"25" Precision) ) Document) Agr."Vote" ZenCrowd" Top"1" Simple  Crowdsourcing   ZenCrowd   AutomaFc  
  20. 20. Lessons  Learnt   •  Crowdsourcing  +  Prob  reasoning  works!   •  But   – Different  worker  communiFes  perform  differently   – Many  low  quality  workers   – No  differences  w/  different  contexts   – CompleFon  Fme  may  vary  (based  on  reward)   22-­‐Apr-­‐12   Gianluca  DemarFni,  eXascale  Infolab   20  
  21. 21. Conclusions   •  ZenCrowd:  ProbabilisFc  reasoning  over  automaFc  and   crowdsourcing  methods  for  enFty  linking   •  Standard  crowdsourcing  improves  6%  over  automaFc   •  4%  -­‐  35%  improvement  over  standard  crowdsourcing   •  14%  average  improvement  over  automaFc  approaches   •  Next  steps   –  Long-­‐term  worker  behavior  analysis   –  More  efficient  and  effecFve  linking  by  LOD  dataset  pre-­‐ selecFon   hVp://diuf.unifr.ch/xi/zencrowd/   22-­‐Apr-­‐12   Gianluca  DemarFni,  eXascale  Infolab   21  

×