The document discusses using crowdsourced human computation to improve interlinking of datasets in the Linked Open Data cloud. It presents CROWDKI, a system that manages microtasks on crowdsourcing platforms to systematically involve humans in assessing potential interlinks and validating automatically generated links. Two use cases are described: 1) having crowds assess the relevance of different interlinking possibilities between resources and 2) having crowds validate and enhance automatically computed links between datasets. The architecture utilizes a crowdsourcing platform API to generate and collect microtasks that can be completed rapidly and at low cost to enhance interlinking over purely automatic methods.

1. Institute for Web Science and Technologies · University of Koblenz-Landau, Germany
Programmatic Access to Crowdsourced
Human Computation for
Designing and Enhancing Interlinking
Cristina Sarasua
csarasua@uni-koblenz.de
ESWC2015 Developers Workshop

2. CROWDKI 2Cristina Sarasua
The Problem

3. CROWDKI 3Cristina Sarasua
Mostly identity links, few prominent interlinking hubs with high
number of in-links, and still 44% of the analyzed datasets do
not contain out-links [Schmachtenberg et al., 2014]
 To improve in heterogeneity and quantity of links we
need:
– To overcome computational limitations of automatic link
discovery methods
– Methods that assist data publishers in deciding the
datasets to target and the way to define the interlinks.
Current LOD

4. CROWDKI 4Cristina Sarasua
CROWDKI: Crowd-Powered Knowledge
Integration

5. CROWDKI 5Cristina Sarasua
 Humans involved systematically in processing data for
interlinking
 Human input is collected via microtask crowdsourcing
– Online marketplaces (e.g. Clickworker)
– Anyone registered (all around the world)
– Economic reward
– Divided into simple tasks (e.g. review a particular link between
two resources)
– Large and dedicated workforce → fast completion time (hours /
days)
 Software to manage the generation and completion of
microtasks related to interlinking:
https://github.com/criscod/CROWDKI
Crowdsourced Human Computation for Interlinking

6. CROWDKI 6Cristina Sarasua
 Input: list of interlinking possibilities and a particular
context
Car wasDesigned by Person
Car wasDriven by Person
Car wasRecommended by Person
 Output: contextual relevance assessment by the
crowd
UC1:Assessing the relevance of different
interlinking possibilities

7. CROWDKI 7Cristina Sarasua
UC1:Assessing the relevance of different
interlinking possibilities

8. CROWDKI 8Cristina Sarasua
 Input: set of candidate links, RDF data
 Output: set of final links (extended / post-processed)
UC2:Validating and Enhancing automatically
computed links:

9. CROWDKI 9Cristina Sarasua
UC2:Validating and Enhancing automatically
computed links:

10. CROWDKI 10Cristina Sarasua
Architecture
CrowdFlower
REST API
Microtasks
templates
& config
Java
Jena, SPARQL
JSON
Guava IO

11. CROWDKI 11Cristina Sarasua
Starting CROWDKI

12. CROWDKI 12Cristina Sarasua
 Communication is key: processing data with typos may be
interpreted wrongly
 Communities of crowd workers are emerging
 Not all interlinking scenarios require human computation (e.g.
country ISO codes) – the challenge is to automatically decide
when it is really worthwhile
 Drawbacks: no real-time crowdsourcing and crowd workers
cannot be selected accurately
 CrowdFlower (the crowdsourcing platform used) provides
more access to more feature via the UI than the API
Lessons Learned

13. CROWDKI 13Cristina Sarasua
Conclusions
 Hybrid approaches (automatic + crowd interlinking) can be
better (P,R) than purely automatic interlinking methods
 CROWDKI could be used in combination with dataset
recommendation methods that analyze the way data is
already interlinked.

14. CROWDKI 14Cristina Sarasua
 Which challenges did you face with state-of-the-art
link discovery tools, what went wrong?
 How do you think human computation can further
help in interlinking?
 What are in your opinion pros and cons of this
approach?
Questions for the audience

15. CROWDKI 15Cristina Sarasua
 Max Schmachtenberg, Christian Bizer, Heiko Paulheim: Adoption of the
Linked Data Best Practices in Different Topical Domains.
13th International Semantic Web Conference (ISWC2014) - RDB
Track, Riva del Garda, Italy, October 2014
References