The document discusses using crowdsourcing to collect semantic annotations by having multiple annotators label examples to capture disagreement. It proposes that collecting disagreement data can help address problems with traditional methods that rely on inter-annotator agreement as a measure of quality. The approach involves developing a crowdsourcing methodology and experimental plan to collect annotation examples from crowds along with a task workflow to evaluate the results and classifier performance.

1. Crowdsourcing Disagreement for
Collecting Semantic Annotation
Anca Dumitrache
@anouk_anca
Extended Semantic Web Conference
PhD Symposium
June 1st
2015

2. annotatorannotation
example

3. annotatorannotation
exampleproblem:

4. annotatorannotation
exampleproblem:
●

5. annotatorannotation
exampleproblem:
●
●

6. annotatorannotation
exampleproblem:
●
●
●

7. annotatorannotation
exampleproblem:
●
●
●
quality = inter-annotator
agreement

8. crowdannotation
example
solution:

9. annotation
example
solution:
●
●
●
crowd

10. annotation
example
solution:
●
●
●
crowd

11. Hypothesis:

12. annotation
example
approach:
crowd

13. annotation
example ambiguity
approach:
crowd

14. annotation clarity
example
approach:
crowd

15. worker qualityannotation
example
approach:

16. annotation
example
approach:
crowd

17. annotation
example
approach:
crowd

18. annotation
example
approach:
●
●
crowd

19. annotation
example
crowd
approach:

20. annotation
example
●
●
crowd
approach:

21. Crowdsourcing Methodology

22. annotation
example
experimental plan:
crowd

23. annotation
example
experimental plan:
crowd

24. annotation
example
experimental plan:
crowd

25. annotation
example
experimental plan:
crowd

26. annotation
example
experimental plan:
crowd

27. annotation
example
crowd
current work:

28. annotation
example
approach:
crowd

29. Crowdsourcing task workflow

30. annotation
example
evaluation:
crowd

31. Classifier evaluation results

32. Try it out!
CrowdTruth.org
github.com/CrowdTruth.org
@anouk_anca
#CrowdTruth