Download to read offline
Uploaded byDavide Ceolin
Semi-automated Assessment of Annotation Trustworthiness
The document proposes a semi-automated system to assess the trustworthiness of annotations created by volunteers for museums. The system calculates user reputations based on annotation accuracy, clusters annotations semantically to improve efficiency, and ranks annotations to identify the most trustworthy contributions from each user based on their reputation. The results of experiments on real datasets showed the system could accurately identify trustworthy annotations with minimum effort in an efficient manner.
Semi-automated Assessment of Annotation Trustworthiness
- 1. Semi-Automated Assessment of Annotation Trustworthiness DavideCeolin, Archana Nottamkandath Wan Fokkink
- 2.
- 3. Museums... Photo: flickr.com/clumsyjim have aproblem. Photo: flickr.com/grrrl
- 4. Let’s recruit somehelp Photo: flickr.com/anirudhkoul
- 5.
- 6.
- 7.
- 8.
- 9.
- 10. Museums are meticolous • Thecrowd alone does not solve the problem. • We need to select only the most trustworthy annotations. • System requirements: ✓reliable ✓with minumum effort ✓efficient
- 11. User Reputation User reputationshould reflect the trustworthiness of their annotations. Tulip Flower Ugly Photo: flickr.com/hz536n
- 12.
- 13. Reputation estimation Tulip Flower Red Pink Purple 0.00.51.01.52.0 Reputation Reputation values ProbabilityDensity 00.1 0.3 0.5 0.7 0.9 1 We compute reputations as in subjective logic. Photo: flickr.com/hz536n
- 14. Reputation estimation Tulip Flower Red Pink Purple Few annotationsper author: ✓ minimum effort 0.00.51.01.52.0 Reputation Reputation values ProbabilityDensity 0 0.1 0.3 0.5 0.7 0.9 1 Photo: flickr.com/hz536n
- 15.
- 16. Interpreting reputation User reputation: 0.6 0.00.51.01.52.0 Reputation Reputationvalues ProbabilityDensity 0 0.1 0.3 0.5 0.7 0.9 1 E0.6
- 17. Interpreting reputation User reputation: 0.6 Shouldwe accept all his annotations? Reject all?
- 18. Interpreting reputation User reputation: 0.6 Shouldwe accept all his annotations? Reject all? No: accept only the best 60%. How?
- 19. Estimating expertise can help Expertise Trainingset New annotation Tulip Flower Red Pink Purple Semantic distance Rose
- 20. We are almostthere... Rank the annotations and evaluate them. But we can improve the efficiency of the system Rose Violet ... ... Tomato 0.9 0.8 ... ... 0.3 } } 60% accepted 40% rejected User reputation: 0.6 Test set
- 21. Semantic clustering Clustering thetraining set semantically helps in improving the system efficiency Tulip Flower Red Pink Purple Rose
- 22. Semantic clustering Clustering thetraining set semantically helps in improving the system efficiency Tulip Flower Red Pink Purple Rose
- 23. Semantic clustering Clustering thetraining set semantically helps in improving the system efficiency Tulip Flower Red Pink Purple Rose
- 24. Semantic clustering ✓efficiency Clustering thetraining set semantically helps in improving the system efficiency Tulip Flower Red Pink Purple Rose
- 25. Results • We runsome experiments with from 5 to 30 annotations per user, on two datasets. • Accuracy: 68% - 84% • Precision: 87% - 88% • Recall: 80% - 96% • Time saved by clustering: 44% - 52%
- 26. Discussion • Results aresatisfactory but they can be further improved • Provenance can help • Image analysis programs can help • Reuse of evaluations can reduce the museum effort
- 27. Recap • We proposea system to evaluate the trustworthiness of museum annotations • Uses reputation, semantic similarity and clustering ✓Reliable ✓With minumum effort ✓Efficient
- 28. Recap • We proposea system to evaluate the trustworthiness of museum annotations • Uses reputation, semantic similarity and clustering ✓Reliable ✓With minumum effort ✓Efficient Thanks!