The document describes techniques used to improve the quality of crowdsourced data from the GEO-Wiki project. It discusses preprocessing steps like blur detection, duplicate detection, and vote aggregation algorithms. Blur detection removed 2% of low-quality images, while duplicate detection based on perceptual hashing removed 6% of redundant votes. Benchmarking algorithms on expert-annotated data showed that majority voting performed comparably to more complex algorithms when there were many accurate volunteers and few spammers. Preprocessing improved results by reducing workload and increasing statistical significance.

1. Vote Aggregation Techniques in
the Geo-Wiki Crowdsourcing Game:
A Case Study
Michael Khachay, Oleg Nurmukhametov
Artem Baklanov
Krasovskii Institute of Mathematics and Mechanics
Postdoctoral Research Scholar, ASA, IIASA
Krasovskii Institute of Mathematics and Mechanics
Steffen Fritz, Carl Salk, Linda See, and Dmitry
Shchepashchenko
IIASA

2. Young Scientists Summer Program
Since 1977, IIASA’s annual 3-­month Young Scientists
Summer Program (YSSP) offers opportunities to
talented young researchers.

3. Crowdsourcing = Crowd + Outsourcing
Crowdsourcing is a new approach to perform
tasks, when a group of distributed worldwide
people in total can substitute an expert.
Example:
GEO-wiki project, ESM, IIASA

4. Geo-Wiki: Cropland Capture Game
5 millions votes
170 000 satellite images
3000 volunteers
6 months
Goal: land cover map
www.geo-wiki.org d

5. Preprocessing of images
Blur detection
Duplicate detection
Benchmarking
of algorithms for vote
aggregation
Outline of presentation
Noise reduction Prediction

6. Preprocessing of images
Duplicate detection
Blur detection
Benchmarking
of algorithms for vote
aggregation
Outline of presentation
Noise reduction Prediction

7. Blur detection algorithm [H Tang, 2012].
Input: image
Output: coefficient of blur in [0, 100]
Detection of blurry images
100%0%

8. High quality images
Probability
0%
10%
30%

9. Blurry images
80% 90% 95%
98% 99%

10. The table and figure are from
Tong, Hanghang, et al. "Blur detection for digital
images using wavelet transform." Multimedia
and Expo, 2004. ICME'04. 2004 IEEE
International Conference on. Vol. 1. IEEE, 2004.
Blur detection algorithm

11. A key idea
• A= a number of Roof-­Structure and
Gstep-­Structure edge points that have lost
their sharpness.
• B= a number of Roof-­Structure and
Gstep-­Structure edge points.
• Blur coefficient = 100*A/B

12. Detection of blurry images
Results
Current campaign:
Noise reduction in dataset.
+ for future campaigns:
Less workload, more joy.
Blur coefficient Number of images
0% (Excellent images) 74 000 (38%)
80% - 100% (Extremely bad images) 3 200 (2%)
Extremely bad images were removed

13. Detection of duplicates
Web Link
Image unique
identifier Link to image
1 http://cg.tuwien.ac.at/~sturn/crop/img_78.1875_52.9958_1000.jpg
2 http://cg.tuwien.ac.at/~sturn/crop/img_78.1875_52.9958_500.jpg
3 http://cg.tuwien.ac.at/~sturn/crop/img_78.1875_52.9958_300.jpg
… …
Binary file Image content

14. Detection of duplicates
Find the 10 differences

15. Detection of duplicates
The pixel-by-pixel differences
Links: different
Binary files: different
Image content: similar

16. pHash (perceptual hash) [Zauner, 2010].
Detection of duplicates

17. pHash. High level description
1. Resize to 32x32 and grayscale;;
2. Perform DCT-­2 transformation and keep
top-­left 8x8;;
3. Calculate the average value of
alternating components;;
4. Set the 64 hash bits to 0 (or 1) on
whether DCT values more (or less) the
average value.

18. Detection of copies
Method Copies count Time to completion
Binary data comparison (MD5) 2 700 Fast (~10 min)
Image content comparison (pHash) 10 000 (6%) Slow(~7 hours)
Results
Current campaign:
Increase of statistically significance;
Reduction of dimensionality.
+ for future campaigns:
Less workload, more joy.
Votes were merged.

19. Preprocessing of images
Duplicate detection
Blur detection
Benchmarking
of algorithms for vote
aggregation
Outline of presentation
Noise reduction Prediction

20. A part of the dataset was annotated by an
expert after the campaign took place:
• 854 images;;
• 1813 volunteers;;
• 16 940 votes.
We sample two subsets for training and testing
70/30 ratio
Expert dataset

21. Baseline algorithms
We use SVD to reduce dimensionality and
10-­fold cross-­validation to fit parameters

22. A publicly available code implements KOS and EM algorithm;;
both are implemented in conjunction with reputation
algorithm (also called Hard penalty).
https://github.com/ashwin90/Penalty-­based-­clustering
[EM] Dawid, A.P., Skene, A.M.: Maximum likelihood estimation of
observer error-­rates using the em algorithm. Applied statistics pp.
20–28 (1979)
[Hard penalty] Jagabathula, S., et al.: Reputation-­based worker
filtering in crowdsourcing. In: Advances in Neural Information
Processing Systems. pp. 2492–2500 (2014)
[KOS] Karger, D.R., Oh, S., Shah, D.: Iterative learning for reliable
crowdsourcing systems. In: Advances in neural information
processing systems. pp. 1953–1961 (2011)

23. Weighted MV Heuristic
• We use weighted MV with weights equal
to reliabilities of volunteers
Reliability= 2 Pcorrect answer
-­ 1
• The heuristic is combined with iterative
removal of a volunteer with the highest
penalty [Hard penalty]. Then recalculate
penalties, and obtain new results for the
weighted MV.

24. Accuracy for ‘crowdsourcing’ algorithms
Baseline: AdaBoost (35 features)
91.08

25. Accuracy for ‘crowdsourcing’
algorithms with image thresholding.
Only images with
at least 10 votes
are left in the
expert dataset:
404 images,
1777 volunteers.

26. Possible explanations
Spammers
Malicious
Annotators
Good
Annotators
Biased
Annotators
Biased
Annotators
ROCs for all
1813 volunteers
on the expert dataset
Raykar, V.C.: Eliminating Spammers and Ranking Annotators for Crowdsourced
Labeling Tasks. JMLR 13, 491–518 (2012)

27. Volunteers’ ROCs
Spammers
Malicious
Annotators
Good
Annotators
Biased
Annotators
Biased
Annotators
Threshold = 12 votes, 262 volunteers.

28. Volunteers’ ROCs
Spammers
Malicious
Annotators
Good
Annotators
Biased
Annotators
Biased
Annotators
Threshold = 100 votes, 24 volunteers.

29. Accuracy
Due to image processing step
• accuracy of MV increased by 2%,
• workload decreased by 6 %.
Using threshold for voters and the heuristic,
we increase accuracy to 95.5%.

30. Conclusions
• Numerical experiments show that MV performs on a par
with all algorithms.
• Possible explanation: high accuracy of frequently voting
volunteers coupled with the absence of spammers.
• Image thresholding by number of votes helps to improve
the results of all algorithms similarly.
• To summarize, good annotators eliminate any advantages
of state of the art algorithms over MV.
• Image preprocessing is the only way to improve accuracy!

31. Future plans
• Development of theory
• More case studies in this field, hopefully
with spammers and malicious voters!
• Elaborate approach to ‘Maybe’ votes

33. The baseline
Ø We apply SVD to the whole dataset to reduce
dimensionality.
Ø We find an appropriate choice for the number of
features: 5, 14, 35.
Ø Transform the feature space of the testing and
training subsets accordingly.
Ø On the basis of 10-­fold cross-­validation of the training
subset, we fit parameters for the AdaBoost and
Random Forest algorithms. For Linear Discriminant
Analysis (LDA), we use default parameters.
Ø The accuracy of the algorithms with fitted parameters
was estimated using the testing subset.

34. Before and after merging of duplicates
Votes per image Unique votes
Repetition of images
(average)
Before [1 … 269] [1 … 88] 4
After [1…6042] [1…942] 13
900 volunteers
saw the same
image 7 times

35. Accuracy for ‘crowdsourcing’
algorithms with image thresholding.
Only images with
at least 4 votes
are left in the
expert dataset:
729 images,
1812 volunteers.

36. Volunteers’ ROCs
Spammers
Malicious
Annotators
Good
Annotators
Biased
Annotators
Biased
Annotators
Threshold = 44 votes, 52 volunteers.