This document describes a crowdsourcing game developed to validate land cover maps. The game allows citizens to evaluate pixels where two land cover maps, GlobeLand30 and DUSAF, disagree on classification. Players validate pixel classifications to assess map accuracy. The game architecture connects a client, database, and geoserver. Accuracy assessments show disagreement percentages between the maps in Milan and Como, Italy. Evaluation will analyze player engagement and the game's ability to reach classification agreement on complicated pixels.

1. A Crowdsourcing-based game for
land cover validation
Brovelli, Maria Antonia(1); Celino, Irene (2); Molinari, Monia (1);
Venkatachalam, Vijay Charan (1)
(1) Politecnico di Milano, (2) CEFRIEL
bit.ly/foss4game

2. Problem Setting
● When different products are compared there are lots of disagreement
● Which product is correct ?
● Disagreement overall

3. Target
● To evaluate the classification quality of GlobeLand30 on the Italian area this
dataset has been compared with more accurate Italian land cover maps.
● To develop an interactive WebGIS with gaming environment for the
crowdsourcing-based validation of the non-coherent pixels.

4. DataSet
DUSAF(Destinazione d’Uso dei Suoli Agricoli e Forestali )
● Scale 1:10'000
● Updated through aerial photo interpretation.
● Currently available in five releases
http://www.ersaf.lombardia.it/servizi/Menu/dinamica.aspx?idArea=16914&idCat=17254&ID=17254
Global Land30
● 30 meters resolution
● Landsat (TM and ETM+) and HJ-1 satellites images .
● pixel-object-knowledge-based (POK-based)
http://globallandcover.com/GLC30Download/index.aspx

5. Citizen Science
● Public involvement
▪ To produce scientific data
▪ To conduct experiments

6. Human Computation
● Human Computation Systems aim to solve the problems that are trivial for
humans, but very challenging for computer programs.
● There are different types of Human computation systems
▪ Incentive by money - Amazon Mechanical Turk
▪ Incentive by fun - Games with a Purpose
▪ You have no other choices - CAPTCHA/reCAPTCHA

7. Land Cover Validation Game - Architecture
Land Cover Validation Game
Client ( JavaScript- Angular-Leaflet directives)
Web Portal
(PHP, Apache webserver with Linux)
Authentication
MySQL DatabaseQGIS
DUSAF
GL30
Disagreement
map
Geoserver
(CGR orthophotos)

8. Accuracy Assessment - Milano
Coherent
pixels
Non-coherent
pixels
Overall Accuracy
[%]
Disagreement
[%]
2000 173219 28758 85.76 14.24
2010 174604 27240 86.50 13.50
Ground Truth DUSAF 1.1
1 2 3 4 5
GL30
(2000)
1 134025 4020 989 26 267
2 11950 38139 4026 31 601
3 4677 2076 456 0 49
4 0 0 0 0 0
5 30 3 13 0 599
Ground Truth DUSAF 4.1
1 2 3 4 5
GL30
(2010)
1 138952 3199 557 0 441
2 12597 34425 2508 0 340
3 6799 635 371 0 37
4 0 0 0 0 0
5 89 26 12 0 856
CONTINGENCY TABLE

9. Coherent
pixels
Non-coherent
pixels
Overall Accuracy
[%]
Disagreem
ent
[%]
Sample
Size
2000 33416 8022 80.60 19.40 1340
2010 33318 8120 80.50 19.60 1208
CONTINGENCY TABLE
Accuracy Assessment - Como
Ground Truth DUSAF 4.1
1 2 3 4 5
GL30 (2010)
1 16629 1340 2626 0 131
2 917 1644 872 29 9
3 1177 835 12824 79 46
4 0 0 0 0 0
5 57 0 2 0 2221

10. LAND COVER VALIDATION GAME

11. Evaluation of Game
(Game ability to engage
Players)
(Game ability to find an
agreement on pixels’ classification)

12. User Analysis
● Reputation of the players

13. Pixel Analysis

14. Complicated Pixels

15. • The land cover validation game is a novel interactive system that allows citizen
scientists to validate land cover maps while having fun.
• The Confidence level of the GlobeLand30 and DUSAF dataset is obtained.
• The More the player , the higher the number of validated pixels.
Conclusion

16. Thank You and Play!
bit.ly/foss4game
maria.brovelli@polimi.it