This document summarizes a presentation about Mechanical Tsar, a web interface for microtask-based crowdsourcing. Mechanical Tsar is a crowdsourcing engine that runs microtasks, collects answers, and aggregates them. It has different front-ends like a web interface and Telegram. The presentation compares Mechanical Tsar to other platforms like TurKit and PyBossa. It also provides an example of how Mechanical Tsar was used to evaluate word relatedness as part of a Russian semantic study. The presentation concludes by encouraging cooperation on Mechanical Tsar and outlining plans to track user activity and try more workflows.

1. A Web Interface for
Microtask-Based
Crowdsourcing
Ilya Sukhopluev, Dmitry Ustalov
NLPub

2. Outline
•Introduction
•Related Work
•Mechanical Tsar
•Case Study:
RSR
•Conclusion

3. Introduction
•Everybody loves crowdsouring:
• data enrichment;
• data validation;
• solving “AI-hard” problems;
• making the world a better place.
•Crowdsourcing needs infrastructure.
•This talk is neither about AI nor
NLP…
…but we have the matter to discuss.

4. Related Work
•TurKit
• http://groups.csail.mit.edu/uid/turkit/
•psiTurk
• https://psiturk.org/
•Troia
• https://github.com/ipeirotis/Troia-
Server
•PyBossa
• http://pybossa.com/

5. Mechanical Tsar
•A crowdsourcing engine.
• Runs microtasks.
• Collects the answers.
• Aggregates them!
•Different front-ends:
• Web, Telegram, etc.
•http://mtsar.nlpub.org/
•https://nlpub.ru/Mechanical_Tsar

6. Architecture
• Mechanical Tsar
is the engine.
• Boyarin is a front-
end application.
• PostgreSQL is
the database.
• Piwik is the
telemetry system.

7. Example: RDT (2016)
•Continuation of the RUSSE study.
• http://russe.nlpub.ru/
•Evaluating word relatedness.
Panchenko A. et al. (2016) Human and Machine Judgements for
Russian Semantic Relatedness. To appear in Springer CCIS vol.

8. Mechanical Tsar: Stages

9. Mechanical Tsar: Setup

10. Boyarin: Annotation

11. Conclusion
•It is free. No reason not to use it.
•Cooperation wanted!!
•Plans:
• track & analyze the user activity;
• try more workflows.
•Cases:
• shared task annotation;
• private crowdsourcing.

12. Thank You!
•Dmitry Ustalov,
dmitry.ustalov@gmail.com
Usually I provide a LinkedIn link
here.
•http://mtsar.nlpub.org/
The reported study was funded by Russian Foundation for Basic
Research according to the research project № 16-37-00354 мол_а
“Adaptive Crowdsourcing Methods for Linguistic Resources”. This work was
supported by the Russian Foundation for the Humanities project № 13-
04-12020 “New Open Electronic Thesaurus for Russian” and project № 16-
04-12019 “RussNet and YARN thesauri integration”. The present work is
also supported by a short-term grant provided by the Deutscher
Akademischer Austauschdienst.