Gaining participants for research is hard! Increasingly asked questions based on my role as Chair of Research Ethics but also as a researcher dealing in digital research methods. 2 key ideas to outline in terms of research realities and ethics – the increasing use of participant platforms in academic research and the use of crowdsourced research data through vehicles such as Amazon’s mechanical Turk
Mturk is an Amazon owned ‘distributed workforce service’ or crowdsourced data gathering. Which connects those who wish to work or their employers with tasks that can be completed on line, very often online surveys and online experiments. Commonly used in Social sciences, psychology, consumer studies, economics, medicine. Third party group involvement whereby payment/organisation goes to a company who then uses its workers to complete the tasks.
Issues – loss of control of sample, routine answering even if meeting sample criteria, High number of Indian, American and Chinese ‘turkers’ – workers. Very low wages per hour ABOUT $3 PERHOUR. – BUT LACK OF TRANSPARENCY IN THIS as surveys are on a piece basis.- as source of employment versus ‘good way to kill time and earn something’ USA. Turkers are frequently not one time participants in a single academic study Mturk data may not be supported through research grants in some institutions in some countries _ IRBs at certain US and Australian and Canadian universities
Provides CHEAP DATA, FAST DATA, ACCEPTABEL QUALITY(??)
Dholakis – Marketing professor - 50 Studies published using this data collection tool – gained new perspective after becoming a Turker… DATA OFTEN REJECTED BY SYSTEM AFTER COMPLETION = NO PAYMENT NO INFOEMDED CONSENT ON SOME SURVEYS POSTED NO RESEARCH ‘OWNER’ WAS IDENTIFIED IN PLACES – COMPANY OR UNI - NO AFFILIATIONS POOR DESIGN OF RESEARCH CREATING QUALTY PROBLEMS – SCALES ETC No feedback or communication possible no dissemination of results Professionalism of turkers… In future will only use for pretests in future?? And exploratory research?????
Mturk raises multiple ethical issues in the research context which are worth highlighting.
To what extent is it possible to inform those participating about how the data will be used and also the outcome of the research?
Type 1.which may have open or closed access and usually requires an university email address Type 2such as the Oxford University Software incubator firm Type 3 or IRBs (Institution Research or Review Boards in the USA)
All have differing types of payment models. Some are free to upload your questionnaire and each response costs the researcher a few pence. Some give the participants a rewards system where they accumulate points and then get a discount voucher .Another type charges per questionnaire completion. Another version is subscription based with differing levels of access of the data created. Paypal often used as payment intermediary.
Many universities are concerned with who owns the data, where the data is stored, the stability of the platforms – data loss or data corruption If intending to use look out for AoiR, ESOMAr, MRS compliance logos. Lookout for participant screening mechanisms also
Self-limiting pool of particpants, may not be representative or might be. Students and the retired Cash referral schemes offered by some platforms are likely to further bias participant pool.
Participant platforms and response data
Pondering Participant Platforms and
getting response data
Questions for researchers
Dr Sarah Quinton 20/20
Mechanical Turk, digital sweatshop
labour or digital research assistance?
• Do those completing the task have free choice in their
• Have those completing the task been informed about the
context of the research and the use of the data?
• Will there be fair payment to those completing the tasks,
when will they be paid etc?
• Can the level of anonymity required be guaranteed?
• Sampling frames may be very distorted and inaccurate or
even unknown to the researchers.
• How can researchers using Mturk as a data collection
tool mitigate the ethical issues?
Third party participant platforms for
• In order of Oxford Brookes’s research ethics preference
• 1) A research institute or university platform designed to
facilitate research (http://www.callforparticipants.com)
• 2) A spin off from number one developed as a small
• 3) Commercial platforms aimed at academic research
which may state authorisation by university ethics