Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
@mattlease
ml@utexas.edu
Beyond Mechanical Turk: An Analysis of
Paid Crowd Work Platforms
Donna Vakharia & Matthew Lease
U...
Roadmap
• A Brief History of Amazon’s Mechanical Turk
– Its rise to fame, enduring popularity, & strengths & limitations
•...
Amazon Mechanical Turk (MTurk)
• Online marketplace for paid crowd work (micro-tasks)
• Human labor on-demand as a program...
The Rise of Mechanical Turk
J. Pontin. Artificial Intelligence, With Help From the
Humans. New York Times, March 2007
Su e...
2008: Gold Rush begins in Comp. Sci.
Snow et al, EMNLP
• Annotating human language
• 22,000 labels for only US $26 !!!
• C...
MTurk for Social & Behavioral Sciences
• A Guide to Behavioral Experiments
on Mechanical Turk
– W. Mason and S. Suri (2010...
MTurk Strengths
• On-demand, 24-7, vast U.S. & Indian workforce
• Traditional economic friction and geographic
work barrie...
But also challenges (& much research)
• Reliability & data quality
– Risk & uncertainty may preclude adoption;
management ...
Amazon Mechanical Turk (MTurk)
• Online marketplace for paid crowd work (micro-tasks)
• Human labor on-demand as a program...
Matt Lease <ml@utexas.edu>
www.behind-the-enemy-lines.com/2010/10/plea-to-amazon-fix-mechanical-turk.html
10/20
Why Eytan Adar hates MTurk Research
(CHI 2011 CHC Workshop)
• Overly-narrow research focus on MTurk
– Distinguish general ...
Crowdsourcing ≠ Mechanical Turk!
Matt Lease <ml@utexas.edu> 12/20
What about volunteer crowdsourcing?
Matt Lease <ml@utexas.edu> 13/20
Many Crowd Work Platforms Today
And More!
JobBoy, microWorkers, MiniFreelance,
MiniJobz, MinuteWorkers, MyEasyTask,
OpTask...
The Future of Crowd Work
Paper @ ACM CSCW 2013 by
Kittur, Nickerson, Bernstein,
Gerber, Shaw, Zimmerman
Lease, & Horton
Ma...
Here: Qualitative Study of 7 Platforms
Matt Lease <ml@utexas.edu>
We shared preliminary findings with personnel from
all p...
Criteria for Assessing
Platform Capabilities
Matt Lease <ml@utexas.edu> 17/20
Capabilities Beyond MTurk (1 of 2)
• Basic platform operations mature beyond “beta”
– e.g. effective task search, preventi...
Capabilities Beyond MTurk (2 of 2)
• Truly global workforce & deeper geographic foci
– Still accepting new international w...
Summary
• Qualitative study of 7 platforms for crowd work
– Detailed analysis & comparison of current state-of-the-art
– H...
Thank You!
ir.ischool.utexas.eduSlides: www.slideshare.net/mattlease
Matt Lease <ml@utexas.edu> 22/20
AAAI HCOMP 2013 Industry Panel
Anand Kulkarni: “How do we
dramatically reduce the complexity of
getting work done with the...
Why Paid Crowd Work?
• Why can’t we just…
– …ask people to volunteer?
– …create fun games which produce work as a by-produ...
CACM August, 2013
Paul Hyman. Communications of the ACM, Vol. 56 No. 8, Pages 19-21, August 2013.
Matt Lease <ml@utexas.ed...
What about ethics?
• Silberman, Irani, and Ross (2010)
– “How should we… conceptualize the role of these
people who we ask...
What about Human Subjects Research?
•
“What are the characteristics of MTurk workers?... the MTurk
system is set up to str...
`
A MTurk worker’s ID is
also their customer
ID on Amazon. Public
profile pages can link
worker ID to name.
Lease et al., ...
Crowdsourcing
• Take a task traditionally
performed by a known agent
(often an employee)
• Advertise it to an undefined,
g...
The Unreasonable Effectiveness of Data
Banko and Brill (2001)
An AI system’s effectiveness in practice is often
limited by...
Matt Lease <ml@utexas.edu> 31/20
Upcoming SlideShare
Loading in …5
×

Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms

1,398 views

Published on

Joint work with Donna Vakharia, paper presentation at the 2015 iConference (March 26, 2015)

Published in: Technology
  • Be the first to comment

Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms

  1. 1. @mattlease ml@utexas.edu Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms Donna Vakharia & Matthew Lease University of Texas at Austin Talk slides: www.slideshare.net/mattlease
  2. 2. Roadmap • A Brief History of Amazon’s Mechanical Turk – Its rise to fame, enduring popularity, & strengths & limitations • A capability analysis of 7 alternative crowd work platforms – Ascertain distinguishing capabilities not found in MTurk Matt Lease <ml@utexas.edu> Take-away: Predominant use of Mechanical Turk in research on paid crowd work has let its unique vagaries & limitations unduly shape the broader crowdsourcing research agenda. Greater awareness of alternative crowd work platforms can accelerate practice, conceptual understanding, & progress. 2/20
  3. 3. Amazon Mechanical Turk (MTurk) • Online marketplace for paid crowd work (micro-tasks) • Human labor on-demand as a programmatic service • Launched in November 2005 Matt Lease <ml@utexas.edu> 3/20
  4. 4. The Rise of Mechanical Turk J. Pontin. Artificial Intelligence, With Help From the Humans. New York Times, March 2007 Su et al., WWW 2007: “a web-based human data collection system that we [call] ‘System M’ ” Matt Lease <ml@utexas.edu> 4/20
  5. 5. 2008: Gold Rush begins in Comp. Sci. Snow et al, EMNLP • Annotating human language • 22,000 labels for only US $26 !!! • Crowd’s consensus labels can replace traditional expert labels “Discovery” sparks rush for “gold” data across areas • Alonso et al., SIGIR Forum (Information Retrieval) • Kittur et al., CHI (Human-Computer Interaction) • Sorokin and Forsythe, CVPR (Computer Vision) Matt Lease <ml@utexas.edu> 5/20
  6. 6. MTurk for Social & Behavioral Sciences • A Guide to Behavioral Experiments on Mechanical Turk – W. Mason and S. Suri (2010). SSRN online. • Crowdsourcing for Human Subjects Research – L. Schmidt (CrowdConf 2010) • Crowdsourcing Content Analysis for Behavioral Research: Insights from Mechanical Turk – Conley & Tosti-Kharas (2010). Academy of Management • Amazon's Mechanical Turk : A New Source of Inexpensive, Yet High-Quality, Data? – M. Buhrmester et al. (2011). Perspectives… 6(1):3-5. – see also: Amazon Mechanical Turk Guide for Social Scientists Matt Lease <ml@utexas.edu> 6/20
  7. 7. MTurk Strengths • On-demand, 24-7, vast U.S. & Indian workforce • Traditional economic friction and geographic work barriers dramatically streamlined – Cheap & fast recruitment & job execution – Hiring/firing, communication, transfer, payment • Programmatic control via intermediary API • Worker pseudonyms and diverse participant pool amenable to human subjects research Matt Lease <ml@utexas.edu> 7/20
  8. 8. But also challenges (& much research) • Reliability & data quality – Risk & uncertainty may preclude adoption; management overhead may offset savings – Limited reputation system (no ratings/ reviews) – Real & perceived fraud in marketplace • Basic usage difficulties (eg. posting & search) • How to accomplish non-trivial tasks? – Lack of worker profiles (skills & interests) – interaction, collaboration, & workflow design • How to process sensitive data? – Private records, trade secrets, military use • How to assess and navigate ethics? Matt Lease <ml@utexas.edu> 8/20
  9. 9. Amazon Mechanical Turk (MTurk) • Online marketplace for paid crowd work (micro-tasks) • Human labor on-demand as a programmatic service • Launched in Nov. 2005, remains in “beta” in 2015 Matt Lease <ml@utexas.edu> 9/20
  10. 10. Matt Lease <ml@utexas.edu> www.behind-the-enemy-lines.com/2010/10/plea-to-amazon-fix-mechanical-turk.html 10/20
  11. 11. Why Eytan Adar hates MTurk Research (CHI 2011 CHC Workshop) • Overly-narrow research focus on MTurk – Distinguish general vs. platform-specific problems – Distinguish research vs. industry concerns • Should researchers really focus on… – “...writing the user’s manual for MTurk ...”? – “…struggl[ing] against the limits of the platform...”? “…by rewarding quick demonstrations of the tool’s use, we fail to attain a deeper understanding of the problems to which it is applied…” Matt Lease <ml@utexas.edu> 11/20
  12. 12. Crowdsourcing ≠ Mechanical Turk! Matt Lease <ml@utexas.edu> 12/20
  13. 13. What about volunteer crowdsourcing? Matt Lease <ml@utexas.edu> 13/20
  14. 14. Many Crowd Work Platforms Today And More! JobBoy, microWorkers, MiniFreelance, MiniJobz, MinuteWorkers, MyEasyTask, OpTask, ShortTask, SimpleWorkers Paid Crowd Work ≠ Mechanical Turk! Matt Lease <ml@utexas.edu> 14/20
  15. 15. The Future of Crowd Work Paper @ ACM CSCW 2013 by Kittur, Nickerson, Bernstein, Gerber, Shaw, Zimmerman Lease, & Horton Matt Lease <ml@utexas.edu> With diminished visibility and communication… workers may be treated as exchangeable and untrustworthy, having …. strong motivations to shirk. Workers may become equally cynical…” “…we surveyed a number of … popular crowd work platforms... Mechanical Turk, oDesk, Freelancer, Crowdflower, MobileWorks, ManPower…” But our platform coverage was pretty superficial… 15/20
  16. 16. Here: Qualitative Study of 7 Platforms Matt Lease <ml@utexas.edu> We shared preliminary findings with personnel from all platforms, requesting & incorporating feedback 16/20
  17. 17. Criteria for Assessing Platform Capabilities Matt Lease <ml@utexas.edu> 17/20
  18. 18. Capabilities Beyond MTurk (1 of 2) • Basic platform operations mature beyond “beta” – e.g. effective task search, preventing task starvation • Payment models beyond per-task completed – e.g., hourly pay or stronger models of contract work • Rich worker profiles & reputation systems – Some rigorously curate and assess workforce – Work quality, desired demographics, task routing • Rich interaction & collaboration support – Support for a variety of specific task verticals Matt Lease <ml@utexas.edu> 18/20
  19. 19. Capabilities Beyond MTurk (2 of 2) • Truly global workforce & deeper geographic foci – Still accepting new international workers… • Supporting confidential & bring-your-own crowds – Ability to crowdsource sensitive data • “Fair trade” models for work with career ladders – Explicit missions & designs for ethical work practices • Legal terms-of-use beyond marketplace broker – e.g., avoid risk of independent contractors vs. employees debate (Wolfson & Lease, ASIS&T’11) Matt Lease <ml@utexas.edu> 19/20
  20. 20. Summary • Qualitative study of 7 platforms for crowd work – Detailed analysis & comparison of current state-of-the-art – Highlighted key capabilities available beyond MTurk • Further inform current practice & future research Matt Lease <ml@utexas.edu> Take-away: Predominant use of Mechanical Turk in research on paid crowd work has let its unique vagaries & limitations unduly shape the broader crowdsourcing research agenda. Greater awareness of alternative crowd work platforms can accelerate practice, conceptual understanding, & progress. 20/20
  21. 21. Thank You! ir.ischool.utexas.eduSlides: www.slideshare.net/mattlease
  22. 22. Matt Lease <ml@utexas.edu> 22/20
  23. 23. AAAI HCOMP 2013 Industry Panel Anand Kulkarni: “How do we dramatically reduce the complexity of getting work done with the crowd?” Greg Little: How can we post a task and with 98% confidence know we’ll get a quality result? Matt Lease <ml@utexas.edu> 23/20
  24. 24. Why Paid Crowd Work? • Why can’t we just… – …ask people to volunteer? – …create fun games which produce work as a by-product? – …design services that produce work invisibly (e.g. re-Captcha)? • Established models of paying work exist for a reason – Even volunteers need to work to survive – Demand will likely always outstrip volunteer supply – There will always be forms of uninteresting but needed work • Volunteer approaches bring their own set of challenges… • But what about ethics? – Are volunteering approaches truly more ethical? – Since paying work isn’t going away, promote awareness & study Matt Lease <ml@utexas.edu> 24/20
  25. 25. CACM August, 2013 Paul Hyman. Communications of the ACM, Vol. 56 No. 8, Pages 19-21, August 2013. Matt Lease <ml@utexas.edu> 25/20
  26. 26. What about ethics? • Silberman, Irani, and Ross (2010) – “How should we… conceptualize the role of these people who we ask to power our computing?” – Power dynamics between parties • What are the consequences for a worker when your actions harm their reputation? – “Abstraction hides detail” • Fort, Adda, and Cohen (2011) – “…opportunities for our community to deliberately value ethics above cost savings.” Matt Lease <ml@utexas.edu> 26/20
  27. 27. What about Human Subjects Research? • “What are the characteristics of MTurk workers?... the MTurk system is set up to strictly protect workers’ anonymity….” Matt Lease <ml@utexas.edu> 27/20
  28. 28. ` A MTurk worker’s ID is also their customer ID on Amazon. Public profile pages can link worker ID to name. Lease et al., SSRN’13 Matt Lease <ml@utexas.edu> 28/20
  29. 29. Crowdsourcing • Take a task traditionally performed by a known agent (often an employee) • Advertise it to an undefined, generally large group of people via an open call Matt Lease <ml@utexas.edu> 29/20
  30. 30. The Unreasonable Effectiveness of Data Banko and Brill (2001) An AI system’s effectiveness in practice is often limited by the amount of available training data Matt Lease <ml@utexas.edu> 30/20
  31. 31. Matt Lease <ml@utexas.edu> 31/20

×