Introduction and Applications<br />
   Topics<br /><ul><li>Overview
Designing HITs
Controlling for quality
Fine print
Research with AMT
Resources</li></li></ul><li>   Topics<br /><ul><li>Overview
Designing HITs
Controlling for quality
Fine print
Research with AMT
Resources</li></li></ul><li>   What’s it?<br />Amazon Mechanical Turk is a crowdsourcing  Internet marketplace that enable...
   https://www.mturk.com/<br />
   Workers get paid to answer stuff.<br />
   Requesters pay to ask stuff.<br />
   Why use it?<br />Advantages of Mechanical Turk*:<br /><ul><li>low cost ($0.10 per 60-second task)
subject pool size
subject pool diversity
faster theory/experiment cycle</li></ul>* Mason, W. and Siddarth, S. 2009. “Conducting Behavioral Research on Amazon’s Mec...
   Topics<br /><ul><li>Overview
Designing HITs
Quality assurance
Fine print
Research with AMT
Resources</li></li></ul><li>   Human Intelligence Tasks (HITs)<br />Requesters can specify: <br /><ul><li>task
Upcoming SlideShare
Loading in …5
×

Introduction to Amazon Mechanical Turk

3,925 views

Published on

I composed this brief introduction to Amazon's crowdsourcing tool, Mechanical Turk (AMT), at the request of a professor who advocates AMT for marketing research.

Published in: Education
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,925
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
85
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • guidelines for pricing
  • Give numbers
  • Turk demographics
  • Introduction to Amazon Mechanical Turk

    1. 1. Introduction and Applications<br />
    2. 2. Topics<br /><ul><li>Overview
    3. 3. Designing HITs
    4. 4. Controlling for quality
    5. 5. Fine print
    6. 6. Research with AMT
    7. 7. Resources</li></li></ul><li> Topics<br /><ul><li>Overview
    8. 8. Designing HITs
    9. 9. Controlling for quality
    10. 10. Fine print
    11. 11. Research with AMT
    12. 12. Resources</li></li></ul><li> What’s it?<br />Amazon Mechanical Turk is a crowdsourcing Internet marketplace that enables computer programmers (Requesters) to coordinate the use of human intelligence to perform tasks which computers are unable to do.<br />
    13. 13. https://www.mturk.com/<br />
    14. 14. Workers get paid to answer stuff.<br />
    15. 15. Requesters pay to ask stuff.<br />
    16. 16. Why use it?<br />Advantages of Mechanical Turk*:<br /><ul><li>low cost ($0.10 per 60-second task)
    17. 17. subject pool size
    18. 18. subject pool diversity
    19. 19. faster theory/experiment cycle</li></ul>* Mason, W. and Siddarth, S. 2009. “Conducting Behavioral Research on Amazon’s Mechanical Turk.” Yahoo! Research.<br />
    20. 20. Topics<br /><ul><li>Overview
    21. 21. Designing HITs
    22. 22. Quality assurance
    23. 23. Fine print
    24. 24. Research with AMT
    25. 25. Resources</li></li></ul><li> Human Intelligence Tasks (HITs)<br />Requesters can specify: <br /><ul><li>task
    26. 26. keywords
    27. 27. expiration date
    28. 28. reward
    29. 29. time allotted
    30. 30. worker qualifications
    31. 31. location
    32. 32. approval rating</li></li></ul><li> Example HITs:<br /> “Identify forward-facing pictures of dogs”<br />“Find and enter a business address”<br />“How attractive are these items?”<br />“Choose the best category for this product”<br />“Read a set of Tweets and decide if they describe an event”<br />
    33. 33. Designing HITs<br />Options:<br />Use template or code within Mechanical Turk <br /><ul><li>requires basic HTML</li></ul> Link to external survey<br /><ul><li>need some kind of verification input on Mechanical Turk side after Workers complete external survey </li></li></ul><li> Designing HITs<br />Options:<br />Use template or code within Mechanical Turk <br /><ul><li>requires basic HTML</li></ul> Link to external survey<br /><ul><li>need some kind of verification input on Mechanical Turk side after workers complete external survey </li></li></ul><li> Topics<br /><ul><li>Overview
    34. 34. Designing HITs
    35. 35. Quality assurance
    36. 36. Fine print
    37. 37. Research with AMT
    38. 38. Resources</li></li></ul><li> Test your HITs<br />Use the Mechanical Turk Requester sandbox <br /><ul><li>view your published your HIT before releasing it to Workers
    39. 39. check for functionality in all browsers</li></li></ul><li> Help Workers work efficiently<br />Don’t make them scroll and click a lot<br />Set pages to open in new windows<br />
    40. 40. Manage Worker accuracy<br />Reject bad work<br />Reward good work with bonuses<br />Filter Workers by qualifications<br /><ul><li>design a test for specific knowledge
    41. 41. location
    42. 42. approvals (percentage or number)
    43. 43. adult content</li></li></ul><li> Topics<br /><ul><li>Overview
    44. 44. Designing HITs
    45. 45. Quality assurance
    46. 46. Fine print
    47. 47. Research with AMT
    48. 48. Resources</li></li></ul><li> Payment information <br />HITs must be prepaid to a Mechanical Turk account<br /><ul><li>Amazon collects 10% on top of what you pay to Workers
    49. 49. Amazon collects 10% of any bonuses you grant
    50. 50. Payment to Workers can be in money or Amazon credit</li></ul>You’re ultimately only charged for approved HITs, but you’re liable for the amount equal to whatever 100% approved HITs would cost.<br />You’ll have to deal with tax stuff if Workers do enough work for you to meet the IRS threshold for taxable income.<br />Click for prepaid terms and tax information.<br />
    51. 51. What You Can’t Do<br />Not warn Workers about explicit or offensive content<br /><ul><li>label HIT appropriately
    52. 52. require and check for Adult Content Qualification in order to complete AND preview HIT</li></ul>Collect personally identifiable information<br />Ask for this:<br /><ul><li>e-mail address
    53. 53. registration at another site
    54. 54. downloading of software</li></ul>Click to view complete policies.<br />
    55. 55. Topics<br /><ul><li>Overview
    56. 56. Designing HITs
    57. 57. Quality assurance
    58. 58. Payments
    59. 59. Research with AMT
    60. 60. Resources</li></li></ul><li> Accurate replications of classics <br />Tversky and Kahneman:<br /><ul><li>framing - “Asian disease problem”
    61. 61. anchoring – “African countries problem”
    62. 62. conjunction fallacy – “Linda problem”</li></ul>Loewenstein and Prelec:<br /><ul><li>fancy French dinner</li></ul>Kirby and Marakovic<br /><ul><li>hyperbolic discounting</li></ul>More at: http://experimentalturk.wordpress.com/<br />
    63. 63. Extensions<br />Easy cross-cultural research<br /><ul><li>tasks can be confined to workers living in specific countries
    64. 64. lots of Indians
    65. 65. Eriksson and Simpson (2010) </li></li></ul><li> Topics<br /><ul><li>Overview
    66. 66. Designing HITs
    67. 67. Quality assurance
    68. 68. Payments
    69. 69. Research with AMT
    70. 70. Resources</li></li></ul><li> References<br />Amazon FAQs and tutorials<br /><ul><li>Mechanical Turk FAQ
    71. 71. Videos & Webinars
    72. 72. Participation Agreement</li></ul>Literature on Mechanical Turk<br /><ul><li>Mason, W. and Siddarth, S. 2009. “Conducting Behavioral Research on Amazon’s Mechanical Turk.” Yahoo! Research.</li></ul>Research using Mechanical Turk<br /><ul><li>See: http://experimentalturk.wordpress.com/</li>

    ×