Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Introduction and Applications<br />
   Topics<br /><ul><li>Overview
Designing HITs
Controlling for quality
Fine print
Research with AMT
Resources</li></li></ul><li>   Topics<br /><ul><li>Overview
Designing HITs
Controlling for quality
Fine print
Research with AMT
Resources</li></li></ul><li>   What’s it?<br />Amazon Mechanical Turk is a crowdsourcing  Internet marketplace that enable...
   https://www.mturk.com/<br />
   Workers get paid to answer stuff.<br />
   Requesters pay to ask stuff.<br />
   Why use it?<br />Advantages of Mechanical Turk*:<br /><ul><li>low cost ($0.10 per 60-second task)
subject pool size
subject pool diversity
faster theory/experiment cycle</li></ul>* Mason, W. and Siddarth, S. 2009. “Conducting Behavioral Research on Amazon’s Mec...
   Topics<br /><ul><li>Overview
Designing HITs
Quality assurance
Fine print
Research with AMT
Resources</li></li></ul><li>   Human Intelligence Tasks (HITs)<br />Requesters can specify: <br /><ul><li>task
Upcoming SlideShare
Loading in …5
×

Introduction to Amazon Mechanical Turk

4,460 views

Published on

I composed this brief introduction to Amazon's crowdsourcing tool, Mechanical Turk (AMT), at the request of a professor who advocates AMT for marketing research.

Published in: Education
  • How You Can Go From Dead Broke To Abundantly Wealthy By Using One Simple Mind Hack... ♣♣♣ https://tinyurl.com/y44vwbuh
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • I made $2,600 with this. I already have 7 days with this... ★★★ https://tinyurl.com/realmoneystreams2019
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Earn $500 for taking a 1 hour paid survey! read more...  https://tinyurl.com/realmoneystreams2019
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Introduction to Amazon Mechanical Turk

  1. 1. Introduction and Applications<br />
  2. 2. Topics<br /><ul><li>Overview
  3. 3. Designing HITs
  4. 4. Controlling for quality
  5. 5. Fine print
  6. 6. Research with AMT
  7. 7. Resources</li></li></ul><li> Topics<br /><ul><li>Overview
  8. 8. Designing HITs
  9. 9. Controlling for quality
  10. 10. Fine print
  11. 11. Research with AMT
  12. 12. Resources</li></li></ul><li> What’s it?<br />Amazon Mechanical Turk is a crowdsourcing Internet marketplace that enables computer programmers (Requesters) to coordinate the use of human intelligence to perform tasks which computers are unable to do.<br />
  13. 13. https://www.mturk.com/<br />
  14. 14. Workers get paid to answer stuff.<br />
  15. 15. Requesters pay to ask stuff.<br />
  16. 16. Why use it?<br />Advantages of Mechanical Turk*:<br /><ul><li>low cost ($0.10 per 60-second task)
  17. 17. subject pool size
  18. 18. subject pool diversity
  19. 19. faster theory/experiment cycle</li></ul>* Mason, W. and Siddarth, S. 2009. “Conducting Behavioral Research on Amazon’s Mechanical Turk.” Yahoo! Research.<br />
  20. 20. Topics<br /><ul><li>Overview
  21. 21. Designing HITs
  22. 22. Quality assurance
  23. 23. Fine print
  24. 24. Research with AMT
  25. 25. Resources</li></li></ul><li> Human Intelligence Tasks (HITs)<br />Requesters can specify: <br /><ul><li>task
  26. 26. keywords
  27. 27. expiration date
  28. 28. reward
  29. 29. time allotted
  30. 30. worker qualifications
  31. 31. location
  32. 32. approval rating</li></li></ul><li> Example HITs:<br /> “Identify forward-facing pictures of dogs”<br />“Find and enter a business address”<br />“How attractive are these items?”<br />“Choose the best category for this product”<br />“Read a set of Tweets and decide if they describe an event”<br />
  33. 33. Designing HITs<br />Options:<br />Use template or code within Mechanical Turk <br /><ul><li>requires basic HTML</li></ul> Link to external survey<br /><ul><li>need some kind of verification input on Mechanical Turk side after Workers complete external survey </li></li></ul><li> Designing HITs<br />Options:<br />Use template or code within Mechanical Turk <br /><ul><li>requires basic HTML</li></ul> Link to external survey<br /><ul><li>need some kind of verification input on Mechanical Turk side after workers complete external survey </li></li></ul><li> Topics<br /><ul><li>Overview
  34. 34. Designing HITs
  35. 35. Quality assurance
  36. 36. Fine print
  37. 37. Research with AMT
  38. 38. Resources</li></li></ul><li> Test your HITs<br />Use the Mechanical Turk Requester sandbox <br /><ul><li>view your published your HIT before releasing it to Workers
  39. 39. check for functionality in all browsers</li></li></ul><li> Help Workers work efficiently<br />Don’t make them scroll and click a lot<br />Set pages to open in new windows<br />
  40. 40. Manage Worker accuracy<br />Reject bad work<br />Reward good work with bonuses<br />Filter Workers by qualifications<br /><ul><li>design a test for specific knowledge
  41. 41. location
  42. 42. approvals (percentage or number)
  43. 43. adult content</li></li></ul><li> Topics<br /><ul><li>Overview
  44. 44. Designing HITs
  45. 45. Quality assurance
  46. 46. Fine print
  47. 47. Research with AMT
  48. 48. Resources</li></li></ul><li> Payment information <br />HITs must be prepaid to a Mechanical Turk account<br /><ul><li>Amazon collects 10% on top of what you pay to Workers
  49. 49. Amazon collects 10% of any bonuses you grant
  50. 50. Payment to Workers can be in money or Amazon credit</li></ul>You’re ultimately only charged for approved HITs, but you’re liable for the amount equal to whatever 100% approved HITs would cost.<br />You’ll have to deal with tax stuff if Workers do enough work for you to meet the IRS threshold for taxable income.<br />Click for prepaid terms and tax information.<br />
  51. 51. What You Can’t Do<br />Not warn Workers about explicit or offensive content<br /><ul><li>label HIT appropriately
  52. 52. require and check for Adult Content Qualification in order to complete AND preview HIT</li></ul>Collect personally identifiable information<br />Ask for this:<br /><ul><li>e-mail address
  53. 53. registration at another site
  54. 54. downloading of software</li></ul>Click to view complete policies.<br />
  55. 55. Topics<br /><ul><li>Overview
  56. 56. Designing HITs
  57. 57. Quality assurance
  58. 58. Payments
  59. 59. Research with AMT
  60. 60. Resources</li></li></ul><li> Accurate replications of classics <br />Tversky and Kahneman:<br /><ul><li>framing - “Asian disease problem”
  61. 61. anchoring – “African countries problem”
  62. 62. conjunction fallacy – “Linda problem”</li></ul>Loewenstein and Prelec:<br /><ul><li>fancy French dinner</li></ul>Kirby and Marakovic<br /><ul><li>hyperbolic discounting</li></ul>More at: http://experimentalturk.wordpress.com/<br />
  63. 63. Extensions<br />Easy cross-cultural research<br /><ul><li>tasks can be confined to workers living in specific countries
  64. 64. lots of Indians
  65. 65. Eriksson and Simpson (2010) </li></li></ul><li> Topics<br /><ul><li>Overview
  66. 66. Designing HITs
  67. 67. Quality assurance
  68. 68. Payments
  69. 69. Research with AMT
  70. 70. Resources</li></li></ul><li> References<br />Amazon FAQs and tutorials<br /><ul><li>Mechanical Turk FAQ
  71. 71. Videos & Webinars
  72. 72. Participation Agreement</li></ul>Literature on Mechanical Turk<br /><ul><li>Mason, W. and Siddarth, S. 2009. “Conducting Behavioral Research on Amazon’s Mechanical Turk.” Yahoo! Research.</li></ul>Research using Mechanical Turk<br /><ul><li>See: http://experimentalturk.wordpress.com/</li>

×