AWS Mechanical Turk Office Hours - Jan 2011


Published on

Slides presented at the Mechanical Turk Office Hours on January 20, 2011.

Video of the presentation can be found on the Mechanical Turk blog:

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

AWS Mechanical Turk Office Hours - Jan 2011

  1. 1. OFFICE HOURS<br />January 20, 2011<br />Presented by:<br />James Willeford, Principal Engineer<br />Adam Bradley, Sr. Engineer <br />Natala Menezes, Sr. Product Manager<br />
  2. 2. Welcome<br />We are here to participate in a conversation.<br />We want to hear your feedback &questions.<br />Ask a question with the questionpanel. <br />We will moderate questions. <br />Agenda<br />Introductions<br />Platform Review <br />Product & API Updates<br />Best Practices: HIT Design<br />Q & A<br />
  3. 3. Introductions<br />
  4. 4. How customers are using mTurk:<br />Categorization<br />Classification<br />Tagging <br />Sentiment Analysis<br />Data Management<br />Business Listing Verification by Phone or Web<br />Data Entry <br />Training Data (Algo) <br />Business Services<br />Product Testing<br />Surveys<br />Web Research <br />Search Relevancy<br />Content & Media<br />Photo & Content Moderation<br />Content Creation<br />Translation<br />Transcription <br />
  5. 5. How Requesters access mTurk:<br />
  6. 6. Applications & Solution Providers<br />Mechanical Turk supports a robust ecosystem. <br />
  7. 7. Product Updates<br />Worker Management <br />Released in October 2010<br />Search by Requester Name<br />Released in December 2010<br />Improved: In-progress Batch Approvals and Rejections<br />Feature extension that brings this functionality to upload/download via CSV in the Website UI<br />Released in January 2011<br />API Updates<br />
  8. 8. Worker Management<br />Makes it easier for you to find Workers who are good at your HITs.<br />Key Features:<br />Workers StatisticsData on who is doing work for you, their performance on that work.<br />Manage Qualification TypesCreate new Qualification Types in the UI and assign them to Workers to create Qualified Worker groups. <br />Worker Management ToolsBonus, Block and assign qualifications for a specific Worker.<br />
  9. 9. Workers Tab <br />See a list of all Workers who have submitted Work to you, their Qualification Types and Block status. Make modifications via CSV. <br />Customize your view by selecting the Qualification Types to be displayed (select up to 5). <br />New Tab! Found under the “Manage” tab. <br />Download a CSV of all your Workers. <br />You can assign a Qualification Type, change a Workers Qualification Score and modify their Block Status in the CSV and then Upload the CSV back to Mechanical Turk to process the changes. <br />Click on a Worker ID to take action on an individual Worker.<br />
  10. 10. Individual Worker Details<br />Access any worker with their ID at this URL:<br />Stats shown represent all work done to date (lifetime) for the specific Requester. . <br />Click the buttons to Bonus or Block this Worker. To send a bonus you will need the ID of an assignment the Worker has submitted to you.<br />Edit a Worker’s Qualification Score. <br />Revoke a Worker’s Qualification by clicking the icon.<br />
  11. 11. Qualifications <br />Create your own Qualification Types so you can route work to specific Workers. <br />See name, ID, # of Workers with the Qualification, creation date & description <br />Qualification Types with tests – can only be created via API. <br />New Tab! Found under the “Manage” tab. <br />Dispose by clicking the icon.<br />Disposing will permanently delete the Qualification. <br />
  12. 12. Search by Requester Name<br />Enter in the Requester’s name exactly. <br />You can also click on the Requesters name in a results set to see all work currently available from that Requester. <br />Bookmark this page to always see HITs available for a specific Requester<br />
  13. 13. API Best Practices<br />We’ve made a number of improvements to the error messages from the Mechanical Turk Requester API:<br />Check your errors! <br />Use Notifications instead of the polling interfaces<br />Dispose your HITS. <br />
  14. 14. Best Practices<br />HIT Design<br />
  15. 15. Best Practices: HIT Design <br />Clear & Concise instructions<br />Add Keywords to make your HIT searchable<br />Design an efficient HIT UX<br />Listen to feedback from Workers<br />Test & Iterate<br />
  16. 16. Same Workdifferent HITs = different results<br />
  17. 17. HIT Design Principles<br />Clear statement of reward (what’s in it for me, the Worker?) <br />Built in Validation – Worker knows when they are done, Requester can prevent incomplete work<br />Detailed & collapsible instructions<br />
  18. 18. How can I get more accurate results? <br />Plurality Asking multiple Workers to do assignments, if their answers agree then the result is validated)<br />Qualifying & Training Workers<br />Bonus payments<br />Known Data Sets <br />
  19. 19. Q & A<br />Your Questions. Answered. <br />
  20. 20. What is an invalid HIT? <br />Require disclosure of the Worker's identity or e-mail address, either directly or indirectly<br />Directly or indirectly promote a site, service, or opinion <br />Violate the terms and conditions of an activity or website (for instance asking Workers to vote for something)<br />Have explicit or offensive content, for instance, nudity, but do not have the Adult Content Qualification<br />Ask Workers to solicit third parties <br />Generate "referred" site visits or click through traffic<br />Ask workers to take action to manipulate a search engine's relationship data<br />Violate copyrights <br />X<br />X<br />X<br />
  21. 21. What can you do about invalid HITs? <br />Don’t do the work. <br />Report them. <br />
  22. 22. JC Goodrich<br />Capture the Worker ID from the URL<br />API Documentation <br />2 Scenarios:<br />Submit to Turk <br />Submit to Requester<br /><br />
  23. 23. How should I price my HIT? <br />Mechanical Turk is a Marketplace, you determine the offer for your own work – and the workers choose what work to do. <br />Test <br />Do it yourself: How long does it take? What would be a reasonable hourly wage? <br />Marketplace Test: Put in a sample HIT at different prices. How long does it take to get done? Quicker velocity = higher price, easy task = lower price <br />Many other factors also influence velocity (reputation, fun factor)<br />Combine results into one HIT – often lowers per item costs and reduces operational costs to review/approve. <br />Ex. Photo Moderation – put 5-50 photos in one HIT. <br />Compare with market pricing <br />
  24. 24. How can I attract good Workers?<br />Establish quick and fair payment. <br />Bonus your best Workers<br />Participate in the forums –<br />
  25. 25. Workers: How do I handle blocks?<br />Blocks prevent Workers from working for a Requester. <br />Mechanical Turk reviews blocks as a signal of quality. <br />
  26. 26. Q & A<br />Your Questions. Answered. <br />
  27. 27. Thank you. <br />@amazonmturk<br /><br /><br />