Slides presented at the Mechanical Turk Office Hours on January 20, 2011. Video of the presentation can be found on the Mechanical Turk blog: http://mechanicalturk.typepad.com

1. OFFICE HOURS January 20, 2011 Presented by: James Willeford, Principal Engineer Adam Bradley, Sr. Engineer Natala Menezes, Sr. Product Manager

2. Welcome We are here to participate in a conversation. We want to hear your feedback &questions. Ask a question with the questionpanel. We will moderate questions. Agenda Introductions Platform Review Product & API Updates Best Practices: HIT Design Q & A

3. Introductions

4. How customers are using mTurk: Categorization Classification Tagging Sentiment Analysis Data Management Business Listing Verification by Phone or Web Data Entry Training Data (Algo) Business Services Product Testing Surveys Web Research Search Relevancy Content & Media Photo & Content Moderation Content Creation Translation Transcription

5. How Requesters access mTurk:

6. Applications & Solution Providers Mechanical Turk supports a robust ecosystem.

7. Product Updates Worker Management Released in October 2010 Search by Requester Name Released in December 2010 Improved: In-progress Batch Approvals and Rejections Feature extension that brings this functionality to upload/download via CSV in the Website UI Released in January 2011 API Updates

8. Worker Management Makes it easier for you to find Workers who are good at your HITs. Key Features: Workers StatisticsData on who is doing work for you, their performance on that work. Manage Qualification TypesCreate new Qualification Types in the UI and assign them to Workers to create Qualified Worker groups. Worker Management ToolsBonus, Block and assign qualifications for a specific Worker.

9. Workers Tab See a list of all Workers who have submitted Work to you, their Qualification Types and Block status. Make modifications via CSV. Customize your view by selecting the Qualification Types to be displayed (select up to 5). New Tab! Found under the “Manage” tab. Download a CSV of all your Workers. You can assign a Qualification Type, change a Workers Qualification Score and modify their Block Status in the CSV and then Upload the CSV back to Mechanical Turk to process the changes. Click on a Worker ID to take action on an individual Worker.

10. Individual Worker Details Access any worker with their ID at this URL: https://requester.mturk.com/bulk/workers/workerID# Stats shown represent all work done to date (lifetime) for the specific Requester. . Click the buttons to Bonus or Block this Worker. To send a bonus you will need the ID of an assignment the Worker has submitted to you. Edit a Worker’s Qualification Score. Revoke a Worker’s Qualification by clicking the icon.

11. Qualifications Create your own Qualification Types so you can route work to specific Workers. See name, ID, # of Workers with the Qualification, creation date & description Qualification Types with tests – can only be created via API. New Tab! Found under the “Manage” tab. Dispose by clicking the icon. Disposing will permanently delete the Qualification.

12. Search by Requester Name Enter in the Requester’s name exactly. You can also click on the Requesters name in a results set to see all work currently available from that Requester. Bookmark this page to always see HITs available for a specific Requester

13. API Best Practices We’ve made a number of improvements to the error messages from the Mechanical Turk Requester API: Check your errors! Use Notifications instead of the polling interfaces Dispose your HITS.

14. Best Practices HIT Design

15. Best Practices: HIT Design Clear & Concise instructions Add Keywords to make your HIT searchable Design an efficient HIT UX Listen to feedback from Workers Test & Iterate

16. Same Workdifferent HITs = different results

17. HIT Design Principles Clear statement of reward (what’s in it for me, the Worker?) Built in Validation – Worker knows when they are done, Requester can prevent incomplete work Detailed & collapsible instructions

18. How can I get more accurate results? Plurality Asking multiple Workers to do assignments, if their answers agree then the result is validated) Qualifying & Training Workers Bonus payments Known Data Sets

19. Q & A Your Questions. Answered.

20. What is an invalid HIT? Require disclosure of the Worker's identity or e-mail address, either directly or indirectly Directly or indirectly promote a site, service, or opinion Violate the terms and conditions of an activity or website (for instance asking Workers to vote for something) Have explicit or offensive content, for instance, nudity, but do not have the Adult Content Qualification Ask Workers to solicit third parties Generate "referred" site visits or click through traffic Ask workers to take action to manipulate a search engine's relationship data Violate copyrights X X X

21. What can you do about invalid HITs? Don’t do the work. Report them.

22. JC Goodrich Capture the Worker ID from the URL API Documentation 2 Scenarios: Submit to Turk Submit to Requester http://www.quora.com/Natala-Menezes/Can-you-explain-more-about-how-a-best-practice-iFrame-HIT-in-Mechanical-Turk-works

23. How should I price my HIT? Mechanical Turk is a Marketplace, you determine the offer for your own work – and the workers choose what work to do. Test Do it yourself: How long does it take? What would be a reasonable hourly wage? Marketplace Test: Put in a sample HIT at different prices. How long does it take to get done? Quicker velocity = higher price, easy task = lower price Many other factors also influence velocity (reputation, fun factor) Combine results into one HIT – often lowers per item costs and reduces operational costs to review/approve. Ex. Photo Moderation – put 5-50 photos in one HIT. Compare with market pricing

24. How can I attract good Workers? Establish quick and fair payment. Bonus your best Workers Participate in the forums – Turkernation.com

25. Workers: How do I handle blocks? Blocks prevent Workers from working for a Requester. Mechanical Turk reviews blocks as a signal of quality.

26. Q & A Your Questions. Answered.

27. Thank you. @amazonmturk Facebook.com/amazonmturk Mechanicalturk.typepad.com