Your SlideShare is downloading. ×
0
AWS Mechanical Turk Office Hours - Jan 2011
AWS Mechanical Turk Office Hours - Jan 2011
AWS Mechanical Turk Office Hours - Jan 2011
AWS Mechanical Turk Office Hours - Jan 2011
AWS Mechanical Turk Office Hours - Jan 2011
AWS Mechanical Turk Office Hours - Jan 2011
AWS Mechanical Turk Office Hours - Jan 2011
AWS Mechanical Turk Office Hours - Jan 2011
AWS Mechanical Turk Office Hours - Jan 2011
AWS Mechanical Turk Office Hours - Jan 2011
AWS Mechanical Turk Office Hours - Jan 2011
AWS Mechanical Turk Office Hours - Jan 2011
AWS Mechanical Turk Office Hours - Jan 2011
AWS Mechanical Turk Office Hours - Jan 2011
AWS Mechanical Turk Office Hours - Jan 2011
AWS Mechanical Turk Office Hours - Jan 2011
AWS Mechanical Turk Office Hours - Jan 2011
AWS Mechanical Turk Office Hours - Jan 2011
AWS Mechanical Turk Office Hours - Jan 2011
AWS Mechanical Turk Office Hours - Jan 2011
AWS Mechanical Turk Office Hours - Jan 2011
AWS Mechanical Turk Office Hours - Jan 2011
AWS Mechanical Turk Office Hours - Jan 2011
AWS Mechanical Turk Office Hours - Jan 2011
AWS Mechanical Turk Office Hours - Jan 2011
AWS Mechanical Turk Office Hours - Jan 2011
AWS Mechanical Turk Office Hours - Jan 2011
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

AWS Mechanical Turk Office Hours - Jan 2011

2,424

Published on

Slides presented at the Mechanical Turk Office Hours on January 20, 2011. …

Slides presented at the Mechanical Turk Office Hours on January 20, 2011.

Video of the presentation can be found on the Mechanical Turk blog:
http://mechanicalturk.typepad.com

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
2,424
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
21
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. OFFICE HOURS<br />January 20, 2011<br />Presented by:<br />James Willeford, Principal Engineer<br />Adam Bradley, Sr. Engineer <br />Natala Menezes, Sr. Product Manager<br />
  • 2. Welcome<br />We are here to participate in a conversation.<br />We want to hear your feedback &amp;questions.<br />Ask a question with the questionpanel. <br />We will moderate questions. <br />Agenda<br />Introductions<br />Platform Review <br />Product &amp; API Updates<br />Best Practices: HIT Design<br />Q &amp; A<br />
  • 3. Introductions<br />
  • 4. How customers are using mTurk:<br />Categorization<br />Classification<br />Tagging <br />Sentiment Analysis<br />Data Management<br />Business Listing Verification by Phone or Web<br />Data Entry <br />Training Data (Algo) <br />Business Services<br />Product Testing<br />Surveys<br />Web Research <br />Search Relevancy<br />Content &amp; Media<br />Photo &amp; Content Moderation<br />Content Creation<br />Translation<br />Transcription <br />
  • 5. How Requesters access mTurk:<br />
  • 6. Applications &amp; Solution Providers<br />Mechanical Turk supports a robust ecosystem. <br />
  • 7. Product Updates<br />Worker Management <br />Released in October 2010<br />Search by Requester Name<br />Released in December 2010<br />Improved: In-progress Batch Approvals and Rejections<br />Feature extension that brings this functionality to upload/download via CSV in the Website UI<br />Released in January 2011<br />API Updates<br />
  • 8. Worker Management<br />Makes it easier for you to find Workers who are good at your HITs.<br />Key Features:<br />Workers StatisticsData on who is doing work for you, their performance on that work.<br />Manage Qualification TypesCreate new Qualification Types in the UI and assign them to Workers to create Qualified Worker groups. <br />Worker Management ToolsBonus, Block and assign qualifications for a specific Worker.<br />
  • 9. Workers Tab <br />See a list of all Workers who have submitted Work to you, their Qualification Types and Block status. Make modifications via CSV. <br />Customize your view by selecting the Qualification Types to be displayed (select up to 5). <br />New Tab! Found under the “Manage” tab. <br />Download a CSV of all your Workers. <br />You can assign a Qualification Type, change a Workers Qualification Score and modify their Block Status in the CSV and then Upload the CSV back to Mechanical Turk to process the changes. <br />Click on a Worker ID to take action on an individual Worker.<br />
  • 10. Individual Worker Details<br />Access any worker with their ID at this URL: https://requester.mturk.com/bulk/workers/workerID#<br />Stats shown represent all work done to date (lifetime) for the specific Requester. . <br />Click the buttons to Bonus or Block this Worker. To send a bonus you will need the ID of an assignment the Worker has submitted to you.<br />Edit a Worker’s Qualification Score. <br />Revoke a Worker’s Qualification by clicking the icon.<br />
  • 11. Qualifications <br />Create your own Qualification Types so you can route work to specific Workers. <br />See name, ID, # of Workers with the Qualification, creation date &amp; description <br />Qualification Types with tests – can only be created via API. <br />New Tab! Found under the “Manage” tab. <br />Dispose by clicking the icon.<br />Disposing will permanently delete the Qualification. <br />
  • 12. Search by Requester Name<br />Enter in the Requester’s name exactly. <br />You can also click on the Requesters name in a results set to see all work currently available from that Requester. <br />Bookmark this page to always see HITs available for a specific Requester<br />
  • 13. API Best Practices<br />We’ve made a number of improvements to the error messages from the Mechanical Turk Requester API:<br />Check your errors! <br />Use Notifications instead of the polling interfaces<br />Dispose your HITS. <br />
  • 14. Best Practices<br />HIT Design<br />
  • 15. Best Practices: HIT Design <br />Clear &amp; Concise instructions<br />Add Keywords to make your HIT searchable<br />Design an efficient HIT UX<br />Listen to feedback from Workers<br />Test &amp; Iterate<br />
  • 16. Same Workdifferent HITs = different results<br />
  • 17. HIT Design Principles<br />Clear statement of reward (what’s in it for me, the Worker?) <br />Built in Validation – Worker knows when they are done, Requester can prevent incomplete work<br />Detailed &amp; collapsible instructions<br />
  • 18. How can I get more accurate results? <br />Plurality Asking multiple Workers to do assignments, if their answers agree then the result is validated)<br />Qualifying &amp; Training Workers<br />Bonus payments<br />Known Data Sets <br />
  • 19. Q &amp; A<br />Your Questions. Answered. <br />
  • 20. What is an invalid HIT? <br />Require disclosure of the Worker&apos;s identity or e-mail address, either directly or indirectly<br />Directly or indirectly promote a site, service, or opinion <br />Violate the terms and conditions of an activity or website (for instance asking Workers to vote for something)<br />Have explicit or offensive content, for instance, nudity, but do not have the Adult Content Qualification<br />Ask Workers to solicit third parties <br />Generate &quot;referred&quot; site visits or click through traffic<br />Ask workers to take action to manipulate a search engine&apos;s relationship data<br />Violate copyrights <br />X<br />X<br />X<br />
  • 21. What can you do about invalid HITs? <br />Don’t do the work. <br />Report them. <br />
  • 22. JC Goodrich<br />Capture the Worker ID from the URL<br />API Documentation <br />2 Scenarios:<br />Submit to Turk <br />Submit to Requester<br />http://www.quora.com/Natala-Menezes/Can-you-explain-more-about-how-a-best-practice-iFrame-HIT-in-Mechanical-Turk-works<br />
  • 23. How should I price my HIT? <br />Mechanical Turk is a Marketplace, you determine the offer for your own work – and the workers choose what work to do. <br />Test <br />Do it yourself: How long does it take? What would be a reasonable hourly wage? <br />Marketplace Test: Put in a sample HIT at different prices. How long does it take to get done? Quicker velocity = higher price, easy task = lower price <br />Many other factors also influence velocity (reputation, fun factor)<br />Combine results into one HIT – often lowers per item costs and reduces operational costs to review/approve. <br />Ex. Photo Moderation – put 5-50 photos in one HIT. <br />Compare with market pricing <br />
  • 24. How can I attract good Workers?<br />Establish quick and fair payment. <br />Bonus your best Workers<br />Participate in the forums – Turkernation.com<br />
  • 25. Workers: How do I handle blocks?<br />Blocks prevent Workers from working for a Requester. <br />Mechanical Turk reviews blocks as a signal of quality. <br />
  • 26. Q &amp; A<br />Your Questions. Answered. <br />
  • 27. Thank you. <br />@amazonmturk<br />Facebook.com/amazonmturk<br />Mechanicalturk.typepad.com<br />

×