Your SlideShare is downloading. ×
AWS Mechanical Turk Office Hours - Jan 2011
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

AWS Mechanical Turk Office Hours - Jan 2011


Published on

Slides presented at the Mechanical Turk Office Hours on January 20, 2011. …

Slides presented at the Mechanical Turk Office Hours on January 20, 2011.

Video of the presentation can be found on the Mechanical Turk blog:

Published in: Technology

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


    January 20, 2011
    Presented by:
    James Willeford, Principal Engineer
    Adam Bradley, Sr. Engineer
    Natala Menezes, Sr. Product Manager
  • 2. Welcome
    We are here to participate in a conversation.
    We want to hear your feedback &questions.
    Ask a question with the questionpanel.
    We will moderate questions.
    Platform Review
    Product & API Updates
    Best Practices: HIT Design
    Q & A
  • 3. Introductions
  • 4. How customers are using mTurk:
    Sentiment Analysis
    Data Management
    Business Listing Verification by Phone or Web
    Data Entry
    Training Data (Algo)
    Business Services
    Product Testing
    Web Research
    Search Relevancy
    Content & Media
    Photo & Content Moderation
    Content Creation
  • 5. How Requesters access mTurk:
  • 6. Applications & Solution Providers
    Mechanical Turk supports a robust ecosystem.
  • 7. Product Updates
    Worker Management
    Released in October 2010
    Search by Requester Name
    Released in December 2010
    Improved: In-progress Batch Approvals and Rejections
    Feature extension that brings this functionality to upload/download via CSV in the Website UI
    Released in January 2011
    API Updates
  • 8. Worker Management
    Makes it easier for you to find Workers who are good at your HITs.
    Key Features:
    Workers StatisticsData on who is doing work for you, their performance on that work.
    Manage Qualification TypesCreate new Qualification Types in the UI and assign them to Workers to create Qualified Worker groups.
    Worker Management ToolsBonus, Block and assign qualifications for a specific Worker.
  • 9. Workers Tab
    See a list of all Workers who have submitted Work to you, their Qualification Types and Block status. Make modifications via CSV.
    Customize your view by selecting the Qualification Types to be displayed (select up to 5).
    New Tab! Found under the “Manage” tab.
    Download a CSV of all your Workers.
    You can assign a Qualification Type, change a Workers Qualification Score and modify their Block Status in the CSV and then Upload the CSV back to Mechanical Turk to process the changes.
    Click on a Worker ID to take action on an individual Worker.
  • 10. Individual Worker Details
    Access any worker with their ID at this URL:
    Stats shown represent all work done to date (lifetime) for the specific Requester. .
    Click the buttons to Bonus or Block this Worker. To send a bonus you will need the ID of an assignment the Worker has submitted to you.
    Edit a Worker’s Qualification Score.
    Revoke a Worker’s Qualification by clicking the icon.
  • 11. Qualifications
    Create your own Qualification Types so you can route work to specific Workers.
    See name, ID, # of Workers with the Qualification, creation date & description
    Qualification Types with tests – can only be created via API.
    New Tab! Found under the “Manage” tab.
    Dispose by clicking the icon.
    Disposing will permanently delete the Qualification.
  • 12. Search by Requester Name
    Enter in the Requester’s name exactly.
    You can also click on the Requesters name in a results set to see all work currently available from that Requester.
    Bookmark this page to always see HITs available for a specific Requester
  • 13. API Best Practices
    We’ve made a number of improvements to the error messages from the Mechanical Turk Requester API:
    Check your errors!
    Use Notifications instead of the polling interfaces
    Dispose your HITS.
  • 14. Best Practices
    HIT Design
  • 15. Best Practices: HIT Design
    Clear & Concise instructions
    Add Keywords to make your HIT searchable
    Design an efficient HIT UX
    Listen to feedback from Workers
    Test & Iterate
  • 16. Same Workdifferent HITs = different results
  • 17. HIT Design Principles
    Clear statement of reward (what’s in it for me, the Worker?)
    Built in Validation – Worker knows when they are done, Requester can prevent incomplete work
    Detailed & collapsible instructions
  • 18. How can I get more accurate results?
    Plurality Asking multiple Workers to do assignments, if their answers agree then the result is validated)
    Qualifying & Training Workers
    Bonus payments
    Known Data Sets
  • 19. Q & A
    Your Questions. Answered.
  • 20. What is an invalid HIT?
    Require disclosure of the Worker's identity or e-mail address, either directly or indirectly
    Directly or indirectly promote a site, service, or opinion
    Violate the terms and conditions of an activity or website (for instance asking Workers to vote for something)
    Have explicit or offensive content, for instance, nudity, but do not have the Adult Content Qualification
    Ask Workers to solicit third parties
    Generate "referred" site visits or click through traffic
    Ask workers to take action to manipulate a search engine's relationship data
    Violate copyrights
  • 21. What can you do about invalid HITs?
    Don’t do the work.
    Report them.
  • 22. JC Goodrich
    Capture the Worker ID from the URL
    API Documentation
    2 Scenarios:
    Submit to Turk
    Submit to Requester
  • 23. How should I price my HIT?
    Mechanical Turk is a Marketplace, you determine the offer for your own work – and the workers choose what work to do.
    Do it yourself: How long does it take? What would be a reasonable hourly wage?
    Marketplace Test: Put in a sample HIT at different prices. How long does it take to get done? Quicker velocity = higher price, easy task = lower price
    Many other factors also influence velocity (reputation, fun factor)
    Combine results into one HIT – often lowers per item costs and reduces operational costs to review/approve.
    Ex. Photo Moderation – put 5-50 photos in one HIT.
    Compare with market pricing
  • 24. How can I attract good Workers?
    Establish quick and fair payment.
    Bonus your best Workers
    Participate in the forums –
  • 25. Workers: How do I handle blocks?
    Blocks prevent Workers from working for a Requester.
    Mechanical Turk reviews blocks as a signal of quality.
  • 26. Q & A
    Your Questions. Answered.
  • 27. Thank you.