Your SlideShare is downloading. ×
AWS Summit 2011: Amazon Mechanical Turk for Enterprises
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

AWS Summit 2011: Amazon Mechanical Turk for Enterprises


Published on

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Introducing Amazon Mechanical Tu r kPresented by:John HoskinsSr. ManagerJune 10, 2011
  • 2. Agenda  Introduction A brief overview of Mechanical Turk  Use Cases & Case Studies Practical applications of Mechanical Turk and examples of how others have used it.  Getting Started Options for implementing with Mechanical Turk© 2011, Inc. or its Affiliates.
  • 3. Business Challenges  Content Moderation Our community is generating 10,000 comments per day – how can we eliminate SPAM?  Product Feedback We just released a new product or service – how do we judge how it’s being received?  Discoverability We have thousands of legacy media assets – how do we make them searchable?© 2011, Inc. or its Affiliates.
  • 4. When to use Mechanical Turk?  Tasks that require human judgment:  Recognition  Reasoning  Context  Prior Knowledge  Algorithmic solutions are inefficient or impractical  Requires flexibility due to unpredictable fluctuations in volume© 2011, Inc. or its Affiliates.
  • 5. What is Mechanical Turk?© 2011, Inc. or its Affiliates.
  • 6. Mechanical Turk is a marketplace for work. Mechanical Turk gives businesses and developers access to an on-demand, scalable workforce.  Flexibility: Scale your workforce up and down quickly  Accuracy: Get high quality, efficient and cost effective results.  Price: Pay only when you are satisfied with the results.  Speed: Start receiving results in minutes© 2011, Inc. or its Affiliates.
  • 7. Workforce 500,000Workers 190+ Countries  Who are the Workers?  Workers are global, available 24x7  Workers choose the work they want to do and work from anywhere, anytime  Workers are dedicated to your tasks and can work in parallel  Valuable source of feedback on how to improve your HITS© 2011, Inc. or its Affiliates.
  • 8. Your Workers  Identify and manage your best Workers • All available Workers All Workers • 500,000 + • 24x7, 365 • Any Worker who Your completes a HIT Workers • Evaluate performance on your HITS Your Trusted • Your best Workers with proven quality Workers and availability© 2011, Inc. or its Affiliates.
  • 9. How are businesses using it?  Data Management  Categorization  Data Verification  Classification  Data Entry & Collection  Tagging  Data De-duplication  Sentiment Analysis  Algorithm Training  Content & Media  Business Services  Moderate Photos & Content  Search Relevancy  Content Creation & Editing  Product Usability Testing  Transcription  Research© 2011, Inc. or its Affiliates.
  • 10. Search Enhancement / Relevance  Search discovery requires categorization and expansion or classification of attributes – keywords that users can use to find content that relates to them.  Adding meta data – such as detailed descriptions, attribute tags and categories can improve discoverability.  How it works: Item description is Content is more Workers add Tags are added as sent to Mechanical easily discovered creative & attributes to the Turk for review by through Web and descriptive tags item Workers Site search© 2011, Inc. or its Affiliates.
  • 11. Moderation of User Generated Content  The volume and speed at which user generated content is being generated – and maintaining site guidelines is a real problem faced by site publishers.  Filtering content for appropriateness and usefulness requires review..  How it works: Content is sent Web site Workers review Inappropriate to Mechanical maintains according to content is Turk for review community rules deleted by Workers expectation© 2011, Inc. or its Affiliates.
  • 12. Data Handling  Resolve Data conflicts directly in your business process  Remove duplicates  Normalize data entries  Resolve conflicts and collisions  Enhance entries  How it works: Items flagged Business Workers review Business for conflict are Process Ingests item and Process sent to Data Elements resolve conflict continues Mechanical Turk© 2011, Inc. or its Affiliates.
  • 13. Advantages of Mechanical Turk Gives you access to an on-demand workforce Eliminates 100% Efficiency Staffing Issues Maintain the flexibility your business demands Use only the capacity you need, when you need it Converts a fixed cost into a variable cost Lowers Significantly reduces ongoing headcount expenses Costs Offers a pay-as-you-go staffing model Eliminates lead time of interviewing and staffing Reduces work Work gets done faster, by working in parallel turnaround time Work continues around the globe, around the clock© 2011, Inc. or its Affiliates.
  • 14. Getting Started Easy, scalable options for organizations of all sizes© 2011, Inc. or its Affiliates.
  • 15. How it works:. “Validate, “Design & Publish” Pay & Go”© 2011, Inc. or its Affiliates.
  • 16. Your Project  Tasks  “Working Backwards”  What are the business results that you want?  How will you use those results?  What format do the results need to be in?  Which tasks benefits from human intelligence?  Break your overall project into smaller “tasks”  Consolidate like items and processes  Define instructions for each task© 2011, Inc. or its Affiliates.
  • 17. Example Workflow  Content Moderation with API integration© 2011, Inc. or its Affiliates.
  • 18. The Key components of your HIT  Instruction  Guidelines for how to do a HIT  Data  “batch” concept What kind of dress is this?  Question(s) Cocktail Sun dress Bridal dress© 2011, Inc. or its Affiliates.
  • 19. HIT Design  Instructions  The guidelines you provide on how to do your HIT  Define what an acceptable answer looks like  Account for ambiguity  Ergonomics – get work done faster  Design for Worker and reviewer efficiency  Make sure all Workers do equal work for equal $  Defensive design – make sure your questions can be answered as expected  Workflow  Don‟t try to make one HIT do all of the work  Split the process into logical stages  Maintain „Like Reward for Like Work‟© 2011, Inc. or its Affiliates.
  • 20. Work Catalog  Workers discover available HITs through the listings catalog  Name of Requester, Name of HIT, Price and Qualifications are displayed© 2011, Inc. or its Affiliates.
  • 21. Adjudication Strategies  Leverage different methods of determining the correct answer by designing for accuracy in your HIT:  Multiple Choice  Plurality  Known Answers (“Gold Standards”)  Free-form  Multiple HIT Workflow‟s where a new Worker “edits” or “grades” previous Worker submissions  Creative Tasks  Bonus Payments© 2011, Inc. or its Affiliates.
  • 22. Balances & Levers  Leverage market dynamics to optimize speed, price and accuracy to meet your business needs. Speed Price Accuracy© 2011, Inc. or its Affiliates.
  • 23. Requester access: Apps & Solution Web UI API & CLT Providers • Easy Access • Programmatic • Experts at • Create HITS in Access Workflow HTML • Multiple SDKS design • Upload and • Massive scale • They do the manage data training and via CSV management of Workers for specific kinds of tasks© 2011, Inc. or its Affiliates.
  • 24. Applications & Solution Providers  Mechanical Turk supports a robust ecosystem of providers that have built or can build solutions to meet your needs.© 2011, Inc. or its Affiliates.
  • 25. Thank you. John Hoskins 206.266.8105 @amazonmturk© 2011, Inc. or its Affiliates.