• Save
AWS Summit 2011: Amazon Mechanical Turk for Enterprises
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

AWS Summit 2011: Amazon Mechanical Turk for Enterprises

on

  • 1,398 views

 

Statistics

Views

Total Views
1,398
Views on SlideShare
1,398
Embed Views
0

Actions

Likes
3
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

AWS Summit 2011: Amazon Mechanical Turk for Enterprises Presentation Transcript

  • 1. Introducing Amazon Mechanical Tu r kPresented by:John HoskinsSr. ManagerJune 10, 2011
  • 2. Agenda  Introduction A brief overview of Mechanical Turk  Use Cases & Case Studies Practical applications of Mechanical Turk and examples of how others have used it.  Getting Started Options for implementing with Mechanical Turk© 2011 Amazon.com, Inc. or its Affiliates.
  • 3. Business Challenges  Content Moderation Our community is generating 10,000 comments per day – how can we eliminate SPAM?  Product Feedback We just released a new product or service – how do we judge how it’s being received?  Discoverability We have thousands of legacy media assets – how do we make them searchable?© 2011 Amazon.com, Inc. or its Affiliates.
  • 4. When to use Mechanical Turk?  Tasks that require human judgment:  Recognition  Reasoning  Context  Prior Knowledge  Algorithmic solutions are inefficient or impractical  Requires flexibility due to unpredictable fluctuations in volume© 2011 Amazon.com, Inc. or its Affiliates.
  • 5. What is Mechanical Turk?© 2011 Amazon.com, Inc. or its Affiliates.
  • 6. Mechanical Turk is a marketplace for work. Mechanical Turk gives businesses and developers access to an on-demand, scalable workforce.  Flexibility: Scale your workforce up and down quickly  Accuracy: Get high quality, efficient and cost effective results.  Price: Pay only when you are satisfied with the results.  Speed: Start receiving results in minutes© 2011 Amazon.com, Inc. or its Affiliates.
  • 7. Workforce 500,000Workers 190+ Countries  Who are the Workers?  Workers are global, available 24x7  Workers choose the work they want to do and work from anywhere, anytime  Workers are dedicated to your tasks and can work in parallel  Valuable source of feedback on how to improve your HITS© 2011 Amazon.com, Inc. or its Affiliates.
  • 8. Your Workers  Identify and manage your best Workers • All available Workers All Workers • 500,000 + • 24x7, 365 • Any Worker who Your completes a HIT Workers • Evaluate performance on your HITS Your Trusted • Your best Workers with proven quality Workers and availability© 2011 Amazon.com, Inc. or its Affiliates.
  • 9. How are businesses using it?  Data Management  Categorization  Data Verification  Classification  Data Entry & Collection  Tagging  Data De-duplication  Sentiment Analysis  Algorithm Training  Content & Media  Business Services  Moderate Photos & Content  Search Relevancy  Content Creation & Editing  Product Usability Testing  Transcription  Research© 2011 Amazon.com, Inc. or its Affiliates.
  • 10. Search Enhancement / Relevance  Search discovery requires categorization and expansion or classification of attributes – keywords that users can use to find content that relates to them.  Adding meta data – such as detailed descriptions, attribute tags and categories can improve discoverability.  How it works: Item description is Content is more Workers add Tags are added as sent to Mechanical easily discovered creative & attributes to the Turk for review by through Web and descriptive tags item Workers Site search© 2011 Amazon.com, Inc. or its Affiliates.
  • 11. Moderation of User Generated Content  The volume and speed at which user generated content is being generated – and maintaining site guidelines is a real problem faced by site publishers.  Filtering content for appropriateness and usefulness requires review..  How it works: Content is sent Web site Workers review Inappropriate to Mechanical maintains according to content is Turk for review community rules deleted by Workers expectation© 2011 Amazon.com, Inc. or its Affiliates.
  • 12. Data Handling  Resolve Data conflicts directly in your business process  Remove duplicates  Normalize data entries  Resolve conflicts and collisions  Enhance entries  How it works: Items flagged Business Workers review Business for conflict are Process Ingests item and Process sent to Data Elements resolve conflict continues Mechanical Turk© 2011 Amazon.com, Inc. or its Affiliates.
  • 13. Advantages of Mechanical Turk Gives you access to an on-demand workforce Eliminates 100% Efficiency Staffing Issues Maintain the flexibility your business demands Use only the capacity you need, when you need it Converts a fixed cost into a variable cost Lowers Significantly reduces ongoing headcount expenses Costs Offers a pay-as-you-go staffing model Eliminates lead time of interviewing and staffing Reduces work Work gets done faster, by working in parallel turnaround time Work continues around the globe, around the clock© 2011 Amazon.com, Inc. or its Affiliates.
  • 14. Getting Started Easy, scalable options for organizations of all sizes© 2011 Amazon.com, Inc. or its Affiliates.
  • 15. How it works:. “Validate, “Design & Publish” Pay & Go”© 2011 Amazon.com, Inc. or its Affiliates.
  • 16. Your Project  Tasks  “Working Backwards”  What are the business results that you want?  How will you use those results?  What format do the results need to be in?  Which tasks benefits from human intelligence?  Break your overall project into smaller “tasks”  Consolidate like items and processes  Define instructions for each task© 2011 Amazon.com, Inc. or its Affiliates.
  • 17. Example Workflow  Content Moderation with API integration© 2011 Amazon.com, Inc. or its Affiliates.
  • 18. The Key components of your HIT  Instruction  Guidelines for how to do a HIT  Data  “batch” concept What kind of dress is this?  Question(s) Cocktail Sun dress Bridal dress© 2011 Amazon.com, Inc. or its Affiliates.
  • 19. HIT Design  Instructions  The guidelines you provide on how to do your HIT  Define what an acceptable answer looks like  Account for ambiguity  Ergonomics – get work done faster  Design for Worker and reviewer efficiency  Make sure all Workers do equal work for equal $  Defensive design – make sure your questions can be answered as expected  Workflow  Don‟t try to make one HIT do all of the work  Split the process into logical stages  Maintain „Like Reward for Like Work‟© 2011 Amazon.com, Inc. or its Affiliates.
  • 20. Work Catalog  Workers discover available HITs through the listings catalog  Name of Requester, Name of HIT, Price and Qualifications are displayed© 2011 Amazon.com, Inc. or its Affiliates.
  • 21. Adjudication Strategies  Leverage different methods of determining the correct answer by designing for accuracy in your HIT:  Multiple Choice  Plurality  Known Answers (“Gold Standards”)  Free-form  Multiple HIT Workflow‟s where a new Worker “edits” or “grades” previous Worker submissions  Creative Tasks  Bonus Payments© 2011 Amazon.com, Inc. or its Affiliates.
  • 22. Balances & Levers  Leverage market dynamics to optimize speed, price and accuracy to meet your business needs. Speed Price Accuracy© 2011 Amazon.com, Inc. or its Affiliates.
  • 23. Requester access: Apps & Solution Web UI API & CLT Providers • Easy Access • Programmatic • Experts at • Create HITS in Access Workflow HTML • Multiple SDKS design • Upload and • Massive scale • They do the manage data training and via CSV management of Workers for specific kinds of tasks© 2011 Amazon.com, Inc. or its Affiliates.
  • 24. Applications & Solution Providers  Mechanical Turk supports a robust ecosystem of providers that have built or can build solutions to meet your needs.© 2011 Amazon.com, Inc. or its Affiliates.
  • 25. Thank you. John Hoskins hoskins@amazon.com 206.266.8105 @amazonmturk Facebook.com/amazonmturk mechanicalturk.typepad.com© 2011 Amazon.com, Inc. or its Affiliates.