Your SlideShare is downloading. ×
  • Like
  • Save
AWS Summit 2011: Amazon Mechanical Turk for Enterprises
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

AWS Summit 2011: Amazon Mechanical Turk for Enterprises

  • 946 views
Published

 

Published in Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
946
On SlideShare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
0
Comments
0
Likes
3

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Introducing Amazon Mechanical Tu r kPresented by:John HoskinsSr. ManagerJune 10, 2011
  • 2. Agenda  Introduction A brief overview of Mechanical Turk  Use Cases & Case Studies Practical applications of Mechanical Turk and examples of how others have used it.  Getting Started Options for implementing with Mechanical Turk© 2011 Amazon.com, Inc. or its Affiliates.
  • 3. Business Challenges  Content Moderation Our community is generating 10,000 comments per day – how can we eliminate SPAM?  Product Feedback We just released a new product or service – how do we judge how it’s being received?  Discoverability We have thousands of legacy media assets – how do we make them searchable?© 2011 Amazon.com, Inc. or its Affiliates.
  • 4. When to use Mechanical Turk?  Tasks that require human judgment:  Recognition  Reasoning  Context  Prior Knowledge  Algorithmic solutions are inefficient or impractical  Requires flexibility due to unpredictable fluctuations in volume© 2011 Amazon.com, Inc. or its Affiliates.
  • 5. What is Mechanical Turk?© 2011 Amazon.com, Inc. or its Affiliates.
  • 6. Mechanical Turk is a marketplace for work. Mechanical Turk gives businesses and developers access to an on-demand, scalable workforce.  Flexibility: Scale your workforce up and down quickly  Accuracy: Get high quality, efficient and cost effective results.  Price: Pay only when you are satisfied with the results.  Speed: Start receiving results in minutes© 2011 Amazon.com, Inc. or its Affiliates.
  • 7. Workforce 500,000Workers 190+ Countries  Who are the Workers?  Workers are global, available 24x7  Workers choose the work they want to do and work from anywhere, anytime  Workers are dedicated to your tasks and can work in parallel  Valuable source of feedback on how to improve your HITS© 2011 Amazon.com, Inc. or its Affiliates.
  • 8. Your Workers  Identify and manage your best Workers • All available Workers All Workers • 500,000 + • 24x7, 365 • Any Worker who Your completes a HIT Workers • Evaluate performance on your HITS Your Trusted • Your best Workers with proven quality Workers and availability© 2011 Amazon.com, Inc. or its Affiliates.
  • 9. How are businesses using it?  Data Management  Categorization  Data Verification  Classification  Data Entry & Collection  Tagging  Data De-duplication  Sentiment Analysis  Algorithm Training  Content & Media  Business Services  Moderate Photos & Content  Search Relevancy  Content Creation & Editing  Product Usability Testing  Transcription  Research© 2011 Amazon.com, Inc. or its Affiliates.
  • 10. Search Enhancement / Relevance  Search discovery requires categorization and expansion or classification of attributes – keywords that users can use to find content that relates to them.  Adding meta data – such as detailed descriptions, attribute tags and categories can improve discoverability.  How it works: Item description is Content is more Workers add Tags are added as sent to Mechanical easily discovered creative & attributes to the Turk for review by through Web and descriptive tags item Workers Site search© 2011 Amazon.com, Inc. or its Affiliates.
  • 11. Moderation of User Generated Content  The volume and speed at which user generated content is being generated – and maintaining site guidelines is a real problem faced by site publishers.  Filtering content for appropriateness and usefulness requires review..  How it works: Content is sent Web site Workers review Inappropriate to Mechanical maintains according to content is Turk for review community rules deleted by Workers expectation© 2011 Amazon.com, Inc. or its Affiliates.
  • 12. Data Handling  Resolve Data conflicts directly in your business process  Remove duplicates  Normalize data entries  Resolve conflicts and collisions  Enhance entries  How it works: Items flagged Business Workers review Business for conflict are Process Ingests item and Process sent to Data Elements resolve conflict continues Mechanical Turk© 2011 Amazon.com, Inc. or its Affiliates.
  • 13. Advantages of Mechanical Turk Gives you access to an on-demand workforce Eliminates 100% Efficiency Staffing Issues Maintain the flexibility your business demands Use only the capacity you need, when you need it Converts a fixed cost into a variable cost Lowers Significantly reduces ongoing headcount expenses Costs Offers a pay-as-you-go staffing model Eliminates lead time of interviewing and staffing Reduces work Work gets done faster, by working in parallel turnaround time Work continues around the globe, around the clock© 2011 Amazon.com, Inc. or its Affiliates.
  • 14. Getting Started Easy, scalable options for organizations of all sizes© 2011 Amazon.com, Inc. or its Affiliates.
  • 15. How it works:. “Validate, “Design & Publish” Pay & Go”© 2011 Amazon.com, Inc. or its Affiliates.
  • 16. Your Project  Tasks  “Working Backwards”  What are the business results that you want?  How will you use those results?  What format do the results need to be in?  Which tasks benefits from human intelligence?  Break your overall project into smaller “tasks”  Consolidate like items and processes  Define instructions for each task© 2011 Amazon.com, Inc. or its Affiliates.
  • 17. Example Workflow  Content Moderation with API integration© 2011 Amazon.com, Inc. or its Affiliates.
  • 18. The Key components of your HIT  Instruction  Guidelines for how to do a HIT  Data  “batch” concept What kind of dress is this?  Question(s) Cocktail Sun dress Bridal dress© 2011 Amazon.com, Inc. or its Affiliates.
  • 19. HIT Design  Instructions  The guidelines you provide on how to do your HIT  Define what an acceptable answer looks like  Account for ambiguity  Ergonomics – get work done faster  Design for Worker and reviewer efficiency  Make sure all Workers do equal work for equal $  Defensive design – make sure your questions can be answered as expected  Workflow  Don‟t try to make one HIT do all of the work  Split the process into logical stages  Maintain „Like Reward for Like Work‟© 2011 Amazon.com, Inc. or its Affiliates.
  • 20. Work Catalog  Workers discover available HITs through the listings catalog  Name of Requester, Name of HIT, Price and Qualifications are displayed© 2011 Amazon.com, Inc. or its Affiliates.
  • 21. Adjudication Strategies  Leverage different methods of determining the correct answer by designing for accuracy in your HIT:  Multiple Choice  Plurality  Known Answers (“Gold Standards”)  Free-form  Multiple HIT Workflow‟s where a new Worker “edits” or “grades” previous Worker submissions  Creative Tasks  Bonus Payments© 2011 Amazon.com, Inc. or its Affiliates.
  • 22. Balances & Levers  Leverage market dynamics to optimize speed, price and accuracy to meet your business needs. Speed Price Accuracy© 2011 Amazon.com, Inc. or its Affiliates.
  • 23. Requester access: Apps & Solution Web UI API & CLT Providers • Easy Access • Programmatic • Experts at • Create HITS in Access Workflow HTML • Multiple SDKS design • Upload and • Massive scale • They do the manage data training and via CSV management of Workers for specific kinds of tasks© 2011 Amazon.com, Inc. or its Affiliates.
  • 24. Applications & Solution Providers  Mechanical Turk supports a robust ecosystem of providers that have built or can build solutions to meet your needs.© 2011 Amazon.com, Inc. or its Affiliates.
  • 25. Thank you. John Hoskins hoskins@amazon.com 206.266.8105 @amazonmturk Facebook.com/amazonmturk mechanicalturk.typepad.com© 2011 Amazon.com, Inc. or its Affiliates.