Tips & Tricks
Your SlideShare is downloading.
AWS Summit 2011: Amazon Mechanical Turk for Enterprises
Like this presentation? Why not share!
by Sebin John
Amazon mechanical turk intro to bpo...
by John Hoskins
Tutorial on Using Amazon Mechanical...
by Ed Chi
20141021 AWS Cloud Taekwon - Custom...
by Amazon Web Servic...
Email sent successfully!
Show related SlideShares at end
AWS Summit 2011: Amazon Mechanical Turk for Enterprises
Amazon Web Services
Jul 05, 2011
Comment goes here.
12 hours ago
Are you sure you want to
Your message goes here
Be the first to comment
4 months ago
1 year ago
2 years ago
Juan Manuel Mogollón
Sales & Marketing VP
4 years ago
Number of Embeds
Flagged as inappropriate
Flag as inappropriate
No notes for slide
Transcript of "AWS Summit 2011: Amazon Mechanical Turk for Enterprises"
1. Introducing Amazon Mechanical Tu r kPresented by:John HoskinsSr. ManagerJune 10, 2011
Agenda Introduction A brief overview of Mechanical Turk Use Cases & Case Studies Practical applications of Mechanical Turk and examples of how others have used it. Getting Started Options for implementing with Mechanical Turk© 2011 Amazon.com, Inc. or its Affiliates.
Business Challenges Content Moderation Our community is generating 10,000 comments per day – how can we eliminate SPAM? Product Feedback We just released a new product or service – how do we judge how it’s being received? Discoverability We have thousands of legacy media assets – how do we make them searchable?© 2011 Amazon.com, Inc. or its Affiliates.
When to use Mechanical Turk? Tasks that require human judgment: Recognition Reasoning Context Prior Knowledge Algorithmic solutions are inefficient or impractical Requires flexibility due to unpredictable fluctuations in volume© 2011 Amazon.com, Inc. or its Affiliates.
What is Mechanical Turk?© 2011 Amazon.com, Inc. or its Affiliates.
Mechanical Turk is a marketplace for work. Mechanical Turk gives businesses and developers access to an on-demand, scalable workforce. Flexibility: Scale your workforce up and down quickly Accuracy: Get high quality, efficient and cost effective results. Price: Pay only when you are satisfied with the results. Speed: Start receiving results in minutes© 2011 Amazon.com, Inc. or its Affiliates.
Workforce 500,000Workers 190+ Countries Who are the Workers? Workers are global, available 24x7 Workers choose the work they want to do and work from anywhere, anytime Workers are dedicated to your tasks and can work in parallel Valuable source of feedback on how to improve your HITS© 2011 Amazon.com, Inc. or its Affiliates.
Your Workers Identify and manage your best Workers • All available Workers All Workers • 500,000 + • 24x7, 365 • Any Worker who Your completes a HIT Workers • Evaluate performance on your HITS Your Trusted • Your best Workers with proven quality Workers and availability© 2011 Amazon.com, Inc. or its Affiliates.
How are businesses using it? Data Management Categorization Data Verification Classification Data Entry & Collection Tagging Data De-duplication Sentiment Analysis Algorithm Training Content & Media Business Services Moderate Photos & Content Search Relevancy Content Creation & Editing Product Usability Testing Transcription Research© 2011 Amazon.com, Inc. or its Affiliates.
Search Enhancement / Relevance Search discovery requires categorization and expansion or classification of attributes – keywords that users can use to find content that relates to them. Adding meta data – such as detailed descriptions, attribute tags and categories can improve discoverability. How it works: Item description is Content is more Workers add Tags are added as sent to Mechanical easily discovered creative & attributes to the Turk for review by through Web and descriptive tags item Workers Site search© 2011 Amazon.com, Inc. or its Affiliates.
Moderation of User Generated Content The volume and speed at which user generated content is being generated – and maintaining site guidelines is a real problem faced by site publishers. Filtering content for appropriateness and usefulness requires review.. How it works: Content is sent Web site Workers review Inappropriate to Mechanical maintains according to content is Turk for review community rules deleted by Workers expectation© 2011 Amazon.com, Inc. or its Affiliates.
Data Handling Resolve Data conflicts directly in your business process Remove duplicates Normalize data entries Resolve conflicts and collisions Enhance entries How it works: Items flagged Business Workers review Business for conflict are Process Ingests item and Process sent to Data Elements resolve conflict continues Mechanical Turk© 2011 Amazon.com, Inc. or its Affiliates.
Advantages of Mechanical Turk Gives you access to an on-demand workforce Eliminates 100% Efficiency Staffing Issues Maintain the flexibility your business demands Use only the capacity you need, when you need it Converts a fixed cost into a variable cost Lowers Significantly reduces ongoing headcount expenses Costs Offers a pay-as-you-go staffing model Eliminates lead time of interviewing and staffing Reduces work Work gets done faster, by working in parallel turnaround time Work continues around the globe, around the clock© 2011 Amazon.com, Inc. or its Affiliates.
Getting Started Easy, scalable options for organizations of all sizes© 2011 Amazon.com, Inc. or its Affiliates.
How it works:. “Validate, “Design & Publish” Pay & Go”© 2011 Amazon.com, Inc. or its Affiliates.
Your Project Tasks “Working Backwards” What are the business results that you want? How will you use those results? What format do the results need to be in? Which tasks benefits from human intelligence? Break your overall project into smaller “tasks” Consolidate like items and processes Define instructions for each task© 2011 Amazon.com, Inc. or its Affiliates.
Example Workflow Content Moderation with API integration© 2011 Amazon.com, Inc. or its Affiliates.
The Key components of your HIT Instruction Guidelines for how to do a HIT Data “batch” concept What kind of dress is this? Question(s) Cocktail Sun dress Bridal dress© 2011 Amazon.com, Inc. or its Affiliates.
HIT Design Instructions The guidelines you provide on how to do your HIT Define what an acceptable answer looks like Account for ambiguity Ergonomics – get work done faster Design for Worker and reviewer efficiency Make sure all Workers do equal work for equal $ Defensive design – make sure your questions can be answered as expected Workflow Don‟t try to make one HIT do all of the work Split the process into logical stages Maintain „Like Reward for Like Work‟© 2011 Amazon.com, Inc. or its Affiliates.
Work Catalog Workers discover available HITs through the listings catalog Name of Requester, Name of HIT, Price and Qualifications are displayed© 2011 Amazon.com, Inc. or its Affiliates.
Adjudication Strategies Leverage different methods of determining the correct answer by designing for accuracy in your HIT: Multiple Choice Plurality Known Answers (“Gold Standards”) Free-form Multiple HIT Workflow‟s where a new Worker “edits” or “grades” previous Worker submissions Creative Tasks Bonus Payments© 2011 Amazon.com, Inc. or its Affiliates.
Balances & Levers Leverage market dynamics to optimize speed, price and accuracy to meet your business needs. Speed Price Accuracy© 2011 Amazon.com, Inc. or its Affiliates.
Requester access: Apps & Solution Web UI API & CLT Providers • Easy Access • Programmatic • Experts at • Create HITS in Access Workflow HTML • Multiple SDKS design • Upload and • Massive scale • They do the manage data training and via CSV management of Workers for specific kinds of tasks© 2011 Amazon.com, Inc. or its Affiliates.
Applications & Solution Providers Mechanical Turk supports a robust ecosystem of providers that have built or can build solutions to meet your needs.© 2011 Amazon.com, Inc. or its Affiliates.
Thank you. John Hoskins email@example.com 206.266.8105 @amazonmturk Facebook.com/amazonmturk mechanicalturk.typepad.com© 2011 Amazon.com, Inc. or its Affiliates.