Amazon Mechanical TurkIt is a marketplace for work. In a traditional marketplace, like the Amazon.com retail site that you’re probably already familiar with, there are buyers who spend money to purchase products and there are sellers who make money by selling their products. This is true in the Amazon Mechanical Turk Marketplace as well. Businesses, or Requesters as they are called on Mechanical Turk are creating an offer for work. Mechanical Turk Workers accept and execute the offer, sell their work, and earn money when the transaction is complete. And just like in the Amazon.com retail experience, Amazon Mechanical Turk takes care of all of the transaction details including getting the work “listed” on the site so Workers see it when they browse for Work as well as completing the payment transaction when work is submitted and approved by you, the Requester. Let’s talk in a bit more detail about how it works.
Requesters design HITs, Human intelligence tasks. These HITs are composed of Assignments which are published to the Mechanical Turk Marketplace catalog. Workers search for and select tasks to complete and submit their assignments to Requesters.Requesters approve or reject the results and pay Workers.
Think of the Workers like customers. They are a large group of individuals, conditioned to come to the marketplace and search for work on a regular basis. The scale and scope of this group is what makes it powerful.
Add Cloud graphic
Mechanical Turk is ideal for tasks that require a human assessment – things that require recognition or reasoning. Context and prior knowledge.Many of our customers use Mechanical Turk in conjunction with an algorithmic process – in areas where the process can’t make a judgment with high confidence – the 80/20 rule where the 20% that doesn’t have a clear answer is sent to a human for resolution.Mechanical Turk, like all of Amazon’s web service is scalable – in both directions, scale up to meet peaks, and back off when resources aren’t needed. Avoid paying for idle work capacity to meet peak demand.
Add Cloud graphic
A large media company has an archive of photos with very cryptic descriptions. They need to categorize them into a taxonomy that helps customers find what they are looking for easily. This requires that a human look at every photo and place it into the proper category. The company sends the photo to Mechanical Turk where a worker views it and places it into the proper category and sub-category. Is it a landscape, person, object, etc.
A large business listing service receives daily data feeds from many sources. In order to apply that data to the proper business, they have to resolve which business the information belongs and in what sequence. Their internal process is able to resolve approximately 80% of the incoming data – what do they do for the other 20%? They create a task on Mechanical Turk to have a human look at the information and make a judgment on its resolution. Given the answer – the ingestion process continues.
Begin with a projectDefine the goals and key components of your project. For example, your goal might be to clean your business listing database so that you have accurate information for consumers. The sub-components of your project might be to categorize the businesses by listing type (i.e., restaurant or service) and verify that the related address and phone number are current. Break it into tasks and design your HITBreak the project into individual tasks so that many Workers can work in parallel to get your project done faster. For example, if you have 1,000 listings to verify, each listing would be an individual task.Next, design your Human Intelligence Tasks (HITs) by writing crisp and clear instructions, identifying the specific outputs and inputs desired and how much you will pay to have work completed.
Mechanical Turk offers a business process that ensures 100% worker productivity and keeping the cost per submitted work order, (HIT) to the lowest level. Mturk relieves businesses from having to staff to peaks and realizing employee downtime as demand lessens. Businesses avoid the fully burdened costs of a permanent workforce. Also, a workforce that performs at a 70-75% productivity range.
Getting Started with Amazon Mechanical Turk - AWS Summit 2012 - NYC
Introducing AmazonMechanical TurkA Brief Overview of What, Why, and HowJohn Hoskins
AgendaWhat is Mechanical Turk?When would you use it?How are others using it - Case StudiesHow – Implementation options
Mechanical Turk is a marketplace for work.Mechanical Turk gives businesses and developers accessto an on-demand, scalable workforce. • Flexibility: Scale your workforce up and down quickly • Accuracy: Get high quality, efficient and cost effective results. • Price: Pay only when you are satisfied with the results. • Speed: Start receiving results in minutes
How it works:.“Design & Publish” “Validate, Pay & Go”
Workforce500,000 Workers190+ CountriesWho are the Workers?• Workers are global• Workers represent numerous skill sets• 24X7 Follow the sun availabilityManaging Your Workforce• You can choose who is allowed to work on your assignments• Can narrow to US based (i.e. when you need western culture competence)• Leverage qualifications to find your best Workers
Business ChallengesOur Community is generating 10,000 comments per day– how can we eliminate SPAM?We pull information from hundreds of sources – how dowe normalize and cleanse?We have thousands of legacy assets – how do we makethem searchable?Our heuristics need jump started – how can I generatethousands of reference data points fast?We need to add a new attribute to our data?
Business ChallengesRequires Human Judgment:Recognition, Reasoning, Context, Prior KnowledgeCannot be completely solved algorithmicallyHas unpredictable fluctuations in volume
How are businesses using it? Data Management Categorization Data Verification Classification Data Entry & Collection Tagging Data De-duplication Keywords Algorithm Training Sentiment Analysis Content & Media Business Services Moderate Photos & Content Search Relevancy Content Creation & Editing Data Gathering Transcription Enhancement
Search EnhancementMake your assets more discoverable• Categorize them into your search tree• Add descriptions, attributes and keywords• Enhance listing dataHow it works:Item description Workers select is sent to Items are placed Content is more the best categoryMechanical Turk in that branch of easily discovered and sub-category for review by the search tree through search for the item Workers
Data HandlingResolve Data conflicts directly in your business process• Remove duplicates• Normalize data entries• Resolve conflicts and collisions• Enhance entriesHow it works: Business Items flagged forProcess Ingests Workers review Business conflict are sentData Elements item and resolve Process to Mechanical from multiple conflict continues Turk sources
Your Project Tasks“Working Backwards”• What are the business results that you want?• How will you use those results?• What format do the results need to be in?• Which tasks benefits from human intelligence?Break your overall project into smaller “tasks”• Consolidate like items and processesDefine instructions for each task
The Key components of your HITInstruction• Guidelines for how to do a HITData What kind of dress is this?• “batch” concept Cocktail Sun dressQuestion(s) Bridal dress
Work Catalog Workers discover available HITs through the listings catalog Name of Requester, Name of HIT, Price and Qualifications are displayed
HIT Example Specific instructions on what to do – and how to do it Providing as much information as available
Adjudication StrategiesLeverage different methods of determining the correct answerby designing for accuracy in your HIT:• Multiple Choice • Plurality • Known Answers (“Gold Standards”)• Free-form • Multiple HIT Workflow’s where a new Worker “edits” or “grades” previous Worker submissions• Creative Tasks • Bonus Payments
Your WorkersIdentify and manage your best Workers • All available Workers All Workers • 500,000 + • 24x7, 365 • Any Worker who completes a HIT Your Workers • Evaluate performance on your HITS Your Trusted • Your best Workers with proven quality and Workers availability
Balances & LeversLeverage market dynamics to optimize speed, price and accuracy tomeet your business needs. Speed Price Accuracy
Example WorkflowContent Moderation with API integration
Requester access: Apps & Solution Web UI API & CLT Providers• Easy Access • Programmatic • Experts at• Create HITS in Access Workflow design HTML • Multiple SDKS • They do the• Upload and • Massive scale training and manage data via management of CSV Workers for specific kinds of tasks
Self Service ApplicationsMechanical Turk supports a robust ecosystem of providersthat have built or can build solutions to meet your needs.
Advantages of Mechanical Turk Gives you access to an on-demand workforce Eliminates 100% EfficiencyStaffing Issues Maintain the flexibility your business demands Use only the capacity you need, when you need it Lowers Converts a fixed cost into a variable cost Costs Significantly reduces ongoing headcount expenses Offers a pay-as-you-go staffing model Eliminates lead time of interviewing and staffing Reduces work Work gets done faster, by working in parallelturnaround time Work continues around the globe, around the clock