1. Introducing Amazon
Mechanical Turk
A Brief Overview of What, Why, and How
John Hoskins

2. Agenda
What is Mechanical Turk?
When would you use it?
How are others using it - Case Studies
How – Implementation options

3. What is
Mechanical Turk?

4. Mechanical Turk is a
Marketplace.

5. Mechanical Turk is a
marketplace for work.
Mechanical Turk gives businesses and developers access
to an on-demand, scalable workforce.
• Flexibility: Scale your workforce up and down quickly
• Accuracy: Get high quality, efficient and cost effective results.
• Price: Pay only when you are satisfied with the results.
• Speed: Start receiving results in minutes

6. How it works:.
“Design & Publish” “Validate,
Pay & Go”

7. Workforce
500,000 Workers
190+ Countries
Who are the Workers?
• Workers are global
• Workers represent numerous skill sets
• 24X7 Follow the sun availability
Managing Your Workforce
• You can choose who is allowed to work on your assignments
• Can narrow to US based (i.e. when you need western culture competence)
• Leverage qualifications to find your best Workers

8. When would you use
Mechanical Turk?

9. Business Challenges
Our Community is generating 10,000 comments per day
– how can we eliminate SPAM?
We pull information from hundreds of sources – how do
we normalize and cleanse?
We have thousands of legacy assets – how do we make
them searchable?
Our heuristics need jump started – how can I generate
thousands of reference data points fast?
We need to add a new attribute to our data?

10. Business Challenges
Requires Human Judgment:
Recognition, Reasoning, Context, Prior Knowledge
Cannot be completely solved algorithmically
Has unpredictable fluctuations in volume

11. How are businesses using it?
 Data Management  Categorization
 Data Verification  Classification
 Data Entry & Collection  Tagging
 Data De-duplication  Keywords
 Algorithm Training  Sentiment Analysis
 Content & Media  Business Services
 Moderate Photos & Content  Search Relevancy
 Content Creation & Editing  Data Gathering
 Transcription
 Enhancement

12. How are others using
Mechanical Turk?

13. Search Enhancement
Make your assets more discoverable
• Categorize them into your search tree
• Add descriptions, attributes and keywords
• Enhance listing data
How it works:
Item description
Workers select
is sent to Items are placed Content is more
the best category
Mechanical Turk in that branch of easily discovered
and sub-category
for review by the search tree through search
for the item
Workers

14. Data Handling
Resolve Data conflicts directly in your business process
• Remove duplicates
• Normalize data entries
• Resolve conflicts and collisions
• Enhance entries
How it works:
Business
Items flagged for
Process Ingests Workers review Business
conflict are sent
Data Elements item and resolve Process
to Mechanical
from multiple conflict continues
Turk
sources

15. How to apply your task to
Mechanical Turk?

16. Your Project  Tasks
“Working Backwards”
• What are the business results that you want?
• How will you use those results?
• What format do the results need to be in?
• Which tasks benefits from human intelligence?
Break your overall project into smaller “tasks”
• Consolidate like items and processes
Define instructions for each task

17. The Key components of your HIT
Instruction
• Guidelines for how to do
a HIT
Data What kind of dress is this?
• “batch” concept Cocktail
Sun dress
Question(s) Bridal dress

18. Work Catalog
Workers
discover
available HITs
through the
listings catalog
Name of
Requester,
Name of HIT,
Price and
Qualifications
are displayed

19. HIT Example
Specific
instructions on
what to do –
and how to do it
Providing as
much
information as
available

20. Adjudication Strategies
Leverage different methods of determining the correct answer
by designing for accuracy in your HIT:
• Multiple Choice
• Plurality
• Known Answers (“Gold Standards”)
• Free-form
• Multiple HIT Workflow’s where a new Worker “edits” or “grades”
previous Worker submissions
• Creative Tasks
• Bonus Payments

21. Your Workers
Identify and manage your best Workers
• All available Workers
All Workers • 500,000 +
• 24x7, 365
• Any Worker who
completes a HIT
Your Workers • Evaluate performance on
your HITS
Your Trusted • Your best Workers with
proven quality and
Workers availability

22. Balances & Levers
Leverage market dynamics to optimize speed, price and accuracy to
meet your business needs.
Speed
Price
Accuracy

23. Example Workflow
Content Moderation with API integration

24. Requester access:
Apps & Solution
Web UI API & CLT
Providers
• Easy Access • Programmatic • Experts at
• Create HITS in Access Workflow design
HTML • Multiple SDKS • They do the
• Upload and • Massive scale training and
manage data via management of
CSV Workers for
specific kinds of
tasks

25. Self Service Applications
Mechanical Turk supports a robust ecosystem of providers
that have built or can build solutions to meet your needs.

26. Custom Solution Providers

27. Advantages of Mechanical Turk
Gives you access to an on-demand workforce
Eliminates 100% Efficiency
Staffing Issues Maintain the flexibility your business demands
Use only the capacity you need, when you need it
Lowers Converts a fixed cost into a variable cost
Costs Significantly reduces ongoing headcount expenses
Offers a pay-as-you-go staffing model
Eliminates lead time of interviewing and staffing
Reduces work
Work gets done faster, by working in parallel
turnaround time
Work continues around the globe, around the clock

28. Thank you.
John Hoskins
hoskins@amazon.com
206.266.8105
@amazonmturk
Facebook.com/amazonmturk
mechanicalturk.typepad.com