Introduction to Amazon Mechanical Turk

Introduction and Applications

Topics
Overview
Designing HITs
Controlling for quality
Fine print
Research with AMT
Resources

What’s it?
Amazon Mechanical Turk is a crowdsourcing Internet marketplace that enables computer programmers (Requesters) to coordinate the use of human intelligence to perform tasks which computers are unable to do.

https://www.mturk.com/

Workers get paid to answer stuff.

Requesters pay to ask stuff.

Why use it?
Advantages of Mechanical Turk*:
low cost ($0.10 per 60-second task)
subject pool size
subject pool diversity
faster theory/experiment cycle
* Mason, W. and Siddarth, S. 2009. “Conducting Behavioral Research on Amazon’s Mechanical Turk.” Yahoo! Research.

Topics
Overview
Designing HITs
Quality assurance
Fine print
Research with AMT
Resources

Human Intelligence Tasks (HITs)
Requesters can specify:
task
keywords
expiration date
reward
time allotted
worker qualifications
location
approval rating

Example HITs:
“Identify forward-facing pictures of dogs”
“Find and enter a business address”
“How attractive are these items?”
“Choose the best category for this product”
“Read a set of Tweets and decide if they describe an event”

Designing HITs
Options:
Use template or code within Mechanical Turk
requires basic HTML
Link to external survey
need some kind of verification input on Mechanical Turk side after Workers complete external survey

Designing HITs
Options:
Use template or code within Mechanical Turk
requires basic HTML
Link to external survey
need some kind of verification input on Mechanical Turk side after workers complete external survey

Topics
Overview
Designing HITs
Quality assurance
Fine print
Research with AMT
Resources

Test your HITs
Use the Mechanical Turk Requester sandbox
view your published your HIT before releasing it to Workers
check for functionality in all browsers

Help Workers work efficiently
Don’t make them scroll and click a lot
Set pages to open in new windows

Manage Worker accuracy
Reject bad work
Reward good work with bonuses
Filter Workers by qualifications
design a test for specific knowledge
location
approvals (percentage or number)
adult content

Topics
Overview
Designing HITs
Quality assurance
Fine print
Research with AMT
Resources

Payment information
HITs must be prepaid to a Mechanical Turk account
Amazon collects 10% on top of what you pay to Workers
Amazon collects 10% of any bonuses you grant
Payment to Workers can be in money or Amazon credit
You’re ultimately only charged for approved HITs, but you’re liable for the amount equal to whatever 100% approved HITs would cost.
You’ll have to deal with tax stuff if Workers do enough work for you to meet the IRS threshold for taxable income.
Click for prepaid terms and tax information.

What You Can’t Do
Not warn Workers about explicit or offensive content
label HIT appropriately
require and check for Adult Content Qualification in order to complete AND preview HIT
Collect personally identifiable information
Ask for this:
e-mail address
registration at another site
downloading of software
Click to view complete policies.

Topics
Overview
Designing HITs
Quality assurance
Payments
Research with AMT
Resources

Accurate replications of classics
Tversky and Kahneman:
framing - “Asian disease problem”
anchoring – “African countries problem”
conjunction fallacy – “Linda problem”
Loewenstein and Prelec:
fancy French dinner
Kirby and Marakovic
hyperbolic discounting
More at: http://experimentalturk.wordpress.com/

Extensions
Easy cross-cultural research
tasks can be confined to workers living in specific countries
lots of Indians
Eriksson and Simpson (2010)

Topics
Overview
Designing HITs
Quality assurance
Payments
Research with AMT
Resources

References
Amazon FAQs and tutorials
Mechanical Turk FAQ
Videos & Webinars
Participation Agreement
Literature on Mechanical Turk
Mason, W. and Siddarth, S. 2009. “Conducting Behavioral Research on Amazon’s Mechanical Turk.” Yahoo! Research.
Research using Mechanical Turk
See: http://experimentalturk.wordpress.com/

Introduction to Amazon Mechanical Turk

by amjludden on Mar 26, 2011

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 1

Statistics

Introduction to Amazon Mechanical Turk Presentation Transcript