• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Using Amazon Mechanical Turk for Management / Psychology
 

Using Amazon Mechanical Turk for Management / Psychology

on

  • 1,141 views

See more on : http://mgto.org/running-experiments-with-amazon-mechanical-turk/

See more on : http://mgto.org/running-experiments-with-amazon-mechanical-turk/

Statistics

Views

Total Views
1,141
Views on SlideShare
511
Embed Views
630

Actions

Likes
0
Downloads
14
Comments
0

2 Embeds 630

http://mgto.org 606
http://wiki.mgto.org 24

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Using Amazon Mechanical Turk for Management / Psychology Using Amazon Mechanical Turk for Management / Psychology Presentation Transcript

    • Using Amazon Mechanical Turk for Data Collection Gilad Feldman (MGMT-HKUST) http://mgto.org/running-experiments-with-amazon-mechanical-turk/
    • Why should I care?    MTurk is an essential tool for PhD students and researchers There are a lot of misconceptions and irrational biases about MTurk You need to atleast know what MTurk is and how to use it so you can decide for yourself 2
    • What can we expect to learn?     Review of where we stand with MTurk Live demonstration of how MTurk is used Tips and Tricks If you want - Hands on experience 3
    • Amazon Mechanical Turk What it is: Online Labor Market Complete task, get money For researchers: Data collection online Why? Very cheap International sample Very quick results 4
    • Turk Workers & Requestors “Turkers” Requestors 500,000 workers 190 countries 60% female* 83.5% white* 32.2 years old* 14.9 years of education* *Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk. Political Analysis, 20,351–368. 5
    • 6
    • Online? You can’t be serious.This gets published? Yes, it certainly does, in the best journals. Reviews (Psyc, JDM, Economics) : Psychology -  Buhrmester,  Kwang, & Gosling (2011, Perspectives on Psychological Science) Amazo    Economics – Horton, Rand & Zeckhauser (2010, Experimental Economics) – The Online Laborato   Management - Paolacci, Chandler & Ipeirotis (2010, Judgment and Decision Making) – Running expe   7
    • Really?! Published where? Examples of recent top-tier publications : •Science •PNAS •JPSP •Psychological Science •But... management conservative. journals are behind and still very 8
    • We don't like reading papers... So, what's the verdict?  Perspective on Psychological Science 2011 Findings indicate that (quotes)      More demographically diverse Participation is affected by compensation rate and task length Realistic compensation rates do not affect data quality Data obtained at least as reliable as those obtained via traditional methods Overall, MTurk can be used to obtain highquality data inexpensively and rapidly 9
    • We don't like reading papers... So, what's the verdict? Behavioral Research Methods 2011 : "Numerous social science experiments replicated on Mturk" 10
    • 11
    • My own experience - Essential tool   Started using Amazon Mechanical Turk in June 2012 Study #50 ran at Oct2013 Summary :  50 studies in 15 months  Used over 50% of data in papers  More reliable results than MGMT students  The core of my thesis and publication CV 12
    • My experience - used for...     Pretests / Real tests Experiments / Correlational Surveys / IAT / Crosswords Real life performance indicators in an online labour market 13
    • What other type of work can I use MTurk for?     Audio transcriptions Proofreading / spell-checking Rating and rankings Consumer choices But can't ...  Force to download software  Reveal private information (e.g. emails, etc.) 14
    • Cooler features       Can give bonuses Can run longitudinal studies Can set the location / performance Block misbehaving participants Scripts - Turk Check / various / and many others "Skilled Workers", but they cost more 15
    • Quick Demonstration MTurk + Qualtrics
    • Problem The following ten corporations were ranked by Fortune magazine to be the largest 500 United States based firms according to sales revenues for 2003 :   Group A: Reebok International, Hilton Hotels, Starbucks, Radioshack, Hershey Foods Group B : CoconoPhillips, American International Group, McKesson, AmerisourceBergen, The Altria Group Which group of organizations (A or B) had the larger total sales revenues in 2003? How do you know?
    • Problem 1 – Availability Heuristic Bias  Co rr ec The following ten corporations were ranked by Fortune magazine to be the largest 500 United States based firms according to sales revenues for 2003 :  Group A: Reebok International, Hilton Hotels, Starbucks, Radioshack, Hershey Foods tA ns w  er -B Group B : CoconoPhillips, American International Group, McKesson, AmerisourceBergen, The Altria Group (Example adopted from “Judgment in Managerial Decision Making” book by Max Bazerman)
    • Turker Interface 19
    • Requestor Interface 20
    • Creating a HIT 21
    • Creating a HIT 22
    • Creating a HIT 23
    • Creating a HIT 24
    • Creating a HIT May request “master workers” of general/photo/category type Location, approval rate, and “mastery” are the only selection criter 25
    • Creating a HIT 26
    • Creating a HIT 27
    • Creating a HIT 28
    • Managing HITs 29
    • Managing HITs 30
    • Linking to 3rd party software 1. Create a survey in 3PS  1. The last question of the survey should disclose an “approval code” Copy the URL of the survey into a HIT  The HIT has one question: “what is the approval code?” 31
    • Linking to 3rd party software 32
    • How I run things Setup I pay 0.01US$/minute + 10% Amazon commission. Most experiments are 3-5 minutes. Participants Indian participants : A dilemma, I sometimes exclude those. Target Turks : Completed atleast 50 HITs before with 95% acceptance 33
    • How I run things Making sure they do it right "Have you done this task before?" Quiz after every scenario or manipulation (Can’t move on without getting it right) Reading checks (“set this bar on twenty seven”) Decoy questions Timers Manipulation checks, more than 1 if possible Funneling section (Intention of study, connection between parts) Ask - "how serious were you answering this questionnaire?" English proficiency self report + test questions. Use the MTurk Sandbox Pre-Test your survey on 20 participants, tweak, then launch. 34
    • Issues with MTurkers 35
    • Issues with MTurkers 36
    • Advanced Use of Online Experiments More advanced things I ran with Mturk: Implicit Association Test (Response time). Crossword Puzzle (Javascript). More advanced use of Qualtrics (1 / 2 / 3). Essentially, you can run whatever you want online. 37
    • Suppose you don't believe this is publishable, then... what else?    Pretest your stimuli Pretest your manipulations Even run the actual study and compare to your final results from what you think is a better sample. It's always good to show your results are generalizable. 38
    • Any other questions? Email : filian@ust.hk
    • Slides/materials credits   MTurk- Online Sampling with Crowdsourcing PPT Experimental Turk blog 40