• Save
Mechanical Turk is Not Anonymous
Upcoming SlideShare
Loading in...5

Mechanical Turk is Not Anonymous



Talk presented at the ID360 Conference (http://identity.utexas.edu/id360), May 1, 2013. Paper: http://ssrn.com/abstract=2228728. Joint work with Jessica Hullman, Jeffrey P. Bigham, Michael S. ...

Talk presented at the ID360 Conference (http://identity.utexas.edu/id360), May 1, 2013. Paper: http://ssrn.com/abstract=2228728. Joint work with Jessica Hullman, Jeffrey P. Bigham, Michael S. Bernstein, Juho Kim, Walter S. Lasecki, Saeideh Bakhshi, Tanushree Mitra, and Robert C. Miller.



Total Views
Views on SlideShare
Embed Views



1 Embed 14

https://twitter.com 14



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    Mechanical Turk is Not Anonymous Mechanical Turk is Not Anonymous Presentation Transcript

    • Amazons Mechanical Turk is Not AnonymousMatt LeaseSchool of Information @mattleaseUniversity of Texas at Austin ml@ischool.utexas.edussrn.com/abstract=2190946
    • Roadmap• What is Mechanical Turk?• Mechanical Turk & Anonymity• The Vulnerability• Potential Risks• Closing Thoughts2
    • What is Mechanical Turk?3@mattlease
    • • Online marketplace for paid crowd work• On-demand, scalable, 24/7 global workforce• Can perform all interactions via programmer’s API• Requestors & Workers are seemingly anonymous…Amazon Mechanical Turk (MTurk)4
    • Use Case 1: Data Processing5J. Pontin. Artificial Intelligence, With Help Fromthe Humans. New York Times (March 25, 2007)
    • Use Case 2: Data Collection(e.g., surveys, demographics, …)AmazonsMechanical Turk:A New Source ofInexpensive, YetHigh-Quality,Data?M. Buhrmesteret al. (2011)6
    • Mechanical Turk & Anonymity7@mattlease
    • Worker PrivacyEach worker is assigned an alphanumeric ID8
    • Requesters see only Worker IDs9
    • Brief Digression: Identity Fraud• Compromised & exploited worker accounts• Sybil attacks: use of multiple worker identities• Script bots masquerading as human workers10Robert Sim, MSR Faculty Summit’12
    • Safeguarding Personal Data•“What are the characteristics of MTurk workers?... the MTurksystem is set up to strictly protect workers’ anonymity….”11
    • The Vulnerability12@mattlease
    • `Amazon profile pageURLs use the sameIDs used on MTurk !Did Anyone Know?13
    • Did Anyone Know About This?• Researchers & Review Boards (IRBs)?– CrowdCamp announcement at ACM CSCW 2013– Reviewed prior published studies– Contacted researchers around the world– Contacted university IRBs• Amazon?– Reviewed website, technical & legal documents,online forums, blog, & interviews– Talked to Amazon’s VP in charge of MTurk• Workers?– Reviewed worker forums & conducted a survey14
    • Broad Perception of Anonymity15ssrn.com/abstract=2190946
    • Fraudulent Abuse of Workers“Do not do any HITs that involve: filling inCAPTCHAs; secret shopping; test our web page;test zip code; free trial; click my link; surveys orquizzes (unless the requester is listed with asmiley in the Hall of Fame/Shame); anythingthat involves sending a text message; orbasically anything that asks for any personalinformation at all—even your zip code. If youfeel in your gut it’s not on the level, IT’S NOT.Why? Because they are scams...”16
    • Workers’ Views: Survey & Forums• “... my reviewer profile is linked to my Mturk number! I hadno idea...”• “...Amazon needs to separate the Mturk numbers fromseller numbers to protect our privacy…”• “I think this is outrageous though. Makes me concernedabout trusting privacy agreements.”• “Mine pulled up my Amazon wish list which revealed myidentity. It seems to me that so called ”anonymous” taskson mTurk (like surveys) are not anonymous after all.”17
    • Potential Risks18@mattlease
    • Risks toWorkers• Inadvertent disclosure of PII or private data• Loss of blind hiring practices online• Greater risk of exploitation, reputation damage,loss of income, or even physical harm…19
    • Risks to Researchers• Exposing participants to undocumented risks• Having disclosed WorkerIDs (e.g., online)• Having not restricted access to the internally– Potential harm to participants– Lack of compliance with Federal/IRB governanceof human subjects research– Being required to discard collected data– Delays or inability to conduct future MTurk studies20
    • Risks to Amazon• Workers/Requesters abandoning MTurk• The Federal Trade Commission (FTC) has recentlybegun to aggressively protect consumers from databreaches by commercial entities, including therelease of supposedly “anonymous” data– Inadequate protection of customer records: BJWC– De-anonymized customer records: AOL, Netflix– Did workers have a reasonable expectation of privacyin their use of MTurk which has been violated? 21
    • Closing Thoughts22@mattlease
    • Human-centered Privacy Protection• Vulnerabilities are not purely technological• Focusing on software is not enough: humanfactors play a significant role in security of today’ssocio-technical, online systems– Insufficient attention to human factors designcan compromise information security, despite havingthe best algorithmic security protocols• Privacy protection should be explicitly-valued inrelation to other competing goals & stakeholderinterests to prevent being ignored or sacrificed23
    • Brief Digression: Information Schools• At 30 universities in N. America, Europe, Asia• Study human-centered aspects of informationtechnologies: design, implementation, policy, …24www.ischools.orgWobbrock etal., 2009
    • The Future of Crowd Work@ ACM CSCW 2013Kittur, Nickerson, Bernstein, Gerber,Shaw, Zimmerman, Lease, and Horton25
    • Matt Lease - ml@ischool.utexas.edu - @mattleaseThank You!Mechanical Turk is NotAnonymousMatthew Lease, Jessica Hullman,Jeffrey P. Bigham, Michael S. Bernstein,Juho Kim, Walter S. Lasecki, SaeidehBakhshi, Tanushree Mitra, andRobert C. MillerSocial Science Research Networkssrn.com/abstract=2190946ir.ischool.utexas.edu/crowd26