Successfully reported this slideshow.

Mechanical Turk is Not Anonymous

1

Share

Loading in …3
×
1 of 26
1 of 26

Mechanical Turk is Not Anonymous

1

Share

Talk presented at the ID360 Conference (http://identity.utexas.edu/id360), May 1, 2013. Paper: http://ssrn.com/abstract=2228728. Joint work with Jessica Hullman, Jeffrey P. Bigham, Michael S. Bernstein, Juho Kim, Walter S. Lasecki, Saeideh Bakhshi, Tanushree Mitra, and Robert C. Miller.

Talk presented at the ID360 Conference (http://identity.utexas.edu/id360), May 1, 2013. Paper: http://ssrn.com/abstract=2228728. Joint work with Jessica Hullman, Jeffrey P. Bigham, Michael S. Bernstein, Juho Kim, Walter S. Lasecki, Saeideh Bakhshi, Tanushree Mitra, and Robert C. Miller.

More Related Content

More from Matthew Lease

Related Books

Free with a 14 day trial from Scribd

See all

Related Audiobooks

Free with a 14 day trial from Scribd

See all

Mechanical Turk is Not Anonymous

  1. 1. Amazon's Mechanical Turk is Not Anonymous Matt Lease School of Information @mattlease University of Texas at Austin ml@ischool.utexas.edussrn.com/abstract=2190946
  2. 2. Roadmap • What is Mechanical Turk? • Mechanical Turk & Anonymity • The Vulnerability • Potential Risks • Closing Thoughts 2
  3. 3. What is Mechanical Turk? 3@mattlease
  4. 4. • Online marketplace for paid crowd work • On-demand, scalable, 24/7 global workforce • Can perform all interactions via programmer’s API • Requestors & Workers are seemingly anonymous… Amazon Mechanical Turk (MTurk) 4
  5. 5. Use Case 1: Data Processing 5 J. Pontin. Artificial Intelligence, With Help From the Humans. New York Times (March 25, 2007)
  6. 6. Use Case 2: Data Collection (e.g., surveys, demographics, …) Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data? M. Buhrmester et al. (2011) 6
  7. 7. Mechanical Turk & Anonymity 7@mattlease
  8. 8. Worker Privacy Each worker is assigned an alphanumeric ID 8
  9. 9. Requesters see only Worker IDs 9
  10. 10. Brief Digression: Identity Fraud • Compromised & exploited worker accounts • Sybil attacks: use of multiple worker identities • Script bots masquerading as human workers 10 Robert Sim, MSR Faculty Summit’12
  11. 11. Safeguarding Personal Data • “What are the characteristics of MTurk workers?... the MTurk system is set up to strictly protect workers’ anonymity….” 11
  12. 12. The Vulnerability 12@mattlease
  13. 13. ` Amazon profile page URLs use the same IDs used on MTurk ! Did Anyone Know? 13
  14. 14. Did Anyone Know About This? • Researchers & Review Boards (IRBs)? – CrowdCamp announcement at ACM CSCW 2013 – Reviewed prior published studies – Contacted researchers around the world – Contacted university IRBs • Amazon? – Reviewed website, technical & legal documents, online forums, blog, & interviews – Talked to Amazon’s VP in charge of MTurk • Workers? – Reviewed worker forums & conducted a survey 14
  15. 15. Broad Perception of Anonymity 15 ssrn.com/abstract=2190946
  16. 16. Fraudulent Abuse of Workers “Do not do any HITs that involve: filling in CAPTCHAs; secret shopping; test our web page; test zip code; free trial; click my link; surveys or quizzes (unless the requester is listed with a smiley in the Hall of Fame/Shame); anything that involves sending a text message; or basically anything that asks for any personal information at all—even your zip code. If you feel in your gut it’s not on the level, IT’S NOT. Why? Because they are scams...” 16
  17. 17. Workers’ Views: Survey & Forums • “... my reviewer profile is linked to my Mturk number! I had no idea...” • “...Amazon needs to separate the Mturk numbers from seller numbers to protect our privacy…” • “I think this is outrageous though. Makes me concerned about trusting privacy agreements.” • “Mine pulled up my Amazon wish list which revealed my identity. It seems to me that so called ”anonymous” tasks on mTurk (like surveys) are not anonymous after all.” 17
  18. 18. Potential Risks 18@mattlease
  19. 19. Risks to Workers • Inadvertent disclosure of PII or private data • Loss of blind hiring practices online • Greater risk of exploitation, reputation damage, loss of income, or even physical harm… 19
  20. 20. Risks to Researchers • Exposing participants to undocumented risks • Having disclosed WorkerIDs (e.g., online) • Having not restricted access to the internally – Potential harm to participants – Lack of compliance with Federal/IRB governance of human subjects research – Being required to discard collected data – Delays or inability to conduct future MTurk studies 20
  21. 21. Risks to Amazon • Workers/Requesters abandoning MTurk • The Federal Trade Commission (FTC) has recently begun to aggressively protect consumers from data breaches by commercial entities, including the release of supposedly “anonymous” data – Inadequate protection of customer records: BJWC – De-anonymized customer records: AOL, Netflix – Did workers have a reasonable expectation of privacy in their use of MTurk which has been violated? 21
  22. 22. Closing Thoughts 22@mattlease
  23. 23. Human-centered Privacy Protection • Vulnerabilities are not purely technological • Focusing on software is not enough: human factors play a significant role in security of today’s socio-technical, online systems – Insufficient attention to human factors design can compromise information security, despite having the best algorithmic security protocols • Privacy protection should be explicitly-valued in relation to other competing goals & stakeholder interests to prevent being ignored or sacrificed 23
  24. 24. Brief Digression: Information Schools • At 30 universities in N. America, Europe, Asia • Study human-centered aspects of information technologies: design, implementation, policy, … 24 www.ischools.org Wobbrock et al., 2009
  25. 25. The Future of Crowd Work @ ACM CSCW 2013 Kittur, Nickerson, Bernstein, Gerber, Shaw, Zimmerman, Lease, and Horton 25
  26. 26. Matt Lease - ml@ischool.utexas.edu - @mattlease Thank You! Mechanical Turk is Not Anonymous Matthew Lease, Jessica Hullman, Jeffrey P. Bigham, Michael S. Bernstein, Juho Kim, Walter S. Lasecki, Saeideh Bakhshi, Tanushree Mitra, and Robert C. Miller Social Science Research Network ssrn.com/abstract=2190946 ir.ischool.utexas.edu/crowd 26

×