Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Crowdsourcing & ethics: a few thoughts and refences.


Published on

Extracts and addendums from an earlier talk, for those interested in ethics and related issues in regard to crowdsourcing, particularly research uses. Slides updated Sept. 2, 2013.

Published in: Technology, Business
  • My experience with essay writing companies was good. Long story’s short: My parents are getting married. I have literally no time to work on my research paper due to wedding preparation and rehearsals. I have no time at all to sit on a computer for an endless amount of time. Thankfully. I discovered this service who enabled me to pass my academic ordeal to one of their writers. My grade was pretty good as a result of this action. I recommend for any of you who are having the same troubles I am.
    Are you sure you want to  Yes  No
    Your message goes here
  • MADE $30 ON MY FIRST DAY! Being a fresh graduate and having lots of free time, I stumbled upon your site when I was searching for work at home opportunities, good thing I did! Just on my first day of joining I already made $30! Now I'm averaging close to $80 a day just for filling out surveys! 
    Are you sure you want to  Yes  No
    Your message goes here
  • Thanks--great content, and I really appreciate the links to papers and news stories. The paper on TDT annotation crowdsourcing that talked about annotators having nightmares about content was news to me
    Are you sure you want to  Yes  No
    Your message goes here

Crowdsourcing & ethics: a few thoughts and refences.

  1. 1. Extracts and Addendums: Ethics & Crowd Computing Matt Lease School of Information @mattlease University of Texas at Austin
  2. 2. What about context? By minimizing context, greater task efficiency can often be achieved – e.g. “Can you name who is in this photo?” 2
  3. 3. Context & Informed Consent • But people need context for informed consent • Who will benefit from it and how? Jonathan Zittrain, Harvard 3
  4. 4. What about requestor fraud? • Refusing to pay for work, installing malware • Using crowds to defeat online metrics/security 4
  5. 5. Robert Sim, MSR Summit’12 5
  6. 6. What about regulation? • Wolfson and Lease. Look before you leap: Legal pitfalls of crowdsourcing. Proceedings of the American Society for Information Science and Technology (ASIS&T), 2011. • As usual, technology is ahead of the law – employment law – patent inventorship – data security and the Federal Trade Commission – copyright ownership – securities regulation of crowdfunding • Take-away: don’t panic, but be mindful – Understand risks of “just in-time compliance” 6
  7. 7. Who are the workers? • A. Baio, November 2008. The Faces of Mechanical Turk. • P. Ipeirotis. March 2010. The New Demographics of Mechanical Turk • J. Ross, et al. Who are the Crowdworkers? CHI 2010. 7
  8. 8. What about ethics? • Six Silberman, Lilly Irani, and Joel Ross. Ethics and tactics of professional crowdwork. XRDS: Crossroads, The ACM Magazine for Students 17.2 (2010): 39-43. – “How should we… conceptualize the role of these people who we ask to power our computing?” – Power dynamics – “Abstraction hides detail” • Fort, Karën, Gilles Adda, and K. Bretonnel Cohen. Amazon mechanical turk: Gold mine or coal mine?. Computational Linguistics 37.2 (2011): 413-420. – “…opportunities for our community to deliberately value ethics above cost savings.” 8
  9. 9. Davis et al. (2010) The HPU. HPU 9
  10. 10. HPU: “Abstraction hides detail” 10
  11. 11. Requester Fraud on MTurk “Do not do any HITs that involve: filling in CAPTCHAs; secret shopping; test our web page; test zip code; free trial; click my link; surveys or quizzes (unless the requester is listed with a smiley in the Hall of Fame/Shame); anything that involves sending a text message; or basically anything that asks for any personal information at all—even your zip code. If you feel in your gut it’s not on the level, IT’S NOT. Why? Because they are scams...” 11
  12. 12. Fraud example • Is this really the requester’s captcha? 12
  13. 13. Captcha Fraud • Severity? 13
  14. 14. • “…not only do malicious crowd-sourcing systems exist, but they are rapidly growing…” 14
  15. 15. Digital Dirty Jobs • The Googler who Looked at the Worst of the Internet • Policing the Web’s Lurid Precincts • Facebook content moderation • The dirty job of keeping Facebook clean • Even linguistic annotators report stress & nightmares from reading news articles! 15
  16. 16. Crowdsourcing and Vigilantism • Search-Chases-Wrong-Guy 16
  17. 17. What about freedom? • Crowdsourcing vision: empowering freedom – work whenever you want for whomever you want • Risk: people being compelled to perform work – Chinese prisoners farming gold… – Digital sweat shops? Digital slaves? – We really don’t know (and need to learn more…) – Traction? Human Trafficking at MSR Summit’12 17
  18. 18. Mechanical Turk is Not Anonymous Matthew Lease, Jessica Hullman, Jeffrey P. Bigham, Michael S. Bernstein, Juho Kim, Walter S. Lasecki, Saeideh Bakhshi, Tanushree Mitra, and Robert C. Miller. Online: Social Science Research Network, March 6, 2013
  19. 19. Worker Privacy Each worker is assigned an alphanumeric ID 19
  20. 20. Requesters see only Worker IDs 20
  21. 21. Safeguarding Personal Data • “What are the characteristics of MTurk workers?... the MTurk system is set up to strictly protect workers’ anonymity….” 21
  22. 22. Broad Perception of Anonymity 22
  23. 23. ` Amazon profile page URLs use the same IDs used on MTurk ! Who knew? 23
  24. 24. Workers’ Views: Survey & Forums • “... my reviewer profile is linked to my Mturk number! I had no idea...” • “...Amazon needs to separate the Mturk numbers from seller numbers to protect our privacy…” • “I think this is outrageous though. Makes me concerned about trusting privacy agreements.” • “Mine pulled up my Amazon wish list which revealed my identity. It seems to me that so called ”anonymous” tasks on mTurk (like surveys) are not anonymous after all.” 24
  25. 25. Risks to Workers • Inadvertent disclosure of PII or private data • Loss of blind hiring practices online • Greater risk of exploitation, reputation damage, loss of income, or even physical harm… 25
  26. 26. Risks to Researchers • Exposing participants to undocumented risks • Having disclosed WorkerIDs (e.g., online) • Having not restricted access to the internally – Potential harm to participants – Lack of compliance with Federal/IRB governance of human subjects research – Being required to discard collected data – Delays or inability to conduct future MTurk studies 26
  27. 27. Risks to Service Provider • Workers/Requesters abandoning MTurk • The US Federal Trade Commission (FTC) has recently begun to aggressively protect consumers from data breaches by commercial entities, including release of supposedly “anonymous” data – Inadequate protection of customer records: BJWC – De-anonymized customer records: AOL, Netflix – Did workers have a reasonable expectation of privacy in their use of MTurk which has been violated? 27
  28. 28. The Future of Crowd Work, CSCW’13 Kittur, Nickerson, Bernstein, Gerber, Shaw, Zimmerman, Lease, and Horton 28
  29. 29. Additional References • Paul Hyman. Software Aims to Ensure Fairness in Crowdsourcing Projects. Communications of the ACM, Vol. 56 No. 8, Pages 19-21, August 2013. • Irani, Lilly C. The Ideological Work of Microwork. In preparation, draft available online. • Irani, Lilly C., and Six Silberman. Turkopticon: Interrupting worker invisibility in amazon mechanical turk. ACM SIGCHI Conference, 2013. • Adda, Gilles, et al. Crowdsourcing for language resource development: Critical analysis of amazon mechanical turk overpowering use. Proceedings of the 5th Language and Technology Conference (LTC). 2011. • Adda, Gilles, and Joseph J. Mariani. Economic, Legal and Ethical analysis of Crowdsourcing for Speech Processing. (2013). • Harris, Christopher G., and Padmini Srinivasan. Crowdsourcing and Ethics. Security and Privacy in Social Networks. 67-83. 2013. • Harris, Christopher G. Dirty Deeds Done Dirt Cheap: A Darker Side to Crowdsourcing. IEEE 3rd conference on social computing (socialcom). 2011. • Horton, John J. The condition of the Turking class: Are online employers fair and honest?. Economics Letters 111.1 (2011): 10-12. 29
  30. 30. A Few Legal References • Felstiner, Alek. Working the Crowd: Employment and Labor Law in the Crowdsourcing Industry. Berkeley Journal of Employment and Labor Law 32.1 (2011). • Felstiner, Alek. Sweatshop or Paper Route?: Child Labor Laws and In-Game Work. Proceedings of CrowdConf (2010). • Zittrain, Jonathan. Ubiquitous human computing. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 366.1881 (2008): 3813-3821. 30
  31. 31. Join the conversation online • Crowdwork-ethics: “an informal and occasional newsletter for researchers interested in ethical issues in crowd work.” – Email Six Silberman to subscribe (manual) 31