What about regulation?
• Wolfson and Lease. Look before you leap: Legal pitfalls of
crowdsourcing. Proceedings of the American Society for
Information Science and Technology (ASIS&T), 2011.
• As usual, technology is ahead of the law
– employment law
– patent inventorship
– data security and the Federal Trade Commission
– copyright ownership
– securities regulation of crowdfunding
• Take-away: don’t panic, but be mindful
– Understand risks of “just in-time compliance”
• A. Baio, November 2008. The Faces of Mechanical Turk.
• P. Ipeirotis. March 2010. The New Demographics of
• J. Ross, et al. Who are the Crowdworkers? CHI 2010.
What about ethics?
• Six Silberman, Lilly Irani, and Joel Ross. Ethics and
tactics of professional crowdwork. XRDS: Crossroads,
The ACM Magazine for Students 17.2 (2010): 39-43.
– “How should we… conceptualize the role of these people
who we ask to power our computing?”
– Power dynamics
– “Abstraction hides detail”
• Fort, Karën, Gilles Adda, and K. Bretonnel Cohen.
Amazon mechanical turk: Gold mine or coal mine?.
Computational Linguistics 37.2 (2011): 413-420.
– “…opportunities for our community to deliberately
value ethics above cost savings.”
Requester Fraud on MTurk
“Do not do any HITs that involve: filling in
CAPTCHAs; secret shopping; test our web page;
test zip code; free trial; click my link; surveys or
quizzes (unless the requester is listed with a
smiley in the Hall of Fame/Shame); anything
that involves sending a text message; or
basically anything that asks for any personal
information at all—even your zip code. If you
feel in your gut it’s not on the level, IT’S NOT.
Why? Because they are scams...”
• Is this really the requester’s captcha?
• “…not only do malicious crowd-sourcing
systems exist, but they are rapidly growing…”
Digital Dirty Jobs
• The Googler who Looked at the Worst of the Internet
• Policing the Web’s Lurid Precincts
• Facebook content moderation
• The dirty job of keeping Facebook clean
• Even linguistic annotators report stress &
nightmares from reading news articles!
Crowdsourcing and Vigilantism
What about freedom?
• Crowdsourcing vision: empowering freedom
– work whenever you want for whomever you want
• Risk: people being compelled to perform work
– Chinese prisoners farming gold…
– Digital sweat shops? Digital slaves?
– We really don’t know (and need to learn more…)
– Traction? Human Trafficking at MSR Summit’12
Mechanical Turk is Not Anonymous
Matthew Lease, Jessica Hullman, Jeffrey P. Bigham, Michael S. Bernstein, Juho Kim,
Walter S. Lasecki, Saeideh Bakhshi, Tanushree Mitra, and Robert C. Miller.
Online: Social Science Research Network, March 6, 2013
Each worker is assigned an alphanumeric ID
Amazon profile page
URLs use the same
IDs used on MTurk !
Workers’ Views: Survey & Forums
• “... my reviewer profile is linked to my Mturk number! I had
• “...Amazon needs to separate the Mturk numbers from
seller numbers to protect our privacy…”
• “I think this is outrageous though. Makes me concerned
about trusting privacy agreements.”
• “Mine pulled up my Amazon wish list which revealed my
identity. It seems to me that so called ”anonymous” tasks
on mTurk (like surveys) are not anonymous after all.”
• Inadvertent disclosure of PII or private data
• Loss of blind hiring practices online
• Greater risk of exploitation, reputation damage,
loss of income, or even physical harm…
Risks to Researchers
• Exposing participants to undocumented risks
• Having disclosed WorkerIDs (e.g., online)
• Having not restricted access to the internally
– Potential harm to participants
– Lack of compliance with Federal/IRB governance
of human subjects research
– Being required to discard collected data
– Delays or inability to conduct future MTurk studies
Risks to Service Provider
• Workers/Requesters abandoning MTurk
• The US Federal Trade Commission (FTC) has
recently begun to aggressively protect consumers
from data breaches by commercial entities,
including release of supposedly “anonymous” data
– Inadequate protection of customer records: BJWC
– De-anonymized customer records: AOL, Netflix
– Did workers have a reasonable expectation of privacy
in their use of MTurk which has been violated? 27
The Future of Crowd Work, CSCW’13
Kittur, Nickerson, Bernstein, Gerber,
Shaw, Zimmerman, Lease, and Horton
• Paul Hyman. Software Aims to Ensure Fairness in Crowdsourcing Projects.
Communications of the ACM, Vol. 56 No. 8, Pages 19-21, August 2013.
• Irani, Lilly C. The Ideological Work of Microwork. In preparation, draft
• Irani, Lilly C., and Six Silberman. Turkopticon: Interrupting worker
invisibility in amazon mechanical turk. ACM SIGCHI Conference, 2013.
• Adda, Gilles, et al. Crowdsourcing for language resource development:
Critical analysis of amazon mechanical turk overpowering use. Proceedings
of the 5th Language and Technology Conference (LTC). 2011.
• Adda, Gilles, and Joseph J. Mariani. Economic, Legal and Ethical analysis of
Crowdsourcing for Speech Processing. (2013).
• Harris, Christopher G., and Padmini Srinivasan. Crowdsourcing and Ethics.
Security and Privacy in Social Networks. 67-83. 2013.
• Harris, Christopher G. Dirty Deeds Done Dirt Cheap: A Darker Side to
Crowdsourcing. IEEE 3rd conference on social computing (socialcom). 2011.
• Horton, John J. The condition of the Turking class: Are online employers
fair and honest?. Economics Letters 111.1 (2011): 10-12.
A Few Legal References
• Felstiner, Alek. Working the Crowd: Employment
and Labor Law in the Crowdsourcing Industry.
Berkeley Journal of Employment and Labor Law
• Felstiner, Alek. Sweatshop or Paper Route?: Child
Labor Laws and In-Game Work. Proceedings of
• Zittrain, Jonathan. Ubiquitous human computing.
Philosophical Transactions of the Royal Society A:
Mathematical, Physical and Engineering Sciences
366.1881 (2008): 3813-3821.
Join the conversation online
• Crowdwork-ethics: “an informal and
occasional newsletter for researchers
interested in ethical issues in crowd work.”
– Email Six Silberman to subscribe (manual)