SlideShare a Scribd company logo
Toward Effective & Sustainable
Online Crowd Work
Matt Lease
School of Information @mattlease
University of Texas at Austin ml@utexas.edu
Matt Lease <ml@utexas.edu>
2
New Capabilities Enable Innovation
But also new “Dirty” Digital Jobs
• Tasks
– Content moderation (Catch-22)
– Solving captchas, SEO, account creation, …
• Goals
– “Legal” fraud, cyber-crime, or cyber-warfare
• Workers
– Unknown or at-risk: refugees, prisoners, children, …
– Typical lack of informed consent
– How do we know work is freely chosen?
• Part of larger, online ecosystem (cf., Robert Sim)
Matt Lease <ml@utexas.edu>
3
Matt Lease <ml@utexas.edu>
4
Debate in ACM
5
Paul Hyman. Communications of the ACM, Vol. 56 No. 8, Pages 19-21, August 2013.
Matt Lease <ml@utexas.edu>
• Contribute to society and human well-being
• Avoid harm to others
• Be honest and trustworthy
• Be fair and take action not to discriminate
• Respect the privacy of others
COMPLIANCE WITH THE CODE. As an ACM member I will
– Uphold and promote the principles of this Code
– Treat violations of this code as inconsistent with membership in the ACM
Financial and economic imperatives …Which approaches are less
expensive and is this sensible? With the advent of outsourcing
and off-shoring these matters become more complex… there
are often related ethical issues concerning exploitation…
6
Matt Lease <ml@utexas.edu>
What about regulation?
• Wolfson & Lease (ASIS&T 2011)
• As usual, technology ahead of the law
– employment law & independent contractors
– patent inventorship
– data security and the Federal Trade Commission
– copyright ownership
– securities regulation of crowdfunding
• Understand risks of “just in-time compliance”
7
Matt Lease <ml@utexas.edu>
Seek Win-Win Solutions
• Pay debate has centered on quality vs. cost tradeoff
• What is “fair” and sustainable in a global economy?
– Is it better to pay nothing? (i.e., volunteers, gamification)
• A meta-approach
– Learn: Find information (e.g., Internet, surveys)
– Share: Publicize information for others
– Act: Find ways to make the “right” thing preferable
Matt Lease <ml@utexas.edu>
8
The Future of Crowd Work, CSCW’13
Kittur, Nickerson, Bernstein, Gerber,
Shaw, Zimmerman, Lease, and Horton
9
Matt Lease <ml@utexas.edu>
Join the conversation!
Crowdwork-ethics, by Six Silberman
http://crowdwork-ethics.wtf.tw
an informal, occasional blog for researchers
interested in ethical issues in crowd work
ir.ischool.utexas.edu
Unanticipated
challenges & risks with
emerging technologies
Lease et al., SSRN’13
11

More Related Content

What's hot

The Rise of Crowd Computing (December 2015)
The Rise of Crowd Computing (December 2015)The Rise of Crowd Computing (December 2015)
The Rise of Crowd Computing (December 2015)
Matthew Lease
 
The Rise of Crowd Computing - 2016
The Rise of Crowd Computing - 2016The Rise of Crowd Computing - 2016
The Rise of Crowd Computing - 2016
Matthew Lease
 
Data Standards and Linked Data: Challenges & Use Cases in Europe and the Unit...
Data Standards and Linked Data: Challenges & Use Cases in Europe and the Unit...Data Standards and Linked Data: Challenges & Use Cases in Europe and the Unit...
Data Standards and Linked Data: Challenges & Use Cases in Europe and the Unit...
Jonathan Pichot
 
Ralph schroeder and eric meyer
Ralph schroeder and eric meyerRalph schroeder and eric meyer
Ralph schroeder and eric meyer
oiisdp
 
Joe keating - world legal summit - ethical data science
Joe keating  - world legal summit - ethical data scienceJoe keating  - world legal summit - ethical data science
Joe keating - world legal summit - ethical data science
Joe Keating
 
Privacy-driven design of Learning Analytics applications – exploring the desi...
Privacy-driven design of Learning Analytics applications – exploring the desi...Privacy-driven design of Learning Analytics applications – exploring the desi...
Privacy-driven design of Learning Analytics applications – exploring the desi...
Tore Hoel
 
AI and Legal Tech in Context: Privacy and Security Commons
AI and Legal Tech in Context: Privacy and Security CommonsAI and Legal Tech in Context: Privacy and Security Commons
AI and Legal Tech in Context: Privacy and Security Commons
professormadison
 
The Rise of Crowd Computing (July 7, 2016)
The Rise of Crowd Computing (July 7, 2016)The Rise of Crowd Computing (July 7, 2016)
The Rise of Crowd Computing (July 7, 2016)
Matthew Lease
 
Data Science and its impact on society
Data Science and its impact on societyData Science and its impact on society
Data Science and its impact on society
Vienna Data Science Group
 
Student vulnerability, agency and learning analytics: an exploration
Student vulnerability, agency and learning analytics: an explorationStudent vulnerability, agency and learning analytics: an exploration
Student vulnerability, agency and learning analytics: an exploration
University of South Africa (Unisa)
 
Crowdsourcing for Search Evaluation and Social-Algorithmic Search
Crowdsourcing for Search Evaluation and Social-Algorithmic SearchCrowdsourcing for Search Evaluation and Social-Algorithmic Search
Crowdsourcing for Search Evaluation and Social-Algorithmic Search
Matthew Lease
 
Feedback on Technology Governance, Strategy, and Funding Proposal: Executive ...
Feedback on Technology Governance, Strategy, and Funding Proposal: Executive ...Feedback on Technology Governance, Strategy, and Funding Proposal: Executive ...
Feedback on Technology Governance, Strategy, and Funding Proposal: Executive ...
Ron Dolin
 
Crowd Computing: Opportunities & Challenges (IJCNLP 2011 Keynote)
Crowd Computing: Opportunities & Challenges (IJCNLP 2011 Keynote)Crowd Computing: Opportunities & Challenges (IJCNLP 2011 Keynote)
Crowd Computing: Opportunities & Challenges (IJCNLP 2011 Keynote)
Matthew Lease
 
INSPIRE Conference 2013 - Environmental Data on the Web
INSPIRE Conference 2013 - Environmental Data on the WebINSPIRE Conference 2013 - Environmental Data on the Web
INSPIRE Conference 2013 - Environmental Data on the Web
Alex Coley
 
Knoesis Student Achievement
Knoesis Student AchievementKnoesis Student Achievement
Knoesis Student Achievement
Artificial Intelligence Institute at UofSC
 
Introduction to data science
Introduction to data scienceIntroduction to data science
Introduction to data science
Tharushi Ruwandika
 
Broad Data
Broad DataBroad Data
Broad Data
James Hendler
 

What's hot (17)

The Rise of Crowd Computing (December 2015)
The Rise of Crowd Computing (December 2015)The Rise of Crowd Computing (December 2015)
The Rise of Crowd Computing (December 2015)
 
The Rise of Crowd Computing - 2016
The Rise of Crowd Computing - 2016The Rise of Crowd Computing - 2016
The Rise of Crowd Computing - 2016
 
Data Standards and Linked Data: Challenges & Use Cases in Europe and the Unit...
Data Standards and Linked Data: Challenges & Use Cases in Europe and the Unit...Data Standards and Linked Data: Challenges & Use Cases in Europe and the Unit...
Data Standards and Linked Data: Challenges & Use Cases in Europe and the Unit...
 
Ralph schroeder and eric meyer
Ralph schroeder and eric meyerRalph schroeder and eric meyer
Ralph schroeder and eric meyer
 
Joe keating - world legal summit - ethical data science
Joe keating  - world legal summit - ethical data scienceJoe keating  - world legal summit - ethical data science
Joe keating - world legal summit - ethical data science
 
Privacy-driven design of Learning Analytics applications – exploring the desi...
Privacy-driven design of Learning Analytics applications – exploring the desi...Privacy-driven design of Learning Analytics applications – exploring the desi...
Privacy-driven design of Learning Analytics applications – exploring the desi...
 
AI and Legal Tech in Context: Privacy and Security Commons
AI and Legal Tech in Context: Privacy and Security CommonsAI and Legal Tech in Context: Privacy and Security Commons
AI and Legal Tech in Context: Privacy and Security Commons
 
The Rise of Crowd Computing (July 7, 2016)
The Rise of Crowd Computing (July 7, 2016)The Rise of Crowd Computing (July 7, 2016)
The Rise of Crowd Computing (July 7, 2016)
 
Data Science and its impact on society
Data Science and its impact on societyData Science and its impact on society
Data Science and its impact on society
 
Student vulnerability, agency and learning analytics: an exploration
Student vulnerability, agency and learning analytics: an explorationStudent vulnerability, agency and learning analytics: an exploration
Student vulnerability, agency and learning analytics: an exploration
 
Crowdsourcing for Search Evaluation and Social-Algorithmic Search
Crowdsourcing for Search Evaluation and Social-Algorithmic SearchCrowdsourcing for Search Evaluation and Social-Algorithmic Search
Crowdsourcing for Search Evaluation and Social-Algorithmic Search
 
Feedback on Technology Governance, Strategy, and Funding Proposal: Executive ...
Feedback on Technology Governance, Strategy, and Funding Proposal: Executive ...Feedback on Technology Governance, Strategy, and Funding Proposal: Executive ...
Feedback on Technology Governance, Strategy, and Funding Proposal: Executive ...
 
Crowd Computing: Opportunities & Challenges (IJCNLP 2011 Keynote)
Crowd Computing: Opportunities & Challenges (IJCNLP 2011 Keynote)Crowd Computing: Opportunities & Challenges (IJCNLP 2011 Keynote)
Crowd Computing: Opportunities & Challenges (IJCNLP 2011 Keynote)
 
INSPIRE Conference 2013 - Environmental Data on the Web
INSPIRE Conference 2013 - Environmental Data on the WebINSPIRE Conference 2013 - Environmental Data on the Web
INSPIRE Conference 2013 - Environmental Data on the Web
 
Knoesis Student Achievement
Knoesis Student AchievementKnoesis Student Achievement
Knoesis Student Achievement
 
Introduction to data science
Introduction to data scienceIntroduction to data science
Introduction to data science
 
Broad Data
Broad DataBroad Data
Broad Data
 

Similar to Toward Effective and Sustainable Online Crowd Work

Crowdsourcing: From Aggregation to Search Engine Evaluation
Crowdsourcing: From Aggregation to Search Engine EvaluationCrowdsourcing: From Aggregation to Search Engine Evaluation
Crowdsourcing: From Aggregation to Search Engine Evaluation
Matthew Lease
 
Crowdsourcing & ethics: a few thoughts and refences.
Crowdsourcing & ethics: a few thoughts and refences. Crowdsourcing & ethics: a few thoughts and refences.
Crowdsourcing & ethics: a few thoughts and refences.
Matthew Lease
 
AI & Work, with Transparency & the Crowd
AI & Work, with Transparency & the Crowd AI & Work, with Transparency & the Crowd
AI & Work, with Transparency & the Crowd
Matthew Lease
 
The Challenge of Benefit-Cost Analysis As Applied to Online Safety & Digital ...
The Challenge of Benefit-Cost Analysis As Applied to Online Safety & Digital ...The Challenge of Benefit-Cost Analysis As Applied to Online Safety & Digital ...
The Challenge of Benefit-Cost Analysis As Applied to Online Safety & Digital ...
Adam Thierer
 
Explainable Fact Checking with Humans in-the-loop
Explainable Fact Checking with Humans in-the-loopExplainable Fact Checking with Humans in-the-loop
Explainable Fact Checking with Humans in-the-loop
Matthew Lease
 
A Lifecycle Approach to Information Privacy
A Lifecycle Approach to Information PrivacyA Lifecycle Approach to Information Privacy
A Lifecycle Approach to Information Privacy
Micah Altman
 
Thierer Internet Privacy Regulation
Thierer Internet Privacy RegulationThierer Internet Privacy Regulation
Thierer Internet Privacy Regulation
Mercatus Center
 
Governing Privacy Commons at Pitt Science 2017 - Madison
Governing Privacy Commons at Pitt Science 2017 - MadisonGoverning Privacy Commons at Pitt Science 2017 - Madison
Governing Privacy Commons at Pitt Science 2017 - Madison
professormadison
 
AI and Robotics policy overview - Adam Thierer (Aug 2022)
AI and Robotics policy overview - Adam Thierer (Aug 2022)AI and Robotics policy overview - Adam Thierer (Aug 2022)
AI and Robotics policy overview - Adam Thierer (Aug 2022)
Adam Thierer
 
“Permissionless Innovation” & the Grand Tech Policy Clash of Visions to Come
“Permissionless Innovation” & the Grand Tech Policy Clash of Visions to Come“Permissionless Innovation” & the Grand Tech Policy Clash of Visions to Come
“Permissionless Innovation” & the Grand Tech Policy Clash of Visions to Come
Mercatus Center
 
Ethical and social issues in information systems
Ethical and social issues in information systemsEthical and social issues in information systems
Privacy & the Internet: An Overview of Key Issues
Privacy & the Internet: An Overview of Key IssuesPrivacy & the Internet: An Overview of Key Issues
Privacy & the Internet: An Overview of Key Issues
Adam Thierer
 
UT Dallas CS - Rise of Crowd Computing
UT Dallas CS - Rise of Crowd ComputingUT Dallas CS - Rise of Crowd Computing
UT Dallas CS - Rise of Crowd Computing
Matthew Lease
 
Age Friendly Economy - Legislation and Ethics of Data Use
Age Friendly Economy - Legislation and Ethics of Data UseAge Friendly Economy - Legislation and Ethics of Data Use
Age Friendly Economy - Legislation and Ethics of Data Use
AgeFriendlyEconomy
 
Regulating code
Regulating codeRegulating code
Regulating code
blogzilla
 
Forging Self-Sovereign Identity in the Age of the Blockchain - Christopher Al...
Forging Self-Sovereign Identity in the Age of the Blockchain - Christopher Al...Forging Self-Sovereign Identity in the Age of the Blockchain - Christopher Al...
Forging Self-Sovereign Identity in the Age of the Blockchain - Christopher Al...
Christopher Allen
 
Ethics in it
Ethics in itEthics in it
Ethics in it
Akash Mathapati
 
Blockchain law and governance: General Conclusion -- Milan, October 2019
Blockchain law and governance: General Conclusion -- Milan, October 2019Blockchain law and governance: General Conclusion -- Milan, October 2019
Blockchain law and governance: General Conclusion -- Milan, October 2019
Tony Lai
 
Mathews blockchain presentation
Mathews blockchain presentationMathews blockchain presentation
Mathews blockchain presentation
Michael Mathews
 
3 - Social Media and Enterprise
3 - Social Media and Enterprise3 - Social Media and Enterprise
3 - Social Media and Enterprise
Raymond Gao
 

Similar to Toward Effective and Sustainable Online Crowd Work (20)

Crowdsourcing: From Aggregation to Search Engine Evaluation
Crowdsourcing: From Aggregation to Search Engine EvaluationCrowdsourcing: From Aggregation to Search Engine Evaluation
Crowdsourcing: From Aggregation to Search Engine Evaluation
 
Crowdsourcing & ethics: a few thoughts and refences.
Crowdsourcing & ethics: a few thoughts and refences. Crowdsourcing & ethics: a few thoughts and refences.
Crowdsourcing & ethics: a few thoughts and refences.
 
AI & Work, with Transparency & the Crowd
AI & Work, with Transparency & the Crowd AI & Work, with Transparency & the Crowd
AI & Work, with Transparency & the Crowd
 
The Challenge of Benefit-Cost Analysis As Applied to Online Safety & Digital ...
The Challenge of Benefit-Cost Analysis As Applied to Online Safety & Digital ...The Challenge of Benefit-Cost Analysis As Applied to Online Safety & Digital ...
The Challenge of Benefit-Cost Analysis As Applied to Online Safety & Digital ...
 
Explainable Fact Checking with Humans in-the-loop
Explainable Fact Checking with Humans in-the-loopExplainable Fact Checking with Humans in-the-loop
Explainable Fact Checking with Humans in-the-loop
 
A Lifecycle Approach to Information Privacy
A Lifecycle Approach to Information PrivacyA Lifecycle Approach to Information Privacy
A Lifecycle Approach to Information Privacy
 
Thierer Internet Privacy Regulation
Thierer Internet Privacy RegulationThierer Internet Privacy Regulation
Thierer Internet Privacy Regulation
 
Governing Privacy Commons at Pitt Science 2017 - Madison
Governing Privacy Commons at Pitt Science 2017 - MadisonGoverning Privacy Commons at Pitt Science 2017 - Madison
Governing Privacy Commons at Pitt Science 2017 - Madison
 
AI and Robotics policy overview - Adam Thierer (Aug 2022)
AI and Robotics policy overview - Adam Thierer (Aug 2022)AI and Robotics policy overview - Adam Thierer (Aug 2022)
AI and Robotics policy overview - Adam Thierer (Aug 2022)
 
“Permissionless Innovation” & the Grand Tech Policy Clash of Visions to Come
“Permissionless Innovation” & the Grand Tech Policy Clash of Visions to Come“Permissionless Innovation” & the Grand Tech Policy Clash of Visions to Come
“Permissionless Innovation” & the Grand Tech Policy Clash of Visions to Come
 
Ethical and social issues in information systems
Ethical and social issues in information systemsEthical and social issues in information systems
Ethical and social issues in information systems
 
Privacy & the Internet: An Overview of Key Issues
Privacy & the Internet: An Overview of Key IssuesPrivacy & the Internet: An Overview of Key Issues
Privacy & the Internet: An Overview of Key Issues
 
UT Dallas CS - Rise of Crowd Computing
UT Dallas CS - Rise of Crowd ComputingUT Dallas CS - Rise of Crowd Computing
UT Dallas CS - Rise of Crowd Computing
 
Age Friendly Economy - Legislation and Ethics of Data Use
Age Friendly Economy - Legislation and Ethics of Data UseAge Friendly Economy - Legislation and Ethics of Data Use
Age Friendly Economy - Legislation and Ethics of Data Use
 
Regulating code
Regulating codeRegulating code
Regulating code
 
Forging Self-Sovereign Identity in the Age of the Blockchain - Christopher Al...
Forging Self-Sovereign Identity in the Age of the Blockchain - Christopher Al...Forging Self-Sovereign Identity in the Age of the Blockchain - Christopher Al...
Forging Self-Sovereign Identity in the Age of the Blockchain - Christopher Al...
 
Ethics in it
Ethics in itEthics in it
Ethics in it
 
Blockchain law and governance: General Conclusion -- Milan, October 2019
Blockchain law and governance: General Conclusion -- Milan, October 2019Blockchain law and governance: General Conclusion -- Milan, October 2019
Blockchain law and governance: General Conclusion -- Milan, October 2019
 
Mathews blockchain presentation
Mathews blockchain presentationMathews blockchain presentation
Mathews blockchain presentation
 
3 - Social Media and Enterprise
3 - Social Media and Enterprise3 - Social Media and Enterprise
3 - Social Media and Enterprise
 

More from Matthew Lease

Automated Models for Quantifying Centrality of Survey Responses
Automated Models for Quantifying Centrality of Survey ResponsesAutomated Models for Quantifying Centrality of Survey Responses
Automated Models for Quantifying Centrality of Survey Responses
Matthew Lease
 
Key Challenges in Moderating Social Media: Accuracy, Cost, Scalability, and S...
Key Challenges in Moderating Social Media: Accuracy, Cost, Scalability, and S...Key Challenges in Moderating Social Media: Accuracy, Cost, Scalability, and S...
Key Challenges in Moderating Social Media: Accuracy, Cost, Scalability, and S...
Matthew Lease
 
Adventures in Crowdsourcing : Toward Safer Content Moderation & Better Suppor...
Adventures in Crowdsourcing : Toward Safer Content Moderation & Better Suppor...Adventures in Crowdsourcing : Toward Safer Content Moderation & Better Suppor...
Adventures in Crowdsourcing : Toward Safer Content Moderation & Better Suppor...
Matthew Lease
 
Designing Human-AI Partnerships to Combat Misinfomation
Designing Human-AI Partnerships to Combat Misinfomation Designing Human-AI Partnerships to Combat Misinfomation
Designing Human-AI Partnerships to Combat Misinfomation
Matthew Lease
 
But Who Protects the Moderators?
But Who Protects the Moderators?But Who Protects the Moderators?
But Who Protects the Moderators?
Matthew Lease
 
Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact...
Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact...Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact...
Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact...
Matthew Lease
 
Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collectio...
Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collectio...Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collectio...
Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collectio...
Matthew Lease
 
Fact Checking & Information Retrieval
Fact Checking & Information RetrievalFact Checking & Information Retrieval
Fact Checking & Information Retrieval
Matthew Lease
 
Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to E...
Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to E...Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to E...
Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to E...
Matthew Lease
 
What Can Machine Learning & Crowdsourcing Do for You? Exploring New Tools for...
What Can Machine Learning & Crowdsourcing Do for You? Exploring New Tools for...What Can Machine Learning & Crowdsourcing Do for You? Exploring New Tools for...
What Can Machine Learning & Crowdsourcing Do for You? Exploring New Tools for...
Matthew Lease
 
Deep Learning for Information Retrieval: Models, Progress, & Opportunities
Deep Learning for Information Retrieval: Models, Progress, & OpportunitiesDeep Learning for Information Retrieval: Models, Progress, & Opportunities
Deep Learning for Information Retrieval: Models, Progress, & Opportunities
Matthew Lease
 
Systematic Review is e-Discovery in Doctor’s Clothing
Systematic Review is e-Discovery in Doctor’s ClothingSystematic Review is e-Discovery in Doctor’s Clothing
Systematic Review is e-Discovery in Doctor’s Clothing
Matthew Lease
 
Multidimensional Relevance Modeling via Psychometrics & Crowdsourcing: ACM SI...
Multidimensional Relevance Modeling via Psychometrics & Crowdsourcing: ACM SI...Multidimensional Relevance Modeling via Psychometrics & Crowdsourcing: ACM SI...
Multidimensional Relevance Modeling via Psychometrics & Crowdsourcing: ACM SI...
Matthew Lease
 
Crowdsourcing Transcription Beyond Mechanical Turk
Crowdsourcing Transcription Beyond Mechanical TurkCrowdsourcing Transcription Beyond Mechanical Turk
Crowdsourcing Transcription Beyond Mechanical Turk
Matthew Lease
 
Crowdsourcing for Information Retrieval: From Statistics to Ethics
Crowdsourcing for Information Retrieval: From Statistics to EthicsCrowdsourcing for Information Retrieval: From Statistics to Ethics
Crowdsourcing for Information Retrieval: From Statistics to Ethics
Matthew Lease
 
Crowdsourcing & Human Computation Labeling Data & Building Hybrid Systems
Crowdsourcing & Human Computation Labeling Data & Building Hybrid SystemsCrowdsourcing & Human Computation Labeling Data & Building Hybrid Systems
Crowdsourcing & Human Computation Labeling Data & Building Hybrid Systems
Matthew Lease
 
Mechanical Turk is Not Anonymous
Mechanical Turk is Not AnonymousMechanical Turk is Not Anonymous
Mechanical Turk is Not Anonymous
Matthew Lease
 
Rise of Crowd Computing (December 2012)
Rise of Crowd Computing (December 2012)Rise of Crowd Computing (December 2012)
Rise of Crowd Computing (December 2012)
Matthew Lease
 
UT Austin @ TREC 2012 Crowdsourcing Track: Image Relevance Assessment Task (I...
UT Austin @ TREC 2012 Crowdsourcing Track: Image Relevance Assessment Task (I...UT Austin @ TREC 2012 Crowdsourcing Track: Image Relevance Assessment Task (I...
UT Austin @ TREC 2012 Crowdsourcing Track: Image Relevance Assessment Task (I...
Matthew Lease
 

More from Matthew Lease (19)

Automated Models for Quantifying Centrality of Survey Responses
Automated Models for Quantifying Centrality of Survey ResponsesAutomated Models for Quantifying Centrality of Survey Responses
Automated Models for Quantifying Centrality of Survey Responses
 
Key Challenges in Moderating Social Media: Accuracy, Cost, Scalability, and S...
Key Challenges in Moderating Social Media: Accuracy, Cost, Scalability, and S...Key Challenges in Moderating Social Media: Accuracy, Cost, Scalability, and S...
Key Challenges in Moderating Social Media: Accuracy, Cost, Scalability, and S...
 
Adventures in Crowdsourcing : Toward Safer Content Moderation & Better Suppor...
Adventures in Crowdsourcing : Toward Safer Content Moderation & Better Suppor...Adventures in Crowdsourcing : Toward Safer Content Moderation & Better Suppor...
Adventures in Crowdsourcing : Toward Safer Content Moderation & Better Suppor...
 
Designing Human-AI Partnerships to Combat Misinfomation
Designing Human-AI Partnerships to Combat Misinfomation Designing Human-AI Partnerships to Combat Misinfomation
Designing Human-AI Partnerships to Combat Misinfomation
 
But Who Protects the Moderators?
But Who Protects the Moderators?But Who Protects the Moderators?
But Who Protects the Moderators?
 
Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact...
Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact...Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact...
Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact...
 
Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collectio...
Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collectio...Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collectio...
Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collectio...
 
Fact Checking & Information Retrieval
Fact Checking & Information RetrievalFact Checking & Information Retrieval
Fact Checking & Information Retrieval
 
Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to E...
Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to E...Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to E...
Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to E...
 
What Can Machine Learning & Crowdsourcing Do for You? Exploring New Tools for...
What Can Machine Learning & Crowdsourcing Do for You? Exploring New Tools for...What Can Machine Learning & Crowdsourcing Do for You? Exploring New Tools for...
What Can Machine Learning & Crowdsourcing Do for You? Exploring New Tools for...
 
Deep Learning for Information Retrieval: Models, Progress, & Opportunities
Deep Learning for Information Retrieval: Models, Progress, & OpportunitiesDeep Learning for Information Retrieval: Models, Progress, & Opportunities
Deep Learning for Information Retrieval: Models, Progress, & Opportunities
 
Systematic Review is e-Discovery in Doctor’s Clothing
Systematic Review is e-Discovery in Doctor’s ClothingSystematic Review is e-Discovery in Doctor’s Clothing
Systematic Review is e-Discovery in Doctor’s Clothing
 
Multidimensional Relevance Modeling via Psychometrics & Crowdsourcing: ACM SI...
Multidimensional Relevance Modeling via Psychometrics & Crowdsourcing: ACM SI...Multidimensional Relevance Modeling via Psychometrics & Crowdsourcing: ACM SI...
Multidimensional Relevance Modeling via Psychometrics & Crowdsourcing: ACM SI...
 
Crowdsourcing Transcription Beyond Mechanical Turk
Crowdsourcing Transcription Beyond Mechanical TurkCrowdsourcing Transcription Beyond Mechanical Turk
Crowdsourcing Transcription Beyond Mechanical Turk
 
Crowdsourcing for Information Retrieval: From Statistics to Ethics
Crowdsourcing for Information Retrieval: From Statistics to EthicsCrowdsourcing for Information Retrieval: From Statistics to Ethics
Crowdsourcing for Information Retrieval: From Statistics to Ethics
 
Crowdsourcing & Human Computation Labeling Data & Building Hybrid Systems
Crowdsourcing & Human Computation Labeling Data & Building Hybrid SystemsCrowdsourcing & Human Computation Labeling Data & Building Hybrid Systems
Crowdsourcing & Human Computation Labeling Data & Building Hybrid Systems
 
Mechanical Turk is Not Anonymous
Mechanical Turk is Not AnonymousMechanical Turk is Not Anonymous
Mechanical Turk is Not Anonymous
 
Rise of Crowd Computing (December 2012)
Rise of Crowd Computing (December 2012)Rise of Crowd Computing (December 2012)
Rise of Crowd Computing (December 2012)
 
UT Austin @ TREC 2012 Crowdsourcing Track: Image Relevance Assessment Task (I...
UT Austin @ TREC 2012 Crowdsourcing Track: Image Relevance Assessment Task (I...UT Austin @ TREC 2012 Crowdsourcing Track: Image Relevance Assessment Task (I...
UT Austin @ TREC 2012 Crowdsourcing Track: Image Relevance Assessment Task (I...
 

Recently uploaded

Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
MichaelKnudsen27
 
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSFAppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
Ajin Abraham
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Hiroshi SHIBATA
 
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Tosin Akinosho
 
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
Edge AI and Vision Alliance
 
Skybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoptionSkybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoption
Tatiana Kojar
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Brandon Minnick, MBA
 
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge GraphGraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
Neo4j
 
Apps Break Data
Apps Break DataApps Break Data
Apps Break Data
Ivo Velitchkov
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
DianaGray10
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
ssuserfac0301
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
panagenda
 
Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
c5vrf27qcz
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
Jakub Marek
 
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
saastr
 
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
saastr
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Alpen-Adria-Universität
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
Miro Wengner
 
June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
Ivanti
 

Recently uploaded (20)

Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
 
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSFAppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
 
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
 
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
 
Skybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoptionSkybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoption
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
 
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge GraphGraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
 
Apps Break Data
Apps Break DataApps Break Data
Apps Break Data
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
 
Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
 
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
 
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
 
Artificial Intelligence and Electronic Warfare
Artificial Intelligence and Electronic WarfareArtificial Intelligence and Electronic Warfare
Artificial Intelligence and Electronic Warfare
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
 
June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
 

Toward Effective and Sustainable Online Crowd Work

  • 1. Toward Effective & Sustainable Online Crowd Work Matt Lease School of Information @mattlease University of Texas at Austin ml@utexas.edu
  • 2. Matt Lease <ml@utexas.edu> 2 New Capabilities Enable Innovation
  • 3. But also new “Dirty” Digital Jobs • Tasks – Content moderation (Catch-22) – Solving captchas, SEO, account creation, … • Goals – “Legal” fraud, cyber-crime, or cyber-warfare • Workers – Unknown or at-risk: refugees, prisoners, children, … – Typical lack of informed consent – How do we know work is freely chosen? • Part of larger, online ecosystem (cf., Robert Sim) Matt Lease <ml@utexas.edu> 3
  • 5. Debate in ACM 5 Paul Hyman. Communications of the ACM, Vol. 56 No. 8, Pages 19-21, August 2013. Matt Lease <ml@utexas.edu>
  • 6. • Contribute to society and human well-being • Avoid harm to others • Be honest and trustworthy • Be fair and take action not to discriminate • Respect the privacy of others COMPLIANCE WITH THE CODE. As an ACM member I will – Uphold and promote the principles of this Code – Treat violations of this code as inconsistent with membership in the ACM Financial and economic imperatives …Which approaches are less expensive and is this sensible? With the advent of outsourcing and off-shoring these matters become more complex… there are often related ethical issues concerning exploitation… 6 Matt Lease <ml@utexas.edu>
  • 7. What about regulation? • Wolfson & Lease (ASIS&T 2011) • As usual, technology ahead of the law – employment law & independent contractors – patent inventorship – data security and the Federal Trade Commission – copyright ownership – securities regulation of crowdfunding • Understand risks of “just in-time compliance” 7 Matt Lease <ml@utexas.edu>
  • 8. Seek Win-Win Solutions • Pay debate has centered on quality vs. cost tradeoff • What is “fair” and sustainable in a global economy? – Is it better to pay nothing? (i.e., volunteers, gamification) • A meta-approach – Learn: Find information (e.g., Internet, surveys) – Share: Publicize information for others – Act: Find ways to make the “right” thing preferable Matt Lease <ml@utexas.edu> 8
  • 9. The Future of Crowd Work, CSCW’13 Kittur, Nickerson, Bernstein, Gerber, Shaw, Zimmerman, Lease, and Horton 9 Matt Lease <ml@utexas.edu>
  • 10. Join the conversation! Crowdwork-ethics, by Six Silberman http://crowdwork-ethics.wtf.tw an informal, occasional blog for researchers interested in ethical issues in crowd work ir.ischool.utexas.edu
  • 11. Unanticipated challenges & risks with emerging technologies Lease et al., SSRN’13 11