The Rise of Crowd Computing
Matt Lease
School of Information @mattlease
University of Texas at Austin ml@utexas.edu
Slides...
“The place where people & technology meet”
~ Wobbrock et al., 2009
“iSchools” now exist at 65 universities around the worl...
Roadmap
• Motivation from Artificial Intelligence (AI)
– Need for Plentiful Labeled Data
– Need for Capabilities Beyond Wh...
Motivation 1:
AI effectiveness is often limited by training data size
Problem: creating labeled data is expensive!
Banko a...
Motivation 2: What do we do when
state-of-the-art AI isn’t good enough?
Crowdsourcing
@mattlease
Crowdsourcing
• Jeff Howe. Wired, June 2006.
• Take a job traditionally
performed by a known agent
(often an employee)
• O...
• Marketplace for paid crowd work (“micro-tasks”)
– Created in 2005 (remains in “beta” today)
• On-demand, scalable, 24/7 ...
Find Jim Gray (February 2007)
The Early Days
• S
• Artificial Intelligence, With Help From the Humans.
– J. Pontin. NY Times, March 25, 2007
• Is Amazon...
The 1st Wave of Crowd Computing:
Data Collection via Crowdsourcing
@mattlease
2008
MTurk “Discovery” sparks rush for “gold” labels across areas
• Alonso et al., SIGIR Forum (Information Retrieval)
• K...
NLP Example – Dialect Identification
14
See work by Chris Callison-Burch. Interface:
http://ugrad.cs.jhu.edu/~sjeblee/arab...
August 12, 2012 15
Social & Behavioral Sciences
• A Guide to Behavioral Experiments
on Mechanical Turk
– W. Mason and S. Suri (2010). SSRN on...
Beyond MTurk
@mattlease
Crowdsourcing ≠ Mechanical Turk!
Many Platforms for Paid Crowd Work
And More!
JobBoy, microWorkers, MiniFreelance,
MiniJobz, MinuteWorkers, MyEasyTask,
OpT...
Why Eytan Adar hates MTurk Research
(CHI 2011 CHC Workshop)
• Overly-narrow research focus on MTurk
– Distinguish general ...
Beyond Mechanical Turk: An Analysis of
Paid Crowd Work Platforms
Vakharia and Lease, iConference 2015
Qualitative assessme...
22
Crowdsourcing Transcription Beyond
Mechanical Turk
With Haofeng Zhou & Denys Baskov
HCOMP 2013 Speech Workshop
A lot of volunteer, Citizen Science too!
Citizen science in the early internet (2000-2001)
24
www.nasaclickworkers.com
Zooniverse
25
Crowd4U (Another Open Platform)
26
27
ESP Game (Gamification)
L. Von Ahn and L. Dabbish (2004)
28
reCaptcha (Repurpose Existing Activity)
von Ahn et al. (2008). In Science.
29
DuoLingo (Education)
30
Tracking Sentiment (Access Resource)
Brew et al., PAIS 2010
• Volunteer-crowd
– Work in exchange for
access to rich conten...
Beat the Machine (Earn Money)
32
The 2nd Wave of Crowd Computing:
Human Computation
@mattlease
What is a Computer?
34
Princeton University Press, 2005
• What was old is new
• Crowdsourcing: A New
Branch of Computer Science
– D.A. Grier, IEE...
The Mechanical Turk
The original, constructed and
unveiled in 1770 by Wolfgang
von Kempelen (1734–1804)
36
J. Pontin. Arti...
The Human Processing Unit (HPU)
Davis et al. (2010)
HPU
37
ACM Queue, May 2006
38
“Software developers with innovative ideas for
businesses and technologies are constrained by the
l...
Creating A New Class of
Intelligent Applications
39@mattlease
“Amazon Remembers”
40
PlateMate (Noronha et al., UIST’10)
41
Ethics Checking: The Next Frontier?
• Mark Johnson’s address at ACL 2003
– Transcript in Conduit 12(2) 2003
• Think how us...
Soylent: A Word Processor with a Crowd Inside
• Bernstein et al., UIST 2010
43
fold.it
S. Cooper et al. (2010)
Alice G. Walton. Online Gamers Help Solve Mystery of
Critical AIDS Virus Enzyme. The Atlan...
MonoTrans:
Translation by monolingual speakers
45
• Bederson et al.,
2010
• See also: Morita & Ishidi, ACM IUI 2009
VizWiz aaaaaaaa
Bingham et al. (UIST 2010)
46Matt Lease - ml@ischool.utexas.edu
Zensors
Laput et al., CSCW 2015
47
Flock: Hybrid Crowd-Machine Learning
Classifiers (2015)
48
@mattlease
HCOMP 2013 Panel
Anand Kulkarni: “How do we
dramatically reduce the complexity of
getting work done with the crowd?”
Greg ...
How to ensure data quality?
• Research on statistical quality control methods
– Online vs. offline, feature-based vs. cont...
52
SQUARE:
A Benchmark
for Research on
Computing
Crowd
Consensus
@HCOMP’13
ir.ischool.utexas.edu/square
(open source)
Is everyone just lazy, stupid, or deceitful?!?
Many published papers seem to suggest this
• “Cheaters”
• “Fraudsters”
• “L...
What is our responsibility?
• Ill-defined/incomplete/ambiguous/subjective task?
• Confusing, difficult, or unusable interf...
Task Decomposition & Workflow Design
55
What about context?
“Best practices” for crowdsourcing design often
minimizes context to maximize task efficiency
– e.g. “...
Importance of Informed Consent +
Potential for Oppression, Crime, & War
Jonathan Zittrain, Minds for Sale
57
Who are
the workers?
• A. Baio, November 2008. The Faces of Mechanical Turk.
• P. Ipeirotis. March 2010. The New Demograph...
What about ethics?
• Silberman, Irani, and Ross (2010)
– “How should we… conceptualize the role of these people
who we ask...
Digital Dirty Jobs
• The Googler who Looked at the Worst of the Internet
• Policing the Web’s Lurid Precincts
• Facebook c...
What about freedom?
• Crowdsourcing vision: empowering freedom
– work whenever you want for whomever you want
• Risk: peop...
The Future of Crowd Work
Paper @ CSCW 2013 by
Kittur, Nickerson, Bernstein, Gerber,
Shaw, Zimmerman, Lease, and Horton 62
Conclusion
• Crowdsourcing is quickly transforming practice
in industry and academia via greater efficiency
• Human Comput...
Matt Lease - ml@utexas.edu - @mattlease
Thank You!
ir.ischool.utexas.edu/crowd
Slides: slideshare.net/mattlease
The Rise of Crowd Computing (December 2015)
Upcoming SlideShare
Loading in …5
×

The Rise of Crowd Computing (December 2015)

767 views

Published on

Talks given at the Qatar Computing Research Institute (QCRI) and Qatar University in December 2015.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
767
On SlideShare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

The Rise of Crowd Computing (December 2015)

  1. 1. The Rise of Crowd Computing Matt Lease School of Information @mattlease University of Texas at Austin ml@utexas.edu Slides: slideshare.net/mattlease
  2. 2. “The place where people & technology meet” ~ Wobbrock et al., 2009 “iSchools” now exist at 65 universities around the world www.ischools.org What’s an Information School? 2
  3. 3. Roadmap • Motivation from Artificial Intelligence (AI) – Need for Plentiful Labeled Data – Need for Capabilities Beyond What AI Can Deliver • The Rise of Crowd Computing – 1st Wave: Crowd-based data labeling • Mechanical Turk & Beyond – 2nd Wave: Crowd-based Human Computation • Delivering beyond state-of-the-art AI applications today • Open Problems 3
  4. 4. Motivation 1: AI effectiveness is often limited by training data size Problem: creating labeled data is expensive! Banko and Brill (2001)
  5. 5. Motivation 2: What do we do when state-of-the-art AI isn’t good enough?
  6. 6. Crowdsourcing @mattlease
  7. 7. Crowdsourcing • Jeff Howe. Wired, June 2006. • Take a job traditionally performed by a known agent (often an employee) • Outsource it to an undefined, generally large group of people via an open call 7
  8. 8. • Marketplace for paid crowd work (“micro-tasks”) – Created in 2005 (remains in “beta” today) • On-demand, scalable, 24/7 global workforce • API lets human labor be integrated into software – “You’ve heard of software-as-a-service. Now this is human-as-a-service.” Amazon Mechanical Turk (MTurk)
  9. 9. Find Jim Gray (February 2007)
  10. 10. The Early Days • S • Artificial Intelligence, With Help From the Humans. – J. Pontin. NY Times, March 25, 2007 • Is Amazon's Mechanical Turk a Failure? April 9, 2007 – “As of this writing, there are [only] 128 Human Intelligence Tasks available via the Mechanical Turk task page.” • Su et al., WWW 2007: “a web-based human data collection system that we [call] ‘System M’ ” 11
  11. 11. The 1st Wave of Crowd Computing: Data Collection via Crowdsourcing @mattlease
  12. 12. 2008 MTurk “Discovery” sparks rush for “gold” labels across areas • Alonso et al., SIGIR Forum (Information Retrieval) • Kittur et al., CHI (Human-Computer Interaction) • Sorokin and Forsythe, CVPR (Computer Vision) Snow et al, EMNLP (NLP) • Annotating human language • 22,000 labels for only US $26 • Crowd’s consensus labels can replace traditional expert labels
  13. 13. NLP Example – Dialect Identification 14 See work by Chris Callison-Burch. Interface: http://ugrad.cs.jhu.edu/~sjeblee/arabic-classification-plus.shtml
  14. 14. August 12, 2012 15
  15. 15. Social & Behavioral Sciences • A Guide to Behavioral Experiments on Mechanical Turk – W. Mason and S. Suri (2010). SSRN online. • Crowdsourcing for Human Subjects Research – L. Schmidt (CrowdConf 2010) • Crowdsourcing Content Analysis for Behavioral Research: Insights from Mechanical Turk – Conley & Tosti-Kharas (2010). Academy of Management • Amazon's Mechanical Turk : A New Source of Inexpensive, Yet High-Quality, Data? – M. Buhrmester et al. (2011). Perspectives… 6(1):3-5. – see also: Amazon Mechanical Turk Guide for Social Scientists 16
  16. 16. Beyond MTurk @mattlease
  17. 17. Crowdsourcing ≠ Mechanical Turk!
  18. 18. Many Platforms for Paid Crowd Work And More! JobBoy, microWorkers, MiniFreelance, MiniJobz, MinuteWorkers, MyEasyTask, OpTask, ShortTask, SimpleWorkers
  19. 19. Why Eytan Adar hates MTurk Research (CHI 2011 CHC Workshop) • Overly-narrow research focus on MTurk – Distinguish general vs. platform-specific problems – Distinguish research vs. industry concerns • Should researchers really focus on… – “...writing the user’s manual for MTurk ...”? – “…struggl[ing] against the limits of the platform...”? “…by rewarding quick demonstrations of the tool’s use, we fail to attain a deeper understanding of the problems to which it is applied…”
  20. 20. Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms Vakharia and Lease, iConference 2015 Qualitative assessment of 7 platforms for paid crowd work
  21. 21. 22 Crowdsourcing Transcription Beyond Mechanical Turk With Haofeng Zhou & Denys Baskov HCOMP 2013 Speech Workshop
  22. 22. A lot of volunteer, Citizen Science too!
  23. 23. Citizen science in the early internet (2000-2001) 24 www.nasaclickworkers.com
  24. 24. Zooniverse 25
  25. 25. Crowd4U (Another Open Platform) 26
  26. 26. 27
  27. 27. ESP Game (Gamification) L. Von Ahn and L. Dabbish (2004) 28
  28. 28. reCaptcha (Repurpose Existing Activity) von Ahn et al. (2008). In Science. 29
  29. 29. DuoLingo (Education) 30
  30. 30. Tracking Sentiment (Access Resource) Brew et al., PAIS 2010 • Volunteer-crowd – Work in exchange for access to rich content • Never-ending Learning – Continual model updates as what is relevant vs. not changes over time 31
  31. 31. Beat the Machine (Earn Money) 32
  32. 32. The 2nd Wave of Crowd Computing: Human Computation @mattlease
  33. 33. What is a Computer? 34
  34. 34. Princeton University Press, 2005 • What was old is new • Crowdsourcing: A New Branch of Computer Science – D.A. Grier, IEEE President • Tabulating the heavens: computing the Nautical Almanac in 18th-century England – M. Croarken (2003) 35
  35. 35. The Mechanical Turk The original, constructed and unveiled in 1770 by Wolfgang von Kempelen (1734–1804) 36 J. Pontin. Artificial Intelligence, With Help From the Humans. New York Times (March 25, 2007)
  36. 36. The Human Processing Unit (HPU) Davis et al. (2010) HPU 37
  37. 37. ACM Queue, May 2006 38 “Software developers with innovative ideas for businesses and technologies are constrained by the limits of artificial intelligence… If software developers could programmatically access and incorporate human intelligence into their applications, a whole new class of innovative businesses and applications would be possible. This is the goal of Amazon Mechanical Turk… people are freer to innovate because they can now imbue software with real human intelligence.”
  38. 38. Creating A New Class of Intelligent Applications 39@mattlease
  39. 39. “Amazon Remembers” 40
  40. 40. PlateMate (Noronha et al., UIST’10) 41
  41. 41. Ethics Checking: The Next Frontier? • Mark Johnson’s address at ACL 2003 – Transcript in Conduit 12(2) 2003 • Think how useful a little “ethics checker and corrector” program integrated into a word processor could be! 42
  42. 42. Soylent: A Word Processor with a Crowd Inside • Bernstein et al., UIST 2010 43
  43. 43. fold.it S. Cooper et al. (2010) Alice G. Walton. Online Gamers Help Solve Mystery of Critical AIDS Virus Enzyme. The Atlantic, October 8, 2011. 44
  44. 44. MonoTrans: Translation by monolingual speakers 45 • Bederson et al., 2010 • See also: Morita & Ishidi, ACM IUI 2009
  45. 45. VizWiz aaaaaaaa Bingham et al. (UIST 2010) 46Matt Lease - ml@ischool.utexas.edu
  46. 46. Zensors Laput et al., CSCW 2015 47
  47. 47. Flock: Hybrid Crowd-Machine Learning Classifiers (2015) 48
  48. 48. @mattlease
  49. 49. HCOMP 2013 Panel Anand Kulkarni: “How do we dramatically reduce the complexity of getting work done with the crowd?” Greg Little: How can we post a task and with 98% confidence know we’ll get a quality result? 50
  50. 50. How to ensure data quality? • Research on statistical quality control methods – Online vs. offline, feature-based vs. content-agnostic – Worker calibration, noise vs. bias, weighted voting • Human factors matter too! – Instructions, design, interface, interaction – Names, relationship, reputation Fair pay, hourly vs. per-task, recognition, advancement 51
  51. 51. 52 SQUARE: A Benchmark for Research on Computing Crowd Consensus @HCOMP’13 ir.ischool.utexas.edu/square (open source)
  52. 52. Is everyone just lazy, stupid, or deceitful?!? Many published papers seem to suggest this • “Cheaters” • “Fraudsters” • “Lazy Turkers” • “Scammers” • “Spammers” But why can’t the workers just get it right to begin with? 53
  53. 53. What is our responsibility? • Ill-defined/incomplete/ambiguous/subjective task? • Confusing, difficult, or unusable interface? • Incomplete or unclear instructions? • Insufficient or unhelpful examples given? • Gold standard with low or unknown inter-assessor agreement (i.e. measurement error in assessing response quality)? • Task design matters! (garbage in = garbage out) – Report it for review, completeness, & reproducibility 54
  54. 54. Task Decomposition & Workflow Design 55
  55. 55. What about context? “Best practices” for crowdsourcing design often minimizes context to maximize task efficiency – e.g. “Are these pictures of the same person?” 56
  56. 56. Importance of Informed Consent + Potential for Oppression, Crime, & War Jonathan Zittrain, Minds for Sale 57
  57. 57. Who are the workers? • A. Baio, November 2008. The Faces of Mechanical Turk. • P. Ipeirotis. March 2010. The New Demographics of Mechanical Turk • J. Ross, et al. Who are the Crowdworkers? CHI 2010. 58
  58. 58. What about ethics? • Silberman, Irani, and Ross (2010) – “How should we… conceptualize the role of these people who we ask to power our computing?” • Irani and Silberman (2013) – “…by hiding workers behind web forms and APIs… employers see themselves as builders of innovative technologies, rather than… unconcerned with working conditions… redirecting focus to the innovation of human computation as a field of technological achievement.” • Fort, Adda, and Cohen (2011) – “…opportunities for our community to deliberately value ethics above cost savings.” 59
  59. 59. Digital Dirty Jobs • The Googler who Looked at the Worst of the Internet • Policing the Web’s Lurid Precincts • Facebook content moderation • The dirty job of keeping Facebook clean • Even linguistic annotators report stress & nightmares from reading news articles! 60
  60. 60. What about freedom? • Crowdsourcing vision: empowering freedom – work whenever you want for whomever you want • Risk: people being compelled to perform work – Digital sweat shops? Digital slaves? – Chinese Prisoners used for online gold farming – We really don’t know (and need to learn more…) – Traction? Human Trafficking at MSR Summit’12 61
  61. 61. The Future of Crowd Work Paper @ CSCW 2013 by Kittur, Nickerson, Bernstein, Gerber, Shaw, Zimmerman, Lease, and Horton 62
  62. 62. Conclusion • Crowdsourcing is quickly transforming practice in industry and academia via greater efficiency • Human Computation enables a new design space for applications, augmenting state-of-the- art AI with human computation to offer new capabilities and user experiences • With people at the center of this new computing paradigm, important research questions span both technological and social/societal challenges
  63. 63. Matt Lease - ml@utexas.edu - @mattlease Thank You! ir.ischool.utexas.edu/crowd Slides: slideshare.net/mattlease

×