Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
The Rise of Crowd Computing:
Crowdsourcing + Human Computation
Matt Lease
School of Information (“iSchool”) @mattlease
Uni...
• Crowd Computing =
Crowdsourcing + Human Computation
• Crowdsourcing enables new levels of efficiency
& scalability in da...
AAAI Human Computation (HCOMP) Conference
www.humancomputation.com
October 30-November 3, 2016 in Austin
• Attend (learn &...
“The place where people & technology meet”
~ Wobbrock et al., 2009
“iSchools” now exist at 65 universities around the worl...
Motivation
@mattlease
AI effectiveness is often limited by training data size
Problem: creating labeled data is expensive!
Banko and Brill (2001)
What do we do when even
state-of-the-art AI isn’t good enough?
Crowdsourcing
@mattlease
Crowdsourcing
• Jeff Howe. Wired, June 2006.
• Take a job traditionally
performed by a known agent
(often an employee)
• O...
10
• Marketplace for paid crowd work (“micro-tasks”)
– Created in 2005 (remains in “beta” today)
• On-demand, scalable, 24/7 ...
12
MTurk: The Requester
• Sign up with your Amazon account
• Purchase prepaid credit
– No minimum or up-front fee
• Amazon co...
14
Jane saw the man with the binoculars
DEMO!
15
Collecting Data from Crowds
MTurk sparks 2008 “gold rush” for ML training data
• Information Retrieval: Alonso et al., SIG...
17
SQUARE:
A Benchmark
for Crowd
Consensus
@HCOMP’13 ir.ischool.utexas.edu/square
(open source)
Beyond Mechanical Turk: An Analysis of
Paid Crowd Work Platforms
Vakharia and Lease, iConference 2015
Qualitative assessme...
Many Volunteer Crowdsourcing Examples
Matt Lease <ml@utexas.edu>
20
ESP Game (Games With a Purpose)
L. Von Ahn and L. Dabbish (2004)
21
22
reCaptcha
L. von Ahn et al. (2008). In Science.
23
DuoLingo (Launched Nov. 2011)
24
Crowdsourcing Beyond
Data Labeling
@mattlease
What is a Computer?
26
Princeton University Press, 2005
• What was old is new
• Crowdsourcing: A New
Branch of Computer Science
– David Alan Grie...
The Mechanical Turk
The original, constructed and
unveiled in 1770 by Wolfgang
von Kempelen (1734–1804)
28
J. Pontin. Arti...
Davis et al. (2010) The HPU.
HPU
29
• PhD Thesis, December 2005
• Law & von Ahn, Book, June 2011 30
LUIS VON AHN, CMU
Human Computation
ACM Queue, May 2006
31
“Software developers with innovative ideas for
businesses and technologies are constrained by the
l...
Designing Crowd-Powered
Applications
@mattlease
PlateMate (Noronha et al., UIST’10)
33
“Amazon Remembers”
34
Zensors
Laput et al., CSCW 2015
35
VizWiz aaaaaaaa
Bigham et al. (UIST 2010)
36Matt Lease - ml@ischool.utexas.edu
Ethics Checking: The Next Frontier?
• Mark Johnson’s address at ACL 2003
– Transcript in Conduit 12(2) 2003
• Think how us...
Soylent: A Word Processor with a Crowd Inside
• Bernstein et al., UIST 2010
38
MonoTrans:
Translation by monolingual speakers
39
• Bederson et al.,
2010
• See also: Morita & Ishidi, ACM IUI 2009
Counting by Hybrid Divide-&-Conquer
JellyBean
Sarma et al.,
HCOMP 2015
40
Scribe (Lasecki et al., 2012)
Real-time Captioning by Non-professionals
41
fold.it
S. Cooper et al. (2010)
Alice G. Walton. Online Gamers Help Solve Mystery of
Critical AIDS Virus Enzyme. The Atlan...
CACM August, 2013
Paul Hyman. Communications of the ACM, Vol. 56 No. 8, Pages 19-21, August 2013.
Matt Lease <ml@utexas.ed...
The Future of Crowd Work
Paper @ CSCW 2013 by
Kittur, Nickerson, Bernstein, Gerber,
Shaw, Zimmerman, Lease, and Horton 44
Summary
• Crowd Computing =
Crowdsourcing + Human Computation
• Crowdsourcing transforms data collection &
processing via ...
Matt Lease - ml@utexas.edu - @mattlease
Thank You!
ir.ischool.utexas.edu/crowd
Slides: slideshare.net/mattlease
Upcoming SlideShare
Loading in …5
×

The Rise of Crowd Computing (July 7, 2016)

210 views

Published on

Presentation to First Bytes Workshop for High School Computer Science teachers (https://apps.cs.utexas.edu/camp/teachers-workshop)

Published in: Technology
  • Be the first to comment

The Rise of Crowd Computing (July 7, 2016)

  1. 1. The Rise of Crowd Computing: Crowdsourcing + Human Computation Matt Lease School of Information (“iSchool”) @mattlease University of Texas at Austin ml@utexas.edu Slides: slideshare.net/mattlease Videos: ir.ischool.utexas.edu
  2. 2. • Crowd Computing = Crowdsourcing + Human Computation • Crowdsourcing enables new levels of efficiency & scalability in data collection & processing • Human Computation lets us build next- generation applications today, providing capabilities beyond state-of-the-art AI Roadmap
  3. 3. AAAI Human Computation (HCOMP) Conference www.humancomputation.com October 30-November 3, 2016 in Austin • Attend (learn & network) • Still opportunities to submit too! (due 8/15) – Industry & Practice (e.g., educational use) – Works-in-progress & Demos
  4. 4. “The place where people & technology meet” ~ Wobbrock et al., 2009 “iSchools” now exist at 65 universities around the world www.ischools.org What’s an Information School (iSchool)? 4
  5. 5. Motivation @mattlease
  6. 6. AI effectiveness is often limited by training data size Problem: creating labeled data is expensive! Banko and Brill (2001)
  7. 7. What do we do when even state-of-the-art AI isn’t good enough?
  8. 8. Crowdsourcing @mattlease
  9. 9. Crowdsourcing • Jeff Howe. Wired, June 2006. • Take a job traditionally performed by a known agent (often an employee) • Outsource it to an undefined, generally large group of people via an open call 9
  10. 10. 10
  11. 11. • Marketplace for paid crowd work (“micro-tasks”) – Created in 2005 (remains in “beta” today) • On-demand, scalable, 24/7 global, paid workforce • API lets human labor be integrated into software – “You’ve heard of software-as-a-service. Now this is human-as-a-service.” Amazon Mechanical Turk (MTurk)
  12. 12. 12
  13. 13. MTurk: The Requester • Sign up with your Amazon account • Purchase prepaid credit – No minimum or up-front fee • Amazon collects a 20-40% commission on workers payments – https://requester.mturk.com/pricing – The minimum commission charge is $0.005 per assignment • Vocabulary: Human Intelligence Task (HIT), Assignment 13
  14. 14. 14 Jane saw the man with the binoculars
  15. 15. DEMO! 15
  16. 16. Collecting Data from Crowds MTurk sparks 2008 “gold rush” for ML training data • Information Retrieval: Alonso et al., SIGIR Forum • Human-Computer Interaction: Kittur et al., CHI • Computer Vision: Sorokin & Forsythe, CVPR • NLP: Snow et al, EMNLP – Annotating human language – 22,000 labels for only US $26 – Crowd’s consensus labels can replace traditional expert labels
  17. 17. 17 SQUARE: A Benchmark for Crowd Consensus @HCOMP’13 ir.ischool.utexas.edu/square (open source)
  18. 18. Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms Vakharia and Lease, iConference 2015 Qualitative assessment of 7 crowd work platforms
  19. 19. Many Volunteer Crowdsourcing Examples Matt Lease <ml@utexas.edu>
  20. 20. 20
  21. 21. ESP Game (Games With a Purpose) L. Von Ahn and L. Dabbish (2004) 21
  22. 22. 22
  23. 23. reCaptcha L. von Ahn et al. (2008). In Science. 23
  24. 24. DuoLingo (Launched Nov. 2011) 24
  25. 25. Crowdsourcing Beyond Data Labeling @mattlease
  26. 26. What is a Computer? 26
  27. 27. Princeton University Press, 2005 • What was old is new • Crowdsourcing: A New Branch of Computer Science – David Alan Grier, 2013 IEEE Society President 27
  28. 28. The Mechanical Turk The original, constructed and unveiled in 1770 by Wolfgang von Kempelen (1734–1804) 28 J. Pontin. Artificial Intelligence, With Help From the Humans. New York Times (March 25, 2007)
  29. 29. Davis et al. (2010) The HPU. HPU 29
  30. 30. • PhD Thesis, December 2005 • Law & von Ahn, Book, June 2011 30 LUIS VON AHN, CMU Human Computation
  31. 31. ACM Queue, May 2006 31 “Software developers with innovative ideas for businesses and technologies are constrained by the limits of artificial intelligence… If software developers could programmatically access and incorporate human intelligence into their applications, a whole new class of innovative businesses and applications would be possible. This is the goal of Amazon Mechanical Turk… people are freer to innovate because they can now imbue software with real human intelligence.”
  32. 32. Designing Crowd-Powered Applications @mattlease
  33. 33. PlateMate (Noronha et al., UIST’10) 33
  34. 34. “Amazon Remembers” 34
  35. 35. Zensors Laput et al., CSCW 2015 35
  36. 36. VizWiz aaaaaaaa Bigham et al. (UIST 2010) 36Matt Lease - ml@ischool.utexas.edu
  37. 37. Ethics Checking: The Next Frontier? • Mark Johnson’s address at ACL 2003 – Transcript in Conduit 12(2) 2003 • Think how useful a little “ethics checker and corrector” program integrated into a word processor could be! 37
  38. 38. Soylent: A Word Processor with a Crowd Inside • Bernstein et al., UIST 2010 38
  39. 39. MonoTrans: Translation by monolingual speakers 39 • Bederson et al., 2010 • See also: Morita & Ishidi, ACM IUI 2009
  40. 40. Counting by Hybrid Divide-&-Conquer JellyBean Sarma et al., HCOMP 2015 40
  41. 41. Scribe (Lasecki et al., 2012) Real-time Captioning by Non-professionals 41
  42. 42. fold.it S. Cooper et al. (2010) Alice G. Walton. Online Gamers Help Solve Mystery of Critical AIDS Virus Enzyme. The Atlantic, October 8, 2011. 42
  43. 43. CACM August, 2013 Paul Hyman. Communications of the ACM, Vol. 56 No. 8, Pages 19-21, August 2013. Matt Lease <ml@utexas.edu>
  44. 44. The Future of Crowd Work Paper @ CSCW 2013 by Kittur, Nickerson, Bernstein, Gerber, Shaw, Zimmerman, Lease, and Horton 44
  45. 45. Summary • Crowd Computing = Crowdsourcing + Human Computation • Crowdsourcing transforms data collection & processing via greater efficiency & scalability • Human Computation lets us build next- generation applications today, providing capabilities beyond state-of-the-art AI
  46. 46. Matt Lease - ml@utexas.edu - @mattlease Thank You! ir.ischool.utexas.edu/crowd Slides: slideshare.net/mattlease

×