Presentation to First Bytes Workshop for High School Computer Science teachers (https://apps.cs.utexas.edu/camp/teachers-workshop)

1. The Rise of Crowd Computing:
Crowdsourcing + Human Computation
Matt Lease
School of Information (“iSchool”) @mattlease
University of Texas at Austin ml@utexas.edu
Slides:
slideshare.net/mattlease
Videos:
ir.ischool.utexas.edu

2. • Crowd Computing =
Crowdsourcing + Human Computation
• Crowdsourcing enables new levels of efficiency
& scalability in data collection & processing
• Human Computation lets us build next-
generation applications today, providing
capabilities beyond state-of-the-art AI
Roadmap

3. AAAI Human Computation (HCOMP) Conference
www.humancomputation.com
October 30-November 3, 2016 in Austin
• Attend (learn & network)
• Still opportunities to submit too! (due 8/15)
– Industry & Practice (e.g., educational use)
– Works-in-progress & Demos

4. “The place where people & technology meet”
~ Wobbrock et al., 2009
“iSchools” now exist at 65 universities around the world
www.ischools.org
What’s an Information School (iSchool)?
4

5. Motivation
@mattlease

6. AI effectiveness is often limited by training data size
Problem: creating labeled data is expensive!
Banko and Brill (2001)

7. What do we do when even
state-of-the-art AI isn’t good enough?

8. Crowdsourcing
@mattlease

9. Crowdsourcing
• Jeff Howe. Wired, June 2006.
• Take a job traditionally
performed by a known agent
(often an employee)
• Outsource it to an undefined,
generally large group of
people via an open call
9

10. 10

11. • Marketplace for paid crowd work (“micro-tasks”)
– Created in 2005 (remains in “beta” today)
• On-demand, scalable, 24/7 global, paid workforce
• API lets human labor be integrated into software
– “You’ve heard of software-as-a-service. Now this is human-as-a-service.”
Amazon Mechanical Turk (MTurk)

12. 12

13. MTurk: The Requester
• Sign up with your Amazon account
• Purchase prepaid credit
– No minimum or up-front fee
• Amazon collects a 20-40% commission on workers payments
– https://requester.mturk.com/pricing
– The minimum commission charge is $0.005 per assignment
• Vocabulary: Human Intelligence Task (HIT), Assignment
13

14. 14
Jane saw the man with the binoculars

15. DEMO!
15

16. Collecting Data from Crowds
MTurk sparks 2008 “gold rush” for ML training data
• Information Retrieval: Alonso et al., SIGIR Forum
• Human-Computer Interaction: Kittur et al., CHI
• Computer Vision: Sorokin & Forsythe, CVPR
• NLP: Snow et al, EMNLP
– Annotating human language
– 22,000 labels for only US $26
– Crowd’s consensus labels can
replace traditional expert labels

17. 17
SQUARE:
A Benchmark
for Crowd
Consensus
@HCOMP’13 ir.ischool.utexas.edu/square
(open source)

18. Beyond Mechanical Turk: An Analysis of
Paid Crowd Work Platforms
Vakharia and Lease, iConference 2015
Qualitative assessment of 7 crowd work platforms

19. Many Volunteer Crowdsourcing Examples
Matt Lease <ml@utexas.edu>

20. 20

21. ESP Game (Games With a Purpose)
L. Von Ahn and L. Dabbish (2004)
21

22. 22

23. reCaptcha
L. von Ahn et al. (2008). In Science.
23

24. DuoLingo (Launched Nov. 2011)
24

25. Crowdsourcing Beyond
Data Labeling
@mattlease

26. What is a Computer?
26

27. Princeton University Press, 2005
• What was old is new
• Crowdsourcing: A New
Branch of Computer Science
– David Alan Grier,
2013 IEEE Society President
27

28. The Mechanical Turk
The original, constructed and
unveiled in 1770 by Wolfgang
von Kempelen (1734–1804)
28
J. Pontin. Artificial Intelligence, With Help From
the Humans. New York Times (March 25, 2007)

29. Davis et al. (2010) The HPU.
HPU
29

30. • PhD Thesis, December 2005
• Law & von Ahn, Book, June 2011 30
LUIS VON AHN, CMU
Human Computation

31. ACM Queue, May 2006
31
“Software developers with innovative ideas for
businesses and technologies are constrained by the
limits of artificial intelligence… If software developers
could programmatically access and incorporate human
intelligence into their applications, a whole new class
of innovative businesses and applications would be
possible. This is the goal of Amazon Mechanical Turk…
people are freer to innovate because they can now
imbue software with real human intelligence.”

32. Designing Crowd-Powered
Applications
@mattlease

33. PlateMate (Noronha et al., UIST’10)
33

34. “Amazon Remembers”
34

35. Zensors
Laput et al., CSCW 2015
35

36. VizWiz aaaaaaaa
Bigham et al. (UIST 2010)
36Matt Lease - ml@ischool.utexas.edu

37. Ethics Checking: The Next Frontier?
• Mark Johnson’s address at ACL 2003
– Transcript in Conduit 12(2) 2003
• Think how useful a little “ethics checker and
corrector” program integrated into a word
processor could be!
37

38. Soylent: A Word Processor with a Crowd Inside
• Bernstein et al., UIST 2010
38

39. MonoTrans:
Translation by monolingual speakers
39
• Bederson et al.,
2010
• See also: Morita & Ishidi, ACM IUI 2009

40. Counting by Hybrid Divide-&-Conquer
JellyBean
Sarma et al.,
HCOMP 2015
40

41. Scribe (Lasecki et al., 2012)
Real-time Captioning by Non-professionals
41

42. fold.it
S. Cooper et al. (2010)
Alice G. Walton. Online Gamers Help Solve Mystery of
Critical AIDS Virus Enzyme. The Atlantic, October 8, 2011.
42

43. CACM August, 2013
Paul Hyman. Communications of the ACM, Vol. 56 No. 8, Pages 19-21, August 2013.
Matt Lease <ml@utexas.edu>

44. The Future of Crowd Work
Paper @ CSCW 2013 by
Kittur, Nickerson, Bernstein, Gerber,
Shaw, Zimmerman, Lease, and Horton 44

45. Summary
• Crowd Computing =
Crowdsourcing + Human Computation
• Crowdsourcing transforms data collection &
processing via greater efficiency & scalability
• Human Computation lets us build next-
generation applications today, providing
capabilities beyond state-of-the-art AI

46. Matt Lease - ml@utexas.edu - @mattlease
Thank You!
ir.ischool.utexas.edu/crowd
Slides: slideshare.net/mattlease