The Rise of Crowd Computing - 2016


Published on

Talk at South-by-southwest (SXSW) on crowdsourcing + human computation = crowd computing., March 11, 2016. See

Global growth in Internet connectivity and participation is driving a renaissance in human computation: use of people rather than machines to perform certain computations for which human competency continues to exceed that of state-of-the-art algorithms (e.g. “AI-hard” tasks such as interpreting text or images). While current AI limitations will certainly improve with time, using human computation lets us bulid applications which deliver superior results today. Just as cloud computing now enables us to harness vast Internet computing resources on demand, new crowdsourcing APIsenable us to build computing systems which integrate human computation at run-time, invoking crowd labor on-demand and at-scale. Moreover, we can achieve the best of both worlds by integrating automated AI with human computation, creating hybrid systems with capabilities greater than the sum of their parts. When AI falls short, not only can human computation meet the immediate end-user need, but the results can be fed back into the system to further improve the AI. As a consequence, AI limitations are no longer a bottleneck to delivering innovative, new applications. Such enhanced capabilities have begun to change how we design and implement intelligent systems. While early work in crowd computing focused only on collecting more data from crowds to better train AI, we are increasingly seeing hybrid, socio-computational system emerge which creatively blend human computation and AI at run-time to solve hard computing problems. As such, we find ourselves today in an exhilarating new design space in which intelligent system capabilities are seemingly limited only by our imagination and creativity in designing new algorithms to compute effectively using crowds as well as silicon.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

The Rise of Crowd Computing - 2016

  1. 1. The Rise of Crowd Computing Matt Lease School of Information @mattlease University of Texas at Austin Slides:
  2. 2. “The place where people & technology meet” ~ Wobbrock et al., 2009 “iSchools” now exist at 65 universities around the world What’s an Information School? 2
  3. 3. • Crowd Computing = Crowdsourcing + Human Computation • Crowdsourcing enables new levels of efficiency & scalability in data collection & processing • Human Computation lets us build next- generation applications today, providing capabilities beyond state-of-the-art AI Roadmap
  4. 4. Motivation @mattlease
  5. 5. AI effectiveness is often limited by training data size Problem: creating labeled data is expensive! Banko and Brill (2001)
  6. 6. What do we do when even state-of-the-art AI isn’t good enough?
  7. 7. Crowdsourcing @mattlease
  8. 8. Crowdsourcing • Jeff Howe. Wired, June 2006. • Take a job traditionally performed by a known agent (often an employee) • Outsource it to an undefined, generally large group of people via an open call 8
  9. 9. • Marketplace for paid crowd work (“micro-tasks”) – Created in 2005 (remains in “beta” today) • On-demand, scalable, 24/7 global workforce • API lets human labor be integrated into software – “You’ve heard of software-as-a-service. Now this is human-as-a-service.” Amazon Mechanical Turk (MTurk)
  10. 10. Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms Vakharia and Lease, iConference 2015 Qualitative assessment of 7 crowd work platforms
  11. 11. Collecting Data from Crowds MTurk sparks 2008 “gold rush” for ML training data • Information Retrieval: Alonso et al., SIGIR Forum • Human-Computer Interaction: Kittur et al., CHI • Computer Vision: Sorokin & Forsythe, CVPR • NLP: Snow et al, EMNLP – Annotating human language – 22,000 labels for only US $26 – Crowd’s consensus labels can replace traditional expert labels
  12. 12. 12 SQUARE: A Benchmark for Crowd Consensus @HCOMP’13 (open source)
  13. 13. Human Computation @mattlease
  14. 14. 14
  15. 15. Zensors Laput et al., CSCW 2015 15
  16. 16. ACM Queue, May 2006 16 “Software developers with innovative ideas for businesses and technologies are constrained by the limits of artificial intelligence… If software developers could programmatically access and incorporate human intelligence into their applications, a whole new class of innovative businesses and applications would be possible. This is the goal of Amazon Mechanical Turk… people are freer to innovate because they can now imbue software with real human intelligence.”
  17. 17. PlateMate (Noronha et al., UIST’10) 17
  18. 18. VizWiz aaaaaaaa Bigham et al. (UIST 2010) 18Matt Lease -
  19. 19. “Amazon Remembers” 19
  20. 20. Ethics Checking: The Next Frontier? • Mark Johnson’s address at ACL 2003 – Transcript in Conduit 12(2) 2003 • Think how useful a little “ethics checker and corrector” program integrated into a word processor could be! 20
  21. 21. Soylent: A Word Processor with a Crowd Inside • Bernstein et al., UIST 2010 21
  22. 22. MonoTrans: Translation by monolingual speakers 22 • Bederson et al., 2010 • See also: Morita & Ishidi, ACM IUI 2009
  23. 23. Counting by Hybrid Divide-&-Conquer JellyBean Sarma et al., HCOMP 2015 23
  24. 24. Scribe (Lasecki et al., 2012) Real-time Captioning by Non-professionals 24
  25. 25. S. Cooper et al. (2010) Alice G. Walton. Online Gamers Help Solve Mystery of Critical AIDS Virus Enzyme. The Atlantic, October 8, 2011. 25
  26. 26. @mattlease
  27. 27. AAAI Human Computation (HCOMP) Conference October 30-November 3, 2016 in Austin • Give a Talk @ Industry Track • Attend (learn & network) • Become a Sponsor!
  28. 28. The Future of Crowd Work Paper @ CSCW 2013 by Kittur, Nickerson, Bernstein, Gerber, Shaw, Zimmerman, Lease, and Horton 28
  29. 29. Summary • Crowd Computing = Crowdsourcing + Human Computation • Crowdsourcing transforms data collection & processing via greater efficiency & scalability • Human Computation lets us build next- generation applications today, providing capabilities beyond state-of-the-art AI
  30. 30. Matt Lease - - @mattlease Thank You! Slides: