Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Crowd Agents: Interactive Crowd-Powered Systems in the Real World

1,422 views

Published on

In this talk, I discuss several interactive crowd-powered systems
that help people address real-world problems. For instance, VizWiz
sends questions blind people have about their visual environment to
the crowd, Legion allows outsourcing of desktop tasks to the crowd,
and Scribe allows the crowd to caption audio in real-time. The
thousands of people have engaged with these systems, providing an
interesting look at how end users want to interact with crowd work.
Collectively, these systems illustrate a new approach to human
computation in which the dynamic crowd is provided the computational
support needed to act as a single, high-quality agent. The classic
advantage of the crowd has been its wisdom, but our systems are
beginning to show how crowd agents can surpass even expert individuals
on motor and cognitive performance tasks.

  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Crowd Agents: Interactive Crowd-Powered Systems in the Real World

  1. 1. Crowd Agents Interactive Crowd-Powered Systems in the Real World Jeffrey P. Bigham University of RochesterUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham
  2. 2. Crowd Agents Interactive Crowd-Powered Systems in the Real World Jeffrey P. Bigham University of RochesterUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham
  3. 3. Introduction VizWiz Crowd Agents ScribeUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham
  4. 4. Introduction VizWiz Crowd Agents Scribe Human Assistance in HistoryWhat the Disability CommunityCan Teach Us About InteractiveCrowdsourcing. Jeffrey P.Bigham and Richard Ladner.Iteractions magazine. July 2011.University of Rochester Human-Computer Interaction Jeffrey P. Bigham
  5. 5. Introduction VizWiz Crowd Agents Scribe ConnectivityUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham
  6. 6. Introduction VizWiz Crowd Agents ScribeUniversity of Rochester Human-Computer Interaction Courtesy Jeffrey P. Brabyn of John Bigham
  7. 7. Introduction VizWiz Crowd Agents Scribe Remote Assistance Video Relay Services Real-time CaptioningUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham
  8. 8. Introduction VizWiz Crowd Agents Scribe Connectivity -> Crowd Mechanical Turk Friends and Family on Social NetworksUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham
  9. 9. Introduction VizWiz Crowd Agents Scribe VizWiz Bigham et al. Nearly Real-Time Answers to Visual Questions. UIST 2010.University of Rochester Human-Computer Interaction Jeffrey P. Bigham
  10. 10. Introduction VizWiz Crowd Agents Scribe Access Technology • Optical Character Recognition • Color Recognizers • Talking GPS • … Problems 1. Limited Scope 2. Unacceptable Error Rate 3. $$$ 4. Not Exactly What Users WantUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham
  11. 11. Introduction VizWiz Crowd Agents Scribe Releasing VizWiz • Released on May 31, 2011 – 5000 users asked more than 50,000 questions – answers in less than a minuteUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham
  12. 12. Introduction VizWiz Crowd Agents Scribe Recruiting Crowd Quickly How many workers do we need? - number of current workers - likelihood of needing more workers Post jobs or remove jobs Turkers answer multiple questions Turkit For $4/hr goes down to under 30s from start to finish. quikturkit.googlecode.com Bigham et al. Nearly Real-Time Answers to Visual Questions. UIST 2010.University of Rochester Human-Computer Interaction Jeffrey P. Bigham
  13. 13. Introduction VizWiz Crowd Agents Scribe Characterization of the Crowd - Workers Come and Go - Some May Do the Wrong ThingUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham
  14. 14. Introduction VizWiz Crowd Agents Scribe Supporting a Continuous Interaction? Where’s the coffee? Walk to end of this hall, turn right. Turn right into the kitchen. Where’s the Soda on left, coffee on the right How do I use this machine? coffee?University of Rochester Human-Computer Interaction Jeffrey P. Bigham
  15. 15. Introduction VizWiz Crowd Agents Scribe Model for Crowd AgentsUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham
  16. 16. Introduction VizWiz Crowd Agents Scribe Model for Crowd Agents Input Mediation LearningUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham
  17. 17. Introduction VizWiz Crowd Agents Scribe Model for Crowd Agents • What interface is being controlled? • How is input mediation done? • Role of automated agents?University of Rochester Human-Computer Interaction Jeffrey P. Bigham
  18. 18. Introduction VizWiz Crowd Agents Scribe ChorusUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham
  19. 19. Introduction VizWiz Crowd Agents ScribeUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham
  20. 20. Introduction VizWiz Crowd Agents Scribe Legion: Control of Any Interface Input Media on Legion Server - video stream Flash Media Server - - task description Input Mediators - - crowd agreement/payment info - video stream quikTurkit - - task description - worker input (key presses, mouse clicks) Worker Interface - mediated input Legion Client 250 8/10 8/10 200 10/10 Explanation of Time (sec) controls, and feedback 150 regarding current bonus level (tied to crowd agreement). 10/10 100 4/10 50 0 Feedback reflecting worker’s Solo Mob Vote Active Leader last key press, and whether the interface last followed multiple workers the crowd or the worker. W. Lasecki, S. White, K. Murray, R. Miller, and J.P. Bigham “Real-Time Control of Existing Interfaces.” UIST 2011.University of Rochester Human-Computer Interaction Jeffrey P. Bigham
  21. 21. Introduction VizWiz Crowd Agents ScribeUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham
  22. 22. Introduction VizWiz Crowd Agents Scribe Crowd Memory W.S. Lasecki, S.C. White, K.I. Murray and J.P. Bigham. “Crowd Memory: Learning in the Collective.” Collective Intelligence 2012.University of Rochester Human-Computer Interaction Jeffrey P. Bigham
  23. 23. Introduction VizWiz Crowd Agents Scribe Crowd MemoryUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham
  24. 24. Introduction VizWiz Crowd Agents Scribe Deployable Activity Recognition W.S. Lasecki, Y. Song, H. Kautz, and J.P. Bigham. “Real-Time Activity Labeling for Deployable Activity Recognition.” Submitted to CSCW 2012. Pervasive 2012 (poster)University of Rochester Human-Computer Interaction Jeffrey P. Bigham
  25. 25. Legion:Scribe Real-Time Captions by Groups of Non-ExpertsUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham
  26. 26. Introduction VizWiz Crowd Agents Scribe Real-Time Captioning Problem: produce text transcript of speech with less than 5-second latency Stenographers ASR expensive cheap difficult to schedule available on demand lack domain expertise Can I can be trained for new vocab help? pretty accurate does not work* NO, you are worse than ASR. * in real settings from an unknown mic with speaker who hasn’t trained the ASRUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham
  27. 27. Introduction VizWiz Crowd Agents Scribe Real-Time Captioning W. Lasecki, C. Miller, A. Sadilek, A. Abumoussa, D. Borrello, R. Kushalnagar, J.P. Bigham. “Real-Time Captioning by Groups of Non-Experts.” UIST 2012.University of Rochester Human-Computer Interaction Jeffrey P. Bigham
  28. 28. Introduction VizWiz Crowd Agents Scribe Input Mediator Multiple Sequence Alignment Online Version Stage 1 the Stage 2 the Stage 3 the now and Graph open open open file Time java java java up Worker 1 open the file now Worker 2 the java fiel Worker 3 open java file up and Baseline open the java file now and W.S. Lasecki, C.D. Miller, D. Borrello and J.P. Bigham. “Online Sequence Alignment for Real-Time Audio Transcription by Non-Experts.” AAAI 2012 (poster).University of Rochester Human-Computer Interaction Jeffrey P. Bigham
  29. 29. Introduction VizWiz Crowd Agents Scribe Scribe Interface Encourages: - real-time input - global coverage - short sequences Co-evolution of Interface and AlgorithmUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham
  30. 30. Introduction VizWiz Crowd Agents Scribe Coverage GraphUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham
  31. 31. Introduction VizWiz Crowd Agents Scribe Tradeoff Failures: “n-factorial”  “in pectoral”University of Rochester Human-Computer Interaction Jeffrey P. Bigham
  32. 32. Introduction VizWiz Crowd Agents Scribe Interesting Qualities • Captionists can be experts – not at captioning but in the subject • Low cost – $30/hour on Mturk (did not optimize) – or free (impossible before) • Recruited on demand – for only as long as neededUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham
  33. 33. Introduction VizWiz Crowd Agents Scribe Scribe ASR Web prefetching is 1 technique that A lactate fencing is one thinking that ressearchers rely on history based to and etc. rely on to improve network. the non history based technique the Phillipe pitching. Anything survived downloaded pages will be scanned and all incident techniques… hyperlinks will be…University of Rochester Human-Computer Interaction Jeffrey P. Bigham
  34. 34. Introduction VizWiz Crowd Agents Scribe Incorporating ASR Coverage Increase: 28% to 55% (single worker case)University of Rochester Human-Computer Interaction Jeffrey P. Bigham
  35. 35. Conclusions General Lessons, Science, and the FutureUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham
  36. 36. Introduction VizWiz Crowd Agents Scribe “What would it take for me to be proud of my daughter being a crowd worker?” - Niki Kittur @ CrowdCamp Currence Bigham after her first running race.University of Rochester Human-Computer Interaction Jeffrey P. Bigham
  37. 37. Introduction VizWiz Crowd Agents Scribe Do Good Connect to help and support. Do Better Do better work than anyone could alone.University of Rochester Human-Computer Interaction Jeffrey P. Bigham
  38. 38. Introduction VizWiz Crowd Agents Scribe hci.cs.rochester.edu @jeffbigham Thanks! Funded by: National Science Foundation Grants (#IIS-1149709, #IIS- 1116051, #IIS-1049080 ), and Google.University of Rochester Human-Computer Interaction Jeffrey P. Bigham
  39. 39. Introduction VizWiz Crowd Agents ScribeUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham

×