Wisdom and Folly of the Crowds


Published on

Published in: Design, Technology, Business
  • Be the first to comment

Wisdom and Folly of the Crowds

  1. 1. Wisdom (and Folly) of the Crowds<br />Jessica Keup<br />
  2. 2. Outline<br />Definitions<br />Uses<br />Pros/Cons<br />Crowdsourcing Marketplaces<br />Crowdsourcing Research<br />
  3. 3. History<br />“Crowdsourcing is a method of solving problems with, and outsourcing work, to the collective online intelligence” <br />James Surowiecki - The Wisdom of Crowds, 2005<br />
  4. 4. History<br />“Remember outsourcing? Sending jobs to India and China is so 2003. The new pool of cheap labor: everyday people using their spare cycles to create content, solve problems, even do corporate R & D.”<br />Jeff Howe - The Rise of Crowdsourcing, 2006<br />
  5. 5. Crowdsourcing vs . . . <br />Outsourcing <br />work given to pre-decided person or group<br />Open Source<br />initiated by public, not a client<br />
  6. 6. Examples?<br />Digg/Reddit <br />iStockPhoto <br />Netflix prize<br />Threadless<br />ReCAPTCHA<br />Google image labeler<br />EquitySplash / Piqqem<br />
  7. 7. Best Uses<br />Image tagging<br />Search relevance evaluation<br />Content generation<br />Product categorization<br />Collecting feedback and reviews<br />Transcriptions<br />
  8. 8. Pros<br />Labor at reduced cost<br />Around-the-clock<br />Huge potential workforce<br />Variety of opinions, ideas, experience<br />Consensus and better answers<br />
  9. 9. Cons<br />Cheating/manipulation <br />Inexperience/ignorance <br />Confidentiality<br />
  10. 10. Fixes<br />Cheating/manipulation<br />corroboration, ratings<br />Inexperience/ignorance<br />qualification tests<br />Confidentiality<br />?<br />
  11. 11. Crowdsourcing Marketplaces<br />CrowdFlower<br />Mechanical Turk<br />Cloud Crowd<br />Bootb<br />crowdSPRING <br />Innocentive<br />TopCoder<br />Kluster<br />
  12. 12. mTurk<br />“Artificial Artificial Intelligence”<br />Human Intelligence Tasks (HITs)<br />$0.00 and up, mostly a few cents<br />SDKs in Java, Ruby, Perl, .NET<br />Or, create through website UI*<br />
  13. 13. Training AI<br />“Computer Music Composition using Crowdsourcing and Genetic Algorithms”<br />DarwinTunes<br />
  14. 14.
  15. 15.
  16. 16. Research Examples<br />collecting creative drawings <br />rating audio and video sample quality <br />tagging location-sensitive queries and points of interest with mobile devices <br />remote massage <br />task delegation in Wikipedia <br />collecting volunteer expertise to help the homeless <br />annotating political campaign ads <br />
  17. 17. More Research examples<br />describing and organizing geometric shapes*<br />tracking popular music <br />rating search result relevance <br />collecting user feedback <br />extracting product and brand information <br />tagging specific objects in an image<br />training speech recognition software <br />
  18. 18. Questions?<br />
  19. 19. References<br />Alonso, O., Rose, D.E., and Stewart, B. (2008). Crowdsourcing for Relevance Evaluation. SIGIR Forum 42(2) 9-15. <br />Amazon.com, Inc. (2010). Amazon Mechanical Turk. https://www.mturk.com/mturk/welcome. <br />CastingWords (2010). Audio transcription services: plus video, CDs, podcasts and more. http://castingwords.com/ <br />Chung, K., Chiu, C., Xiao, X., & Chi, P. (2009). Stress outsourced: A haptic social network via crowdsourcing. Paper presented at the CHI EA '09: Proceedings of the 27th International Conference Extended Abstracts on Human Factors in Computing Systems, Boston, MA, USA. 2439-2448. <br />DarwinTunes (2010).Welcome to DarwinTunes. http://darwintunes.org/<br />Ganjisaffar, Y., Javanmardi, S., & Lopes, C. (2009). Leveraging crowdsourcing heuristics to improve search in wikipedia. Paper presented at the WikiSym '09: Proceedings of the 5th International Symposium on Wikis and Open Collaboration, Orlando, Florida. 1-2.<br />Harper, F. M., Raban, D., Rafaeli, S. & Konstan, J. A. (2008). Predictors of answer quality in online Q & A sites. Paper presented at the CHI '08: Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems, Florence, Italy. 865-874.<br />Hintikka, K. A. (2008). Web 2.0 and the collective intelligence. Paper presented at the MindTrek '08: Proceedings of the 12th International Conference on Entertainment and Media in the Ubiquitous Era, Tampere, Finland. 163-166.<br />Howe, J. (2006). The Rise of Crowdsourcing. Wired, 14(6). http://www.wired.com/wired/archive/14.06/crowds.html. <br />
  20. 20. More References<br />Hsueh, P., Melville, P., & Sindhwani, V. (2009). Data quality from crowdsourcing: A study of annotation selection criteria. Paper presented at the HLT '09: Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing, Boulder, Colorado. 27-35.<br />Jagadeesan, A. P., Lynn, A., Corney, J. R., Yan, X. T., Wenzel, J., Sherlock, A., et al. (2009). Geometric reasoning via internet CrowdSourcing. Paper presented at the SPM '09: 2009 SIAM/ACM Joint Conference on Geometric and Physical Modeling, San Francisco, California. 313-318.<br />Kittur, A., Chi, ,E. H., & Suh, B. (2008). Crowdsourcing user studies with mechanical turk. Paper presented at the CHI '08: Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems, Florence, Italy. 453-456. <br />Koblin, A. M. (2009). The sheep market. Paper presented at the C&C '09: Proceeding of the Seventh ACM Conference on Creativity and Cognition, Berkeley, California, USA. 451-452.<br />Krieger, M., Stark, E. M., & Klemmer, S. R. (2009). Coordinating tasks on the commons: Designing for personal goals, expertise and serendipity. Paper presented at the CHI '09: Proceedings of the 27th International Conference on Human Factors in Computing Systems, Boston, MA, USA. 1485-1494.<br />Ledlie, J., Odero, B., Minkov, E., Kiss, I., & Polifroni, J. (2009). Crowd translator: On building localized speech recognizers through micropayments. SIGOPS Operating.Systems Review, 43(4), 84-89.<br />Li, C., Buyuktur, A. G., Hutchful, D. K., Sant, N. B., & Nainwal, S. K. (2008). Portalis: Using competitive online interactions to support aid initiatives for the homeless. Paper presented at the CHI '08: CHI '08 Extended Abstracts on Human Factors in Computing Systems, Florence, Italy. 3873-3878.<br />
  21. 21. Even More References<br />Mannes, A. E. (2009). Are we wise about the wisdom of crowds? the use of group judgments in belief revision. Management Science, 55(8), 1267-1279. <br />Mason, W., & Watts, D. J. (2009). Financial incentives and the performance of crowds. Paper presented at the HCOMP '09: Proceedings of the ACM SIGKDD Workshop on Human Computation, Paris, France. 77-85. <br />Milne, D., & Witten, I. H. (2008). Learning to link with wikipedia. Paper presented at the CIKM '08: Proceedings of the 17th ACM Conference on Information and Knowledge Management, Napa Valley, California, USA. 509-518.<br />Roman, D. (2009). Crowdsourcing and the question of expertise. Commnuications of the.ACM, 52(12), 12-12. <br />Stewart, O., Huerta, J. M., & Sader, M. (2009). Designing crowdsourcing community for the enterprise. Paper presented at the HCOMP '09: Proceedings of the ACM SIGKDD Workshop on Human Computation, Paris, France. 50-53.<br />Su, Q., Pavlov, D., Chow, J., & Baker, W. C. (2007). Internet-scale collection of human-reviewed data. Paper presented at the WWW '07: Proceedings of the 16th International Conference on World Wide Web, Banff, Alberta, Canada. 231-240.<br />Surowiecki, J. (2005). The wisdom of crowds. Anchor.<br />Tanaka, A., Tokui, N., & Momeni, A. (2005). Facilitating collective musical creativity. Paper presented at the MULTIMEDIA '05: Proceedings of the 13th Annual ACM International Conference on Multimedia, Hilton, Singapore. 191-198.<br />Von Ahn L., & Dabbish, L. (2004). Labeling images with a computer game. Paper presented at the CHI '04: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Vienna, Austria. 319-326.<br />