For designers, the initial reaction to crowd sourcing is almost an instant rejection. Crowd sourcing does seem to de-value the essence of design. While working on her PhD in HCI, Jessica Keup has studied the advantages and disadvantages of crowd sourcing. Enjoy her talk from Big Design Conference 2011 talk, “The Wisdom (and Folly) of the Crowds”.
3. History
“Crowdsourcing is a method of solving
problems with, and outsourcing work, to
the collective online intelligence”
James Surowiecki - The Wisdom of
Crowds, 2005
4. History
“Remember outsourcing? Sending jobs
to India and China is so 2003. The new
pool of cheap labor: everyday people
using their spare cycles to create
content, solve problems, even do
corporate R & D.”
Jeff Howe - The Rise of Crowdsourcing,
2006
5. Crowdsourcing vs . . .
Outsourcing
◦ work given to pre-decided person or
group
Open Source
◦ initiated by public, not a client
Request
Submissions/
Work
Reward
12. mTurk
“Artificial Artificial Intelligence”
Human Intelligence Tasks (HITs)
$0.00 and up, mostly a few cents
SDKs in Java, Ruby, Perl, .NET
Or, create through website UI*
13. Training AI
“Computer Music Composition using
Crowdsourcing and Genetic
Algorithms”
DarwinTunes
14.
15.
16. Research Examples
collecting creative drawings
rating audio and video sample quality
tagging location-sensitive queries and
points of interest with mobile devices
remote massage
task delegation in Wikipedia
collecting volunteer expertise to help
the homeless
annotating political campaign ads
17. More Research examples
describing and organizing geometric
shapes*
tracking popular music
rating search result relevance
collecting user feedback
extracting product and brand
information
tagging specific objects in an image
training speech recognition software
19. References
Alonso, O., Rose, D.E., and Stewart, B. (2008). Crowdsourcing for Relevance Evaluation.
SIGIR Forum 42(2) 9-15.
Amazon.com, Inc. (2010). Amazon Mechanical Turk. https://www.mturk.com/mturk/welcome.
CastingWords (2010). Audio transcription services: plus video, CDs, podcasts and more.
http://castingwords.com/
Chung, K., Chiu, C., Xiao, X., & Chi, P. (2009). Stress outsourced: A haptic social network via
crowdsourcing. Paper presented at the CHI EA '09: Proceedings of the 27th International
Conference Extended Abstracts on Human Factors in Computing Systems, Boston, MA,
USA. 2439-2448.
DarwinTunes (2010). Welcome to DarwinTunes. http://darwintunes.org/
Ganjisaffar, Y., Javanmardi, S., & Lopes, C. (2009). Leveraging crowdsourcing heuristics to
improve search in wikipedia. Paper presented at the WikiSym '09: Proceedings of the 5th
International Symposium on Wikis and Open Collaboration, Orlando, Florida. 1-2.
Harper, F. M., Raban, D., Rafaeli, S. & Konstan, J. A. (2008). Predictors of answer quality in
online Q & A sites. Paper presented at the CHI '08: Proceeding of the Twenty-Sixth Annual
SIGCHI Conference on Human Factors in Computing Systems, Florence, Italy. 865-874.
Hintikka, K. A. (2008). Web 2.0 and the collective intelligence. Paper presented at the
MindTrek '08: Proceedings of the 12th International Conference on Entertainment and Media
in the Ubiquitous Era, Tampere, Finland. 163-166.
Howe, J. (2006). The Rise of Crowdsourcing. Wired, 14(6).
http://www.wired.com/wired/archive/14.06/crowds.html.
20. More References
Hsueh, P., Melville, P., & Sindhwani, V. (2009). Data quality from crowdsourcing: A study of
annotation selection criteria. Paper presented at the HLT '09: Proceedings of the NAACL HLT
2009 Workshop on Active Learning for Natural Language Processing, Boulder, Colorado. 27-
35.
Jagadeesan, A. P., Lynn, A., Corney, J. R., Yan, X. T., Wenzel, J., Sherlock, A., et al. (2009).
Geometric reasoning via internet CrowdSourcing. Paper presented at the SPM '09: 2009
SIAM/ACM Joint Conference on Geometric and Physical Modeling, San Francisco, California.
313-318.
Kittur, A., Chi, ,E. H., & Suh, B. (2008). Crowdsourcing user studies with mechanical turk.
Paper presented at the CHI '08: Proceeding of the Twenty-Sixth Annual SIGCHI Conference
on Human Factors in Computing Systems, Florence, Italy. 453-456.
Koblin, A. M. (2009). The sheep market. Paper presented at the C&C '09: Proceeding of the
Seventh ACM Conference on Creativity and Cognition, Berkeley, California, USA. 451-452.
Krieger, M., Stark, E. M., & Klemmer, S. R. (2009). Coordinating tasks on the commons:
Designing for personal goals, expertise and serendipity. Paper presented at the CHI '09:
Proceedings of the 27th International Conference on Human Factors in Computing
Systems, Boston, MA, USA. 1485-1494.
Ledlie, J., Odero, B., Minkov, E., Kiss, I., & Polifroni, J. (2009). Crowd translator: On building
localized speech recognizers through micropayments. SIGOPS Operating.Systems
Review, 43(4), 84-89.
Li, C., Buyuktur, A. G., Hutchful, D. K., Sant, N. B., & Nainwal, S. K. (2008). Portalis: Using
competitive online interactions to support aid initiatives for the homeless. Paper presented at
the CHI '08: CHI '08 Extended Abstracts on Human Factors in Computing Systems, Florence,
21. Even More References
Mannes, A. E. (2009). Are we wise about the wisdom of crowds? the use of group judgments
in belief revision. Management Science, 55(8), 1267-1279.
Mason, W., & Watts, D. J. (2009). Financial incentives and the performance of crowds. Paper
presented at the HCOMP '09: Proceedings of the ACM SIGKDD Workshop on Human
Computation, Paris, France. 77-85.
Milne, D., & Witten, I. H. (2008). Learning to link with wikipedia. Paper presented at the CIKM
'08: Proceedings of the 17th ACM Conference on Information and Knowledge
Management, Napa Valley, California, USA. 509-518.
Roman, D. (2009). Crowdsourcing and the question of expertise. Commnuications of
the.ACM, 52(12), 12-12.
Stewart, O., Huerta, J. M., & Sader, M. (2009). Designing crowdsourcing community for the
enterprise. Paper presented at the HCOMP '09: Proceedings of the ACM SIGKDD Workshop
on Human Computation, Paris, France. 50-53.
Su, Q., Pavlov, D., Chow, J., & Baker, W. C. (2007). Internet-scale collection of human-
reviewed data. Paper presented at the WWW '07: Proceedings of the 16th International
Conference on World Wide Web, Banff, Alberta, Canada. 231-240.
Surowiecki, J. (2005). The wisdom of crowds. Anchor.
Tanaka, A., Tokui, N., & Momeni, A. (2005). Facilitating collective musical creativity. Paper
presented at the MULTIMEDIA '05: Proceedings of the 13th Annual ACM International
Conference on Multimedia, Hilton, Singapore. 191-198.
Von Ahn L., & Dabbish, L. (2004). Labeling images with a computer game. Paper presented
at the CHI '04: Proceedings of the SIGCHI Conference on Human Factors in Computing