Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Evaluating crowdsourcing websites


Published on

NDF Barcamp, Hamilton City Library, Hamilton, New Zealand

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

Evaluating crowdsourcing websites

  1. 1. How effectively are crowdsourcingwebsites supporting volunteerparticipation and quality contribution?Donelle McKinleyPhD candidate, School of Information ManagementVictoria University of WellingtonNDF Barcamp, Hamilton, 21 June @donellemckinley
  2. 2. Crowdsourcing: a short versionCrowdsourcing outsources taskstraditionally performed by specificindividuals to a group of people orcommunity through an open call (Howe,2009).
  3. 3. Crowdsourcing: a long version“Crowdsourcing is a type of participative online activity in which anindividual, an institution, a non-profit organization, or companyproposes to a group of individuals of varying knowledge,heterogeneity, and number, via a flexible open call, the voluntaryundertaking of a task. The undertaking of the task, of variablecomplexity and modularity, and in which the crowd shouldparticipate bringing their work, money, knowledge and/orexperience, always entails mutual benefit. The user will receive thesatisfaction of a given type of need, be it economic, socialrecognition, self-esteem, or the development of individual skills,while the crowdsourcer will obtain and utilize to their advantagethat what the user has brought to the venture, whose form willdepend on the type of activity undertaken.”(Estellés-Arolas & González-Ladrón-de-Guevara, 2012)
  4. 4. Potential benefits of crowdsourcingfor the institutionO Continue tradition of volunteerismO Market and stimulate interest in collectionsO Signal the institutions openess and approachabilityO Achieve goals otherwise too costly and labour-intensiveO Better reflect the diversity of visitorsO Tap into expertise outside the institutionO Engage visitors in new waysO Raise the profile of researchO Demonstrate relevanceO Enable new research questions to be explored
  5. 5. Common motivations for volunteer participationO The size of the challengeO The necessity for volunteer contributionO Collaboration with prestigious institutionsO Contribution to researchO EducationO Mental stimulationO Being part of a communityO Personal research interestsO Enhancing a resource from which they will benefit
  6. 6. The potential of the crowdDigitalkoot (National Library of Finland)O First 51 days of the projectO 31, 816 visitors to the siteO 15% participatedTranscribe Bentham (University College London)O First 6 months of the projectO 1,207 visitors registered to participateO 21% participated
  7. 7. Three common scenariosThe website either:O Follows the structure of the underlyingtechnology or the organisationO Adheres to familiar conventionsO Is the product of personal preferenceGarrett, J. (2011). The Elements of User Experience: User-centereddesign for the web. USA: Aiga/New Riders.
  8. 8. How effectively does your website…?O Define its objectivesO Reflect visitor motivationsO Align visitor motivations with relevantincentivesO Minimize sources of frustration andconcern
  9. 9. How are they impacting on the effectivenessof your website?O ContentO LanguageO ReadabilityO Website navigationO Arrangement of page elementsO ConsistencyO Visual appearanceO Page load speedO Number and complexity of processes to completethe desired action
  10. 10. O Prototype: an early sample or model built to test a conceptor processO Pilot: a small-scale preliminary experiment conducted toevaluate feasibility, time, cost, adverse events, and improvedesign prior to the launch of a full-scale projectO Soft-launch: the release of a website to a limited audience,in order to (beta) test and tweak a design before beinglaunched to a wider audienceO Beta testing: user testing by a limited audience to ensurethe website/software has few faults or bugs, and gatheruser feedbackO Optimization: increasing the percentage of visitors thatfulfill the objective of a webpage or website
  11. 11. User evaluation: A different kind of userengagementO Focus groupsO SurveysO Usability testingO Ongoing feedback channels
  12. 12. Analytics:How is website design impacting on…?O The number of online visitorsO Time spent on siteO The number of online visitors who register toparticipateO The number of online visitors who actuallyparticipateO The number of abandoned and completed tasksO The number of return visitors
  13. 13. OCR text correction: Trove, National Library of Australia
  14. 14. Evaluation of the Trove crowdsourcinguser interface involved:O Asking potential volunteers to comment on the prototypeO Sitting potential volunteers in front of the computerscreen and asking them how they would complete thetaskO A soft launch and beta testingO Gathering feedback on the beta version from over 600users over the course of five months, via a survey, anonline contact form, direct observation of user activity,analytics, online comments, and direct contact withusers via email and phone
  15. 15. Evaluation of the Transcribe Bentham taskinterface involved:O Beta testingO A user surveyO Website analyticsO Analysis of user interaction statisticsO Comparisons with studies oncrowdsourcing volunteer behaviour
  16. 16. UK Reading Experience Database (UK-RED)
  17. 17. Evaluation of the UK-RED task interfaceinvolved:O Heuristic evaluationO A survey of current and potentialcontributorsO Comparison with other crowdsourcingtask interfacesO Comparison with recommended practiceas outlined in crowdsourcing and HCIliterature
  18. 18. Usability and functionality requirements fora NZ-RED task interfaceO Minimize user effortO Support integration of the task with researchprocessesO Enable new visitors and contributors to understandwhat the task involves quickly and easilyO Support accurate and controlled data entryO Be easy to use for people reasonably confident withthe WebO Support flexible, structured data entryO Support bilingual data entry
  19. 19. Challenges impacting on evaluationO FundingO ExpertiseO Organisational culture
  20. 20. What could a more effective website meanfor the project?O More online visitors participateO Tasks are completed more efficientlyO Tasks are completed with greateraccuracyO The task is more enjoyableO Volunteers participate more oftenO The project is more cost-effectiveO More volunteers are willing to participatein future projects
  21. 21. Thanks!For references and other great reads visit and slides will be available at crowdsourcing research updates @donellemckinley