Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Eric Mayer and Kathryn Eccles, Oxford Internet Institute


Published on

How to measure success: Understanding and monitoring impact

Published in: Technology, Education
  • Be the first to comment

Eric Mayer and Kathryn Eccles, Oxford Internet Institute

  1. 1. How to measure success: Understanding and monitoring impact<br />Eric T. Meyer & Kathryn Eccles<br />Oxford Internet Institute, University of Oxford<br />JISC<br />6th September 2011<br />@etmeyer<br />#tidsr<br />#oess<br />
  2. 2. What is impact and why consider it?<br />What do we mean by impact?<br /><ul><li> Reaching intended audience
  3. 3. Reaching new audiences
  4. 4. Attracting users
  5. 5. Attracting new usage
  6. 6. Enabling new research questions
  7. 7. Enabling new approaches to education
  8. 8. Embedding a resource within a community
  9. 9. Embedding a resource within an educational environment</li></li></ul><li>Measuring usage and impact<br />What can you measure?<br />Users<br />Types of use<br />Awareness<br />Citation practices<br />Marketing strategies<br />Embedding<br />
  10. 10. JISC funded project<br />July 2008-April 2009<br />Looked at five specific JISC-funded resources<br />Designed to test the TIDSR methods and review them for the TIDSR toolkit<br />TIDSR: The first usage and impact study<br />
  11. 11. Methods<br />Quantitative methods<br />• Webometrics<br />• Web Analytics<br />• Log file analysis<br />• Scientometrics / bibliometrics<br />Content Analysis<br />Qualitative methods<br /> Interviews<br /><ul><li> Focus groups
  12. 12. User feedback
  13. 13. Referrer analysis
  14. 14. Content Analysis</li></li></ul><li>Project 1 – Online Historical Population Reports (OHPR/Histpop)<br />
  15. 15. Survey: Low Awareness<br />
  16. 16. Survey: High Importance to Users<br />
  17. 17. Log Files: Non UK Activity<br />
  18. 18. Log File Analysis<br />Top Search Phrases: Histpop<br />
  19. 19. Histpop: User Communities<br />Perception: Specific niche community<br />Well known by target audience<br />Transforming access and usage patterns<br />User surveys:<br />Embedded in educational resources<br />Enhanced access to primary sources<br />‘Histpop made it possible to do a completely different project’<br />Continuing education, online resources, non-traditional learners<br />
  20. 20. Project 2 – British Library 19th Century Newspapers<br />
  21. 21. Citation Habits<br />Have you ever published a piece based on your work in this collection? <br />If so, how did you cite the collection? <br />
  22. 22. Webometric results<br /><ul><li>Highest numbers for original British Library resource (analogue)
  23. 23. 19th Century British Library Newspapers registers strong links for a project page
  24. 24. Note: Importance of comparator sites when using webometrics</li></li></ul><li>Blog Evidence<br />
  25. 25. Project 3 – British Library Archival Sound Recordings<br />
  26. 26. Interviews, Group Interviews, Focus Groups<br />Time intensive, but productive if you are careful about what you ask!<br />Different stakeholders:<br />Project team: Positive view of the work only<br />Broader stakeholders: While the digital project was good, it also introduced tensions in the broader setting of the library<br />New kinds of serendipity, wide range of users<br />
  27. 27. News<br />Engagement officer<br />
  28. 28. Awareness of Resource by Country<br />
  29. 29. How did you find this resource?<br />
  30. 30.<br />
  31. 31.
  32. 32. Some recommendations:<br />Think about impact from the beginning<br />Set your goals – what steps will you need to take?<br />Identify connections:<br />Which resources do you see as successful in terms of audience and impact?<br />Is your resource connected to a community of resources? How can you use these connections?<br />
  33. 33. Slides at:<br />Eric T. Meyer<br /><br />Kathryn Eccles<br /><br />Oxford e-Social Science Project<br />Project work funded by:<br />