The document discusses impact assessment of digital resources. It defines impact as reaching intended audiences, enabling new research questions, and enabling new approaches to education. It recommends identifying key stakeholders and goals to ensure impact. Metrics for measuring impact include usage analytics, citations, and embedding in other resources. The TIDSR project studied five digital resources and tested impact assessment methods. Interviews found resources transformed access and usage patterns, allowing more comprehensive searches and serendipitous findings. Resources are now part of broader transformations in research workflows, allowing more efficient discovery of sources.

1. Impact as a process: considering the
reach of resources from the start
Eric T. Meyer & Kathryn Eccles
Oxford Internet Institute, University of Oxford
@etmeyer
Digital Humanities@Oxford Summer School #tidsr
#dhoxss
5th July 2012
Slides at: http://www.slideshare.net/etmeyer/tidsrdhoxss2012

2. What is impact and why consider it?
What do we mean by impact?
•Reaching intended audience
•Reaching new audiences
•Attracting users
•Attracting new usage
•Enabling new research questions
•Enabling new approaches to education

3. Where to begin?
• Identify your audience and key stakeholders
• Set your goals. What types of impact do you
envisage your resource having?
• What steps are you taking to ensure these types of
impact?
• Identify connections
• What resources do you see as successful in terms
of audience and impact?
• Do you see your resource as part of a network of
connected resources?

4. Digitisation and Impact

5. Measuring usage and impact
What to measure?
• Users
• Types of use
• Awareness
• Citation practices
• Marketing strategies
• Embedding

6. TIDSR: The first usage and impact study
• JISC funded project
• July 2008-April 2009
• Looked at five specific JISC-funded resources
• Designed to test the TIDSR methods and review
them for the TIDSR toolkit

7. Methods
Quantitative methods Qualitative methods
• Webometrics • Interviews
• Web Analytics • Focus groups
• Log file analysis • User feedback
• Scientometrics / • Referrer analysis
bibliometrics • Content Analysis
• Content Analysis

9. 100% 2% 2% 1% 2%
3% 7% 7%
7% 7%
90% 7% Use it regularly or
18% frequently
80% 22%
27% 28%
70%
60%
Use it on occasion
35%
50%
88%
40%
69% Have seen it, but don't
30% 63% 63%
use it
20% 39%
10%
I haven't heard of it
0%
HistPop BOPCRIS BL News BL Sounds Med Backfiles

10. 100% 96%
90%
90%
86%
84% 82%
79%
80% 76%
71% 72%
69%
70%
60% 61%
60% Important to my research
51%
49%
50%
Important to my teaching
40%
40%
35% 36% Important to field
34% 33%
29%
30%
Would recommend
20%
10%
0%
HistPop BOPCRIS BL News BL Sounds Med Backfiles

12.  Top Search Phrases: Histpop

13.  Perception: Specific niche community
 Well known by target audience
 Transforming access and usage patterns
 User surveys:
 Embedded in educational resources
 Enhanced access to primary sources
▪ ‘Histpop made it possible to do a completely different project’
 Continuing education, online resources, non-
traditional learners

16. Have you ever published a piece
If so, how did you cite the collection?
based on your work in this
collection? Original version Original + URL Online version Other
46% HistPop 9% 55% 36% 0%
29% BOPCRIS 36% 36% 21% 7%
35% BL News 53% 38% 6% 2%
BL
20% 38% 50% 13% 0%
Sounds
Med
43% 43% 48% 10% 0%
Backfiles
60% 40% 20% 0% 0% 20% 40% 60% 80% 100%

17. • Highest numbers for original British Library resource
(analogue)
• 19th Century British Library Newspapers registers
strong links for a project page
• Note: Importance of comparator sites when using
webometrics

20.  Time intensive, but productive if you are
careful about what you ask!
 Different stakeholders:
 Project team: Positive view of the work only
 Broader stakeholders: While the digital project
was good, it also introduced tensions in the
broader setting of the library
 New kinds of serendipity, wide range of users

21. Engagement officer

24. • Some resources are available through multiple outlets
• Webometrics can capture comparative awareness
• These results show how powerful known resources
and/or publishers can be

27.  Wellcome Medical Journals Backfiles project page records strong links, links to
Pub Med for WMJB material impossible to trace

28.  Historians? (would be looking at older articles)
 Not typical PubMed users
 Search interface issues / limited search
 Clinicians? (would be looking at newer articles)
 Not typically reading 100 year old articles
 Other users?
 Paths of discovery?

29.  Majority of downloads targeted more recent
material – opening up of new resources to
clinicians
 More thorough and comprehensive searches
 Historians reported more comprehensive search
results (quantitative results)
 Also reported increased browsing, greater
serendipity, due to time saved finding articles

30. House of Commons Parliamentary Papers 48%
83%
British Library Newspapers 50%
77%
British Periodicals 56%
71%
Old Bailey Online 36%
62%
Imperial War Museum 32%
58%
Archival Sounds 24%
54%
BOPCRIS 27%
51%
Internet Lib of Early Journals 38%
47%
Historical Directories 32%
45%
Wellcome Medical Journal Backfiles 29%
34%
Fine Rolls 18%
31%
Chronicling America 30%
17%
Histpop 9% Non-UK Awareness
16%
Sciper 11%
12% UK Awareness
Cornell Animal Sounds 13%
7%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

31. 50%
45%
40% HistPop
35% BOPCRIS
30%
25% BL News
20% BL Sounds
15% Med Backfiles
10%
5%
0%

32. http://microsites.oii.ox.ac.uk/tidsr/

33. Coming this autumn….
Project page: Impact of Digitised Resources
http://www.kdcs.kcl.ac.uk/innovation/impact.html
Simon Tanner’s blog post about the project
http://simon-tanner.blogspot.co.uk/2012/03/new-approach-to-
measuring-impact-for.html

35. University of Oxford Podcasts

36. Proceedings of the Old Bailey Online

37. British History Online

39. Siobhan Davies RePlay

40. http://www.rin.ac.uk/humanities-case-studies
Bulger, M., Meyer, E.T., de la Flor, G., Terras, M., Wyatt, S., Jirotka, M., Eccles, K., Madsen, C.

41. The Case Studies

42. Browsing and Searching
79% Google
66% Google Scholar
Libraries 59% Visit the library
55% Browse library materials online
62% Search library materials online
83% Citation chaining
Journals 48% Browse printed journals
76% Browse online journals
Peers
95% Consult peers and experts

43. Reconfiguring Resources

44. “ Old Bailey Online hasn’t replaced
anything for me or displaced anything for
me, but it is part of this general
transformation of how I do what I do.

45. “ The amount of time I now spend doing the
very mechanical, laborious, time-
consuming work is much smaller. You can
now do things in 5 seconds which it took
you 3 months to do a few years ago.

46. Transformations in Use

47. “
It’s a huge change. You can do things much
more quickly, read much more
widely, find connections…it’s very, very
important.

48. “
With something like the Burney Collection, 5
years ago for writing an article I would need to
review the newspapers, I would have gone
into the British Library and done it on
microfilm.
20 years ago, I would have gone into the
British Library and done it with the actual
paper in front of me. Now I sit at home
and I do a keyword search.

49. “ Asking new questions?

50. “
I’m not sure all of this raises the quality of
anybody’s work. I think it would be quite daft
to pretend that all of this makes us better
scholars, or makes our books or papers of higher
quality. I don’t know if that is true by any
means, but it certainly makes it easier and I
suppose makes the quantity of stuff that you can
produce greater.

51. “
What might take you several months if not years
of research, you could do in hours, days, a week.
So I think that means that it makes the nature
of your research different because it allows
you quantitative information much more
quickly, which then allows you to maybe
think about how you might use that
information differently, because you’ve got
so much more time.

52. Slides at: http://www.slideshare.net/etmeyer/tidsrdhoxss2012
Eric T. Meyer Kathryn Eccles
eric.meyer@oii.ox.ac.uk kathryn.eccles@oii.ox.ac.uk
http://www.oii.ox.ac.uk/people/?id=120 http://www.oii.ox.ac.uk/people/?id=138
Oxford e-Social Science Project
Project work funded by: