Presented at the IIPC Web Archiving Conference, 6-7th June 2019, Zagreb, Croatia. http://netpreserve.org/ga2019/programme/wac/ This paper presents the results of a study to examine, determine and propose the optimal approach to develop impact assessment indicators for the UK Web Archive (UKWA). In the United Kingdom, legal deposit libraries collaboratively operate a nationwide web archiving project, the UKWA, which has collected over 500 TB of data and is growing by approximately 60–70 TB a year. At the same time, UK publicly funded organisations face reduced funding and the challenge of convincing funders to finance their archival function by undergoing evaluations of their services’ values. Under such circumstances, a proper assessment of the values and impacts of web archiving is a point of discussion for cultural heritage organisations. To the best of the authors’ knowledge, there has not yet been a comprehensive assessment or evaluation of the UKWA conducted. Thus, this paper seeks to answer the research question: “What would the indicators of impact assessment for the UKWA be?” As a result, we propose a set of impact assessment indicators for the UKWA (and web archiving in general) with broad strategic perspectives including social, cultural, educational and economic impact. This study examines and proposes the optimal approach to develop impact assessment indicators for the UKWA. The research began by analysing the literature of impact assessment frameworks for digital resources and the types of impact in related fields. Primarily drawing from Simon Tanner’s Balanced Value Impact Model (BVI Model), this research then proposes impact indicators for the UKWA and develops an impact assessment plan consisting of three stages: context setting, indicator development, and indicator evaluation. This paper will present the method and results of the study. Firstly, it identified the UKWA’s foundational context, the mission, the principal values and the key stakeholder groups. The research project prioritised focal areas for the archive that seem most advantageous for stakeholders and aligned with Tanner’s Value Lenses. Secondly, we proposed the UKWA impact assessment indicators; scrutinising existing indicators and various evidence collection methods. In the third stage, the developed indicators’ functionality was checked against set quality criteria and then tested through semi-structured interviews and survey submissions with 8 UKWA staff members. Finally, the paper presents the thirteen potential indicators for the UKWA. Based on the lessons learned, presenters will also make recommendations for organisations which recognise the necessity of undertaking impact assessments of their web archives.

1. Simon Tanner
King’s College London
simon.tanner@kcl.ac.uk
@SimonTanner
Julie Fukuyama
National Diet Library
ju-fukuy@ndl.go.jp
IIPC Web Archiving Conference 2019, Zagreb, Croatia, 7 June 2019
&
Developing impact assessment indicators:
Making a proposal for the UK Web Archive

2. Background

3. What would the indicators of
impact assessment for the UKWA be?
Research Question

4. The Balanced Value Impact Model
http://simon-tanner.blogspot.com/2017/10/BVI-Model-V2.html
Approach to develop the UKWA IA indicators

5. Approach to develop the UKWA IA indicators

6. The UKWA’s mission gathered from the website, the government’s guidance on
the regulations, and the Agency for the Legal Deposit Libraries (ALDL)
Sorted out mission of the UKWA
Activity Objects How For whom
UKWA
- Collect
- Capture
- Save
- Preserve
- Give permanent
access
- All of the UK Web Space
- Key UK websites
- Assets: html pages, images, pdf’s, video’s
- Information that is only available on the Web
- Significant witness to our times
- Highly ephemeral:
altered, disappear
- Annually
- Comprehensively
- Future generations
- Current & future
researchers
(academic or
otherwise)
Guidance
- Archive
- Preserve
- National collection of
non-print publications
- Efficient - Future generations
Agency
- Collect
- Preserve
- Make it available
- Nation’s published output
- Nation’s intellectual record
- Future published heritage
- Systematically
- Future generations
- Readers within the
deposit libraries

7. User segments and their benefits

8. User segments and their benefits

9. Balancing perspectives and value lenses for the UKWA

10. Focus areas of the UKWA: areas to be assessed

11. Approach to develop the UKWA IA indicators

12. Focus areas and elements of success criteria

13. Focus areas and elements of success criteria

14. Elements of success criteria Success criteria
Effectiveness/ease of information
search
Users find it effective/easy to search
Collection coverage of existing websites The UKWA collects the whole UK websites
Usability of datasets Users find the datasets as usable/meaningful
Trustworthiness of archived websites Archived websites prove to be unchanged/secure
Meaningful/interesting collections Users find them meaningful/interesting collections
Outcomes of R&D
Researchers: find research questions, use data, publish the results
Developers: find issues, develop tools, tools become widespread
Economic benefits
The UKWA becomes seeds of grants, produces direct/indirect
outcome/benefits
Influence on web archiving community Stakeholders adopt/apply/base the UKWA technologies/research results
Detailed success criteria of the UKWA

15. Questions for the UKWA

16. List of the impact assessment indicators

17. Test of the quality of the impact assessment indicators

18. Test of the quality of the impact assessment indicators

19. Feedback collection

20. Feedback from UKWA staffs

21. Feedback from UKWA staffs

22. Feedback from UKWA staffs
2,50
1,06

23. Feedback from UKWA staffs

24. The final impact assessment indicators list

25.  To adopt our indicators as a starting point to
consider impact indicators
 To discuss among partners and with stakeholders
 To focus upon measuring what you can measure
by conducting small scale internal pilot impact
assessments
Recommendations

26. Thanks to:
• Nicola Bingham (The British Library)
• Helena Byrne (The British Library)
• The staff of the UK Web Archive
Acknowledgements