www.kcl.ac.uk/ddh/

Digital Humanities:
the application of digital technology to humanities disciplines
reflection upon th...
Measuring the Impact of
Digitized Resources:
The Balanced Value Model
http://www.kdcs.kcl.ac.uk/innovation/impact.html
“the measurable outcomes arising from the existence of a
digital resource that demonstrate a change in the life or life
op...
The case for Impact
We are more effective and efficient in delivering change
and tangible benefits (Internal Impact);
Our ...
www.kdcs.kcl.ac.uk/innovation/impact.html
www.kdcs.kcl.ac.uk/innovation/impact.html
Impact taskforce
Simon Tanner
Aggregated
• 2200+ content
providers
• 173 aggregators
• 26.9 million objects
Network
• 600+ individual
members, working
in taskforces and
on strategy

Breakdown
Galleries: 2
Libraries: 111
Archives: ...
Stakeholder Survey and Mapping
A person, group, community, or organization who
affects or can be affected by the ecosystem...
Stakeholders – the kinds of broad groups
Consumers
One Stop Consumers
Partners and Collaborators
Paymasters
Producers and ...
Stakeholder Survey and Mapping
Stakeholder mapping is a powerful tool to
enable planning
It enables a list of stakeholders...
Stakeholder Mapping
Knowledge

Context
Setters
Champions &
Activists

Influencers
Bystanders
Influence / Initiative
Stakeholder Mapping: Example
Knowledge
Champion
Fund project
Govt. Funder

Gain IP agreement

CEO
Govt.

Publisher

Influe...
Exercise: Draw your own
Knowledge

Context
Setters
Champions &
Activists

Influencers
Bystanders
Influence / Initiative
www.kdcs.kcl.ac.uk/innovation/impact.html
PERSPECTIVE
+ VALUE
DRIVER
OBJECTIVES

STAKEHOLDERS

AREAS MEASURED

METHODS

INDICATORS

CURRENT
USE (SOCIAL)

Those with...
PERSPECTIVE +
VALUE DRIVER
OBJECTIVES

STAKEHOLDERS

AREAS MEASURED

METHODS

INDICATORS

CURRENT USE
(SOCIAL)

Community
...
PERSPECTIVE
+ VALUE
DRIVER

OBJECTIVES

STAKEHOLDERS

AREAS MEASURED

METHODS

INDICATORS

POTENTIAL
USE
(INNOVATION)

Use...
PERSPECTIVE
+ VALUE
DRIVER

OBJECTIVES

STAKEHOLDERS

AREAS MEASURED

METHODS

INDICATORS

INTERNAL

How have the
staff of...
PERSPECTIVE
+ VALUE
DRIVER
OBJECTIVES

STAKEHOLDERS

AREAS MEASURED

METHODS

INDICATORS

Google Analytics
review
Site sur...
Thanks!

Thank you to all the folks at the Wellcome Library
for allowing me to share this with you.
Contacts
Alexander Gre...
The case for Impact
We are more effective and efficient in delivering change
and tangible benefits (Internal Impact);
Our ...
www.kdcs.kcl.ac.uk/innovation/impact.html
With thanks to Alice Maggs for the Impact illustrations
alice.100@hotmail.com
Introduction to Implementing the Balanced Value Impact Model - Workshop for NDF 2013
Introduction to Implementing the Balanced Value Impact Model - Workshop for NDF 2013
Introduction to Implementing the Balanced Value Impact Model - Workshop for NDF 2013
Introduction to Implementing the Balanced Value Impact Model - Workshop for NDF 2013
Introduction to Implementing the Balanced Value Impact Model - Workshop for NDF 2013
Introduction to Implementing the Balanced Value Impact Model - Workshop for NDF 2013
Introduction to Implementing the Balanced Value Impact Model - Workshop for NDF 2013
Introduction to Implementing the Balanced Value Impact Model - Workshop for NDF 2013
Upcoming SlideShare
Loading in …5
×

Introduction to Implementing the Balanced Value Impact Model - Workshop for NDF 2013

1,389 views
1,274 views

Published on

The Balanced Value Impact Model is intended to aid the thinking and decision making of those wishing to engage in Impact Assessment. It also acts as a guide through the process of Impact Assessment to enable the core values most appropriate to the assessment to be brought to the fore and given a balanced consideration when evaluating outcomes. It presumes that the assessment will be measuring change within an ecosystem for a digital resource.

For the purposes of this Model, the definition of Impact is: The measurable outcomes arising from the existence of a digital resource that demonstrate a change in the life or life opportunities of the community.

Who should use the BVI Model?

The aim of this workshop is to provide key information and a strong model for the following primary communities of use:

Memory institutions and cultural heritage organizations, such as libraries, museums and archives.
Funding bodies who wish to promote evidence-based impact assessment of activities they support.
Holders and custodians of special collections.
Managers, project managers and fundraisers who are seeking to justify further investment in digital resources.
Academics looking to establish digital projects and digital scholarship collaborations with collection owners.
Publishing, media and business sectors which may be considering the best means to measure the impact of their digital resources and are looking to collaborate and align with collection owners, with academia or with memory institutions.
Impact Assessment practitioners considering an Impact Assessment of a digital resource.
What the workshop will cover:

Where the value and impact can be found in digital resources,
Who are the beneficiaries gaining from the impact and value,
How to measure change and impact for digital resources,
How to do an Impact Assessment using the Balanced Value Impact Model, and
How to present a convincing evidence-based argument for digital resources?
The Workshop will include case studies of how the BVI Model is being implemented at present.

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,389
On SlideShare
0
From Embeds
0
Number of Embeds
155
Actions
Shares
0
Downloads
14
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Introduction to Implementing the Balanced Value Impact Model - Workshop for NDF 2013

  1. 1. www.kcl.ac.uk/ddh/ Digital Humanities: the application of digital technology to humanities disciplines reflection upon the impact of digital media upon humanity > 50 academics & researchers ~ £2.5 million research income per annum 5+ million digital objects in 107+projects 200+ million hits over the last 5 years
  2. 2. Measuring the Impact of Digitized Resources: The Balanced Value Model http://www.kdcs.kcl.ac.uk/innovation/impact.html
  3. 3. “the measurable outcomes arising from the existence of a digital resource that demonstrate a change in the life or life opportunities of the community” www.kdcs.kcl.ac.uk/innovation/impact.html
  4. 4. The case for Impact We are more effective and efficient in delivering change and tangible benefits (Internal Impact); Our organisation is gaining strategic advantage through the innovation inherent in this digital activity (Innovation Impact); We are delivering a strong economic benefit to our community that demonstrate the worth and value of our endeavours in clear monetary terms (Economic Impact); and the community has been changed by the resource in beneficial ways that can be clearly identified (Social Impact)
  5. 5. www.kdcs.kcl.ac.uk/innovation/impact.html
  6. 6. www.kdcs.kcl.ac.uk/innovation/impact.html
  7. 7. Impact taskforce Simon Tanner
  8. 8. Aggregated • 2200+ content providers • 173 aggregators • 26.9 million objects
  9. 9. Network • 600+ individual members, working in taskforces and on strategy Breakdown Galleries: 2 Libraries: 111 Archives: 26 Museums: 60 National Aggs: 22 Publishers: 2 Creative Ind: 5 Research: 78 Ministries: 9 Other: 174
  10. 10. Stakeholder Survey and Mapping A person, group, community, or organization who affects or can be affected by the ecosystem of the digital resource to be assessed. Questions to be answered in establishing and identifying stakeholders include: Have all primary stakeholders been listed? Have all secondary stakeholders been listed? Have all potential supporters and opponents been identified? Have the interests of vulnerable or minority groups been identified? What new primary or secondary stakeholders are likely to emerge?
  11. 11. Stakeholders – the kinds of broad groups Consumers One Stop Consumers Partners and Collaborators Paymasters Producers and Creators? Commentators Marginalised Leavers Non-users Champions Competitors
  12. 12. Stakeholder Survey and Mapping Stakeholder mapping is a powerful tool to enable planning It enables a list of stakeholders to become an active tool in three simple steps The steps: Name your stakeholders and put them in categories Map where the stakeholder is now in relation to your organisation or the initiative for which you want their support Now map where you need the stakeholder to be to enable your organisation or initiative to succeed. Use this information to plan actions/timing etc.
  13. 13. Stakeholder Mapping Knowledge Context Setters Champions & Activists Influencers Bystanders Influence / Initiative
  14. 14. Stakeholder Mapping: Example Knowledge Champion Fund project Govt. Funder Gain IP agreement CEO Govt. Publisher Influence / Initiative
  15. 15. Exercise: Draw your own Knowledge Context Setters Champions & Activists Influencers Bystanders Influence / Initiative
  16. 16. www.kdcs.kcl.ac.uk/innovation/impact.html
  17. 17. PERSPECTIVE + VALUE DRIVER OBJECTIVES STAKEHOLDERS AREAS MEASURED METHODS INDICATORS CURRENT USE (SOCIAL) Those with an interest in the intellectual content find it useful to their research Google Users of the 1. Discovery Code2. Engagement Analytics review breakers 3. Usefulness Site surveys resource. sampling users on the Codebreakers microsite. Tracking of recommendatio ns to others or reviews. Users User panels. Citations of content on the site. 1. a. b. c. Discovery of resource: Web visits/visitors Views to digitised content Relative use to historical use figures (where applicable) 1. Engagement with resource: a. Average time spent on digitised content b. Repeat visitors to Codebreakers c. Downloads of content d. User journeys across Codebreakers 1. Usefulness of resource: a. Site surveying to collect reported usage and utility of Codebreakers resource b. Citation indices – this can only be a very long term measure due to research and publishing timescales c. User panel – recruited from actual users of the Codebreakers resource. Engage in enquiry of the function, content and discoverability of Codebreakers. d. Desk research to find innovative use of the content e. Online media monitoring to capture people’s mentions and recommendations of Codebreakers.
  18. 18. PERSPECTIVE + VALUE DRIVER OBJECTIVES STAKEHOLDERS AREAS MEASURED METHODS INDICATORS CURRENT USE (SOCIAL) Community Peer organisation s and members of our professional community have changed their policy or practice concerning digitisation projects. Practitioners 1. , peer organisation s and 2. members of our professional community 3. who have been influenced by the project Awareness of the project Take up of methods/ approaches / standards Impact of take up on partner and peer organisatio ns Survey of partner organisations who worked on the project Survey of event attendees / key peer organisations Desk research 1. a. b. c. d. 1. a. b. 1. Awareness: Number attending digitisation open days/events held at Wellcome Library Contacts from peer organisations/practitioners Online media monitoring for blogs, conference presentations, events, workshops, open days, social media etc. Citations/references to the project Take up: Survey of partner and peer organisations, and practitioners identified in stage 1. Desk research to identify stakeholders influenced by the Codebreakers project (e.g. Ronan Deazley work on archives and copyright) Impact: a. Survey of partner and peer organisations, and practitioners identified in stage 1. b. Desk research
  19. 19. PERSPECTIVE + VALUE DRIVER OBJECTIVES STAKEHOLDERS AREAS MEASURED METHODS INDICATORS POTENTIAL USE (INNOVATION) Users The Codebreakers project has enabled new potential activities and research methods for those interested in the intellectual content. Potential users of the Codebreakers resource 1. 2. 3. Community The extent to which Codebreakers has created new possibilities for organisations and professional members of the cultural heritage community. Practitioners, peer organisations and members of our professional community who may be influenced by the project 1. 2. 3. 4. Delivery of the planned functionality/ usability of the Codebreakers resource. User understandin g of the new research enabled by Codebreakers . Unforeseen potentials for new research. Evaluation of functional capabilities of the finished site against our initial goals. Heuristic evaluation of Codebreakers resource. Site survey. Focus group with potential users 1. a. Desk work to determine accessibility of technical and process developments. Awareness of Qualitative measures such as access to developments questionnaires, desk research, . structured Uptake of interviews. A practices initiated in the comparison of our initial goals with the project as final site. industry standard. Unforeseen 1. a. The accessibility of technical developments . b. 1. a. b. c. d. 1. b. 1. a. 1. a. 1. a. b. Delivery: Desk work to check the functionality of the site against our initial specification. Heuristic evaluation of usability User understanding: Site survey of users on Codebreakers microsite Focus group research Usability research with potential users Heuristic evaluation Unforeseen potentials: Staff interviews to discover where unexpected benefits occurred during the build of the site. Focus group recruited from potential Codebreakers users to discuss new opportunities for researchers. Accessibility of developments: Desk work to check accessibility of technical developments to the peer community Availability of documentation of process/organisational developments Awareness of access: Survey of partner and peer organisations, and practitioners identified in Current Community Awareness. Uptake as industry standards: As described in Current Community uptake Unforeseen potentials: Survey of partner and peer organisations, and practitioners identified in Current Community Awareness. Staff interviews to discover where unexpected benefits occurred during the build of the site.
  20. 20. PERSPECTIVE + VALUE DRIVER OBJECTIVES STAKEHOLDERS AREAS MEASURED METHODS INDICATORS INTERNAL How have the staff of the Wellcome Trust had their skills, abilities, capacity and knowledge Development enhanced by developing Codebreakers Inheritance / Bequest How does Codebreakers represent the inheritance of the Wellcome Trust Library’s activities since collecting began and how does it prepare the Library for the future and bequeath benefits to future generations? Staff of the Wellcome Library and Trust. 1. 2. Changes in individual knowledge or skills. Changes in working practices and behaviours 3. Staff and members of the Wellcome Trust. Change in usage enabled by Codebreaker s resource. Value for future digitisation activity Benchmarkin g against peer organisations . Comparison to historical strategic direction of Library. 2. 3. 4. 1. a. b. 1. a. b. 1. a. Changes in organisation al capacity or ability. 1. Survey of Trust staff connected with the project. Interviews with line-managers. Interviews with senior managers. Google Analytics Interviews with senior managers Review of peer organisation activity Desk research Individual knowledge: Survey of Trust staff connected to Codebreakers activity. Interviewing line-managers of staff involved in the Codebreakers project. Working practices and behaviours: Survey of Trust staff connected to Codebreakers activity. Interviewing line-managers of staff involved in the Codebreakers project. Changes to organisation: Interviews with senior managers. Change in usage: a. See Current Users usage Value for future digitisation activity: a. Interviews with senior managers b. Data from Internal Development Benchmarking: a. Review of peer organisations – desk research and interviews to compare the Wellcome Library’s digital status in comparison with its peers worldwide. Comparison to historical strategic direction: a. Desk research b. Interviews with senior managers
  21. 21. PERSPECTIVE + VALUE DRIVER OBJECTIVES STAKEHOLDERS AREAS MEASURED METHODS INDICATORS Google Analytics review Site survey User panels. This will be based on methodology developed by the British Library in their 2013 economic evaluation. The full British Library report is available here. It will include: 1. Comparison of Codebreakers usage with archive usage records over the last 5 years with an assessment of the cost of use. 2. User time spent on the Codebreakers resource 3. Users’ geographic location. 4. Equivalent cost implication for users consulting across collections previously held in physically separate locations. 5. Contingent valuation questions included in site survey and user panels. ECONOMIC What is the net economic effect of making the content freely available online? Users Users of the Economic gain Codebreaker to individual s resource. users of the resource. Economic value generated for organisations that are endusers of the resource.
  22. 22. Thanks! Thank you to all the folks at the Wellcome Library for allowing me to share this with you. Contacts Alexander Green Email: A.Green@wellcome.ac.uk Christy Henshaw Email: C.Henshaw@wellcome.ac.uk
  23. 23. The case for Impact We are more effective and efficient in delivering change and tangible benefits (Internal Impact); Our organisation is gaining strategic advantage through the innovation inherent in this digital activity (Innovation Impact); We are delivering a strong economic benefit to our community that demonstrate the worth and value of our endeavours in clear monetary terms (Economic Impact); and the community has been changed by the resource in beneficial ways that can be clearly identified (Social Impact)
  24. 24. www.kdcs.kcl.ac.uk/innovation/impact.html
  25. 25. With thanks to Alice Maggs for the Impact illustrations alice.100@hotmail.com

×