Applying SROI in the evaluation of Libraries for Life


Published on

Paper presented at 9th Northumbria International Conference on Performance Measurement in Libraries and Information Services, York, UK, 22-25th August 2011.

Outlines the Libraries for Life strategy in Wales from 2008-11 and the use of SROI as part of the evaluation of the strategy for two of the work strands.
(SROI = Social return on investment)

Published in: Education, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • We are both very pleased to be able to present to this international conference and hope you will find our research interesting and useful. Libraries in Wales are sometimes the envy of others – we have a single library web portal, single searchable cross-sector library catalogue, joint marketing initiatives, bursaries for staff training, funding for refurbishment of public libraries and much more besides. But we’re less good at broadcasting our achievements, so today is a good opportunity to do that in terms of the currently under-used SROI method of measuring impact.
  • Will provide a bit of background context to Wales to help you put our work in context. Then I will briefly outline what the library strategy was that we evaluated using a variety of methods, then the other Alison will discuss SROI in detail and we’ll end with issues to consider, and leave time for questions.
  • Wanted to give a flavour of Wales, its size, to show you what the strategy encompasses. We had a cross-sector approach in the strategy, and involved most of these libraries, although health, prison, school and workplace libraries on the periphery and were not involved in the programme.
  • CyMAL Cymru: Museums Archives and Libraries Wales Also means ‘a joint’, as in your elbow joint, in Welsh Other countries mainly separate but Norway has joint organisation Norwegian Archive, Library and Museum Authority Roles include: advise the Minister; develop policy; deliver grant programmes; provide advice and support for sector; provide free training programme; networking; research; publish magazine. Key role to develop and delivery strategy – two library ones so far. Set up in 2004
  • LfL was a complex multi-strand strategy, with significant amount of Government funding of £10.5m over 3 years. Had six work strands, and each strand quite different from the other in terms of size, approach, management, objectives. We wanted to no only evaluate the use made of government money, to be see whether it had had an impact on library staff, library users, non users and library services. And librarians are very keen to capture evidence of the impact of libraries and library services. We also wanted to put Wales in a bigger context of other library strategies in other countries. So, we designed the evaluation framework with all this in mind
  • External appointment for Scotinform to undertake the 16 month Evaluation from December 2009 to March 2011, and Associates in Wales. Provided us with two Interim reports– helped with early planning stages of next library strategy - Libraries Inspire Key features of the evaluation process: Literature review to place Welsh library strategy development in context of other countries Extensive consultations with library staff at all levels, and with library users and library non-users. Provided very useful information. Heads of services questionnaire, staff survey and focus groups. 300+ staff 700+ users – online and paper survey and focus groups c.600 non –users – f2f interviews and focus groups SROI analysis was undertaken on two work programmes which Alison will now discuss. SROI – chosen because of its ability to measure social benefit and ability to translate outcomes into tangible monetary values which make effective headlines for attracting the attention of policy makers and budget holders.
  • At the outset of the evaluation, we were asked the question – is SROI a suitable tool for measuring impact of the Libraries for Life programme. I thought it would be useful to give a brief explanation of what SROI is. Social Return on Investment (SROI) is a framework for measuring and accounting for the concept of value developed by the Cabinet of the Third Office. It measures change in ways that are relevant to the people or organisations that experience or contribute to it. It tells the story of how change is being created by measuring social, environmental and economic outcomes and uses monetary values to represent them. SROI is about value, rather than money. Money is simply a common unit and as such is a useful and widely accepted way of conveying value.
  • As Alyson has mentioned, SROI analysis was undertaken on two work strands It was decided that a full and rigorous SROI assessment could not be carried out on all strands of the Libraries for Life Strategy given the size of the programme of work and the complexity of the work being undertaken within each of the strands. Therefore it was decided to concentrate the SROI element of the evaluation on two of the strands: the Community Learning Libraries Capital Investment Programme and the Workforce Development strands of the Strategy.   It was agreed with CyMAL that the assessment of the CLLP would focus on the specific activities within the CLLP through 2 case studies:   Pontarddulais Library (South Wales) Wrexham Library (North Wales)   These two case studies were selected because they represent the breadth of types of libraries being refurbished in Wales. One is in the north of the country, one in the south; one was refurbished at the beginning of the programme of work and one in the final year; one is a large city based library and one a small rural library.
  • The process was challenging and although many stakeholders and indicators were identified it was not always possible to find a financial proxy to show the benefit or value leading to the assumption that the figures given could be higher The SROI process is underpinned by involvement by and consultation with stakeholders – which proved very difficult with such a large programme I would suggest that the answer to the question that was originally posed – is that SROI was not the most appropriate tool for measuring value for a programme of this size and complexity. However, it provided a valuable insight into the wider impacts and benefits that the investment in the library sector in Wales has had, and has highlighted areas of impact that would not normally be focussed on in a purely quantitative evaluation of spend against output. Where I do feel SROI could be a really useful tool is when trying to measure the wider impact of smaller discreet library projects and investment where it would be possible to have the involvement of all stakeholders
  • [ALYSON] The full Evaluation was very positive about the LfL Strategy – it represented excellent value for money, and had an holistic approach. Partnerships were seen as beneficial. For policy development for future library strategies in Wales the findings are exceedingly useful in providing evidence of the use, value and impact of libraries, especially the views of non users or lapsed users. This will help inform future policy development.
  • The need for clear and robust data gathering framework in place from day 1 Is SROI the most appropriate tool for large complex programmes? ROI project under way with Wales, Scotland and Northern Ireland libraries – developing online toolkit for public libraries We have had performance measurement in terms of the WPL Standards for public libraries in Wales for about 10 years – considering what might be appropriate for when current set finish in 2015. The research provided us with the opportunity to test out the use of SROI across a range of development activities – The MLA had identified this as the preferred impact methodology for the museums, libraries and archives sector but it still hasn’t been deployed much as far as I’m aware in the library sector. The significance of the findings will be of interest across the library world.
  • Allow 10 mins for Questions
  • Applying SROI in the evaluation of Libraries for Life

    1. 1. Corporate slide master With guidelines for corporate presentations Applying SROI in the evaluation of Libraries for Life CyMAL: Amgueddfeydd Archifau a Llyfrgelloedd Cymru CyMAL: Museums Archives and Libraries Wales Dr Alyson Tyler Libraries Development Adviser CyMAL, Welsh Government @libalyson Alison Kennedy ASK Consulting
    2. 2. Outline of talk <ul><li>Wales context </li></ul><ul><li>Libraries for Life strategy and evaluation </li></ul><ul><li>Social Return on Investment </li></ul><ul><li>Issues to consider </li></ul><ul><li>Questions </li></ul>Grandad Bibli – part of marketing campaign
    3. 3. Wales context 3 million people, 20% speak Welsh 1 National Library of Wales 22 local authority library services c. 12 HE libraries (mergers ongoing) c. 23 FE libraries (mergers ongoing) 50+ health libraries 5 Prison libraries School libraries Workplace libraries Image from CC Flickr - giveawayboy
    4. 4. What is CyMAL? <ul><li>Policy division of Welsh Government </li></ul><ul><li>(Cymru Museums Archives and Libraries) </li></ul><ul><li>Cross-domain approach </li></ul><ul><li>Policy development, grant programmes, advice and support </li></ul><ul><li>Library strategies: </li></ul><ul><ul><li>2005-07 @ your library </li></ul></ul><ul><ul><li>2008-11 Libraries for Life </li></ul></ul>Government building in Aberystwyth
    5. 5. Libraries for life 2008-11 <ul><li>Funding £10.5 m </li></ul><ul><ul><li>2008-09 £1m + £1.5m capital </li></ul></ul><ul><ul><li>2009-10 £1m + £3m capital </li></ul></ul><ul><ul><li>2010-11 £1m + £3m capital </li></ul></ul><ul><li>Work strands </li></ul><ul><ul><li>Community Learning Libraries Programme </li></ul></ul><ul><ul><li>Online content and resources </li></ul></ul><ul><ul><li>Regional Library Partnerships framework </li></ul></ul><ul><ul><li>National marketing strategy </li></ul></ul><ul><ul><li>Developing workforce skills </li></ul></ul><ul><ul><li>Welsh Public Libraries Standards Framework Three </li></ul></ul>
    6. 6. Evaluation framework <ul><li>Independent research company Scotinform, with Associates </li></ul><ul><li>16 month evaluation process – interim reports </li></ul><ul><li>Qualitative research: </li></ul><ul><ul><li>Literature review – country comparator </li></ul></ul><ul><ul><li>Interviews with CyMAL staff </li></ul></ul><ul><ul><li>Interviews with strand leaders and key stakeholders </li></ul></ul><ul><ul><li>Survey of heads of library services </li></ul></ul><ul><ul><li>Staff: survey (online), focus groups (online and face to face) </li></ul></ul><ul><ul><li>Users: survey (online and paper), focus groups (online and face to face) </li></ul></ul><ul><ul><li>Non-users: interviews (face to face) and focus groups (online and face to face) </li></ul></ul><ul><li>SROI analysis on two programme strands </li></ul>
    7. 7. What is SROI? <ul><li>  </li></ul><ul><li>  </li></ul><ul><li>Framework for measuring and accounting for value </li></ul><ul><li>Developed by Cabinet Office of the Third Sector </li></ul><ul><li>It measures relevant change </li></ul><ul><li>Value rather than money </li></ul>
    8. 8. <ul><li>2 work strands assessed </li></ul><ul><li>Workforce Development </li></ul><ul><li>CLLIP – 2 case study libraries </li></ul><ul><li>Pontarddulais (near Swansea, south Wales) </li></ul><ul><li>Wrexham (north Wales) </li></ul>SROI - Process
    9. 9. <ul><li>Establishing scope and identifying key stakeholders </li></ul><ul><li>Mapping outcomes </li></ul><ul><li>Evidencing outcomes and giving them a value </li></ul><ul><li>Establishing impact </li></ul><ul><li>Calculating the SROI </li></ul><ul><li>Reporting, using and embedding </li></ul>SROI – 6 Stage Process
    10. 10. <ul><li>Focus groups held with library staff and users </li></ul><ul><li>SROI relevant questions included in evaluation surveys </li></ul><ul><li>CyMAL reports and management information analyzed </li></ul>SROI – Information Gathering
    11. 11. Outcomes Identified <ul><li>Outcomes </li></ul><ul><li>Workforce Development: £3.12* of SROI </li></ul><ul><li>Pontarddulais : £3.42* of SROI </li></ul><ul><li>Wrexham: £2.16* of SROI </li></ul><ul><li>“ SROI is not just a number but a story about change which includes narrative, qualitative, quantitative and financial aspects” </li></ul><ul><li>* = unverified </li></ul>
    12. 12. Findings from the SROI Process <ul><li>Staff and users benefited; increased usage and users and improved staff working environments </li></ul><ul><li>Staff benefitted from increased levels of skills and confidence and higher earning potential </li></ul><ul><li>Provided valuable insight into wider impacts and benefits of the investment in the library sector in Wales </li></ul>
    13. 13. SROI and Measuring Impact of Libraries for Life <ul><li>Challenging process </li></ul><ul><li>Many stakeholders and indicators identified </li></ul><ul><li>Not always possible to find a financial proxy to show the benefit (materiality) </li></ul><ul><li>Consulting with large number of stakeholders biggest challenge </li></ul><ul><li>Not an appropriate measurement tool for a programme of this size </li></ul><ul><li>Tool would be more appropriate for smaller, discreet projects </li></ul>
    14. 14. Other findings and conclusions <ul><li>“… excellent value for money…” </li></ul><ul><li>Wide reach impact of an holistic approach to investment in libraries </li></ul><ul><li>Value of partnerships across the library sector </li></ul><ul><li>Intrinsic ‘value’ of libraries, even from non-users </li></ul>
    15. 15. Future issues <ul><li>Data gathering framework vital </li></ul><ul><li>SROI suitability for large programmes? </li></ul><ul><li>Other economic impact tools e.g. joint ROI project 2011-12 </li></ul><ul><li>Other options for measuring performance in libraries (cf our Public Library Standards) </li></ul>
    16. 16. Contact details <ul><li>Alyson Tyler </li></ul><ul><li>@libalyson </li></ul><ul><li> </li></ul><ul><li> - Evaluation report available here </li></ul><ul><li>Alison Kennedy </li></ul><ul><li> </li></ul><ul><li>Scotinform </li></ul>