Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

BIG Assist programme - OPM Evaluation

1,010 views

Published on

Evaluation of the BIG assist investment in national infrastructure programme July 2015

Published in: Government & Nonprofit
  • Be the first to comment

  • Be the first to like this

BIG Assist programme - OPM Evaluation

  1. 1. OPM Evaluation of the Assist Investment in National Infrastructure Programme Evaluation of the Assist Investment in National Infrastructure Programme Final Report Report to NCVO and Big Lottery Fund July 2015
  2. 2. OPM Evaluation of the Assist Investment in National Infrastructure Programme Client NCVO and Big Lottery Fund Company OPM Title Evaluation of the Assist Investment in National Infrastructure Programme Subtitle Final Report Dates Last revised 23 Mar. 2016 Status Version: Final draft Classification Restricted Internal Project Code 8933 Author(s) Karen Naya, Oliver Ritchie, Sophie Wilson Quality Assurance by Dr. Heather Heathfield Main point of contact Karen Naya Telephone 020 7239 7813 Email Knaya@opm.co.uk If you would like a large text version of this document, please contact us. OPM 252B Gray’s Inn Road 0845 055 3900 London www.opm.co.uk WC1X 8XG info@opm.co.uk
  3. 3. OPM Evaluation of the Assist Investment in National Infrastructure Programme Contents Summary ....................................................................................................................1 The evaluation ........................................................................................................1 Outputs and experience: infrastructure organisations ...........................................2 Outputs and experience: suppliers.........................................................................2 Peer to peer experience: face to face and online ..................................................3 Impacts on IOs........................................................................................................4 How well has BIG Assist met the outcomes for evaluation?..................................4 The future and recommendations ..........................................................................6 Introduction................................................................................................................7 The context for infrastructure .................................................................................7 What is BIG assist and how does the programme work? ......................................8 The evaluation of BIG Assist..................................................................................9 Findings....................................................................................................................12 How to navigate this section.................................................................................12 The Marketplace ......................................................................................................13 Customer outputs and experience .......................................................................13 Finding out about the programme ........................................................................13 Motivations for engaging with BIG Assist and key outputs..................................14 Use of the online platform to apply for vouchers..................................................17 Accessing a supplier.............................................................................................18 Receiving support.................................................................................................19 Making the most of vouchers ...............................................................................21 The customer experience: conclusions................................................................23 Suppliers: outputs, experience and impacts .......................................................25 Motivations for engaging with BIG Assist and uptake/outputs.............................25 Suppliers: Reflections on the programme design ................................................28 Impact for suppliers ..............................................................................................30 The Supplier experience, outputs and impacts: conclusions...............................30 Peer to peer offers: Outputs, experience and impact .........................................32 Motivations for engaging and uptake/ outputs .....................................................33 The peer to peer experience: conclusions ...........................................................36
  4. 4. OPM Evaluation of the Assist Investment in National Infrastructure Programme Impacts on infrastructure organisations ..............................................................38 Extra help and added impact................................................................................39 Greater efficiency .................................................................................................45 Other and longer term changes............................................................................47 Factors influencing the level of impact on infrastructure organisations: drivers and barriers 48 Conclusions: Impacts on infrastructure organisations .........................................52 Overall conclusions and recommendations ........................................................53 Summary ..............................................................................................................53 Conclusions: Has BIG Assist met the outcomes for evaluation?.........................53 Recommendations................................................................................................55 Appendices ..............................................................................................................57 Appendix 1: Pathways to Outcomes Model .........................................................57 Appendix 2: Evaluation activity and programme milestones ...............................58 Appendix 3: Sampling for customer and supplier interviews ...............................59 Appendix 4: Summary of output data used in this evaluation..............................61 Collation of output data used in this report...........................................................61 Appendix 5: Interview Topic Guides.....................................................................64
  5. 5. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 1 of 87 Summary BIG Assist BIG Assist is a £6million programme funded by the BIG Lottery Fund and delivered by the National Council for Voluntary Organisations (NCVO). The programme has piloted new ways of offering a targeted support programme to help voluntary sector infrastructure organisations (IOs) be more efficient, effective and sustainable. Support is offered across the following areas: - Strategy, planning and managing change. - Financial sustainability. - Innovation, new products and ways of working. - Marketing and building strategic relationships. - Supporting and developing people and organisational change Infrastructure organisations access relevant support via an online marketplace of approved suppliers, and pay for the support using vouchers supplied by the BIG Assist programme. A key element of the programme is also to promote peer to peer learning and the exchange of ideas between IOs. The evaluation OPM conducted an independent evaluation of the BIG Assist programme between December 2012 and April 2015. The evaluation focuses on the impact of the BIG Assist Programme and the extent to which the following four programme outcomes have been met: 1. Infrastructure organisations perceive and can evidence that they provide higher quality support to customer and frontline VCSE organisations. 2. National VCS and private sector support providers develop better and more sustainable models of providing support services to infrastructure organisations. 3. IOs value and feel they benefit from the opportunities for peer to peer learning and support. 4. BIG Assist develops and shares learning about how demand-led models of national support services could work in a local and national context and in a more market oriented way. The evaluation drew upon qualitative interviews with IOs, suppliers and the BIG Assist team, as well as routinely collected programme output data which are largely quantitative.
  6. 6. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 2 of 87 Outputs and experience: infrastructure organisations BIG Assist support to IOs has been extensive - the programme has issued 846 vouchers, with a total value of £3,545,9501. Over 700 IOs have completed diagnostic review to access voucher support and 576 IOs have been awarded vouchers2. The average voucher award was £8,040.70. The IOs that engaged in BA were motivated and wanted to make changes that would secure their survival in a challenging world. Programme changes especially the increase in value of the voucher awards encouraged more to engage. Planning for a more sustainable future was their key focus. IOs wanted to develop new and efficient ways of working. They looked to BIG assist to identify new sources of funding and be more effective at generating income. The Marketplace experience has worked well for IOs in terms of: - The online platform to apply for vouchers – Many found the experience of applying to BIG Assist a simple and positive experience. Review calls provided an important opportunity for them to reflect on their needs as an organisation. - Accessing a supplier - IOs were largely able to access a choice of supplier, although it could sometimes be hard to differentiate between supplier offers. Some IOs in rural areas and with specialist needs experienced issues with supplier availability. - Receiving support – IOs were very satisfied with the support they received. The vast majority gave their supplier the highest rating available. They particularly appreciated their expertise and understanding of IO issues and wider context. - Making the most of vouchers – Larger vouchers also helped. IOs generally reflected that larger vouchers had enabled them to conduct more comprehensive, far reaching projects. Many benefitted from the opportunity to apply again for support and vouchers. A few smaller customers pooled their vouchers, using flexibility within the programme, to get more value from support. - Advantages to a voucher system - It is less time consuming for customers to administer and because vouchers were for specific, ring fenced work, funds for support were secure and could not be diverted for other purposes. Outputs and experience: suppliers There are 223 approved suppliers in the Marketplace and they are a diverse group of organisation type including companies (106), VCS organisations (69), sole traders (34) and others (14). 1 Source of all output data: BIG Assist programme data. Reported 22nd May 2015 2 These figures include IOs resubmitting to the programme for further support.
  7. 7. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 3 of 87 There is variation in the volume of projects they have undertaken. One supplier has completed 37 projects, to a value of £196,000. 124 Suppliers, 53% have completed at least 1 completed project. Suppliers were motivated to join BIG assist for a variety of reasons. It was an important new source of funding their work; it helped them reach new clients who needed their support; and many hoped that being an approved supplier to Big Assist would raise their profile across the sector. Some also felt that approved supplier status was a ‘badge of recognition’ for their experience and quality of support. The self-assessment and review call meant that customers had a good knowledge of their support needs before work started. Some suppliers however still had to work with customers to identify what could be realistically delivered for the voucher value. Suppliers feel the Marketplace is an efficient model for delivering support. They understand that sustainability is an issue for BIG Assist. Without ongoing funding the Marketplace would soon disappear, although they hope that the new contacts they’ve made may endure. Peer to peer experience: face to face and online Peer to peer opportunities have developed along with the programme. Awareness and involvement have increased over time, as the numbers involved in BIG Assist have grown and feedback and information has spread. ConnectSpace, offering sponsored visits (and mentoring, now suspended) has seen significant uptake. 229 visitors have participated in supported visits. IOs are positive about the visit experience. They have made new contacts, gained confidence and practical knowledge about the areas they need to develop and in some cases, have found new collaborators for projects. ShareSpace, offering online discussion forms around key topics, has seen very considerable volumes of traffic. Some live discussions have had over 10,000 views. IOs access the forum and are interested in the topics raised. We have little evidence about how active participation is and how IOs are using ShareSpace to build their own capacity. The BIG Assist Library has built up over 600 resources for IOs. We have little evidence about how active participation is and how IOs are using the library to build their own capacity. The library resources are viewed on average 26 times per month, with the following pages being the most popular: 1. Six challenges for infrastructure organisations 2. Changing role not just for volunteer centres 3. Five ways to stay afloat without more revenue 4. The perils of ignoring infrastructure 5. Small charities big impact
  8. 8. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 4 of 87 Impacts on IOs Extra help and added impact. BIG Assist is a valued source of funded support in a time of great need for IOs. Important change projects were unlikely to have happened without BIG Assist funded resources and expertise from suppliers, or support would have been of lesser quality and/ or delayed, limiting impact. New sources of funding. IOs have won new sources of grant funding or contracts to deliver work. This has sometimes been because BIG Assist has helped them identify new funding sources to bid for and/ or because IOs can demonstrate they operate more effectively, which has been important to funders. Improved opportunities for consortia bidding. This is an emerging impact of the programme. Using vouchers to set up consortia has allowed IOs to identify and work more effectively with partners and they can now bid effectively as a group. New ideas for generating income. This includes IOs developing new chargeable services or products. Increased ability or capacity to adapt to change. BIG Assist has been a catalyst for necessary change in many organisations. BIG Assist projects have helped to secure staff and Trustee engagement and buy-in for change. Having independent, expert support has built staff confidence in the solutions put forward. Clearer focus on impactful and/or sustainable activities. IOs feel it is likely that BIG Assist support will make a long term difference to them. Some benefits won’t be felt for some time however, and sustainability over the longer term is subject to many variables, including the tough economic environment. How well has BIG Assist met the outcomes for evaluation? 1. Many infrastructure organisations perceive that they can provide higher quality support to customer and frontline VCSE organisations, or will be in able to in time. This programme has had extensive reach. Many IOs have gained new knowledge and skills that are of direct relevance to supporting frontline organisations. By improving their offer and range of support to the frontline, IOs hope to become more sustainable. In addition, BIG Assist has helped IOs extend their networks and find collaborators for new projects. Some organisations made rapid changes to the way they work and provide services, or saw immediate results from applying new skills. Others feel that the impact of some changes and projects are likely to be manifested over the medium to longer term. The wider context remains very challenging and may be limiting the capacity of infrastructure organisations to maximise the full potential of their BIG Assist support.
  9. 9. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 5 of 87 However without BIG Assist support, projects are unlikely to have happened in many cases, despite being of high importance to the organisation. 2. We are unable conclude that National VCS and private sector suppliers have developed better and more sustainable models of providing support services to infrastructure organisations. However, we have found much positive evidence that: This model is effective. Suppliers have delivered high volumes of quality support through the programme, which matches their own areas of expertise and IOs’ needs. The Marketplace has worked well for those who use BIG Assist as a means to deliver support both to existing clients and networks, and new clients. Suppliers have successfully extended their reach through the programme. IOs are exercising choice from a selection of quality suppliers but there is room for suppliers to improve the ways they respond to customers, market their expertise and raise their profiles in the Marketplace. 3. IOs do value and feel they benefit from opportunities for peer to peer learning and support. Peer to peer offers have grown with the programme. There are some very high levels of interest and participation. Sponsored visits are of great benefit to visitors and hosts, with positive feedback and emerging case studies suggesting immediate and medium term impacts. IOs are choosing to view ShareSpace live forum discussions in very large numbers. The potential for knowledge exchange is considerable given the interest, although there is limited evidence of how IOs are using their participation to support change in their own organisations. IOs value opportunities to exchange knowledge with others in similar positions to themselves, who are perhaps working in different ways. Time constraints on IO staff appear to be a limiting factor to more active participation. 4. Big Assist develops and shares learning about how demand-led models of national support services could work in a local and national context and in a more market oriented way, although there are opportunities to do more. BIG Assist has established a range of channels for developing and sharing learning about this pilot. Events, both face to face and online, are reaching a large audience. Since its launch, BIG Assist has used feedback from stakeholders, self-reflection and learning from evaluation to develop and refine the programme offer. There are opportunities to engage a wider group of stakeholders, connecting with infrastructure organisations that do not actively participate in BIG Assist offers. There is also potential to share learning with funders and others who influence VSC policy.
  10. 10. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 6 of 87 The future and recommendations The BIG Assist 3 year contract has been extended for a further year in order to develop four work areas: 1. A further £1 million investment in the infrastructure through the BIG Assist platform. 2. An extension of the evaluation. 3. A programme of outreach and consultation with funders and other stakeholders to secure additional investment to support the longer term sustainability of the BIG Assist platform. 4. A programme of engagement with leading infrastructure organisations to learn from and share their vision for the future. These work areas are strongly supported by the findings and conclusions of this evaluation. This evaluation makes the following recommendations for the programme: 1. Continue to offer demand led support for IOs through a marketplace model, retaining the features of self-assessment and range of topic areas for support. 2. Revisit opportunities to strengthen the provision of support for follow-up and implementation of projects, through approved suppliers and peer infrastructure organisations. 3. Explore options for making the programme sustainable over the long term by opening up the BIG Assist platform to partnership with other funders. 4. Work to engage infrastructure organisations in BIG Assist who have not yet participated in order to (i) better understand obstacles to access and (ii) broaden impact across the sector and demonstrate the potential reach, or limitations, of this demand-led model. 5. Maintain the existing high levels of programme flexibility and responsiveness to feedback from infrastructure organisations in order to sustain change and to ensure that the programme stays closely aligned to the needs of the sector. 6. Track impact within individual infrastructure organisations over an extended timescale in order to (i) provide evidence of longer term programme impact and (ii) better understand the impact of wider contextual barriers and levers for change.
  11. 11. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 7 of 87 Introduction The context for infrastructure Changes in the voluntary sector environment have meant that more infrastructure organisations (IOs) than ever are seeking extra support and advice to help become more efficient, effective and sustainable. Austerity measures in the national budget have led to significant cuts across local government.3 As a result, there has been a large reduction in the amount of money that councils are able to make available to support infrastructure organisations through grant funding. Competition for the remaining grants has become increasingly tough, and infrastructure organisations are being forced to find new ways to fund themselves, either by working with local government on a contract basis, by finding external funding sources, or by raising money themselves.4 The combined effects of the economic downturn and reduced spending on welfare and public services have also led to some front line charities experiencing increased demand for their services, at a time when some charities are also struggling to maintain their funding streams. In some cases, this has placed additional pressure on infrastructure organisations because the charities that they work with are asking for increased support, or because their support needs are changing. As a result of these pressures, a number of infrastructure organisations have downsized or been forced to close down entirely over the past few years. As part of the BIG Assist programme, voluntary sector infrastructure organisations were asked to mark themselves on a map in order to build a picture of the infrastructure sector. 898 infrastructure organisations were identified by August 2015. These organisations had an income of around £811 million in 2013/14. This was a drop of around £113 million in cash terms from 2010/11.5 3 Local Government Association. (April 2014). Under Pressure: How councils are planning for future cuts. Available http://www.local.gov.uk/documents/10180/5854661/Under+pressure.pdf/0c864f60-8e34-442a-8ed7- 9037b9c59b46 (accessed 17/04/2015) 4 Justin Davis-Smith. (2013). The Future of Infrastructure. Speech at 3rd July 2013 BIG Assist Conference. Available http://blogs.ncvo.org.uk/2013/07/03/six-challenges-for-infrastructure-organisations/ (accessed 17/04/2015) 5 NCVO Almanac. Available http://data.ncvo.org.uk/data/voluntary-sector-infrastructure/
  12. 12. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 8 of 87 What is BIG assist and how does the programme work? The £6 million programme BIG Assist, aims to provide support that meets sector needs in these challenging times. Since 2012 NCVO has delivered the BIG Assist programme under contract to the BIG Lottery Fund. BIG Assist is testing new ways of delivering support to infrastructure organisations in the voluntary sector to adapt and change how they work to meet the challenges of a much changed operating environment. The programme has been piloting a ‘demand led’ model of support by awarding a voucher that the organisation uses to select support of their choice through an on line market place of approved suppliers of support. The BIG Assist programme offers a wide range of support for infrastructure organisations in addition to awarding vouchers, through a large peer to peer programme. BIG Assist opportunities include:  Marketplace: where infrastructure organisations can browse and get in touch with BIG Assist approved suppliers.  ShareSpace: an online discussion forum where infrastructure organisations are encouraged to engage with each other by sharing thoughts, ideas and other information.  ConnectSpace: opportunities to get involved in sponsored visits and BIG Assist events (both face to face events around the country and online participation).  The BIG Assist Library, giving access to handpicked resources that are relevant to infrastructure. This includes opportunities to proactively edit and add content. The aim of BIG Assist is to help infrastructure organisations be more effective, sustainable and better able to adapt to change’. As such BIG Assist offers organisational support in the following topic areas:  Strategy, planning and managing change.  Financial Sustainability.  Innovation, new products and ways of working.  Marketing and building strategic relationships.  Supporting and developing people and organisational culture. To apply for BIG Assist vouchers, IOs, or customers, organisations that access support through BIG Assist go through a 3 step process: firstly, they answer a set of on-line pre- qualifying questions to determine they are eligible for an award. Next, they must submit an on-line self-assessment which is a more detailed set of questions about the organisation. Finally, they have a review call with a BIG Assist customer consultant, who will have read through the on line self- assessment and undertaken desk research about the organisation.
  13. 13. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 9 of 87 The review call is used to identify areas of need and assess the organisations ability to implement change. Support priorities and clear outcomes for the support are agreed. Review calls take approximately 1 hour. Once the focus of support and voucher value of support is agreed between the customer and BIG Assist, customers go to the on line Marketplace to review suppliers and select a supplier of their choice. Customers make contact with suppliers by email or telephone and once a project is agreed, BIG assist is notified and work can commence. On completion of the project, the supplier submits their invoice for payment and customers are asked to rate and comment on the support provided by their supplier. Reviews are available online for other customers to see. Suppliers apply online. The supplier applications are assessed by independent assessors to determine if they are approved as a BIG Assist supplier to deliver support though the programme. Approval includes assessment of relevant experience, prior work and references. Infrastructure organisations can be both customers and suppliers through BIG Assist. Support from the BIG Assist team is available to both customers and suppliers at each stage, over the telephone, on-line and through guidance documents and information. The evaluation of BIG Assist Aims and objectives OPM has been asked to conduct an independent evaluation of the Assist programme. The evaluation will focus on the impact of Assist: the extent to which the following outcomes have been met: 1. Infrastructure organisations perceive and can evidence that they provide higher quality support to customer and frontline VCSE organisations because through Assist, they:  Gain knowledge and skills to implement change and new ways of working.  Over the medium to longer term, have implemented changes to the way they work and provide service.  Feel increased confidence that they will be more sustainable in the future. 2. National VCS and private sector support providers develop better and more sustainable models of providing support services to infrastructure organisations because they:  Supply support which matches their own areas of quality practice and IOs needs.  Learn from, and can respond to, IOs exercising choice from a selection of quality suppliers.
  14. 14. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 10 of 87  Learn from and make changes to the support they provide in response to IO feedback. 3. IOs value and feel they benefit from the opportunities for peer to peer learning and support. 4. Assist develops and shares learning about how demand-led models of national support services could work in a local and national context and in a more market oriented way. Evaluation activities and methods The evaluation was conducted between December 2012 and April 2015. Appendix 2 shows how evaluation activities aligned with key milestones in the programme. Evaluation employed a range of activities consisting of:  A ‘pathways to outcomes’ model: developed to understand the programme and provide a guiding framework for the evaluation priorities and questions.  In depth qualitative interviews with 50 customers which engaged with the programme. These were grouped into 2 sets of informants. Each set was interviewed initially after they have been involved in the programme for 6 months to 1 year (first interview) and again when they exited the programme or support has been delivered (follow up interview).  In depth qualitative interviews with 5 IOs who are eligible for the programme but who have not engaged with it; - In depth qualitative interviews with 47 approved suppliers. These were grouped into in 2 sets of informants. The first group were interviewed at the start of the programme and the second group as projects were being delivered, towards the end of the programme - Three group interviews with the BIG Assist team: developed to understand the learning from voucher awards and peer to peer support opportunities IOs were sampled to cover a range of support needs, regions, and focus for their own work. Suppliers were sampled from all regions of England, and include sole traders, companies and VCS suppliers. Details of the sampling framework and interviews conducted are provided in Appendix 3 Interview data have been analysed thematically, i.e. all interview notes were reviewed and categorized, to identify patterns and developing themes. This evaluation also draws upon BIG Assist reports and output data: - These data are sourced largely from the BIG Assist online administration platform.
  15. 15. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 11 of 87 - Data are submitted directly by customers and other infrastructure organisations, or are created automatically by the on-line system. - BIG Assist supplied additional quantitative data, which it routinely collects, to OPM. - Programme data have been selected and interpreted by OPM and verified by BIG Assist. We employed descriptive statistics to understand these data, describing the main features of the data and providing summaries where appropriate, i.e. frequencies and some basic measures of central tendency. A table collating the quantitiive programme data used in this report is provided in Appendix 4. Considerations around the evaluation It is useful to note the following considerations:  Whilst the evaluation has focused around the impacts of BIG Assist, this is a largely formative evaluation. BIG Assist started from scratch and we should recognise that the experiences of customers and suppliers have changed, as the programme developed.  Not all offers were fully in place from the start of the programme (notably peer to peer support) and important changes have been implemented along the journey, for example phasing from the pilot Beta online platform to full online functionality, and increasing the maximum voucher value. We acknowledge the effect on interviewee experiences and when interpreting the programme data.  Impacts of the programme are still emerging and it is not possible to fully understand the extent of some changes within the scope of the current evaluation.  BIG Assist is being delivered in a very challenging environment for the VCS. IOs are experiencing high demand on staff time and in many cases there has been significant staff turnover. This has influenced the experience of the programme and its impact, but has also meant that it has not always been possible to conduct evaluation interviews with the target informant, or someone who can discuss the ‘big picture’ of the BIG Assist context and journey for an IO.
  16. 16. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 12 of 87 Findings How to navigate this section In this section we discuss the experiences and outputs of IOs and suppliers at various stages in their journey with BIG Assist. We also present findings around the impacts of BIG assist on IOs. This is a complex programme. To present findings and conclusions systematically, we have structured them as follows: We start with the Marketplace:  Customers: We first present the customer experience of the Marketplace and voucher system, with key customer outputs and impact. This is prefaced with an overview of these findings. We offer brief conclusions at the end of this subsection.  Suppliers: We then present the supplier experience of the Marketplace and key supplier outputs. This is prefaced with an overview of these findings. We offer brief conclusions at the end of this subsection. We move on to peer to peer offers: online and face to face:  Here we present the experience and outputs of the various peer to peer offers. This is prefaced with an overview of these findings. We offer brief conclusions at the end of this subsection. We then present impacts for IOs:  Impacts on IOs, attributable to various offers of the BIG Assist programme, are identified and discussed. This is prefaced with an overview of these findings. We offer brief conclusions at the end of this subsection.
  17. 17. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 13 of 87 The Marketplace Customer outputs and experience Overview of outputs and experience  IOs were motivated to engage with BIG Assist to make changes that would secure their survival in a challenging world. Financial sustainability was their key focus. IOs wanted to develop new and efficient ways of working. They looked to BIG assist to identify new sources of funding and be more effective at generating income.  BIG Assist support has been extensive: the programme has issued 846 vouchers, with a total value of £3,545,9506. The maximum number of vouchers awarded to an IO is 5. The average is 1.91 per IO, with an average total value per IO of £8,040.70.  The process of applying for vouchers in itself added value for some IOs, because it gave them an opportunity to discuss and clearly identify their needs through the diagnostic review.  Customers were able to access a choice of supplier and the range of suppliers was good, for most customers.  Customers were generally very satisfied with the support they received. 72% gave their supplier the highest rating possible. Suppliers are knowledgeable and experienced. They also demonstrated commitment to helping IOs and offered good value.  They report that larger vouchers allowed more comprehensive work to take place. Many had received larger vouchers and had also resubmitted to the programme, for further voucher support. This was often a way of embedding the impact of the initial project. Finding out about the programme BIG Assist was promoted widely through a series of face to face engagement events. Nearly 1,400 people attended events held between October 2012 and September 2014. 15 events took place in Year 1 of the programme, 11 in Year 2 and 3 in Year 3. The BIG assist National Summer Conference in 2013 was attended by 162 delegates. 6 Source for all output data: BIG Assist. Data reports extracted 22nd May 2015.
  18. 18. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 14 of 87 The majority of IOs that we spoke to found out about BIG Assist through email alerts, particularly through the NCVO and NAVCA mailing lists. Additional ways in which IOs found out about the programme include networking events or colleague contacts. Motivations for engaging with BIG Assist and key outputs IOs were seeking support to secure their survival A number of organisations applied to the BIG Assist programme because they recognised a fundamental need for support which would help them to survive in a challenging world. In this context, a number of infrastructure organisations have hailed the BIG Assist programme as a source of much needed support for a struggling sector. “There were no other sources of support around. Even realising that things were not right you put everything down to the external environment – you are a product of your environment, but it is you that can change it, react to it, take yourself out of it. That is hard to do when you are so engrossed in that. […] BA were able to do that in partnership with us. We know we had to do something, but were not sure what. […] It has really helped.” IO, round 1, first interview 7557 IOs completed on line diagnostic the second stage of applying for voucher support. Key outputs showing response to demand We will describe IO motivations for applying in more detail below, but here it is important to understand the scale and type of support BIG Assist has provided, in response to IOs: Overall, BIG Assist has issued 846 vouchers to support infrastructure, with a value of £3,545,950. 576 IOs received voucher awards8. The table below shows the allocation of BIG Assist vouchers for support, by topic area and sub topic. 7 This figure includes re-submissions and applications since withdrawn. 8 This figure includes re-submissions
  19. 19. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 15 of 87 Topic area Sub-topic Number issued Total value Financial sustainability Cost efficiency and savings 1 £3,000 Financial management 8 £30,000 Income strategy and new business models 153 £620,750 Managing your assets 6 £28,000 Innovation, new products and ways of working Developing new products, services and ways of working 130 £560,500 Innovating culture 2 £11,000 Marketing and strategic relationships Collaboration and partnerships 27 £113,000 Marketing and communications 131 £533,500 Strengthening and creating effective relationships 23 £91,000 Strategy, planning and managing change Business planning 103 £398,000 Leading change 17 £84,200 Organisational strategy 80 £330,500 Planning, assessing and communicating impact 52 £217,500 Supporting and developing people and organisational culture Developing skills and adapting to change 75 £363,000 New structures and ways of working 34 £146,000 Organisational culture 4 £16,000 TOTAL 846 £3,545,950 We can see that the biggest single sub topic area for both number and value of voucher awards was around income strategy and developing new business models, in the topic area of Financial Sustainability, Over 150 vouchers were issued for this sub-topic with a total value of £620,750. Marketing and strategic communications, and Developing new products,
  20. 20. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 16 of 87 services and ways of working were the next greatest sub-categories in terms of number of vouchers issued. Planning for a more sustainable future Seeking support to adapt to the tough economic landscape was a major driver for IOs to apply to BIG Assist for support. Many organisations saw assistance from the BIG Assist programme as a means of planning for a more sustainable financial future. Several organisations were reaching the end of a grant cycle and applied to use the voucher to secure a future beyond this funding: “Where are we going to get money from? We looked around…and out of the blue popped up the Big Assist Offer which we grabbed with both hands”. Round 2, first interview with IO Infrastructure organisations commented that support for voluntary and community infrastructure organisations has become increasingly intermittent, and mentioned that other CVSs have disappeared. As a result, many IOs recognised the need to ‘come up with a different approach’, and used the programme to help them to implement this. “It is a great resource, any money you can get as an infrastructure organisation, it’s always more difficult to get money as a second-tier organisation – so I think it’s important to know about”. Round 2, first interview with IO Developing more efficient ways of working Many IOs hoped that BIG Assist support would help them to streamline their ways of working, in order to use available resources more efficiently Some wanted to use the vouchers to increase the scope and reach of their organisation, so that they could provide more support, or reach a greater number of frontline organisations: “This will benefit the community and voluntary organisations because if we are bigger they are better”. Round 2, first interview with IO Many IOs also talked about being motivated to apply to BIG Assist to become more streamlined and accessible in order to serve their present members in an increasingly efficient and timely manner. In several cases, this involved developing more comprehensive online materials, or modifying the services they deliver to ensure they meet the needs of their user groups. IOs also reflected that increasing their own efficiency would enable them to minimise costs for services to front line organisations, or even allow them to deliver some support at no charge.
  21. 21. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 17 of 87 Use of the online platform to apply for vouchers The application process could be a positive experience and provide space to focus on the future BIG Assist required the creation of an entirely new online platform. Many customers found the experience of applying to BIG Assist a simple and a positive experience, This was either because they viewed it as being less burdensome than other applications that they had taken part in, or because it was actively helpful, providing support to structure their thinking about what they need to do as an organisation. For a number of IOs, the application process was an important opportunity to take time to focus clearly on their needs and future. Finding this time and making it a priority was sometimes difficult to do in a busy organisation, driven by immediate demands. Review calls add value for infrastructure organisations We predominantly gathered feedback on customer review calls in the baseline round 1 and baseline round 2 interviews. Customers were particularly positive about their review calls, which took place as part of the diagnostic review process. Often customers felt the interview helped them to prioritise issues and understand the needs of their organisation better, by providing a space for constructive reflection. Many commented that the interview process added value in itself, and indeed one applicant identified the interview and subsequent report as the most valuable part of the BIG Assist programme: “One of the best interviews I’ve ever had; informative and constructive”. Round 2, first interview with IO “The guidance was brilliant, they were professional, they asked lots of questions, really took time to understand our organisation”. Round 2, first interview with IO Many interviewees added that the interviewer had been able to provide them with helpful suggestions and advice, informed by a thorough understanding of the needs of the sector. Some interviewees described how the interviews could be further improved. This included having more information in advance about what information would be needed during the interviews so that they could prepare, and there being more clarity about whether the interview would affect whether they got vouchers, or whether it was purely diagnostic of their needs. It should be noted that a minority of interviewees, particularly those in early rounds of interviews, felt that the online application process was time-consuming and placed a significant demand on the IO to complete. This theme lessened as the initial online pilot site, was phased out and replaced from January 2013 onwards. Based on user feedback the BIG Assist team modified and improved the on line diagnostic including amending the wording of questions and providing guidance information such as Question and Answer guides, to help
  22. 22. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 18 of 87 IOs through the process as smoothly as possible. This suggests that ongoing work to improve the website and application process has been successful To understand how the adoption of an online system might affect the impacts of BIG Assist, and to identify potential areas for programme improvement, in the second round of interviews with customers we asked participants about their self-assessed IT ability. Whilst this requires the respondent to judge, it is perhaps helpful context about their general levels of capacity and ‘comfort’ around using online systems. Responses to this question showed that there are a number of individuals in the sector who find IT difficult in general. This may provide one explanation for the issues reported by some organisations. A consistent theme throughout the evaluation found that where people had difficulties using the online platform, the BIG Assist team were able to support people effectively and remedy difficulties quickly. “I do have to say that my human contact and IT based contact are two very different experiences”. Round 2, first interview with IO The BIG Assist team are responsive and able to provide solutions A strong theme was that interviewees provided positive feedback about the responsiveness of the BIG Assist team. When issues had arisen (for a minority), phone or email contact had minimised or negated the impact of any difficulties they had been experiencing. Such comments were made in all rounds of interviews, especially the later rounds, possibly indicating that contact from the team has been maintained throughout the customer’s BIG Assist journey and did not tail off after the initial application process. Accessing a supplier Customers were able to exercise choice of supplier Feedback on the availability of suppliers was generally positive. Most customers reported that they were able to find a suitable supplier. Approaches to identifying a supplier varied greatly between different infrastructure organisations. While many customers browsed the information on the online marketplace to identify a supplier, others looked for organisations they were familiar with, or even encouraged their preferred supplier to sign up to the BIG Assist programme if they were not already on the approved supplier list. With over 220 suppliers available (across all topics for support, nationally) customers frequently described how it could be difficult to differentiate between suppliers on the online marketplace. It was felt this was because many suppliers had put up similar information about themselves, or had ticked the same range of boxes as to what types of support that
  23. 23. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 19 of 87 they could deliver. Ratings systems and online feedback were valued as a source of additional information to help make a decision, and this information built up over the course of the programme. Some customers nevertheless reported that more advanced sorting and search functions would have been helpful to refine their selection. Many customers initially approached a range of suppliers (those we interviewed typically contacted approximately 3-5) before making a decision about which to select. These approaches varied considerably in formality from sending out a quick email or ringing up possible providers for a discussion, to producing ‘invitation to tender’ documents asking suppliers to produce bids for the work. Customers told us they did not always receive responses from all of their contacts to suppliers, which some felt had limited their options. Not all customers used the on line market place to find suppliers. A few applied to the BIG Assist programme already knowing which supplier they planned to use, perhaps because they had worked with them before. In some cases, they encouraged a supplier to register on the programme so that they could use them. The BIG Assist Marketplace was therefore used by some as a vehicle for working with established contacts. Additionally, many customers who were involved in more than one round of support chose to use the same suppliers again a second time, rather than go back to the marketplace and make a new selection. Some said that this was because they knew that they would be satisfied with the support that they received, while others said it was because it meant that the supplier would already know their background context, making the support delivery more efficient. A few customers expressed a preference for working with VCS suppliers, feeling that where possible ‘money’ should be kept within the sector. The marketplace enabled these customers to choose VCS suppliers for their support. However a few organisations with specific needs reported some difficulties. Additionally, some customers in more rural locations felt that many of the suppliers seemed to be based in cities. One customer based in a very remote area said that after discussions with suppliers, some had declined working with them due to the distances involved, which they felt had limited the options available to them. Despite this minority experience, BIG Assist have an understanding gained from interactions with customers and suppliers that suppliers are travelling and extending their range to deliver projects. Evaluation interviews with suppliers also suggest this happens in some cases. Receiving support Customers were very satisfied with the support they received Customers had the opportunity to rate the support that they had received from 1 to 5, as part of a feedback process in the online systems. Levels of satisfaction with the supplier were very high overall. 72% (487 / 676) of customers gave a 5 rating – the maximum possible - for the support that they received as part of the programme. 162/ 676 customers awarded a rating of 4, indicating good satisfaction.
  24. 24. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 20 of 87 Interviewees and online reviews show that organisations giving high scores to suppliers often highlighted the supplier’s excellent knowledge of the operating context for IOs and their responsiveness to the specific issues they faced. “Very experienced and knowledgeable consultant who quickly grasped our operating context, culture and challenges. ” IO, online feedback Several IOs reported that they are already working with the same supplier on another project, or hope to do so if they can obtain more funding because of their high levels of satisfaction with the support that they received. “We couldn’t have afforded it [without BIG Assist funding] even though it was only a small amount – we’re only a small organisation, and the work wouldn’t have got done at all. […] This piece of work was invaluable.” Round 1, follow up interview with IO IOs as new clients for a consultancy model of support In fact, some IOs have changed their views of using external consultancy support as a result of their positive experiences of BIG Assist. A comment made by a few infrastructure organisations was that prior to the BIG Assist programme, they had not seriously considered the possibility of bringing consultancy support to help them. The programme had therefore given them experience of a new way of working. Some of these organisations said that if they could afford to do so they would consider engaging a consultant again in the future as a result of their experiences. One organisation told us about a bad experience that they had had with consultants in the past, which had made them initially a little wary of the support model. However, the positive support that they received through the BIG Assist programme made them more open to the possibility of using a consultant again in the future. Some respondents who awarded a lower score of 3/5 said that this was due to the need for additional internal capacity of their own to take the support forward and achieve outcomes, rather than because the support had not been high quality. For this small group, it is hard to conclude if the supplier may have made unrealistic assumptions or demands around the customer’s capacity, which might explain the lower score and would in effect, be associated with to the quality of supplier support. Only a tiny minority (5 ratings, less than 1%) awarded low scores of 2 or 1 to their supplier. Where customers gave a low score, they commonly explained that this was due to how well the supplier understood their local context, or how easy it was to communicate with the supplier, for example how easy it was to organise dates for training. “[They] had no knowledge of the local issues in [the local area]. Their advice was of a standard nature and the research they intended to undertake was already known to us.” IO, online feedback
  25. 25. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 21 of 87 Commitment and value from suppliers During interviews, some organisations also noted that their supplier went ‘above and beyond’ the amount of work they would normally do for that amount of money. This dedication was greatly appreciated by the IOs. However a few also noted that due to the small size of each voucher, they might not have been able to fulfil their aims if suppliers hadn’t worked over and above in this way. “We got more from the consultant than expected. The extra value they added was going above and beyond the call of duty; they did more than we paid them for. […] If [the consultant] had only done what they’d been paid to do I don’t think we’d have got the kind of results we got.” Round 1, follow up interview with IO Making the most of vouchers For high numbers of interviewees, the use of a voucher-based system ran smoothly and had not presented any particular issues. Larger vouchers allowed more comprehensive work to take place In May 2014, towards the end of the programme, the maximum voucher size available to infrastructure organisations increased. In the final rounds of interviews we spoke to a number of organisations which had been awarded larger vouchers (up to approximately £9k each, in our sample) It was felt that these larger vouchers had allowed more comprehensive work to take place, meaning that the level of input from suppliers has enabled customers to take forward new ideas into practice, rather than just delivering information about them. There is anecdotal evidence that this has done a good deal to increase the impact of the support, as some organisations had previously struggled to find the resources to make full use of the strategic guidance that they had received from their support. Use of multiple vouchers Customers were able to receive more than one voucher, either through one application and award, or by re-applying to the programme for additional support. A voucher is awarded for each area of support. It is common for customers to need support in a number of different areas. Customers received on average 2 voucher awards. 162 organisations reapplied for voucher awards. The average award over the whole programme was £6,258. One organisation received 5 voucher awards to the value of £18,000, this was the highest award. Where two vouchers were awarded at one time, the same supplier was generally, although not always, used for both vouchers.
  26. 26. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 22 of 87 Resubmitting for additional support Many organisations we spoke to in follow up interviews said they had applied to the programme a second time and received additional support. Organisations had done this as a way to further embed the impacts of the initial support. For example, organisations initially applied for support developing a new strategy, and then later applied for a second voucher to help implement this. Others applied for additional vouchers in order to meet support needs that had been identified through the initial support. Pooling vouchers with other organisations The opportunity to pool vouchers with other organisations was a flexibility offered by the programme. Pooling arrangements were intended to allow IOs to get added value from combined support. Some customers were supported by the BIG Assist team to do this. Others sorted this out themselves. “Working with another organisation worked well- a lot of shared learning can happen across that. A £2,500 grant doesn’t pay for much consultant time- 5 or 6 days. By working together and combining projects, it feels like we got better value.” Round 1, second interview with IO The programme has details of the situations where the team have supported the pooling of vouchers but it has not been able to record all situations where organisations have themselves shared vouchers. So we cannot quantify the uptake of this opportunity. Interviewees described several reasons for pooling vouchers. Some did this because they were trying to find ways to work more efficiently with one another or wished to set up a consortia. Others did it because they realised that they had similar needs, and wanted to be able to afford a bit more support. In such situations, one piece of feedback that we received during round 1 interviews was that while BIG Assist processes did not seem to have been explicitly designed to accommodate voucher pooling, the BIG Assist team were flexible about customers’ pooling arrangements and were helpful in making them a possibility. Vouchers can benefit IOs not directly involved in BIG Assist Some organisations spent their voucher in a way that was beneficial to another organisation, even if that other organisation had not also applied to the BIG Assist programme. In one such example, two CVSs were considering merging, and one of them used support from the BIG Assist programme to help consult stakeholder views on this. Advantages to using a voucher system Some IOs, especially smaller ones or those with less back office support, welcomed the fact that they did not have to process payments to suppliers themselves, as this can use up
  27. 27. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 23 of 87 valuable admin time. It also made budgeting easier for some because the money did not have to pass through their systems at all. Some interviewees thought that receiving the support as a voucher for a specific element of their work meant it had a greater impact than if they had simply been given the money. One reason for this is that the programme helped them to identify what their needs were, and then provided them with funding that was ring fenced to address these. Some interviewees said that if they had been given cash, it might have been harder to justify using the money to support longer term investment in this way, rather than simply using it to help balance to books in the short or medium term. We also received feedback that the use of vouchers could lead to a better working relationship between customers and suppliers because payment was assured through the BIG Assist process. Just one customer who we spoke to in the first round of interviews had a different experience: a delay in drawing down the voucher meant that their supplier was paid late, and they were worried that this might have affected their relationship. Challenges around the use of vouchers: timescales A minority of infrastructure organisations suggested that deadlines for customers to choose and engage suppliers were not long enough. This observation is supported by the fact that in the round one follow up interviews, over half of interviewees reported having asked for an extension to finish spending their voucher. Suppliers also commented on the tight timescales, with vouchers expiring after 4 months, explaining that the time left to complete the work after being approached and contracted was sometimes insufficient to do a really good job. However, one interviewee from an infrastructure organisation said that they had found the deadlines extremely helpful in keeping her on task. She explained that without them, it would have been harder to prioritise organising the support, and it would have taken her much longer to engage a provider. The BIG Assist team have reflected on this learning. They state their experience is that the longer timescales given at the early stages of the programme led to a lack of momentum around selecting suppliers and starting the work. These delays in starting were perceived by the BIG assist team as a limiting factor on impact and the reduced timescales introduced during the programme encouraged customers to get on with the work and take action. The customer experience: conclusions  BIG Assist has delivered important support to help IOs survive in a very challenging landscape.  Outputs are very significant: there are great numbers of IO applying for, and receiving support. Whilst we are unable to make definitive conclusions about
  28. 28. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 24 of 87 the reach of BIG Assist, due to uncertainties of mapping the VCS infrastructure sector.  Feedback and programme data around the allocation of vouchers suggests that BIG Assist projects are closely aligned to the needs of IOs.  Most IOs are highly satisfied with the support they have received.  Applying to BIG Assist has given IOs an opportunity to consider their future and identify needs in a focused way, aided by review calls with the BIG Assist team  They have exercised their choice of supplier.  Customers can manage the online process and come back to the programme for more support.  The BIG Assist team have developed much insight into the needs of IOs.  We note that a process using cash based transactions could also identify needs, determine the focus of support and set time limits for undertaking the project in the same way. Apart from any efficiencies around reduced IO administration through a voucher as opposed to managing a cash award, use of a voucher process for support may be a ‘psychological’ benefit over a cash based process.
  29. 29. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 25 of 87 Suppliers: outputs, experience and impacts Overview of outputs and experience  Suppliers were quick to engage with BIG Assist. There are 223 approved suppliers in the Marketplace.  Suppliers are a diverse group, the majority are companies (106), followed by VCS organisations (69), Sole traders (34) and others (14).  There is variation in the volume of projects they have undertaken. One supplier has completed 37 projects, to a value of £196,000. The 124 Suppliers with at least 1 completed project have delivered an average of 5.7 projects each.  Suppliers were motivated to join BIG assist because it was a new source of funding their work with infrastructure. They expected BIG Assist to be the vehicle for delivering the bulk of infrastructure support and felt they needed to be on board.  They hoped that being an approved supplier would raise their profile across the sector and introduce them to new clients. They wanted BIG Assist to help them to reach clients who needed their support.  Suppliers think the voucher system is an efficient way of working although would sometimes like more time to complete projects, or to deliver an extended period of support to help embed change. Finding out about the programme Suppliers heard about BIG assist through their own networks of contacts, including news updates from key VCS organisations and through promotional material produced by BIG Assist. In some cases, infrastructure organisations told us that they had asked suppliers to sign up to the programme so that they could get support from them. Motivations for engaging with BIG Assist and uptake/outputs A new source of funding their work with infrastructure Many suppliers saw the BIG Assist programme as the means by which they could access funding for infrastructure support work within the wider context of austerity. They were motivated by being part of a programme that enables them to exchange support and share expertise. Key outputs regarding suppliers are shown below:
  30. 30. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 26 of 87 The Marketplace attracted a large number of suppliers 223 approved suppliers are currently in the Marketplace. Suppliers were quick to engage with the programme. Around 200 applied to be approved suppliers in the first application round when the programme opened. Approved suppliers are a diverse group. A breakdown from programme data shows:  Companies: 106.  Other: 14.  Sole traders: 34.  Voluntary or community organisations 69. The amount of support delivered by various suppliers differs greatly. There is variation in the volume of projects completed by suppliers. An average of 3.15 projects have been undertaken per supplier (if all approved suppliers are included). However, the 124 suppliers with completed projects9 have delivered an average of 5.7 projects each. Some suppliers have undertaken a great deal of projects - the greatest number of completed projects by any supplier was 37, with a total voucher value of £196,000. This diversity makes variation in suppliers’ experiences of the programme inevitable. However, themes did emerge from the interviews conducted throughout the programme. These are reported below: Big expectations of BIG Assist When we spoke to suppliers in December 2012 – January 2013, some thought that most, or all, infrastructure support might end up being delivered through the BIG Assist Programme. A number therefore saw it as essential to register on the programme in order to ‘stay in the game’. A very large number signed up at the first opportunity when the programme opened, with over 200 applications received. This view of BIG Assist as the main source of work for delivering infrastructure support was not discussed to the same extent in the second round of interviews, conducted after suppliers had started to deliver support from June 2014. Uncertainty about the volume of new work Although suppliers appreciated the scale of BIG Assist early on, they were not always sure how much work they would end up delivering through it. Expectations were particularly unclear to suppliers in early interviews. In the round two interviews we found that 9 The BIG assist online system shows projects as completed when supplier invoicing and other requirements have been met, so this would not include suppliers with their first projects pending or underway
  31. 31. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 27 of 87 experiences had been quite varied, with some organisations managing to deliver a good deal more than anticipated and others a little less. Opportunity to work with new contacts and/ or work in new ways Suppliers saw BIG Assist as an opportunity to reach out to new clients. Suppliers hoped the programme would expand their networks of contacts and clients and we heard from many that they had achieved this aim. In some cases, BIG Assist projects brought suppliers into contact with clients from slightly different backgrounds from their usual client base. This was more of a theme for suppliers who were companies, who despite demonstrating relevant experience, were in some cases used to working predominantly with slightly larger organisations than BIG Assist customers. Raising their profile Across both sets of interviews, suppliers were attracted by the potential for the programme to raise their profile. Several suppliers said they felt ‘proud’ to be on the scheme and felt they would benefit from being associated with the BIG Lottery and the NCVO brands. Suppliers felt that involvement in the programme would ensure their visibility to IOs in the sector. For example, one stated reason for getting involved was: ‘So people who need me, can find me’ (Round 1 supplier interviews). Some suppliers also thought that becoming an approved supplier would act as a mark or standard of their quality and so would help them be accredited for the work they’d done previously. Going forward, they hoped that Approved Supplier status would help them to win work outside of BIG Assist. Feedback from some suppliers was that they were not always clear on how they could market themselves most effectively through the online marketplace, especially if they did not come up on the first few search pages, or had become involved in the programme later. Suppliers joining the programme later told us they felt marketing themselves was more difficult because other suppliers already had a good deal of positive reviews from customers, which made it harder for them to get their first piece of work through the programme. Others said that they had not tried very hard to market themselves, either because they were getting as much work through the programme as they wanted, or because they did not have the time to do so due to other commitments. The BIG Assist team have encouraged suppliers to improve their marketing and presence on the online platform, and have issued information to support them to do this more effectively.
  32. 32. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 28 of 87 Suppliers: Reflections on the programme design Some suppliers also changed their practice by expanding their geographical working area, or stated that they were available to work across a wider area. We do not have data on how many of those who wished to do so may not have been selected because customers prefer a local supplier. Knowledge of customer’s support needs A positive piece of feedback from suppliers was that because infrastructure organisations went through a rigorous application process, customers often had a clear idea of what support they wanted, making it easier to deliver what was needed. However not all suppliers had this experience: some said that they often had to do some work initially to help customers work out what could reasonably be achieved with the budget. Timescales for delivery Like customers, some suppliers reported challenges around programme timescales. Suppliers articulated their issues as sometimes finding it quite demanding to finish work by the deadline set for completion. This was particularly an issue where customers had taken a long time deciding which supplier to use. However, in such situations solutions were often found. In some cases suppliers report that it was possible to get an extension from the Big Assist team, in others they report treating the deadlines slightly informally, for example by conducting some unpaid follow up work after being paid on the official deadline. Timing of payments A number of smaller suppliers (especially sole traders) said that the voucher payment system could cause them cash flow difficulties because the money is only released after the project is complete. Follow-up support to maximise impact Like some customers, suppliers reported the need for follow up support after the work had been delivered, in order to ensure maximum impact. One individual reported that in order to fund an extended period of support, they charged slightly higher rates for the period of ‘official’ support, on the assumption that they would then provide additional input for no further charge over subsequent months. BIG Assist is an efficient way to deliver support A few suppliers also commented on the efficiency of the BIG Assist model. One commented that a marketplace is an effective way for them to spend money because it allows them to
  33. 33. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 29 of 87 only pay for the exact support that is delivered: if they were to use in house employees to deliver the support there might be additional overhead costs involved. “From Assist’s point of view it’s a low risk model, they’re not employing me, if people don’t choose to work with me there’s no consequence re cost, there’s no management with regard to me on BIG Assists part, they only pay me for something I deliver. […] It’s a new way of working, a leaner way […] so this means the money stretches further and the frontline services can continue.” Supplier interview, 2013 In addition, some suppliers conceptualised the support that they were providing as being part of a cascade model: they would help to upskill infrastructure organisations, which in turn could pass that learning on to the organisations that they support. One supplier described this as ‘developing 10 birds with one stone’ (supplier, round 1 interview.) Potential of an open marketplace A minority of third sector organisations (VCS suppliers and BIG Assist customers) felt strongly that where possible, money for infrastructure support should be kept ‘within the sector’. Some suppliers with this view said that they hoped that customers would choose them over commercial suppliers for this reason. Some suppliers (and some customers) noted that certain VCS organisations could be quite ‘territorial’, only asking for or delivering services to organisations on their patch. A number of suppliers said that they thought the online marketplace might help to encourage IOs to look at a wider range of possible options to meet their support needs. There is evidence from BIG Assist team learning that while geography remained an important consideration for many customers, some did report engaging in suppliers who they wouldn’t have considered otherwise as a result of the programme. Sustainability of BIG Assist Suppliers (and customers) reflected that the BIG Assist programme had created a somewhat artificial market for support services because while infrastructure organisations have a good deal of need for support, they do not typically have the resource to buy it in themselves. It was therefore observed that the while a useful market had been created, if the financial injection provided by the BIG Assist programme was to cease, then the market would quickly collapse. This observation was not typically meant as a criticism: it was simply an observation that positive influences of the programme on the market may not last long beyond the end of the programme
  34. 34. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 30 of 87 Impact for suppliers New markets and new work for suppliers Taking part in the Assist programme has had a number of positive impacts on suppliers, including helping some to access new markets, and providing them with a source of income. We spoke to suppliers at two different stages of the programme. In the first round of interviews, they had provided little or no support as yet, while the second cohort of suppliers that we spoke to had all delivered some. As discussed previously, suppliers’ experiences of the programme were varied, with some delivering much larger amounts of support than others. There was consequently a degree of variation in the types of impact that suppliers experienced. When we spoke to suppliers in the round 1 interview, a number said that they were hoping to use the programme to become involved in new markets, either in terms of the type of work that they were doing, the type of organisation they were supporting or the geographical region in which they were working. During the second round of interviews, many suppliers said that they had indeed managed to access new markets in this way. “For me it has expanded my geography. When I started working on BA most of my clients were in one area, and I have now had them all over the country.” Round two, supplier interview For some suppliers, the support in accessing new markets, or increasing their profile in existing ones was considered a greater impact than any profits that they made as a result of the work. Indeed, some suppliers told us that they delivered work through BIG Assist for very little or no profit, as a way to expand their client base. There is some supporting evidence for this from customers, who said that their supplier had delivered work ‘over and above’ the value of the voucher. The Supplier experience, outputs and impacts: conclusions  BIG Assist has successfully created a Marketplace of suppliers.  Suppliers recognise the scale and importance of BIG Assist to the sector.  They are using the opportunity to reach out to old and new clients, although there is room for some to improve their visibility in the Marketplace.  Some suppliers have found BIG Assist to be a strong source of work, delivering multiple projects through voucher support.  Suppliers identify the challenges IOs face around implementing support over the longer term. They see a need to offer longer periods of support to IOs, or additional capacity to maximise the impact of BIG Assist projects.
  35. 35. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 31 of 87  There is limited evidence that suppliers have changed their ways of working as a result of BIG Assist. We cannot assess the extent to which suppliers’ expressed willingness to work in different ways (for example in new geographical areas or with new types of client) has translated into practice. There is some evidence that a number of suppliers have varied their geographical range of operation.  Despite the huge volumes of support work carried out through BIG Assist, we cannot know the extent to which BIG Assist has become the main vehicle through which IO support is delivered, as some suppliers anticipated. Projects provided additional work that suppliers would not have otherwise undertaken.
  36. 36. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 32 of 87 Peer to peer offers: Outputs, experience and impact A key element of the programme is around facilitating peer to peer learning and exchange between IOs. This section discusses the peer to peer support provided by BIG Assist, beyond the Marketplace and voucher system. It is important to note that much of this support was offered and promoted to infrastructure organisations that had not received a voucher to engage a supplier, as well as those who had received vouchers. The evidence reported here draws upon programme data (output data around on-line and other peer to peer activity) and a secondary analysis of the reports submitted by those participating in Sponsored visits. We also report themes emerging from interviews with voucher recipients. Overview of findings: outputs, experience and impact  Peer to peer opportunities have developed along with the programme. Awareness and involvement have increased over time, as the numbers engaged in BIG Assist has grown and feedback and information has spread.  ConnectSpace, offering face to face and online events, sponsored visits (and mentoring, now suspended) has seen significant uptake.  229 visits have taken place, by 129 organisations.  IOs are positive about their visit experience.  Impacts are emerging. IOs have made new contacts, gained confidence and practical knowledge about the areas they need to develop. Some have found new collaborators for projects. IOs feel positive impacts will emerge from visits.  ShareSpace, offering online discussion forms around key topics, has attracted considerable volumes of ‘traffic’. Some live discussions have had over 10,000 views.  There have been 29 events over the 3 years of the programme, the largest with 162 delegates.
  37. 37. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 33 of 87 Motivations for engaging and uptake/ outputs ConnectSpace: Face to face and online events, sponsored visits and mentoring The programme team has promoted opportunities for sponsored visits and, until spring 2014, mentoring opportunities. In all, programme metrics show that 229 visitors participated. There has been significant uptake of sponsored visits as the programme has run, probably due to word of mouth reporting through networks, feedback about visits online and efforts from the BIG Assist team to promote sponsored visits. The BIG Assist team has communicated about visits actively via a bi-monthly newsletter, social media and through case studies demonstrating the benefits of visits, which may have also increased interest. Sponsored visits have been welcomed by IOs and feedback is highly positive. They report the main benefits of visits are: Time out to focus IOs valued ‘time out’ to exchange knowledge in a focused way. Some felt that time for fact- finding and networking visits with a clear purpose can be easier to justify, in a context where resource constraints can make it difficult to take time away from day to day delivery. Value of connecting with others in a similar position Many interviewees reported finding it particularly useful to speak to organisations which worked in a similar context to them (e.g. do they have a membership model in a local area, or provide a particular service nationally). This was more important for many interviewees than visiting a geographically close organisation, or one in a similar stage of development. For example, visitors were keen to have the chance to speak to others who had already initiated a similar charging process for their services. They reported finding it incredibly helpful in informing the design of their own processes. However it was commented on by many that meeting organisations who were approaching things a bit differently to them was one of the more useful aspects of the visits. Value of seeing new ways of working Visitors state that they had benefited by seeing new ways of working in practice. They find it helpful to use alternative approaches being adopted by organisations in a similar position to themselves. “We came away with a very clear picture about elements we need to work on in order to move to charging for development support, including determining what
  38. 38. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 34 of 87 services to charge for, pricing, publicity and managing relationships with donors and service users. We will share our learning with the rest of our staff team and trustees. We will produce a menu of services and publicity about the new approach. We will examine our systems (admin / finance) and update where necessary to support this new approach.” IO, online feedback about a sponsored visit to an organisation, focusing on developing a model for charging for support. Value of developing networks In addition, IOs report how visits have developed their networks of contacts. They have identified potential for future collaborations on projects and some collaboration is emerging at this stage. “This has supported the group to establish a more coherent regional offer, share practice and investigate how we can share resources and a more national offer in the future. We have identified a number of areas for joint delivery [with the host organisation] and the format for a business plan. Additionally we will be facilitating more specific sessions to follow up this work.” IO, online feedback Potential for more IOs to become engaged Many in the round two follow up interviews said that lack of time had held them back from making greater use of the ConnectSpace offer. Some planned to make greater use of it in the future. “One of the things that we haven’t tapped into is the site visits and the stuff on the website. You can only use so much stuff at once. I think it is just time.” Round 2, second interview with IO The positive experience of visiting encouraged one host to engage further in BIG Assist, by applying both for a voucher for support themselves, and as a potential supplier of support to others. ShareSpace: Online discussion forums Programme data show the traffic on ShareSpace, the BIG Assist platform for online discussion forums. ShareSpace: number of topics and posts per forum Forum Topics Posts Innovation One live discussion achieved over 11k views. 24 550
  39. 39. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 35 of 87 People and culture 20 41 Financial sustainability 16 137 Marketing and relationships 3 live discussions with over 7k to over 10k views each 22 504 Strategy and change One live discussion with nearly 5k views Live Q&A with nearly 8k views 26 237 General 47 96 We can see that certain forums, notably Innovation and Marketing and relationships have over 500 posts, so have found an audience and there are active contributions being made. However the greatest levels of engagement came from live on line discussions. Many of these discussions have a very high number of views (with 2 over 10 thousand). Live discussions with the greatest number of views reflect IO interest in developing innovative ways of working and reaching new audiences/ markets, or working in more effective ways with the market they have. This is an extensive level of interest in live discussions. Audiences are clearly able to access the content and topics selected for discussion are resonating with those involved, in their thousands, in many cases. However some caution must be taken interpreting the data, as we cannot know how many views are by single individuals, or by people viewing in groups. Some individuals may be logging in and out repeatedly to discussions, rather than joining in continuously. Far fewer organisations actively post or contribute and it is therefore difficult to infer what impact participation has at this stage. It may be worth exploring the impact of forums more, if the programme is to continue offering this opportunity. Of course, in the early rounds of evaluation interviews, the peer to peer offer and supporting platforms were still in development. We explored interviewee knowledge and use of ShareSpace in more detail in the round two follow up interviews. While awareness of the online forums was high at that point, many interviewees said that they had not yet made much use of them. This finding is difficult to reconcile with the high viewing rates for online discussions. Our interviewees gave reasons for why they hadn’t yet made use of the opportunity. They described how they were too busy or had not yet had a chance to engage with Sharespace [the online forum]. Many interviewees in the round two follow up interviews commented that receiving and implementing support from the BIG Assist programme had increased their workloads, and that once the support was over and they had a bit more free time they would be more likely to engage online. However, a couple of
  40. 40. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 36 of 87 individuals were unsure how long they would still have access to these resources, either because they were no longer actively receiving support from the programme, or because of uncertainty over how long the programme would continue to exist. BIG Assist Library: resources including articles and wikis The BIG Assist Library contains over 600 articles and wikis, including case studies to share experience and learning10 .This extensive Library and its resources were mentioned rarely in interviews. When prompted, there was limited awareness of the offer amongst our interviewees. The BIG Assist team have worked to drive up ‘traffic’ and engagement in the Library by promoting it widely and ensuring content is updated and responds to current issues for infrastructure. From online analytics, the library resources are viewed on average 26 times per month, with the following pages being the most popular: 1. Six challenges for infrastructure organisations 2. Changing role not just for volunteer centres 3. Five ways to stay afloat without more revenue 4. The perils of ignoring infrastructure 5. Small charities big impact Events for shared learning BIG Assist has delivered a range of events as an opportunity for stakeholders to find out more about BIG Assist and share learning about how it can support them. Events were held between October 2012 and March 2015. 15 events took place in year 1, 11 in year 2 and 3 in year 3. The BIG Assist National Summer Conference in 2013 had the greatest attendance with 162 delegates. IOs told us that they value events as an opportunity to network face to face. People also came to hear about innovative practice and practical ideas for taking them forward. A few reported it was easier, or felt more justifiable, for them to take time out for learning at a designated event than for other forums for knowledge exchange. The peer to peer experience: conclusions  Peer to peer offers are a way for very large numbers of IOs to access support that is designed with their needs in mind.  IOs are eager to exchange knowledge with their peers, particularly those in similar circumstances who are implementing new ways of working. Interest and 10 Data extracted from online platform on 22nd May 2015
  41. 41. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 37 of 87 engagement in some offers, e.g. events, have been apparent from the start of the programme.  Sponsored visits are very well received and are showing promising impacts for IOs. Increased knowledge, confidence and partnerships are leading to new opportunities for income generation and/ or improving services offered to the frontline. There is evidence from post-visit reviews and interviews that we can expect other changes to emerge as a result of active participation in visits.  ShareSpace, offering live discussions, is attracting huge volumes of traffic for some outputs. This indicates that a great many IOs are able to access materials and find the topics of interest. This level of views suggests enormous potential for shared learning.  However, we cannot currently determine how active ShareSpace participation is and how IOs are using ShareSpace to build their own capacity.  Despite huge viewing figures, conversations with voucher recipients (who made limited use of on-line peer to peer resources) suggest that there is further room for increased uptake of the on-line peer to peer offer.  At this point, it is difficult to conclude what the impacts of Share Space will be.  At this point, there is little evidence for us to draw conclusions about the BIG Assist Library, in terms of the use experience or possible impacts.  It can be difficult for IOs to make time for reflection and learning in such times of constraint and demand, despite an awareness of the need to do so. Funding time for knowledge exchange appears to be an enabling factor but having a clear focus for how that time will be used, for example through a dedicated event, with practical outcomes, may give IOs the ‘permission’ they need to step back and take a wider view.
  42. 42. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 38 of 87 Impacts on infrastructure organisations Summary of findings around impact  BIG Assist has been a catalyst for necessary change in many organisations. Change projects and the resulting impacts were unlikely to have happened without BIG Assist funded resources and expertise from suppliers, or support would have been of lesser quality, and/ or would have taken IOs longer to get around to.  Impacts are emerging in critical areas for IOs.  The support received through BIG Assist has helped a number of IOs to win new sources of grant funding or contracts to deliver work.  IOs are identifying new ideas for generating income. This includes the development of chargeable services and/ or products.  Improved consortia bidding is an emerging impact of the programme. Using vouchers to set up consortia has allowed IOs to identify and work more effectively with partners and they can now bid effectively as a group.  IOs report that the BIG Assist project has helped to secure staff and Trustee engagement and buy-in for change, due to the external nature of the support.  Some IOs feel that the main impacts of the programme are of a longer term nature and are likely to emerge in the future.  Enablers of change include:  Voucher value (although impact is seen with smaller voucher awards)  Relevant, tailored support  Role of external, independent support  Barriers to change include:  Organisational capacity and timing  Wider context In this section, we describe the key impacts of BIG Assist on IOs. We also use examples from specific organisations to illustrate these impacts.
  43. 43. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 39 of 87 Extra help and added impact A source of funded support in a time of great need Many organisations said that they would not have been able to get support via other means, so the impacts seen through the programme would have not taken place. They lacked capacity internally and could not have found resources to bring in a consultant. Some said that they would have tried to do similar work of their own accord, but this would not have been done as quickly, efficiently, or to the same quality. Very few organisations said that they would have managed to get the same quality of support, or the same outcomes that they described in the absence of the BIG Assist programme. New sources of funding Improved fundraising abilities Organisations described how BIG Assist projects helped them to demonstrate the quality of their work, and hence strengthen their ability to win bids. Projects which improved core internal processes, such as financial management and staff development, also helped customers to secure funding, More awareness of funding options and new sources of funding Case study: new sources of funding One organisation has accessed multiple sources of new funding following support from the BIG Assist programme. Their first vouchers award focused on income generation and innovative ways of working encouraged them to apply for workplace development funding from the Department for Business, Innovation and Skills. This has provided the IO with steady funding for the next three years. “The biggest impact has been the new funding stream. Without this I think we would be having a very different conversation if we didn’t have this contract. We would be a different second tier organisation by this point.” This contract has had a significant impact on the organisation as a whole, allowing them to develop expertise in education and training:  Growth in the number of staff members from 1 to 17.  New organisational structure including an operations manager.  Two new dedicated spaces in London and the North.  Accredited existing training.
  44. 44. OPM Evaluation of the Assist Investment in National Infrastructure Programme Page 40 of 87  Developed 2 new training qualifications. The original vouchers also encouraged the IO to look at a donation model for increasing funding; they have since raised £60,000 in donations. The IO has since reapplied and received more vouchers from the BIG Assist looking at sustainability in response to the IO having to frontload costs to deliver the programme. These new vouchers have supported the IO to manage their growth ‘mindfully’, without over extending beyond their means: “we are not expanding or creating an operational cost we can’t sustain”. The IO felt the BIG Assist programme had met their needs “100%”. “It’s enabled us to develop and grow, and I think in this current climate that is quite unusual.” Improved consortia bidding Some organisations also used the vouchers to support the organisation to set up consortia or other, more informal, arrangements to help multiple smaller providers to work together to deliver work. This has allowed the voluntary and community sector to compete more effectively with private organisations and larger national providers. Individuals who we interviewed were often positive that the new arrangements would help the organisations involved to win funding. One organisation described how they had used the support to help them to improve their processes of working with other organisations, which allowed them to bid more effectively as a consortium. They said that a direct impact from this was that they had been able to win much larger grants and contacts from local funders: “It helped to secure £1m from the local authority over two years, and an additional £42k a year from the CCG. Helped join up our work and bring us more closely together. The council have always told us there would be one contract- the BIG Assist helped us to get there. It has given us the database to do the work together and do the monitoring and provide the evidence that funders need.” Round 2 second interview with IO Case study: Training for front line organisations One London based IO used BIG Assist to develop training in partnership with a supplier organisation. This was designed to support front line organisations to work with the health and social care sector. The training sessions revolved around:  Procurement and commissioning skills.  Managing a voluntary consortium. The training and events have helped to bring voluntary organisations together into consortia, allowing them to bid for £700,000 new funding from the local CCG. In addition to

×