Evaluation Update
•Download as PPT, PDF•
1 like•428 views
The evaluation update summarizes the progress of a large-scale evaluation from 2014 to 2021 that is assessing the change in outcomes for older people in the UK and aims to identify programs that work and why. The evaluation is being led by Ecorys UK with partners and includes a process evaluation, impact evaluation, and economic evaluation along with communications and learning activities. This year's focus areas include developing a common measurement framework, an impact measurement approach, planning ethical approval, and initial sharing learning activities. Progress so far includes piloting the common measurement framework and an impact survey with further qualitative work planned for autumn 2015.
More Related Content
What's hot
What's hot (18)
Similar to Evaluation Update
Similar to Evaluation Update (20)
Evaluation Update
- 2. • Evaluation team led by Ecorys UK with Brunel University and Bryson Purdon Social Research • Large scale evaluation from 2014 until 2021 • Aiming to assess the change in outcomes for older people • To identify and provide evidence about what works and why? Evaluation update
- 3. • Process evaluation • Impact evaluation • Economic evaluation • Coupled with… • Supporting communications and sharing learning activities Coverage of evaluation
- 5. This year, a focus on key areas: •Development of Common Measurement Framework •Development of impact measurement approach •Process for ethical approval •Planning initial sharing learning activities Progress so far
- 6. • CMF piloting and go live – July/August • Pilot impact survey – August • Main stage impact survey – Sept-October • Planning mini impact studies, CBA and qualitative work – Autumn 2015 Steps for the rest of the year
- 7. • Ageing.better@ecorys.com • Move towards account manager relationships • Big Lottery Fund’s online community • Social media #ageingbetterlearn #ageingbetter Keep in touch
Editor's Notes
- Team: The team for the evaluation is led by Ecorys, working with Christina Victor, Professor of Gerontology and Public Health at the Brunel Institute for Ageing Studies from Brunel University and with Susan Purdon and Caroline Bryson of Bryson Purdon Social Research. Each partner will lead on specific aspects. Timing: It’s a large scale evaluation – started in Jan 2014 and ending in 2021 after projects have ended. Aims: The evaluation aims to assess the scale of change in outcomes for older people and communities that is different to what would have happened without the fund’s investment. The evaluation needs to provide evidence and continuous learning to help in identifying what works amongst the approaches being taken forward, and contribute to broader aims around disseminating learning and influencing public policy. As evaluators are aim is to work with partnerships to maximise the opportunity that this investment provides to strengthen the evidence base for interventions to address social isolation. Next slide
- We have designed a quantitative and qualitative evaluation that runs alongside the funded projects for up to 12 months after the last project finishes to ensure effective dissemination of the results. Process evaluation will gather evidence about “ how the investment was delivered” & how the process including systems change is working from the perspective of partnerships, delivery organisations and users. Building up a good overall picture about what the investment is being used for – what the local arrangements are and what projects/services and being supported. How the fund is integrating with wider activity and harnessing funding and in-kind support. How partnership is building in flexibility to adapt and evolve over time. Evaluation will build up evidence on models of delivery and good practice focussing on: Older people being more engaged in the design and delivery of services that help reduce their isolation and how this has benefited them (e.g. improve confidence) as well as services. Better planning, co-ordination and delivery of services designed to reduce isolation amongst older people. Impact evaluation will aim to measure “what difference of all changes have made for older people” – what are the characteristics of older people being supported, what are the risk factors and in the long run are older people less isolated as a result of the investment? This involves measuring outcomes for older people but also for older people that have not been affected by Ageing Better investment so that we can measure statistically the difference that the investment has made. We are also tasked with carrying out an economic evaluation. This involves setting the costs of the investment alongside the reductions in isolation (e.g. including costs saved in emergency care). This work is pioneering and we are keen to hear more about local approaches. The assessment of economic benefits and any fiscal savings as well as potential for replicability, scaling up and sustainability beyond this 8 year programme will be important for future case making. An older people’s advisory group will be set up during autumn 2014 and will inform and co-design aspects of our work. We would like to recruit older people through your networks so their inputs are fully integrated with wider work. Specifically, older people will be asked to comment on research tools for example. The evaluation is not just about research. Together we will build up the evidence base and this needs to be made available to influence the services that help reduce isolation for older people in the future. So, our work programme includes lots of sharing learning activity such as action learning and communications activities. We’d like to hear more about your ambitions for learning and communications and how we can support you with this. Next slide
- This diagram gives an overview of all the tasks that make up the evaluation. Basically it involves: A research and design stage, which is taking place from now until when the national evaluation design is finalised. Our timetable for finalising the national evaluation design is a step behind the timetable for you to refine your planned outcomes and impact because naturally we’ll be designing an evaluation that matches what you tell us about what change you are planning for. Then two main workpackages, one for qualitative and one for quantitative research. A work package for project or area level support which the Ecorys team have already started giving to projects and which will continue throughout the programme. Then work packages for reporting and project management. There will be annual programme level and project evaluation reports starting in 2015. Next slide
- We will design a robust impact assessment which meets BIG’s objectives for a high standard of evidence that will stand up to external scrutiny. In those Ageing Better areas where there is an expectation that outcomes for many older people in the local area will be improved over the lifetime of the investment, the national evaluation will measure this through a series of local surveys of the population eligible for Ageing Better interventions. Large-scale longitudinal survey of older people; Around 5,250 older people at baseline Interviewed at baseline; after 2 years; after 4 years (3,000 at year 4) The survey will enable them to track changes among a cohort of older people and map their transitions into or out of social isolation. To assess the counterfactual, comparisons will be made with a matched group of older people from existing national data sets e.g. English Longitudinal Survey of Ageing; Understanding Society; National Travel Survey to generate the counterfactual For areas to be eligible for this part of the impact study the prevalence of social isolation and loneliness in our survey cohort must be reduced in each area by at least four or five percentage points. (Without this level of change we will not have statistically significant results.) The implication is that the Ageing Better programme must be planning to reach, and measurably affect, at least 1 in 20 older people in the area. So, in reality it must be planning to reach even more than 1 in 20 older people through either directly supported activities or more probably through changing the way that services are designed and delivered. We will also carry out project impact studies focusing on: Projects where involvement by the older people is relatively intense, so that there is a realistic expectation of our finding an impact on older people’s outcomes with a fairly small sample size; Projects that look to be working well, but where there is a need to provide evidence of that; Projects that are relatively expensive, and there is a need to provide evidence that the cost is justified; Projects that are either being run across several areas, or where there is interest from multiple areas on whether they work. The design of these studies will be established on a project-by-project basis. We will identify potential projects in consultation with local areas, but would be very happy for areas to put forward ideas. We will take decisions about the approach and the balance between area-based and project level work after examining your plans and discussing these issues with you and the fund. Next slide
- We will design a robust impact assessment which meets BIG’s objectives for a high standard of evidence that will stand up to external scrutiny. In those Ageing Better areas where there is an expectation that outcomes for many older people in the local area will be improved over the lifetime of the investment, the national evaluation will measure this through a series of local surveys of the population eligible for Ageing Better interventions. Large-scale longitudinal survey of older people; Around 5,250 older people at baseline Interviewed at baseline; after 2 years; after 4 years (3,000 at year 4) The survey will enable them to track changes among a cohort of older people and map their transitions into or out of social isolation. To assess the counterfactual, comparisons will be made with a matched group of older people from existing national data sets e.g. English Longitudinal Survey of Ageing; Understanding Society; National Travel Survey to generate the counterfactual For areas to be eligible for this part of the impact study the prevalence of social isolation and loneliness in our survey cohort must be reduced in each area by at least four or five percentage points. (Without this level of change we will not have statistically significant results.) The implication is that the Ageing Better programme must be planning to reach, and measurably affect, at least 1 in 20 older people in the area. So, in reality it must be planning to reach even more than 1 in 20 older people through either directly supported activities or more probably through changing the way that services are designed and delivered. We will also carry out project impact studies focusing on: Projects where involvement by the older people is relatively intense, so that there is a realistic expectation of our finding an impact on older people’s outcomes with a fairly small sample size; Projects that look to be working well, but where there is a need to provide evidence of that; Projects that are relatively expensive, and there is a need to provide evidence that the cost is justified; Projects that are either being run across several areas, or where there is interest from multiple areas on whether they work. The design of these studies will be established on a project-by-project basis. We will identify potential projects in consultation with local areas, but would be very happy for areas to put forward ideas. We will take decisions about the approach and the balance between area-based and project level work after examining your plans and discussing these issues with you and the fund. Next slide