Going the Extra Mile in Monitoring and Evaluation – the GEM Story
•Download as PPTX, PDF•
0 likes•223 views
Presentation from CCRI seminar December 3rd 2020 detailing the 'GEM' project.
Recommended
Recommended
More Related Content
Similar to Going the Extra Mile in Monitoring and Evaluation – the GEM Story
Similar to Going the Extra Mile in Monitoring and Evaluation – the GEM Story (20)
More from Countryside and Community Research Institute
More from Countryside and Community Research Institute (20)
Recently uploaded
Recently uploaded (20)
Going the Extra Mile in Monitoring and Evaluation – the GEM Story
- 1. Going the Extra Mile in Monitoring and Evaluation- the GEM story Leonie Burton and Paul Courtney University of Gloucestershire
- 2. Plan for the session An introduction to GEM Evaluation and monitoring within the GEM project and the team The methods of the GEM evaluation-and why chosen Process evaluation findings Outcomes evaluation findings Considerations for GEM in final months Next Steps for the Evaluation Any questions
- 3. Introduction The Going the Extra Mile (GEM) Project is a social inclusion project committed to helping people overcome challenges to employment and move them closer towards or into work, including self-employment. The GEM Project is jointly funded by the National Lottery Community Fund and the European Social Fund Individuals Partners Employers
- 4. Going The Extra Mile (GEM)
- 6. The M&E Research Team Paul Courtney: pcourtney@glos.ac.uk Colin Baker: cmbaker@glos.ac.uk Leonie Burton: lburton@glos.ac.uk Wendy Russell wrussell@glos.ac.uk Isabel Fielden ifielden@glos.ac.uk Ab Gardiner agardner@glos.ac.uk Fahimeh Malekinezhad fmalekinezhad@glos.ac.uk
- 7. The GEM Evaluation model
- 8. Formative (Process) Evaluation Focuses on processes linked to planning, management and delivery of the project, and extent to which planned activities are carried out Ultimate aim is to ascertain whether any changes are needed to the improve the project design and implementation
- 9. We wanted to find out… Whether any changes are needed to the improve the project How people’s lives have changed as a result of GEM
- 10. We asked… What difference did GEM make? What worked well? What were the challenges? Is the project on track?
- 11. We spoke to… Participants Navigator developers Delivery partners GEM staff team Our steering group Executive management team Opportunity Hunters Business sector leads
- 12. We used a variety of methods… Project information Interviews with the range of stakeholders Focus group meetings Evaluations of meetings Peer evaluation Creative and inclusive methods
- 14. Process evaluation findings: 2017-2020 Overall, every year feedback from everyone has been very positive and the project itself is overwhelmingly highly valued. “ I feel like I can lean back and I’m not going to fall’ (participant)
- 16. Green Impact of GEM Green Impact is an internationally recognised benchmark environmental assessment tool run by SOS-UK (Students Organising for Sustainability UK). As part of the GEM Service Agreement, the 25 GEM partner organisations committed to achieving Silver level accreditation in the Green Impact scheme. This provides a way for GEM to report back to its stakeholders how it has met its sustainability and environmental targets and is about helping organisations to embed sustainability into their everyday workplace practices
- 17. 2020 process evaluation findings Innovative response The focus that the N/Ds have given towards supporting the emotional wellbeing of their participants has been vital Relevant support to participants in need Commitment across the board Support for health and wellbeing Excellent support from GEM team and improvements in team operations during the pandemic
- 18. The GEM Evaluation model
- 19. Summative (Outcomes) Evaluation Examines the Outcomes and Impact of a project with the aim of determining overall effectiveness How has people’s lives changed as a result of GEM? Outcomes, not just Outputs Effectiveness determined quantitatively and qualitatively, through metrics of change and using an SROI Social outcomes distance travelled survey (1000+ Reponses) Inclusive evaluation approaches including digital storytelling, peer researcher and other creative approaches
- 20. GEM Theory of Change World Café event December 2016 Mixed methods including Inclusive Evaluation approaches integral to the M&E framework from the start
- 21. Social Return on Investment (SROI) Government recognised framework to exploring, measuring and valuing change Stakeholder driven and is focused on outcomes SROI = Value of benefits i.e. 1:3 Investment Triple bottom line – economic, social and environmental How much can outcome change be attributed to GEM? What would have happened anyway?
- 22. Measuring change in the outcomes – distance travelled SECTION A: To what extent do you agree with the following statements? Please place a tick under the picture which best describes what your life is like. STATEMENTS Strongly Disagree Disagree Not Sure Agree Strongly Agree 1. I find it easy to obtain help and advice about employment or training. 2. I often get a chance to learn new things.
- 23. GEM ToC Outcomes and indicators Outcome Survey Question (Statement) Change Score N=704 Improved sense of competence, engagement and purpose I often get a chance to learn new things I feel that I am making a difference to my life +32 Reduced barriers to receiving help and advice I find it easy to obtain help and advice about employment or training. +32 Increased levels of interest and take up in training, volunteering and vocational opportunities I regularly volunteer or get involved in activities to help others. +21 Improved confidence and motivation to engage with and access new opportunities I will try something even if there is a chance it might not work. +12 Improved resilience and self esteem In general I feel very positive about myself. +25 Improved financial situation of target participants I have a wide range of skills to help me at work, in training or in my daily life. I feel that I can manage my money and finances. +16 Improved employability through job- ready knowledge, skills and attributes I feel ready to take up a new opportunity - for example, training / work / volunteering. +14 Improved physical and mental health I feel generally healthy and have good energy levels. +17 Improved positive functioning and coping strategies I feel generally healthy and have good energy levels. +15 Reduced social isolation I often meet socially with friends, relatives or colleagues. +13 Improved financial literacy and ability to manage finances I feel that I can manage my money and finances. +12 Improvement in life and work skills I have a wide range of skills to help me at work, in training or in my daily life. +19
- 24. Societal Return of GEM The Social Return on Investment (SROI) model forecasts that GEM is set to return £2.50 for every £1 invested in the programme. Evaluative model to be produced in Autumn 2021.
- 26. Inclusive evaluation – Peer Research findings
- 27. Inclusive evaluation - Digital Stories Digital storytelling enables those with challenging histories around employment, addiction and mental health to be listened to. The stories explain how central GEM has been to their lives and the digital storytelling method is a tool for both gauging impact and for individuals’ empowerment.
- 28. Inclusive Participant Stories – Digital Storytelling
- 29. Inclusive Participant Stories – Digital Storytelling
- 30. Partnership assessment - logic model There was widespread support for the principle of partnership working and its potential to address complex social challenges through the intentional bringing together of multiple stakeholders. Resources and advice were a key benefit of the GEM partnership including staff expertise, knowledge and mutual support.
- 32. Future considerations for GEM Economic challenges ahead and the impact on job seekers Impact of recession on social and emotional needs of communities in the county Continued backdrop of uncertainty until the vaccine is delivered Looming end of project – how to sustain progress Never has a project such as this been more needed Informing the design of a successor to GEM Nationally – GEM has proven employment inclusivity is preferable to one size fits all.
- 33. Next steps for the GEM evaluation Evaluation cycle to continue with focus on learning lessons for a post GEM environment Remain flexible and agile in response to current circumstances as they evolve SROI and cost saving to the state Building a detailed Partnership assessment Formal analysis of mental health using WEMWBS Inclusive evaluation methods to be re-introduced post pandemic.
Editor's Notes
- The GEM partnership was was formed in August 2016. This sort of partnership is unique as together the partners have developed a vision for Gloucestershire and through the GEM project been able to realise the employment and learning ambitions of hundreds of vulnerable people across the county. Today we will be focusing specifically on the work of the monitoring and evaluation team to share some details of how we have worked with the project since 2016 and to tell you about some of the findings that have been revealed trough this ambitious partnership project.
- The stakeholders Individuals – GEM refers to the people who access services as their participants and aims to reconnect people/ GEM participants with their local services through their communities and ultimately help them achieve their desired outcome, whether through a working or educational related route. The GEM Project understands that everyone is unique, with different needs and requirements, impacting on their step into the world of employment, education or training. The target participants benefitting from the GEM Project will be: hard to reach individuals with multiple and complex needs furthest from the labour market. With multiple barriers to work eg mental health, addiction, homelessness, history of offending, those with caring responsibilities, physical or learning disabilities, those on low incomes, young people (including those with low financial literacy), older working age people and BAME communities across the county. Across the entire Gloucestershire LEP area but will focus particularly on those living in areas of disadvantage including: Areas where unemployment is higher than the England average: : Areas where individuals experience socio-economic vulnerability, including: Partners- the project has 46 local delivery partners in Gloucestershire all of whom collaborate and who have agreed to host a post paid for by GEM called a Navigator Developer. These NAV Devs provide a unique one to one service to the people who access GEM, acting as Guides, coaches, mentors and support. As one participant told me this year, ‘I feel I can lean back and I’m not going to fall.’ The GEM Project will make full use of all partners' resources to ensure that the project has a positive impact on the environment, economy and people’s living conditions. More about this in regards Green Impact. The project also works with a wider community across Gloucestershire that also contributes to the success of the work. Employers. The Central GEM team have 2 Opportunity Hunters who make strong connections with employers in the county to bring to the table a host of opportunities for GEM participants to access.
- Gloucestershire County Council (GCC) is the lead organisation for the GEM Project and the Gloucestershire Gateway Trust manages the day to day running of the project through the GEM Management team. UoG has been in the unique and privileged position of co-writing the bid for GEM in 2016 and has been involved from the outset. GEM Project planning began with the launch of the Programme Development Fund which brought together over 150 voluntary sector organisations and public sector bodies to discuss options for responding to the call. A GEM Development Team was formed including UoG to design a proposal which identified the linkages and gaps in activity across the county. The benefits this has provided for the project has been the joining of a very a strong model/ proposal, informed by the grass roots experiences of those organisations with day to day contact with target participants with the establishment of a theoretical framework to wrap around the project and measure project outcomes successes and indicators for change. Indicators of change for each of the outcomes were devised by the external evaluation team, drawing on their experience of capturing and measuring social value Within this, approaches used by the ONS, European Social Survey and the New Economic Foundation for capturing employability-related change was also consulted. The indicators were refined and tested through the development of appropriate bespoke distance-travelled survey questions and tools, distinguished by the four impact pathways and delivered via the appropriate gateways to impact. Stakeholders and participants complete distance-travelled surveys periodically, using a mix of on-line and paper based tools with appropriate professional support. Data gathered through these bespoke tools have been used to populate an SROI model to estimate and value social, economic and environmental impact, incorporating the use of appropriate regional and national secondary data (such as, for example, the ONS Labour Force Survey and Personal Well-being questions in the Annual Population Survey) to estimate additionality.
- Explain the GEM journey. At the heart of the GEM project is the participant journey from referall to being allocated their own Navigator Developer to work with. The ND helps to open up many opportunities to take forward ambitions that the participant has and to help resolve barriers that may be holding them back. This can be very varied including resolving financial, health and housing issues. Not surprisingly, the relationship between an N/D and participant is based on high levels of trust. N/Ds work hard to establish this through regular contact and being a consistently reliable person in the participant’s life. This year, all of the participants I interviewed felt their N/D had provided an amazing service. One participant described meeting their ND as a very big deal as previously she had been so isolated and the perception was the Nav Dev gave emotional and psychological support; ‘The project is very good at scaffolding. I’ve been very impressed with the contact over the lockdown. It makes it easier to know that there is someone there and I’m not judged’ (participant). The results of this perceived supportive one to one relationship is that it can lead to many positive outcomes which we will come on to later.
- Introduction to the team…
- Formative Evaluation = Process (led by LB) Summative Evaluation = Outcomes (Led by PC) But in the series of evlaition cycles and feedback processes, process and outcomes have been considered together
- Explain missed methods and why…..
- The following information is an overview and full details will be contained in the evaluation reports.
- The intensive and tailored one-to-one support one-to-one support that GEM affords is highly valued by delivery partners, navigator/developers and participants alike. There was also recognition and value attributed to the difference in the way the project operates and the feeling of being connected and treated in a very kind humane way which for people battling mental health issues is an essential. Most participants spoke about the confidence they had gained from being part of GEM. ‘Volunteering and education opportunities break the cycle-otherwise I’m dealing with it in my own head and in my own four walls and having someone to talk to deals with panic attacks and is essential. It is a long process to get well and to get work and I have to be pushed as well. GEM is good as there are no deadlines and it’s about connection versus isolation.’ (participant) Participants have reported -increased confidence and sense of purpose -feeling more positive about themselves reduced barriers to obtaining help and advice Organisational benefits, they are beginning to see how beneficial the GEM work is in how they engage with their target groups and how their Organisational practices can learn from this and in addition they are seeing benefits in how their Organisation reaches out to the local community. Extra Mile funding Participants Council- greatly appreciated and very positive Gem was useful meeting the team and doing activities. Particularly joining the Participants Council, every 2 months I take part in meetings. We go to different venues in Gloucestershire. I’ve been on for 21 Months- at first I wasn’t massively confident but as time has gone on I’ve become more confident speaking my ideas.’ Every year with the report we build in time for reflection on the finding with GEM Management to see how the ideas fed back can positively change and tweak how the project is run. Most of the time the report will re-enforce ideas that are already in the pipeline eg last year we were talking about the cap on how much could be applied for with Extra Mile fund for example and whether flex could be built in to this- which was already under consideration. The intention is to support the project to operate the best that it can and to not stay static.
- This year’s entries demonstrated a high level of environmental awareness and commitment to the programme. The results were commendable and exceeded the original target - with 9 partners achieving Gold, 13 Silver, 1 Bronze and 1 working towards Bronze. The Awards Ceremony took place online via Zoom in July 2020.
- The project faced a huge challenge in early 2020 due to the Covid-19 global pandemic. However, this has only served to strengthen our partnership further as our staff and partners adapted to the enforced changes of an unprecedented national lockdown. Each person played their part in making sure that GEM continued to support individuals, keep them well and help prevent social isolation The effect of the UK lockdown created the need for GEM project to be transferred to an online setting. The GEM project responded quickly initially to bring all the tailored one to one support online conducted via zoom and telephone calls. The GEM office set up a programme of online support through a weekly programme of support published on their website. In addition laptops were delivered to participants in need to enable them to access the online services and not be disadvantaged. GEM online- a comprehensive programme of employment support and wellbeing activities, art, yoga and much more. It is clear that the programme has responded very appropriately to the crisis thrown up by the Covid pandemic. During lockdown the programme worked quickly and innovatively, grasping the gravity of the situation and putting in measures to ensure that the GEM programme support and focus would be relevant to participants needs at the time. Since emerging from lockdown, many aspects of the programme are working well and have resumed support for participants in job seeking, training and volunteering. Across the board, there has been a great deal of commitment to the GEM project. There has been some issues of motivation and engagement caused by the pause and stillness created by lockdown. However, the flexibility to respond to the needs of GEM participants and the willingness of GEM staff to work collaboratively to support participants in their journeys has been unwavering. This aspect of the programme seems all the more important given the challenges in the wider external context. Of particular note this year has been the enormous impact GEM has had in terms of supporting the health and wellbeing of participants. N/Ds have reported that during lockdown they have dealt with a lot more participants struggling with anxiety and more severe mental health episodes. This has been reinforced by feedback from the participants interviewed. This has demonstrated the significant impact on the lives of participants that the GEM project has had in 2020. GEM online has been a very positive development, providing a lifeline to many. This has been coupled with the provision of IT equipment where it was needed. This action has significantly tackled a barrier that faced many who were disadvantaged as they could not access online support in its many guises; zoom calls with N/Ds, GEM online classes and online training opportunities. This strand of development that has been new in 2020 has been overwhelmingly welcomed by both NDs and participants who would wish this to continue going forwards. Some external issues were recognised as problematic, including the new normal of social distancing. Many NDs found that it took longer to onboard participants to the programme working at a distance. Whilst the technology in some instances could help, it was more difficult to build rapport, trust and ensure even meetings were attended online. In addition during lockdown, for some participants, the lure of other pursuits such as online gaming or other pulls on their time, caused them to withdraw. Some barriers have been quite testing for example, where participants have lacked confidence to use Zoom and access the online services. N/Ds have had to think creatively but have still not seen much progress for example, working with young people whereby lockdown has made it difficult to help them progress and there has been little interest in pursuing job. As inevitably the UK faces a difficult winter as the second wave of Covid cases approaches, this is a situation that NDs may face again and is worth recognising in preparing for the months ahead. Closer to home, frustrations with internal processes were apparent including paperwork, still being a very time consuming process. However, these frustrations have largely been outweighed by the overwhelming positive feedback internally on the project including excellent support from the GEM office, the innovations that have been supported across the teams, the huge improvements in staff communications including the 10@10 weekly calls and use of social media apps for N/Ds to keep in touch and forge bonds. Considering the broader economic and therefore employment context, the issue concerning supporting participants not only in the transition to employment but during their employment seems critical. Keeping both participants safe whilst on the GEM programme and getting them in to employment with employers with safe working practices is something that N/Ds report as being important and challenging. The Participants’ Council provides participants with a voice and the ability to influence the programme and should be recognised as an important part of the programme. This has been on hold this year and consideration to how this might be supported going forwards would be worthwhile. In terms of legacy, the network of N/Ds and delivery partner organisations has really bonded in lockdown through the sharing of information, expertise and resources that can hopefully continue and endure beyond the life of the programme
- 18. Reminder Formative Evaluation = Process (led by LB) Summative Evaluation = Outcomes (Led by PC) But in the series of evaluation cycles and feedback processes, process and outcomes have been considered together
- Very interested in stories as well as metrics and models, which brings things to life.
- Using Nefs journey of change model, which I’ve used a lot over the years., Map out what they thought the outcomes could be for participants and to consider how one outcome leads to another, in a chain of events. Also considered methods. Felt strongly that we would need to reach out to the extremities of what is essentially a harder to reach group of individuals.
- From the start decided to base the M&E framework around an SROI approach or framework. Because - stakeholder driven – fits the inclusive and co-produced nature of GEM, and is focused on outcomes – clear at the bidding stage that a focus on outcomes was a perquisite for a successful employment inclusivity programme.
- To give you an idea about how we have measured change - or distance travelled - in the outcomes. When a participant is inducted onto the programme they complete a social outcomes survey, and then they complete it again when exiting, and sometimes also at a mid-point in their GEM journey. Principal way that we are collected data to inform the SROI model and has been through various iterations to improve it over the last 4 years. Very careful about language and accessibility, and in fact survey greatly improved by Fahimeh since she joined the M&E team because she was actually a GEM participant herself, so we can say that in a very real and useful way this has been an exercise in co-production.
- Self-reported change in outcomes identified in the ToC. Top two have had consistently high DT scores, which validates GEM as a programme, but the consistency also validates the survey and the methods themselves. changing the lives of its participants: the competence, engagement and purpose of participants through offering them a chance to learn new things the removal of barriers to receiving help and advice, with participants self-reporting a mean change of 32% in both outcomes SROI increased to £2.50 for every £1 spent participants have experienced a 25% improvement in resilience self-esteem and a 17% improvement in physical and mental health as a result of their involvement in GEM.
- SROI is a forecast or prospective model that is updated every year. Survey results have been used to populate a forecast Social Return on Investment (SROI) model during every evaluation cycle since 2017, demonstrating that GEM has generated a range of psycho-social benefits for participants to the value of between £1.57 and £2.39 for every £1 invested since the programme began. Results from the 2020 distance travelled analysis have again been used to update the forecast SROI model, which demonstrates a slight improvement in the prospective societal return of GEM, with the forecast Benefit-to-Investment Ratio (BIR) rising to 1: 2.50.
- Headlines from the outcomes evaluation: GEM shown to be Agile and Resilient in meeting the needs of its participants during the Covid-19 pandemic Consistent success in improving the competence, engagement and purpose and other motivational attributes of participants, and in reducing barriers to help and advice Highlighted differences in the trajectory of outcome change with respect to characteristics such as gender, disability, ethnicity and the length of time that participants have been engaged on the programme. For example, we have have found females to have self reported great levels of change across the majority of psycho-social outcomes compared to males, and that the greatest level of impact is associated with participants which have been on GEM between 3 and 6 months. And this has all fed back into the management and delivery of GEM as part of the annual evaluation cycle.
- Inclusive evaluation, has had three main elements – -Some initial pilots to test out things like scrapbooks, photography and informal and outdoor approaches to data collection. To do that we teamed up with one of GEMs partners Anna Bonallack from Creative Sustainability, a CIC in Stroud which will be working with us again next year in these aspects. -A peer researcher approach where participants have interviewed each other, which has been led by Fahimeh previously mentioned. -And a digital storytelling method led by UoG’s Abigail Gardner which has involved 3 participants to date and more will be getting involved in the final year of the evaluation. So the peer researcher approach involved 12 indepth 9ntevriews which were transcribed and analysed in Nvivo. And these were the three main themes that emerged from that: We have a 20 page report on this element itself, and submitting this to a journal in the next few weeks.
- The way that DS works is that participants bring along images that are personal to them – and in this case pertinent to their GEM journey – along to a series of workshops where they develop and record a script of their story with the images forming the backdrop to the story. This is a very powerful way of getting across participant experiences, but is also empowering in itself as throughout the process of the DS the power literally goes from the facilitator to the participant – which is actually great for their self-esteem. Four half day Digital Storytelling workshops formed the foundation of the pilot phase. The first introduced the concept of digital storytelling, the following two focused on story circles and scripting and the fourth was a technical workshop. The script was developed by the facilitator and clients outside of the formal workshop and the second session comprised the recording. When GEM participants made digital stories, they were being listened to - sometimes for the first time. The impact of this on mental health, self-esteem and development cannot be overstated. The digital storytelling process facilitates the emergence of powerful experiential narratives of being involved in GEM. Co-production is key to both GEM and digital storytelling - it opens up a productive space where experience and knowledge can be exchanged. Key to the process is how the balance of power shifts from the facilitator (at the start) to the participant (towards the end).
- One participant did a stained glass window course through GEM, and this is his work. His story of journeying from reluctance and reservation, to leadership, achievement and improved confidence illustrates the power of digital storytelling in spotlighting individual impact.
- Similarly another revealed how GEM had reconfigured his life goals and expectations, specifically in relation to family, drugs and crime. Obviously you need to listen to the stories themselves to get a sense of the impacts, and how powerful they are
- A more recent addition to the GEM methodological party is a partnership assessment, being led by Colin Baker of the School of Sport and Exercise. And of course with 30+ partners in GEM it is important for us to capture the range of impacts that have happened as a result of such a large and vibrant partnership. So this diagram shows the themes that emerged from the initial online workshops to explain the experiences and perception of the partnership and help develop the logic model that now sits alongside our ToC. This element of the evaluation will be developed further next year and be an important feature of the final evaluation report in Autumn 2021.
- So in the same way here are the headlines form the inclusive evalaiton work presented in Inforgraphic form, with the word cloud there ilustraing some of the lngauge that has been used hwne participnats have interviewed each other.
- So as we look forward from the 2020 evaluation cycle, our attention has necessarily turned more towards what is going to happen post GEM when the funding ends, so an important role for the evaluation findings here is in helping to inform the design of a successor to GEM, and also to put forward recommendations on how the GEM experience can inform and influence what happens nationally. One thing GEM has proved is an inclusive and tailored approach is considerably more preferable to the one size fits all approach of DWP that is primarily focused on getting people off benefits and into employment. And of cours as evaluators we very much hope that the both the impacts of GEM and the experience of the programme will have a legacy, for Gloucestershire and beyond.
- In regards what we’ve learned in terms of the evaluation and what next.. Alongside the GEM programme itself the evaluation will seek to remain flexible and agile in responding to any specific events in what will hopefully prove to be the autumn of the Covid-19 pandemic. -The evaluation cycle will be undertaken as in previous years, providing an internal lens on management and practices as the programme is implemented and delivered. However it will also look to focus on the lessons learned for the implementation of GEM principles and practices in a post GEM environment. -The forecast Social Return on Investment (SROI), which has been updated annually throughout the GEM programme, will comprise an evaluative model towards the end of the programme in 2021. The research team will also seek to extend and deepen the metrics of social value with an indication of cost-savings to the state resulting from programme engagement. -A detailed partnership assessment will be pursued building on the initial scoping and logic model reported here, and emerging themes including training and development and planning for sustainability. This work will also have relevance for partnership working beyond GEM, and for the design and implementation of future partnership based programmes – in Gloucestershire and beyond. -Subject to the collection of sufficient participant data, a formal analysis of the mental health impacts of GEM will be undertaken, which will again have potential implications for future employment inclusivity, social inclusion and community health programmes. -Following delays caused by the Covid-19 situation, inclusive elements of the evaluation will reach new heights in 2021, incorporating methodological lessons learned from previous inclusive evaluation pilots, and from the inclusive participant stories reported here.