Successfully reported this slideshow.
450                                                                    400                                   Average house...
Contents   • Overview of the meta-evaluation       – Objectives       – Phases       – Methods   • Planning and prioritisa...
Study objectives   Comprehensive and systematic meta-evaluation of the Impacts and Legacy   of the London 2012 Olympic Gam...
Overview of the 3 phases   Phase 1: May 2010 – April 2011        Phase 2 : Feb 2011 – March 2012        Phase 3: April 201...
Contents   • Overview of meta-evaluation   • Setting up and planning the meta-evaluation       – Project scope       – Log...
Defining scope   • Legacy initiatives:      – New public programmes and investments developed as a result of        Games ...
Logic chainsRationale and Objectives                                                                     Outcomes/impacts ...
Research questions   Designed to tease out what the impacts have been in legacy areasparticipation in sport and physical  ...
This is a meta-evaluation • Requires robust programme evaluations, that meet needs of Meta-Evaluation,   so… 1. Identify g...
Contents   • Overview of meta-evaluation   • Setting up and planning the meta-evaluation   • The evaluation approach      ...
Framework for assessing impact   • Builds upon existing evaluation frameworks:       –   Treasury Green Book input – outpu...
Sport: Logic chain   Rationale and Objectives                                                                   Outcomes/i...
NATIONAL EVALUATIONSData sources:     National evaluations with a regional dimension                                 eg. C...
Examples   • Sport and social themes:       Amendments to national culture and sport surveys in        England, facilitat...
Valuing benefits                                                                           2012 Games   Techniques include...
Thematic challenges   • Sport      – Measuring the sustainability of participation (or use of proxies)      – Methodologie...
Broader challenges  • Questions have been prioritised, but scope still vast and complex  • Internal organisation and syste...
Contents   • Overview of meta-evaluation   • Planning the meta-evaluation   • The evaluation approach   • Key areas for fu...
Areas for further research:Theme                  Primary Data RequirementsEast London            • Household survey and f...
How you can get involved   • Share knowledge on existing and planned 2012 evaluations and     research   • Share findings ...
Questions…             22
Meta evaluation presentation 010611secondpresentation
Upcoming SlideShare
Loading in …5
×

Meta evaluation presentation 010611secondpresentation

427 views

Published on

Published in: Education, Spiritual, Technology
  • Be the first to comment

  • Be the first to like this

Meta evaluation presentation 010611secondpresentation

  1. 1. 450 400 Average house prices (£ 000s) England 350 Five Host Boroughs 300 Greenwich 250 Hackney 200 London Newham 150 Tower Hamlets 100 Waltham Forest 50 0 1995-01 1996-01 1997-01 1998-01 1999-01 2000-01 2001-01 2002-01 2003-01 2004-01 2005-01 2006-01 2007-01 2008-01 2009-01 2010-01Meta-Evaluation of the Impacts and Legacy of theLondon 2012 Olympic Games and Paralympic GamesJonathan France and Anna WoodhamJune 2nd 2011
  2. 2. Contents • Overview of the meta-evaluation – Objectives – Phases – Methods • Planning and prioritisation • The evaluation approach • Key areas for further research 2
  3. 3. Study objectives Comprehensive and systematic meta-evaluation of the Impacts and Legacy of the London 2012 Olympic Games and Paralympic Games • Considering Sport, Economic, Social and East London, plus Disability and Sustainability • Outputs, intermediate results, and outcomes, and the additional impact and VFM of legacy investments • Tangible and intangible/intended and unintended effects • Lessons learnt around planning and delivery of a lasting legacy • Influencing role (interim evaluation) • Inform and advance the evidence base on hosting mega-events • To develop and enhance methods of meta-evaluation (ESRC supported) 3
  4. 4. Overview of the 3 phases Phase 1: May 2010 – April 2011 Phase 2 : Feb 2011 – March 2012 Phase 3: April 2012 – March 2013 Project Initiation Report 3 (Baseline) • Project Initiation Document • Baseline Report 6 (Post Games Evaluation) • Project Plan • Counterfactual • Baseline & counterfactual update • Project set-up tasks • Evaluation synthesis Report 4 (Interim Evaluation) • Primary research Report 1 (Scope) • Evaluation synthesis • Economic modelling • Identification of project scope • Primary research • VFM • Research questions • Initial modelling • Lessons learnt • Logic chains • Impacts and lessons learnt to date • Longer-term impacts • Data strategy • Meta-evaluation methods Report 2 (Methodology) • Meta-evaluation methodology • Lessons learnt Phase 4 planned for (March 2013-2020) 4
  5. 5. Contents • Overview of meta-evaluation • Setting up and planning the meta-evaluation – Project scope – Logic chains – Research questions • The evaluation approach • Key areas for further research 5
  6. 6. Defining scope • Legacy initiatives: – New public programmes and investments developed as a result of Games – Existing programmes and investments refocused, expanded, or accelerated – Private and third sector investments inspired by the Games • Distributional impacts: – Geography (East London, Nations and Regions) – Groups (disabled, BME, young people, gender) • Temporal impacts: – Pre, during and post Games  Thoughts refined through desk research, DCMS workshop, stakeholder consultation and logic chains. 6
  7. 7. Logic chainsRationale and Objectives Outcomes/impacts Gross to Net ConversionLondon 2012 involves a major regeneration programme providing jobs, homes, - Economic benefits – -Need to take account of additionality - infrastructure, training and opportunities for UK businesses. This helps overcome increased GVA, job extent to which opportunities would not a coordination failure whereby firms and individuals are unwilling to invest in the creation and jobs have been provided or not be available area even though if they all did together they would benefit. from other sources in the counterfactual safeguarded scenario.The Games will also: - Better health- act as a showcase for British expertise and capabilities, overcoming similar outcomes -Need some consideration of extent to problems with coordinated promotion to encourage inward investment, tourism - Better quality of life/ which programme of intervention is and improve export potential; improved wellbeing responsible for observed outcomes.- provide a focal point for national identity, representing Britain and British culture to - More cohesive and the world and creating civic and national pride that inspires new attitudes and -Adjustments for behaviours; inclusive communities displacement, leakage, substitution, cro- inspire greater participation in sport, providing positive externalities through - More sustainable wding out and multiplier effects, as improved health. communities. relevant.Activities Outputs Results- Sport - Harnessing the UKs passion for - Engagement with individuals - New participants in sport and cultural activity sport (sports, culture and employment - People into jobs and gaining qualifications support) - More new businesses and more contracts traded with- Economy - Exploiting opportunities for existing businesses economic growth - Improvements to sports - New inward investment to East London and UK infrastructure- Social - Promoting social engagement - Increased visitor numbers to London and UK - Engagement with businesses - Greater sense of community cohesion and inclusion and participation - Creation of opportunities - More sustainable behaviour of individuals and- East London - Driving the regeneration (volunteering, disabled people etc) businesses of East London. - Marketing and awareness campaigns - Improved accessibility for disabled people in accessing - Building of venues according to new sport, culture employment and transport. sustainable construction standards. 7
  8. 8. Research questions Designed to tease out what the impacts have been in legacy areasparticipation in sport and physical CROSS CUTTING activity the development of competitive and elite sport Changes in attitudes to disability; increased participation of disabled people in sport, economy, volunteering and culture economic Contribution to sustainable development impacts, particularly in Effects on well-being terms of employment Effects on the international profile and reputation of the UK, and GVA London and East London Benefits to target groups/communities social impacts, particularly in terms of volunteering and development of the big society value for money Whether impacts have been sustained Lessons learned about how to maximise the benefits to contribution to the the host country and city from the staging of mega-events, regeneration of particularly in terms of organisational lessons and change East London
  9. 9. This is a meta-evaluation • Requires robust programme evaluations, that meet needs of Meta-Evaluation, so… 1. Identify gaps or lack of granularity 2. Influence evaluations • Evaluation Steering Group • Work with Legacy Boards • Internal and external seminars 3. Ongoing assessment of relevance and robustness of project-level evaluations 4. Last resort develop own plans for primary research 9
  10. 10. Contents • Overview of meta-evaluation • Setting up and planning the meta-evaluation • The evaluation approach – Framework – Value for money – Challenges • Key areas for further research 10
  11. 11. Framework for assessing impact • Builds upon existing evaluation frameworks: – Treasury Green Book input – output – outcome model – Regional Development Agency Impact Assessment Framework – Magenta Guidance – PWC 2012 Games Evaluation Framework • Each logic model grounded in theory of change and common outcome measures, which can then be tested out through evaluation evidence • Issues of deadweight must be tested and will apply to all themes; displacement and substitution more to economic themes • Relevant impact assessment frameworks then devised for each theme, but top down and bottom up analysis required in all cases 11
  12. 12. Sport: Logic chain Rationale and Objectives Outcomes/impacts Gross to Net Conversion - Increased participation in Need to take account of additionality - The 2012 Games provides an unparalleled opportunity to catalyse investment in new in active sport extent to which opportunities would not and improved sports infrastructure, as well as inspiring people to do more and - Increased life expectancy have been provided or not be available achieve more in sport. Social and economic benefits can be derived from health and better health from other sources in the counterfactual benefits. Elite achievement in sport can deliver important benefits in terms of - Increased excellence and scenario. boosting national pride, providing sport role models, encouraging community and elite participation in sport, and improving the UKs reputation and influence abroad. elite sport achievements - Increased sporting Need some consideration of extent to influence abroad which programme of intervention is The GOE’s Sports Legacy Strategy Paper, set out five promises: responsible for observed outcomes. - Places: Transforming the places where people play sport, making the benefits of London 2012 visible in cities, towns and villages across the country - People: Inspiring people to make sport happen at the local level - Play: create the sporting opportunities and challenges that give everyone the chance to become part of the mass participation legacy. Activities Outputs Results Legacy activity is to take place in three areas: - Increased access to sport opportunities, sport - Youths involved in Olympic and Paralympic satisfaction, club membership style school sport competitions - New participants in sport and less drop-off -Places: Upgrading up to a thousand local sports clubs - Increased confidence of pupils, higher school - Adults and young people involved in and facilities;· investing in iconic multi-sport facilities; attendance, and academic motivation community sport and physical activity protecting and improving hundreds of playing fields. - Better pathways and support to elite sport programmes -People: Recruiting, training and deploying 40,000 - Higher level of sport participation for disabled - Coaches trained to support sports sports leaders - Increased access to sport volunteering and - Hours of volunteering for sports -Play: Motivate over 100,000 adults to test themselves higher interest and commitment in volunteering - Investment in elite sport and NGBs in multiple Olympic and Paralympic sports;; investment - Increased influence abroad - Young people engaged in international sport in sport for disabled people. - Mega sporting event bids won - Bids for other sporting events - Support for Elite and Internationalsport 12
  13. 13. NATIONAL EVALUATIONSData sources: National evaluations with a regional dimension eg. CompeteFor collation; QA;synthesis; and PRIMARY AND SECONDARY DATA aggregation Secure additional questions in surveys Commission studies if needed PROJECT MONITORING Project/programme monitoring information Input data per project MACRO ECONOMIC MODELLING Impacts of construction expenditure Defining counterfactual Estimating secondary (indirect and induced) impacts QUALITATIVE DATA Case studies 13
  14. 14. Examples • Sport and social themes:  Amendments to national culture and sport surveys in England, facilitating statistical tests of association between sports participation and variables measuring engagement in the Games; medals performance; time-series data/scenarios for volunteering  Bottom-up evidence from programme evaluations; post Games venue planning; selective primary research for Games Makers (survey), disability (media analysis) and sustainability (case studies) • Economy and East London • Mix of statistical techniques (differences in differences/control areas; economic modelling); amendments to national tourism surveys; UK trade and investment data • Bottom-up evidence from programme evaluations; stakeholder interviews (SAV); academic studies; resident and business surveys 14
  15. 15. Valuing benefits 2012 Games Techniques include: Benefits/Outputs (with legacy) • Benefits transfer (other studies) 2012 Games (Without legacy) • Econometric modelling No 2012 Games Baseline • Revealed preference techniques • Stated preference techniques On benefits transfer, some benchmarks available through the DCMS CASE programme: • Doing sport once a week generated Subjective Well Being equivalent to £11,000 increase in household income; • Health cost savings of doing sport are £5,000 for a 30-39 year old. 15
  16. 16. Thematic challenges • Sport – Measuring the sustainability of participation (or use of proxies) – Methodologies to analyse elite sport networks are underdeveloped – Top-down sport for development highly criticized and under-evaluated • Social – Lack of participant research – access to LOCOG Games Maker database – Wider influences, e.g. Big Society • Economy – Aggregating the short-term and ongoing impact of the Games on economy – Assessing the longer-term influence of the Games on image/competitiveness; – need to influence tourism agencies to capture impacts over time – wider impact of the Games on inward investment and export trends (longitudinal approach preferred) 16
  17. 17. Broader challenges • Questions have been prioritised, but scope still vast and complex • Internal organisation and systems • What is additional? • Influencing, building support (e.g. LOCOG, academic community), access • Synthesis or aggregation of outcome data? • Assessing additionality • Developing counterfactuals without evaluations • Developing counterfactuals with evaluations • Geographical and E&D sensitivities • Mega-events have not previously been measured in such a comprehensive way: no real precedent 17
  18. 18. Contents • Overview of meta-evaluation • Planning the meta-evaluation • The evaluation approach • Key areas for further research 18
  19. 19. Areas for further research:Theme Primary Data RequirementsEast London • Household survey and focus groups with residents • Private sector leverage • Stakeholder lessons from park conversion and convergenceEconomic • Visitor experience survey • FDI attracted • Sustainability of employment outcomesSport • Impact of infrastructure investments • Elite Sport contributionCommunity Engagement • Inspire Programme outcomes • Games Maker surveyDisability and • Perceptions of disabilitySustainability • Additional small scale primary researchNations and Regions • Pre-Games Training Camps impact • Local Authority 2012 activities and outcomes 19
  20. 20. How you can get involved • Share knowledge on existing and planned 2012 evaluations and research • Share findings of 2012 project level evaluations and research • Influence surveys and research work planned • Views and input on methodology and alternative approaches • Contact: anna.woodham@culture.gsi.gov.uk 20
  21. 21. Questions… 22

×