Programme Evaluation training slides Osvaldo Nestor Feinstein

669 views

Published on

Programme Evaluation training slides Osvaldo Nestor Feinstein

Published in: Education, Business, Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
669
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
13
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Programme Evaluation training slides Osvaldo Nestor Feinstein

  1. 1. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Programme Evaluation for Strategic Management Osvaldo Néstor Feinstein Program Evaluation Workshop Ankara, 28-29 November 2013
  2. 2. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Themes 1. Programme Evaluation and Strategic Management 2. Performance Management & Rating systems 3. Institutionalization of Evaluation in the Public Sector 4. Assumptions Based Programme Evaluation Framework 2
  3. 3. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Programme Evaluation and the Strategic Management Cycle • Policy Review  • Strategic Planning and Selection of Interventions/strategies • Implementation (delivery of strategies) • Performance reporting /Monitoring • Performance auditing/EVALUATIONWider PROGRAMME EVALUATION • Policy Review 
  4. 4. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Programme Evaluation and Effective Governance • Public programmes are interventions that attempt to achieve results, outputs, outcomes and impact, using public resources. • Are government interventions achieving their intentions? Are they producing the intended results? Can they be improved? • Programme Evaluation provides answers to these questions, contributing to effective governance.
  5. 5. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Relations between Monitoring and Evaluation • MONITORING: observing and collecting data on the implementation of a programme, including inputs, outputs and outcomes, & contextual factors • EVALUATION: on the basis of monitoring and other data, assessing in a systematic way programme effectiveness ( achievement of outputs & outcomes),efficiency, relevance and sustainability • Evaluation uses monitoring data and evaluation requirements determines which data should be monitored ME & EM
  6. 6. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Performance monitoring and reporting and their relation to evaluation • Performance monitoring is a continuous process of collecting and analyzing data to compare how well a programme or policy is being implemented against expected results • Performance reporting is an instrument to convey the results of performance monitoring • Performance monitoring and reporting are inputs for Programme Evaluation, which includes an assessment of programmes.
  7. 7. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Differences with Auditing and Investigation • AUDITING verifies if resources were allocated for their intended uses. It does not assess objectives, which takes as a given. • INVESTIGATION assesses the concerns and complaints with respect to the program implementation, for example, in violating rights of minorities or allegations of fraud.
  8. 8. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Key purposes of evaluation • Evaluations for accountability: to show for which purposes resources were used. Were the programme‟s objectives (expected results) achieved? Ex post • Evaluations for learning: to draw lessons from experience and apply them to improve interventions. What knowledge or lessons can be drawn from the program‟s experience? (Janus)
  9. 9. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Programme Evaluation as a tool to learn from experience Looking to the future, with a view to the past THE PAST THE FUTURE
  10. 10. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Program Evaluation and Strategic Decision Making • Decisions concerning – which programs to close or cut, on the basis of performance, results achieved or not achieved (accountability function of evaluation) – which programs should be supported, and/or improved, in light of results achieved and strategic orientations (learning function of evaluation) • Decisions influenced by political factors and by evidence on results provided by evaluations
  11. 11. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Programme evaluation and strategic planning • Decisions among alternative ways (technologies) to achieve programme‟s objectives . Ex-ante use of evaluation (for example, cost benefit analysis) • Identifying contextual factors that may require programme adaptation • Improving stakeholder coordination during preparation of strategic plans 11
  12. 12. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Program evaluation and public management reform • Program Evaluation (PE) as an instrument of public management reform, to improve its implementation by drawing lessons from the implementation experience. • PE provides evidence on effective and ineffective programs so as to guide resource allocation (but beware with mechanical links) 12
  13. 13. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Programs, projects and policies • Projects: achieving objectives (e.g. reducing rural poverty) through a set of instruments or components (e.g. microfinance) • Programmes: set of projects or projects with several components. They include objectives, resources and activities Programmes to be evaluated could be interventions included in Institutional Strategic Plans. • Policies: objectives that are implemented through a set of programmes or projects • POLICIES  PROGRAMMES  PROJECTS
  14. 14. SOME EXAMPLES AMONG THE 25 PROGRAMS INCLUDED IN THE 10th NATIONAL DEVELOPMENT PLAN Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir • PROGRAM ON REDUCING THE INFORMAL ECONOMY • THE COMMERCIALIZATION PROGRAMME IN PRIORITY TECHNOLOGY AREAS • IMPROVING THE HEALTH TOURISM PROGRAM • TRANSFORMATION PROGRAM FROM TRANSPORTATION TO LOGISTICS • URBAN TRANSFORMATION PROGRAM ENHANCING COMPETITIVENESS AND SOCIAL COHESION • INCREASING THE EFFECTIVENESS OF THE LABOUR MARKET PROGRAM 14
  15. 15. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Programme evaluation and the evaluation of the NDP • Turkey‟s 10th National Development Plan indicates that the coordination of monitoring and evaluation of the progress of the Development Plan will be performed by the Monitoring and Direction Committee of the Development Plan, which will be headed by the Undersecretary of the Ministry of Development and comprised of high level administrators from related ministries. • In the process of planning and implementing new investments, the NDP states that there is a need for enhancing the capacity of public institutions in preparation, implementation, monitoring and evaluation issues • Discussion question: could it be convenient for strategic management to separate evaluation from monitoring?
  16. 16. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Strategic Plans and Programme Evaluation • Strategic Plans include programmes although sometimes without calling them “programmes” (like the French gentleman who spoke prose without knowing it). “Implicit programmes”. Some examples – SP of the Ministry of Energy, its “Aims” can be evaluated as programmes (ex. “providing diversity in resources by giving priority to domestic resources) – SP of the Ministry of Justice, its “Objectives” can be evaluated as programmes • Performance indicators of strategic objectives are usually outputor input level indicators. Outcome indicators are rarely applied. In many cases, performance indicators are measured by the level of required inputs • For discussion: how could better performance indicators support strategic management? 16
  17. 17. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Strategic Plans and Monitoring & Evaluation • Monitoring and evaluation is one of the least developed components of SPs among the pilot institutions • Performance Plans do not include any section for the follow up and evaluation of the SP implementation. • For discussíon: why is it important for strategic management to include an appropriate section dealing with evaluation? 17
  18. 18. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir • • • • • • Different types of evaluation Ex-ante evaluation (appraisal) Process evaluation Mid-term evaluation Completion evaluation Ex-post evaluation – Impact evaluations Formative (to improve)  Summative (to judge) – “when the cook tastes the soup, that's formative; – when the guests taste the soup, that's summative“ M.Scriven 18
  19. 19. Costs and Benefits of different types of evaluation Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir • Impact evaluations more expensive and less timely for decision making • Higher benefits of evaluations when their timing is linked to decisions that will have to be made at certain moments or periods • Choose approaches that will allow to deliver the evaluation in time for decision making See “Monitoring and Evaluation: Some Tools, Methods and Approaches” http://www.worldbank.org/ieg/ecd/tools/
  20. 20. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir • • • • • Public expenditure tracking surveys (PETS) Public expenditure tracking surveys (PETS) track the flow of public funds and determine the extent to which resources actually reach the target groups. The surveys examine the manner, quantity, and timing of releases of resources to different levels of government, particularly to the units responsible for the delivery of social services such as health and education (“follow the money”). PETS are often implemented as part of larger service delivery and facility surveys which focus on the quality of service. Can be used to diagnose problems in service delivery quantitatively, to identify best practices and to provide Providing evidence on delays, “leakage,” and corruption. Supports the pursuit of accountability when little financial information is available and improves management by pinpointing bureaucratic bottlenecks in the flow of funds for service delivery. http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTSOCIALDEVELOPME NT/EXTPCENG/0,,contentMDK:20507700~pagePK:148956~piPK:216618~theSite PK:410306,00.html for more details, examples and other methodologies 20
  21. 21. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Cost-Benefit and Cost-Effectiveness Analysis • Tools for assessing whether or not the costs of a programme can be justified by its results. Cost-benefit analysis measures both inputs and outputs in monetary terms. Cost-effectiveness analysis estimates inputs in monetary terms and outcomes in non-monetary quantitative terms (such as improvements in student reading scores). • Can be used to inform decisions about the most efficient allocation of resources and/or to identify programmes or projects that offer the highest rate of return on investment. • Useful for convincing policy-makers and funders that the benefits justify the programme but data for cost-benefit calculations may not be available, and results may be highly dependent on assumptions made. • http://ec.europa.eu/regional_policy/information/evaluations/guidance_en.cf m 21
  22. 22. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Different evaluation approaches/techniques • Objectives based evaluation • Goal-free evaluation • Qualitative methods • Quantitative methods • Mix of quantitative and qualitative methods • Triangulation 22
  23. 23. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Evaluation Criteria and Evaluation Questions • Relevance: were the objectives consistent with beneficiaries‟ requirements, organizational and country priorities & policies? • Effectiveness: to which extent the program‟s objectives were achieved, or are expected to be achieved? • Efficiency: how economically resources/inputs (funds, expertise, time, etc.) were converted into results? how reasonable were the costs? 23
  24. 24. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Four additional evaluation criteria • Sustainability: will the results achieved be durable? Is it likely that those results will be sustainable? • Innovativeness: were innovative approaches introduced by the program? • Replicability and Scaling-up: to what extent the program‟s approaches have been (or are likely to be) replicated and scaled up? • Equity: how equitable by gender, social group (or other attributes) is the distribution of benefits? 24
  25. 25. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Case study exercise (I) Evaluation of the Northern Someland Rural Development Program • With support from the XFund, the 10,000,000 euros Northern Someland Rural Development Program aimed at providing low prized fertilizers to 5,000 farmers in the northern region of Someland, a land-locked country, where small farmers are fruit producers. • The objective of the program was to improve the welfare of the farmers and their families. A program implementation unit was set up to organize the delivery of fertilizers and to establish links between the farmers and local institutions involved in the provision of complementary services. • How would you evaluate this program? What would you do to evaluate the program? Please discuss with your colleagues and prepare a 5 minutes presentation 25
  26. 26. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Alternative approaches to program evaluation • Goal-free evaluation – Its strength in avoiding a limited focus on objectives can be compensated by objectives-based evaluation with an explicit focus on “unintended consequences” – Weak on accountability. • Impact evaluation with control groups – Explicit counterfactual to model the without program situation 26
  27. 27. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Counterfactual in evaluation • What would have happened WITHOUT the program? (had there been no program) • Note that the program is “factual”, it is a fact, as a program was implemented, whereas the counterfactual is a situation that is not observed, it has to be reconstructed. • Explicit and implicit counterfactuals, not theoretical but practical importance, rationale for the program. 27
  28. 28. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Comparisons Before/ After With/Without Without With Before 100 100 After 150 120 Look first only at the “with” column 28
  29. 29. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Comparisons Before/ After With/Without Without Before After With 100 100 80 100 Look first only at the “with” column
  30. 30. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Establishing the “Before” & “Without” • “Before” , or at the beginning of the program. Need for a BASELINE, which could be part of the diagnosis. Sometimes available census and/or household surveys can be used. • “Without”, requires a comparison group; “control group” is a particular case. Statistical methods. Comparative analysis. 30
  31. 31. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir The “General Elimination Method” GEM • Simple and powerful, introduced by Michael Scriven in 2008 • Two key steps: 1. consider the set of “possible plausible causes” (including components of the contexts and/or other programs or policies) 2. eliminate the less plausible possible causes • See application and references in p.3 of • http://www.undp.org/evaluation/documents/ADR/ADR_ Reports/Chile/ADR-Chile.pdf 31
  32. 32. How to manage and conduct program evaluations Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir • The first strategic decision is choosing WHAT to evaluate. Decide (if you have degrees of freedom) on the basis of expected value added for accountability and/or learning • The terms of reference should clarify expectations concerning the evaluation that will be carried out. Answers to WHY, HOW, and WHEN TO EVALUATE? – See “Writing Terms of Reference for an Evaluation: A HowTo Guide” – https://ieg.worldbankgroup.org/evaluations/evaluationcapacity-development-ecd 32
  33. 33. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Which type of evaluation should be used? • Selecting the type(s) of evaluation to be used • Methods adequate to the circumstances (purpose, availability of time and resources). Not only “fit-topurpose”. Pragmatic • See “Monitoring and Evaluation: Some Tools, Methods and Approaches” • http://www.worldbank.org/ieg/ecd/tools 33
  34. 34. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Consultation with stakeholders • Consulting with stakeholders – Why? To get their insights, and – to nurture their interest in the evaluation (which will facilitate the use of the evaluation). • Identifying key stakeholders: – Partners, intended beneficiaries, government officers, civil society representatives 34
  35. 35. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Reviewing relevant documentation • In addition to PROGRAMME DOCUMENTS, reports on the country, internet searches for thesis. other evaluations and papers on the project/program area and on the instruments used (e.g., rural microfinance or agricultural extension in the area) 35
  36. 36. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir APPLYING Evaluation Criteria • Applying evaluation criteria – Relevance, Effectiveness, Efficiency, Sustainability, Innovativeness, Replicability and Scaling-up, Performance of partners – Ratings for the criteria – Rating overall project achievement – Provide a justification for the ratings 36
  37. 37. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Use of evaluation reports • Using evaluation reports – Evaluation reports are important INPUTS for evaluations, as indicated before under “reviewing documentation”. – Evaluation reports are also key OUTPUTS of the evaluation process. Show in a clear way the links between findings and recommendations to facilitate the use of the evaluation report. 37
  38. 38. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Case studies • For in-depth understanding of how a program has operated in a specific context • Problem of eventual lack of representativeness • Typology of contexts/situations, and case studies corresponding to the different contexts/situations. • Combining case studies with surveys 38
  39. 39. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Use of surveys • Search for available surveys • If surveys are needed and are not available, ensure that professionals with the required expertise are involved; possibility of subcontracting • The methodology ACTUALLY used should be described in detail (for replication & interpretation) • Pilot trials are important 39
  40. 40. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Semi-structured interviews • Identify key informants corresponding to different perspectives • Elaborate a brief interview guide • Team should be opened to comments, opinions and judgements not considered in the guide • Systematize the results of the interviews 40
  41. 41. Case study exercise (II) Evaluation of the Youth Employment Program for the Eastern Region Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir • Given the importance of youth unemployment a program to increase the employability of the young was designed and implemented in the Eastern region of the country. The program included training in agricultural activities, as well as training courses for the production of furniture, which is an important economic activity in that region. In addition there were also training courses on informatics. An evaluation team was tasked to evaluate the program. • Discuss with your colleagues how you would conduct the evaluation of the program. Prepare a brief presentation with the conclusions of your discussion. 41
  42. 42. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir e-reference materials • Glossary of Key Terms in Evaluation and Results Based Management – http://www.oecd.org/dac/evaluation/evaluationandaideffectivenes sseries.htm – TURKISH VERSION: – http://www.oecd.org/dac/evaluation/deerlendirmevesonuodaklyon etimdeanahtarterimlersozluu.htm – Monitoring and Evaluation: Some Tools, Methods and Approaches • http://www.worldbank.org/ieg/ecd/tools • Impact Evaluation in Practice (2011) Eng/Fr/Sp. – http://siteresources.worldbank.org/EXTHDOFFICE/Resources/54 85726-1295455628620/ Impact_Evaluation_in_Practice.pdf 42
  43. 43. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir E-evaluation guides from the EC • On the evaluation of innovations • http://ec.europa.eu/regional_policy/sources/docgener/e valuation/pdf/eval2007/innovation_activities/inno_activit ies_guidance_en.pdf. • On impact assessment http://ec.europa.eu/governance/impact/commission_gui delines/commission_guidelines_en.htm • Several guides can be accessed at • http://ec.europa.eu/regional_policy/information/evaluati ons/guidance_en.cfm 43
  44. 44. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir • • • • • Performance Management Performance Management, Performance Measurement, Performance Monitoring Measure to Monitor & Monitor to Manage Two frequent (and partially deceiving) quotes: – “What gets measured gets done” Tom Peters – “What gets measured gets managed” Drucker • Measurement facilitates Management 44
  45. 45. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Performance measurement and management • Performance measurement – What is happening with the programs? – How are the programs performing? – Are programs achieving their expected results? (the “why” question answered by evaluations) • Performance management – making decisions about programs and project portfolio: allocation of additional resources, stopping programs, lessons for new programs, etc. 45
  46. 46. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir PM in practice & Exercise III • What should be measured and managed? • Comparing what is actually measured with what should be measured, which are the expected results? • The results chain: inputs--(processes)outputsoutcomes  impacts • Exercise III: QUESTIONS FOR DEBATE: • Are inputs “results” in the results chain? And outputs? • What are the implications for accountability of different definitions of results? 46
  47. 47. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Reviewing a portfolio of programs • Identification of “PM gaps” (expected vs actual) • In which (type of ) programs is performance higher (lower) than expected? • • • • What worked? What did not work? Why? (an important evaluation question) What influence had the context ? Any patterns? 47
  48. 48. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Purposes of PM • Accountability and learning (improvement) • Whose accountability? • Whose learning? • Complementarity or trade-offs? 48
  49. 49. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Lessons on PM Systems (PMS) • Importance of ownership of the PMS • WHO are the intended users of PMS? • Convenience of involving intended users in the design and/or redesign of the PMS • Opportunities created by PMS to facilitate coordination and enhance effectiveness 49
  50. 50. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Lessons on PM Risks • Nonuse and/or misuse (GAO‟s findings of nonuse of performance information to manage; commitment by leaders and communication of that commitment is crucial) • Mismeasuring and/or not measuring, not counting what counts, leaving out what is important • Inducing short-termism • Bureaucratization, with PM being perceived as a mechanical exercise, as a burden 50
  51. 51. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Issues of PM • Participation of intended PM users in the review of the system • Programs, projects and portfolio, the trees and the forest • Partial achievement: PM when reporting period does not correspond to “gestation period”. Actual costs and likely benefits. 51
  52. 52. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Year Issue of Partial Achievement 2011 Output (reported in year…) 2A 2012 2A 6,000 3,000 Total 4A 16,000 4,000 52 Cost $ Unitary cost $/A 10,000 Revised 5,000 6,000 A:2-2x.5
  53. 53. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Issues of PM • PM and its use for resource allocation, performance informed budgeting • Quality control of data • Performance Measurement & Management • Links of PM with the M&E system • PM, comparisons and benchmarking (with baselines, targets, other organizations) 53
  54. 54. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir • • • • • • Rating Systems Trending theme: S&Poor Why ratings? Experience of WB & IFAD - ECG Validation of self-evaluation ratings, Evaluation of ratings, rating the raters Contestability (dealing with disagreements) 54
  55. 55. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Program or Project Ratings Relevance of anticipated outcomes Effectiveness (achievement of outcomes)and Efficiency on a six-point rating scale Highly Satisfactory, Satisfactory, Moderately Satisfactory. • Moderately Unsatisfactory, Unsatisfactory., Highly Unsat. • (risks to) sustainability of outcomes based on an assessment of four dimensions: financial, sociopolitical, institutional framework & governance, and environmental • • • • 55
  56. 56. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir A six- point rating system for the evaluation criteria • A six-point rating system for the evaluation criteria as a way to quantify qualitative judgements of evaluators – 6: highly satisfactory 5: Satisfactory – 4: moderately satisfactory – -----------------------------------------------– 3: moderately unsatisfactory – 2: unsatisfatory 1: highly unsatisfactory 56
  57. 57. Ratings for innovation, scaling-up and partners‟ performance Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir • Good practice of IFAD‟s evaluation office providing explicit ratings for the creativity, innovation and up-scaling potential of innovative projects and distinguishing between agency and partners‟ performance ratings and outcome ratings • Promotion of pro-poor innovation,replication and scaling up: The extent to which IFAD development interventions have: (i) introduced innovative approaches to rural poverty reduction; and (ii) the extent to which these interventions have been (or are likely to be) replicated and scaled up by government authorities, donor organizations,the private sector and others agencies. • Performance of partners: assess the contribution of partners to project design, execution, monitoring and reporting, supervision and implementation support, and evaluation. The performance of each partner will be assessed on an individual basis with a view to the partner‟s expected role and responsibility in the project life cycle. 57
  58. 58. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Remembering the performance evaluation criteria • Relevance : The extent to which the objectives of a program are consistent with beneficiaries‟ requirements and country needs, institutional priorities and policies. • Effectiveness: The extent to which the program‟s objectives were achieved, or are expected to be achieved, taking into account their relative importance. • Efficiency: A measure of how economically resources/inputs (funds, expertise, time, etc.) are converted into results. 58
  59. 59. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Clarifications • A satisfactory outcome means that relevant project objectives are expected to be achieved with no, minor or moderate shortcomings at the time of evaluation. • A likely sustainability rating means that the project is considered likely to generate continued benefits after project implementation - with no or only moderate risks. • (definition of sustainability: The likely continuation of net benefits from a program beyond the phase of external funding support. It also includes an assessment of the likelihood that actual and anticipated results will be resilient to risks beyond the program‟s life). 59
  60. 60. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir SMART criteria for indicators • Specific relating results to the achievement of one and only one objective. • Measurable. all parties agree on what indicators cover and on practical ways to measure them • Achievable feasible targets • Realistic levels of performance that are ambitious but feasible.. • Time-Bound, not open ended but with dates 60
  61. 61. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Use (and Misuse) of ratings • As an input, but not the only input, for decisionmaking, for performance management • For comparisons, benchmarking • For assessing and enhancing learning & quality • For aggregation of information 61
  62. 62. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Issues concerning Ratings • Identifying critical dimensions applying “Occam‟s razor” d1, d2, d3, …., dn • Weighting the dimensions w1d1, w2d2, w3d3 • Partial orders limits to aggregation: partial comparisons (Amartya Sen); • Transparency (avoiding “secret algorithms”) 62
  63. 63. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Synthesis • Performance management 1. 2. 3. Measurement, availability of performance information is not enough (possible nder-use, non-use and misuse); Focus on expected results, identify PM gaps and patterns, present data with a brief narrative Involve intended users in (re)design of PMS • Rating systems 1. 2. 3. Different approaches to ratings Validation of ratings Transparency of ratings and benchmarking • Links between performance management and rating systems 1. Ratings provide management with comparative data on performance that can be used for aggregation, learning and accountability 63
  64. 64. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Exercise IV: Case study example Performance Measurement System for a Health Program • A country wide health program was launched with the objective of contributing to improve primary health care of the population in urban and rural areas. Different types of health interventions were considered. You have been asked to develop a performance measurement system for such a program. • Discuss with your colleagues and prepare with your group a 5 minutes presentation. 64
  65. 65. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Learning from others World Bank Independent Evaluation Group (IEG) http://ieg.worldbankgroup.org/content/ieg/en/home.html Project perf.assessment reports(PPAR); M&E http://www.ifad.org/evaluation/process_methodology/do c/manual.pdf • http://www.thegef.org/gef/eo_office • https://wpqr1.adb.org/LotusQuickr/ecg/Main.nsf/$defaul tview/0906CAABDA17E70A48257731002A0656/$File/ PublicSectorGPS-revised.pdf?OpenElement • (GPS Evaluation Cooperation Group) • • • • 65
  66. 66. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Additional reference materials • DAC/IEG Sourcebook for evaluating global and regional programs http://web.worldbank.org/WBSITE/EXTERNAL/EXTOED/EXTGLOREGPAR PROG/0,,contentMDK:21178261~menuPK:4426473~pagePK:64829573~pi PK:64829550~theSitePK:4426313,00.html • Program Assessment Rating Tool (PART) US Gov.http://web.worldbank.org/WBSITE/EXTERNAL/EXTOED/EXTEVACAP DEV/0,,contentMDK:22321269~menuPK:4585748~pagePK:64829573~piP K:64829550~theSitePK:4585673,00.html Note: PART has been discontinued but it‟s worthwhile • A critique of the PART in http://www.businessofgovernment.org/report/performance-managementrecommendations-new-administration • Up-to-date information on the US gov. case: http://www.businessofgovernment.org/blog/business-government/ombunveils-its-performancegov-website 66
  67. 67. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir And some other useful references • US (speech by OMB A.Director) http://www.state.gov/s/d/rm/rls/rm/2010/147424.htm • http://web.hbr.org/email/archive/managementtip.php?date=060611 (brief note) • UNEG Good Practice Guidelines for Follow up to Evaluations • http://www.uneval.org/papersandpubs/documentdetail.jsp?doc_id= 610 • http://www.eipa.eu/files/repository/eipascope/20101022101218_Ei pascope_2010_2_Article2.pdf • Patria de Lancer Julnes et.al. (2008) “International Handbook of Practice-Based Performance Management” Sage • http://www.sustainability.com/library/rate-the-raters-phase-three 67
  68. 68. Institutionalization of Evaluation in the Public Sector - Demand Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir • Demand for evaluation – from the Executive (Prime Minister Office, Ministry of Development, MOF, Other M.) – from the Legislative , Parliament, from Civil Society • Types of demand: – Actual demand – Potential demand (there is an interest on evaluation but funds are lacking) – Latent demand (there is no awareness of the role of evaluation but an interest on what evaluation can provide) 68
  69. 69. Institutionalization of Evaluation in the Public Sector - Supply Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir • Actual Supply of Evaluation – Public sector managing the evaluations – Public sector conducting evaluations – Public sector outsourcing evaluations • Potential Supply of Evaluations – (organizations or professionals so far not involved in evaluations but with capacities which can be used for evaluation) • Private consultants, universities, think tanks conducting evaluations 69
  70. 70. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Institutionalization of Program Evaluation in the Public Sector • Who would lead a program evaluation initiative in the public sector? • A central Ministry, like the Ministry of Development in Turkey, the Ministry of Planning or of Finance in other countries • Who will manage, conduct and use program evaluations? Sector Ministries (Health, Education, Labor…) 70
  71. 71. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir International Experiences in Program Evaluation • Frequently the experiences of New Zealand, USA, Canada, the Nordic countries and the UK are highlighted. We will make reference to another OECD case more relevant for Turkey: the case of MEXICO and some other relevant experiences • Mexico has developed a well functioning system of evaluation of social programs, introducing some innovations that are worthwhile to take into account 71
  72. 72. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir The Mexican Experience on social program evaluation • The Mexican system for social program evaluation is described in http://www.coneval.gob.mx/Evaluacion/Paginas/Eva luation-and-monitoring-en.aspx • The National Council for the Evaluation of Social Development Policy (CONEVAL), as set forth in the General Law of Social Development, has the aim to regulate and coordinate the evaluation of the Social Development Policy and Programs implemented by public dependencies. 72
  73. 73. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir More on the Mexican Evaluation Experience • Mexico has also been developing evaluation systems at the subnational level, with support from CONEVAL, and the capital city of Mexico developed an evaluation system on its own. • For experiences of institutionalization of impact evaluation see www.worldbank.org/ieg/ecd/docs/inst_ie_framework _me.pdf 73
  74. 74. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir The experience of Austria • From a recent diagnosis of Austrian Public Administration: input rather than output (or outcome). Concern with who gets how much, instead of what has to be the outcome • Shift in 2009 towards an outcome orientation. Managing public adminstration based on contributions to achieve outcomes (performance management) 74
  75. 75. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir The experience of Malaysia • Malaysia public sector management reform shaped by best practices • Framing a reform as „best practice‟ was a way of neutralizing potential opposition from the very different and sometimes antagonistic stakeholders that the reform had to satisfy • Successive reforms have built on previous ones: past failures have been the paradoxical foundation for present success 75
  76. 76. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir More on the Malaysian experience • The current prime minister, or his deputy, has individual meetings with the ministers responsible for the seven national key result areas every month, and with the rest of the Cabinet every six months,to review their performance against key performance indicators (KPIs) set by the prime minister himself • Need to emphasize the importance of the implementation stage of reform, and of an appropriate diagnosis 76
  77. 77. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir The evaluation of Turkey‟s Vocational Training Programs • The evaluation of Turkey‟s Vocational Training Programs sought to assess the effectiveness of improving vocational training programs in the context of the rapid expansion of these programs in Turkey. • Concept Note, Baseline Report and • Evaluation Report available at • http://web.worldbank.org/WBSITE/EXTERNAL/EXTAB OUTUS/ORGANIZATION/EXTHDNETWORK/EXTHDO FFICE/0,,contentMDK:23177755~pagePK:64168445~p iPK:64168309~theSitePK:5485727,00.html 77
  78. 78. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Exercise V: Debate on the Institutionalization of Evaluation • Discuss the institutionalization of evaluation in the public sector of Turkey and rate demand for and supply of evaluation, as well as the links between demand and supply of evaluation, and the use of evaluation on: – a four point rating scale, with 1 – as the lowest rating and 4 the highest. • Demand: Supply: 78 Links: Use:
  79. 79. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Summing-up: 12 steps to conduct program evaluation 1. Identify the program‟s objective (s) 2. Become familiar with the modalities of implementation 3. Identify the basic assumptions (the logic of the intervention) 4. Decide which will be the key indicators (could be those included in the program, eventually adding some more) 5. Search and read bibliography and review the first four steps in light of your reading 79
  80. 80. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Steps for conducting program evaluation (continued) 6. Search for and review available information sources. 7. Determine needs of additional information, if any 8. Decide which methods/techniques to use 9. Apply the evaluative criteria 80
  81. 81. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Steps for conducting program evaluation (final) 10. Be open to the possibility of unintended consequences, positive and/or negative 11. Analyse and synthesize results by implementation modality and by area of implementation, identifying best and worst 12. Triangulate results (cross-checking different information sources) 81
  82. 82. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir The end? “This is not the end, nor the beginning of the end, but the end of the beginning” - Winston Churchill 82
  83. 83. Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse edilmektedir Thank you! teşekkür ederim ofeinstein@yahoo.com 83

×