Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Rotary Foundation Cadre Training: Monitoring and Evaluation


Published on

During this session we will:
*Review importance of monitoring and evaluation
*Share overview of grant model evaluation plan
*Review methodologies used in previous evaluations
*Share plans for future evaluation methodologies

Published in: Government & Nonprofit
  • Hey guys! Who wants to chat with me? More photos with me here 👉
    Are you sure you want to  Yes  No
    Your message goes here

Rotary Foundation Cadre Training: Monitoring and Evaluation

  1. 1. Monitoring and Evaluation TRF Cadre Training 12 January 2015
  2. 2. CADRE TRAINING | 2 PRESENTERS Philip Silvers •Cadre Chairman 2014-15 •Past RI Director and board liaison to the Trustees •44-year career as a program evaluation and survey research professional •10 years professor of graduate-level Research Methods James Robinson •Director, Strategy Innovation and Evaluation •Former Division Manager Humanitarian Grants and Future Vision Pilot •RI-USAID Steering Committee Korhan Atilla •Cadre Technical Coordinator 2008- 11 •Civil Engineer •Conducted sustainability check site visits •District Governor Elect 2014-15, D- 2430
  3. 3. CADRE TRAINING | 3 Cadre’s Efforts to Support Sustainability Phil Silvers
  4. 4. CADRE TRAINING | 4  The New Grants Model Is Working Fairly Well – TRF Manuals Are Helpful – Clubs and Districts are working together – The Cadre Is an Important Resource • Technical assistance • Stewardship (accountability)  Proposal Writers and Sponsors are struggling – Online application system is a challenge – Effective needs assessments are not being done – Sustainability—still an elusive concept – Monitoring and Evaluation (M&E) Who will do it? – Average time to grant approval: 3 months YEAR ONE – WHAT DID WE LEARN?
  5. 5. CADRE TRAINING | 5 Core Effects: Sustainable Change/ Lasting Results Process Results (‘Doing Good’) Training Cultural Experiences Adventure Travel Friendships Humanitarian Supplies Equipment Externalities (The ‘Sizzle’) Challenges Rotary “Credits”
  6. 6. CADRE TRAINING | 6 • Real sustainability begins with the Community Needs Assessment—by talking to the grass-roots people before you come up with the solution. – Need to comprehend their view of the world—their fears, their challenges, their dreams for their kids • Needs Assessment, Sustainability, and M&E are not independent concepts—they must be integrated into one piece by one team. • Depending on what you are trying to accomplish, the following slide gives you some options on measuring Sustainability (or ‘red flags’ if you omit them) BASIC PRINCIPLES OF SUSTAINABILITY
  7. 7. CADRE TRAINING | 7 Security— •People •Premises •Equipment Security— •People •Premises •Equipment Dimensions of Sustainability Ecological Human Capital Organizational Systems Behavioral Change Mechanical Systems New Technologies (Tools) Cultural Ethos Legacy DIMENSIONS OF SUSTAINABILITY Economic New Policy/ Legislation Cultural Ethos/ Legacy
  8. 8. CADRE TRAINING | 8 Security refers to the protection against harm to people, to premises, and to equipment. Other than a good needs assessment, this is the most fundamental issue in Sustainability. Are the participants secure from coercion, threats to their reputation, privacy, and physical/mental well-being? Are the premises secure from unwanted intruders? Are the equipment and materials safe from theft and vandalism?
  9. 9. CADRE TRAINING | 9 Economic refers to the financial viability of the project over the long haul. Are revenue streams planned to support the program after the grant funding ends?  Operating costs?  Equipment maintenance and replacement? Who will be responsible for managing the revenue streams? Are contingency plans in place if the planned funding streams fail to deliver? Economic
  10. 10. CADRE TRAINING | 10 Cultural ethos refers to what is acceptable or not acceptable behavior within society, e.g., smoking in common areas, young girls’ access to education, women’s role in government, getting tested for HIV/AIDS, tribal enmities, xenophobia. Have project planners done a ‘force-field’ analysis of the factors inhibiting and fostering change? What rewards, incentives, modeling, or strategies might help shift prevailing attitudes? What various measures will reveal whether a change has happened? Cultural Ethos/ Legacy
  11. 11. CADRE TRAINING | 11 SUSTAINABILITY How Else Can You Say It?  Game changer  Systemic change  Lasting results  A permanent difference  Self-perpetuating  Continuity  Can continue without outside help  Can exist on its own indefinitely  The ‘buck’ does not stop here!
  12. 12. CADRE TRAINING | 12 Evaluation Plan Overview James Robinson
  13. 13. CADRE TRAINING | 13 EVALUATION PLAN OVERVIEW M & E Activities 13- 14 14- 15 15- 16 16- 17 17- 18 18- 19 19- 20 20- 21 21- 22 Areas of Focus X X X Strategic Partners X X X Grant Model X X X Sustainability Assessments X X X  Multi-year evaluation  Site visits for 5-10% of Global Grants  Study common project types in each area of focus  Replicate sustainability check methodology
  14. 14. CADRE TRAINING | 14 M & E Framework
  15. 15. CADRE TRAINING | 15 TERMINOLOGY Monitoring typically refers to the routine and ongoing process of collecting data as agreed upon before project implementation. Evaluation generally involves a post-project assessment of the degree to which the project goals were met and the impact the project activities had on beneficiaries. A Measure is a quantitative or qualitative description of a project’s activities or impact. Slide 15
  19. 19. CADRE TRAINING | 19 Slide 19 Is monitoring and evaluation data required to close a global grant? Yes. Sponsors must provide the minimum standard (at least one measure per AoF) for M & E data in the final report. This information is required to close the grant. If Sponsors promised more M & E data than they can provide at the time of reporting, can close the grant if the minimum standard has been met. How should M & E funds budgeted in a global grant be processed as part of grant closure? Sponsors can utilize up to 10% of the grant budget for M & E. Provide accounting of expenditures at time of final report. If applicable, sponsors may provide a plan to expend remaining funds post closure and report M & E data into online system GRANT REPORTING REQUIREMENTS
  20. 20. CADRE TRAINING | 20 CHARACTERISTICS OF GOOD M & E PLANS 1. Less is More Rotarians should be modest in the number of measures they will track…if it’s in the application, we’ll expect to see it in the report. Advise Rotarians to limit the number of areas of focus they select in the application as this will multiply the number of measures required. Slide 20
  21. 21. CADRE TRAINING | 21 2. Measures match core project components Rotarians do NOT need to measure every project component. Instead, they should select the most important components and focus on measuring their success in those. Measures should be relevant to the main goals of the project and should indicate success in reaching the goals. Slide 21 CHARACTERISTICS OF GOOD M & E PLANS
  22. 22. CADRE TRAINING | 22 3. Mixture of output & outcome measures Counting activities and participants is a good place to start when developing an M/E Plan, but falls short of demonstrating quality. Strong M/E plans include measures that demonstrate the quality of the project in terms of changes in the behavior or knowledge of the beneficiaries. Slide 22 CHARACTERISTICS OF GOOD M & E PLANS
  23. 23. CADRE TRAINING | 23 4. Reasonable attribution – Measures should be closely and obviously connected to activities conducted and the project participants. – Grant officers can advise Rotarians on what is reasonable attribution of project outputs to outcomes. Slide 23 CHARACTERISTICS OF GOOD M & E PLAN
  24. 24. CADRE TRAINING | 24 5. Resources are dedicated to M/E activities Eligible items include: – Contracted labor of a local org or paying stipends to community members to collect data. – Consulting costs to technical experts to review and validate data. – To purchase data sets and other information necessary to establish a baseline. – Purchasing equipment such as phones, GPS equipment, and laptops to monitor activities. – Local travel costs, such as car rental, fuel, and train fare to conduct site visits. – Cost of electronic surveys; meeting costs for focus groups for evaluation purposes. – Printing costs for manuals and other materials needed for data collection. Slide 24 CHARACTERISTICS OF GOOD M & E PLAN
  25. 25. CADRE TRAINING | 25 5. Resources are dedicated to M/E activities Ineligible items include: – International travel to visit the project site. – Accommodations, meals and per diems. – Producing project promotional or recognition materials. – Any measurement expenses that are incurred prior to the grant approval date, such as the costs of conducting a community assessment. Slide 25 CHARACTERISTICS OF GOOD M & E PLAN
  26. 26. CADRE TRAINING | 26 Sustainability
  27. 27. CADRE TRAINING | 27 SUSTAINABILITY For The Rotary Foundation, sustainability means: ensuring that grants provide long-term solutions to community needs that the benefiting community can maintain after grant funding ends.
  28. 28. CADRE TRAINING | 28 Legacy Grants; Giving A Man A Fish SUSTAINABILITY
  29. 29. CADRE TRAINING | 29 Rotary Foundation ensures Sustainable impact of goods or services. Clubs and Districts ensure sustainable impact of goods and services Individuals & organizations benefit from sustainable impact of goods and services Community or organization ensures sustainable impact of goods and services TRF ensures sustainable impact of goods and services SUSTAINABILITY Global Grants; Teaching A Man to Fish
  30. 30. CADRE TRAINING | 30 Sustainability Framework
  31. 31. CADRE TRAINING | 31 SUSTAINABILITY FRAMEWORK Community Needs/ Strengths Materials/ Technology Funding • Community Needs Assessment • Meeting beneficiary needs • Involving community partners • Confirm local funding for long-term support • Compensate project participants to ensure continuity of work • Purchase equipment locally • Involve community in selecting technology • Train community to operate and maintain equipment
  32. 32. CADRE TRAINING | 32 SUSTAINABILITY FRAMEWORK Knowledge Motivation Monitoring/ Evaluation • Provide training to beneficiaries • Create plan to transfer knowledge to new beneficiaries • Collaborate with local organizations with expertise • Give incentives to beneficiaries to provide ongoing support • Identify leaders committed to outcomes • Prepare the community to own the project when funds expended • Develop measurable objectives • Identify ways to collect data • Establish baseline data to track the impact of the project
  33. 33. CADRE TRAINING | 33 SUSTAINABILITY FRAMEWORK Community Needs/ Strengths Materials/ Technology Funding Knowledge Motivation Monitoring/ Evaluation High incidence of waterborne illness among children under 5 in a village
  34. 34. CADRE TRAINING | 34 Abt Sustainability Check
  35. 35. CADRE TRAINING | 35 Average Lifecycle Scores For MGs & GGs 1.00 1.50 2.00 2.50 3.00 3.50 4.00 Application & Proposal Contract Point In Time/Final Lifecycle Stage AverageScore(1-4Possible) AverageOf All Grants AverageOf All GGs Averageof All MGs Mid-Point ABT SUSTAINABILITY CHECK
  36. 36. CADRE TRAINING | 36 Lifecycle Sustainability Scores; All Grants 1.00 1.50 2.00 2.50 3.00 3.50 4.00 1 2 3 Lifecycle Stage Score(1-4Possible) India GG25034 Kenya GG25081 India GG25027 Honduras GG25065 Honduras GG25363 India MG74944 India MG74942 Kenya MG70520 Honduras MG71320 Honduras MG71446 Honduras MG71940 Mid-Point Cooperating Organizations ABT SUSTAINABILITY CHECK
  37. 37. CADRE TRAINING | 37 Sustainability Matrix
  38. 38. CADRE TRAINING | 38 SUSTAINABILITY MATRIX Eight Factors of Sustainability Evaluated Grant Management Outcomes Financial Community Beneficiary Organizations Cooperating Organizations Training Software/Equipment
  39. 39. CADRE TRAINING | 39 SUSTAINABILITY MATRIX Methodology Assessment is comprised of eight sections Each section has 15-30 indicator questions Each grant is assigned a score for each indicator within the applicable sections  Score is between 1-4 Each grant is evaluated independently
  40. 40. CADRE TRAINING | 40 SUSTAINABILITY MATRIX Grant Management The Grant Management section focuses on overall project management Key indicators include:  Does the project have precise and finite objectives?  Is there an implementation workplan including milestones and benchmarks?  Were sustainable activities incorporated into the original project design?  Were the required progress reports submitted on time?  Was a sustainability plan included in the final report?
  41. 41. CADRE TRAINING | 41 SUSTAINABILITY MATRIX Measuring Outcomes The Measuring Outcomes section focuses on the measurability of a project, and whether or not appropriate M&E was conducted during and after the project Key indicators include:  Does the project have clear and measurable outcomes?  Does the monitoring system include community involvement?  Are the project outcomes shared with the community and other stakeholders in ways meaningful for sustained project support?
  42. 42. CADRE TRAINING | 42 SUSTAINABILITY MATRIX Financial The Financial section focuses on budget planning and the financial role of the cooperating organization Key indicators include:  Does the projected budget indicate sufficient revenue to fund the workplan?  Does the project Spending Plan (Budget) align with the implementation workplan line items (Activity Plan)?  Does the project budget indicate sufficient revenue to fund the sustainability activities beyond the end of the grant?
  43. 43. CADRE TRAINING | 43 SUSTAINABILITY MATRIX Community The Community section focuses on community involvement and buy-in for the project Key indicators include:  Was a community needs assessment completed? Were recommendations made?  Does the workplan include activities to create or promote local community involvement and ownership?  Do grant reports describe community relationships developed which support the continuation of project outcomes beyond the life of the grant?  Does the grant identify community champions willing to assume leadership roles for sustaining project outcomes after the life of the grant?
  44. 44. CADRE TRAINING | 44 SUSTAINABILITY MATRIX Beneficiary Organization The Beneficiary Organization section focuses on the project’s relationship with the beneficiary organization and the organization’s capacity to take ownership of the project Key indicators include:  Do written agreements define the terms of cooperation, roles and responsibilities of the beneficiary organizations?  Does the documentation identify personnel willing to assume leadership roles for sustaining project outcomes?  Does the local Club/District demonstrate strong partnerships with beneficiary organizations as appropriate to objectives?
  45. 45. CADRE TRAINING | 45 SUSTAINABILITY MATRIX Cooperating Organization The Cooperating Organization section focuses on the project management plan and structure of the relationship between the project sponsors and cooperating organization Key indicators include:  Does the documentation include analysis and/or justification of cooperating organization(s) selection based on demonstrated expertise or experience?  Does the design integrate into the cooperating organization(s) regular systems or operations?  Does the cooperating organization agree in writing to participate in program financial reviews?  Does the documentation include a succession plan to hand over the project to the partner organization(s) at the end of the grant lifecycle?
  46. 46. CADRE TRAINING | 46 SUSTAINABILITY MATRIX Training The Training section focuses on the training and follow up associated with the project Key indicators include:  Does the training plan address gaps identified in the assessment of beneficiaries targeted for training (doctors, teachers, unskilled labor)?  Are the training course design (delivery and materials) locally and culturally relevant (e.g., language, practical examples)?  Is there a measurement process to assess learner outcomes after training?  Is there a plan for keeping skills up-to-date?  Has the project resulted in demand for training outside the targeted beneficiary group (organization, individuals)?
  47. 47. CADRE TRAINING | 47 SUSTAINABILITY MATRIX SECTIONS Software/Equipment The Software/Equipment section assesses whether culturally appropriate technology was used during the project Key indicators include:  Does the documentation include a justification of why equipment is needed?  Does the proposed equipment respect local social and cultural norms?  Was the availability of equipment on the local market considered?  Is there an inventory management system?  Do user guidelines exist for equipment use?  Are local spare parts and replacement equipment available?  Is equipment being used as intended?
  48. 48. CADRE TRAINING | 48 Questions?
  49. 49. CADRE TRAINING | 49 Areas of Focus James Robinson
  50. 50. CADRE TRAINING | 50 AREA OF FOCUS EVALUATIONS – FUTURE VISION PILOT Global Grants: Sample of all Global Grants approved during pilot, 35 site visits across 5 areas of focus Nov 2013- Feb 2014 Nine countries on four continents Focused on VTT and Humanitarian Projects, not Scholarships 10 Cadre and 4 Area of Focus Managers
  51. 51. CADRE TRAINING | 51 Pilot Evaluation - Area of Focus Site Visits Korhan Atilla
  53. 53. CADRE TRAINING | 53 MATTERS OF SUSTAINABILITY 1- Financial Sustainability 2- Maintenance 3- Proper Operation 4- Precautions during installation 5- Rotarian Involvement 6- Benefitting Community Involvement
  54. 54. CADRE TRAINING | 54
  55. 55. CADRE TRAINING | 55
  56. 56. CADRE TRAINING | 56
  57. 57. CADRE TRAINING | 57
  58. 58. CADRE TRAINING | 58
  59. 59. CADRE TRAINING | 59
  60. 60. CADRE TRAINING | 60
  61. 61. CADRE TRAINING | 61
  62. 62. CADRE TRAINING | 62
  63. 63. CADRE TRAINING | 63
  64. 64. CADRE TRAINING | 64
  65. 65. CADRE TRAINING | 65
  66. 66. CADRE TRAINING | 66
  67. 67. CADRE TRAINING | 67
  68. 68. CADRE TRAINING | 68
  69. 69. CADRE TRAINING | 69
  70. 70. CADRE TRAINING | 70
  71. 71. CADRE TRAINING | 71
  72. 72. CADRE TRAINING | 72
  73. 73. CADRE TRAINING | 73
  74. 74. CADRE TRAINING | 74
  75. 75. CADRE TRAINING | 75
  76. 76. CADRE TRAINING | 76
  77. 77. CADRE TRAINING | 77
  78. 78. CADRE TRAINING | 78 Questions?