Self-Assessment of Organizational Capacity in Monitoring & Evaluation

3,955 views

Published on

Presentation that captures self-assessments of two teams of Ethiopian health officers (most of whom have M&E responsibilities): those from SNNP Regional Health Bureau and those from the Sidama Zonal Health Department.

Self-Assessment of Organizational Capacity in Monitoring & Evaluation

  1. 1. SELF-ASSESSMENT OF ORGANIZATIONALCAPACITY IN MONITORING & EVALUATION by SNNP REGIONAL HEALTH BUREAU SIDAMA ZONAL HEALTH DEPARTMENT PRESENTATION of FINDINGS 05 December 2012
  2. 2. Proposed Time Schedule for Discussion10 Minutes Introductions PowerPoint Presentations20 Minutes Terms & Concepts on: * Capacity Building * Organizational Development * SCORE ME org. self-assessment tool of M&E capacity30 Minutes SNNP Regional Health Bureau Presentation of Findings from use of SCORE ME30 Minutes Sidama Zonal Health Department Presentation of Findings from use of SCORE ME30 Minutes Questions, Comments, Next Steps
  3. 3. DEFINITIONS OF CAPACITY and CAPACITY BUILDING CAPACITY: Set of abilities and resources to support a level of desired performance – whether by an individual or an organization of any size CAPACITY BUILDING: Processes that improve the ability of a person, group, organization, or system to meet objectives and/or to perform better
  4. 4. CAPACITY BUILDING: SOMETHING TO THINK ABOUT• Common approach is to provide training, funding, material resources, and technical assistance• Results would be more effective with consideration of other factors influencing performance of an individual or organization  behavioral  cultural  systemic
  5. 5. MEASURE EVALUATION’S FOCUS in CAPACITY BUILDINGSupport the ability of its partners to independently andsustainably carry out activities in monitoring and evaluation(M&E) and routine health information systems (RHIS)Phase I (1998 – 2003) Phase III (2009 – 2014)• Design of new tools ** Knowledge management• Data analysis ** Organization developmentPhase II (2004 – 2009)• Routine health information systems strengthening• Data demand & use• Data quality• M&E technical training
  6. 6. FOCAL AREAS for CAPACITY BUILDING in M&EOperational Functions: • Management and development of human resources in RHIS and M&E • Material and financial resources, and systems to manage them • Technical RHIS and M&E skills • Communication and facilitation of use of information
  7. 7. FOCAL AREAS for CAPACITY BUILDING in M&ELeadership and Relationship Building: • Promotion of culture favoring information use • Strategic leadership • Management of stakeholder relations
  8. 8. MEASURE EVALUATION’s CONCEPT of ORGANIZATION DEVELOPMENT (OD) to SUPPORT M&EOD Goal: Improve organization’s effectiveness in managing its own efforts to achieving objectivesStrategy: Motivate individuals and groups within an organization to seek and sustain best possible performance
  9. 9. MEASURE EVALUATION’s CONCEPT ofORGANIZATION DEVELOPMENT (OD) to SUPPORT M&EMethod — Assist Organization to: • Identify individual and institutional needs for obtaining desired results and levels of achievement • Identify its current strengths and challenges • Increase capacity of those at management and leadership positions to support and champion performance improvement and positive responsiveness to change • Improve M&E performance of organization to regularly assess current strategies, structures, processes, achievements, results
  10. 10. MEASURE EVALUATION’s CONCEPT ofORGANIZATION DEVELOPMENT (OD) to SUPPORT M&ENeeds of Organization:• Clarity of organizational vision• Comprehensive strategic planning• Effective organizational structure• Strong governance structure• Strong capacity among leaders for promoting teamwork and managing change• Strong accountability systems (human and financial resource management, inventory control)
  11. 11. MEASURE EVALUATION’s CONCEPT of ORGANIZATION DEVELOPMENT (OD) to SUPPORT M&EResults — Organization is able to Self-Assess and Improve its Accountability Systems for:1. Planning and executing operational functions and achieving pre-set objectives2. Actively undertaking problem solving3. Committing to its own organizational development as ongoing process4. Learning and sharing knowledge and perceptions with organizations in and across sectors5. Assessing and accessing means for remaining operationally sustainable and relevant to future needs and demands
  12. 12. MEASURE EVALUATION’s CONCEPT ofORGANIZATION DEVELOPMENT (OD) to SUPPORT M&EOutcome — Improved organizational management andperformance in: • Use of M&E methods and tools • Collection and analysis of routine and strategic information • Use of routine information for program M&E • Use of strategic information for organizational planning • Evidence-based decision making by organization’s leadership and management • Sustainability of all the above performance areas
  13. 13. Self-Assessment ofCapacity regardingOrganization’sResponse inM&E SCORE ME
  14. 14. Organization’s Functional Capacities for Self-Assessment in M&E Performance• Strategic Planning• Budgeting and Managing Finances• Implementation Planning• Documenting & Reporting of Work Activities & Results for Monitoring Oversight• Documenting & Reporting of Work Activities & Results for Evaluation Assessment• Managing Facilities and Physical Assets• Managing and Developing Staff• Promoting Continuous Quality Improvement
  15. 15. Features of SCORE ME: Tool for Self-Assessment in M&E Performance Developed as a self-administered tool for an M&E Department/Unit of any agency (government/non- governmental) to appraise its own level of functional capacity in eight areas Designed so that its use on a periodic basis would promote M&E skills and competencies Self-assessment in each functional area determined by number of elements by which agency M&E performance meets conditions (with reference made if supporting evidence is available or not)
  16. 16. Features of SCORE ME (cont’d) Agency can customize the use and wording of M&E performance elements to better suit its purpose for self-assessment in M&E capacity Agency can select those functional areas of greater interest on which to first focus its capacity building efforts Agency can develop plans for M&E capacity building that are based on a formalized pre-assessment of its M&E performance
  17. 17. Introduction of SCORE ME to Health Sectors in SNNP Region• Review of content and use of SCORE ME instrument• Organizational and individual self-assessments of M&E capacity, with on-site technical assistance, as required• Formulation of improvement plan for organizational capacity in M&E, as based on self-assessment findings
  18. 18. SNNP REGIONAL HEALTH BUREAU FUNCTIONAL PROCESSES• CORE Processes – Disease Prevention Services and Good Health Practices Promotion – Curative Health Services – Regulatory Oversight – Public Health Emergency Response Services – HIV/AIDS Preventive Health Services and Disease Control• SUPPORTING Processes – Health Program Planning and Health Program M&E – Disease Diagnoses through Laboratory Services – Medical Research – Human Resources Management – Financial Administration
  19. 19. Memorable Quotes "Measurement is the first step that leads to control and eventually to improvement. If you cant measure something, you cant understand it. If you cant understand it, you cant control it. If you cant control it, you cant improve it." — H. James Harrington "What gets measured gets done, what gets measured and fed back gets done well, what gets rewarded gets repeated." — John E. Jones "Without a standard there is no logical basis for making a decision or taking action." — Joseph M. Juran "Nothing changes until you change; everything changes once you change." — posted at Sidama Zonal Health Department
  20. 20. SCORE ME Self-Assessment of M&E Capacity SNNP Regional Health Bureau 28-29 November 2012
  21. 21. Organization’s Functional Capacities for Self-Assessment in M&E PerformanceA Strategic PlanningB Budgeting and Managing FinancesC Implementation PlanningD Documenting/Reporting Work Activities/Results for Monitoring OversightE Documenting/Reporting Work Activities/Results for Evaluation AssessmentF Managing Facilities and Physical AssetsG Managing and Developing StaffH Promoting Continuous Quality ImprovementSNNP Regional Health Bureau
  22. 22. D Documenting/Reporting Work Activities/Results for Monitoring Oversight Elements of Capacity Condition Building/Organizational Met? Evidence Available?Development Performance in If “Yes”, Specify M&E by Organization/Unit Check  Unit has written policies / procedures and docu- mented evidence for monitoring on a quarterly or more frequent basis: We have: written business process re-engineering documents; facility standard documents; expenditure reports;1. Work inputs  woreda-based plans; balanced score card; standard reporting format; standard check-list; routine performance review; database generated summary data; program-based reporting Information, Education, Communication/ Behavior Change Communication design, distribution, dissemination; integrated2. Work activities  supportive supervision feedback; quarterly HMIS review feedback to zonesSNNP Regional Health Bureau
  23. 23. D Documenting/Reporting Work Activities/Results for Monitoring Oversight Elements of Capacity Condition Building / Organizational Met? Evidence Available?Development Performance in If “Yes”, Specify M&E by Organization / Unit Check  Review guidelines; process maturity assessment; standard integrated supportive supervision; standard guidelines for3. Work processes  reviewing, licensing, and inspecting health facilities; different service delivery guidelines; laboratory external quality assurance Quarterly feedback on program-specific activities; monthly, quarterly, semi-annual, and annual reports (physical and4. Work outputs  financial); payroll reports; attendance sheets; various models on in-and-out assets; epidemic monitoring charts; expanded program for immunizations monitoring chartsSNNP Regional Health Bureau
  24. 24. D Documenting/Reporting for Monitoring Oversight Unit has written Condition policies/procedures and Met? Evidence Available? documented evidence for If “Yes”, Specify responsiveness to Check  ensuring:5. Collection of all  required work data6. Timely collection and transmission of all  required work data See the abovelisted justifications. Performance review team, guidelines, reporting time frame,7. High quality of all  LQAS needs further improvement required work data8. Analysis of all X required work dataSNNP Regional Health Bureau
  25. 25. D Documenting/Reporting for Monitoring Oversight Unit has written Condition policies/procedures and Met? Evidence Available? documented evidence for If “Yes”, Specify responsiveness to ensuring: Check  Board minutes book for hospitals; guidelines and manuals for HIV/AIDS; prevention board and council Management9. Unit’s leadership/Board  standards for woreda health offices, zonal health Department, and Regional Health Bureau Management standards and management minutes book for:10. Unit’s management  hospitals, zonal health department, district health offices and health centers Procedures for fund allocations, fund liquidation; project11. Unit’s funders and management manual; MOA, standard reporting, formal  supporters (financial and physical); standard checklist for project appraisal12. Unit’s target population Community-facility forum guidelines; human development  army guidelines representativesSNNP Regional Health Bureau
  26. 26. D Documenting/Reporting for Monitoring Oversight Unit has written Condition policies/procedures and Met? Evidence Available? documented evidence for If “Yes”, Specify regular review and use of Check  work data findings by: Management standards13. Unit’s leadership/Board  Board minutes book Management standards14. Unit’s management  Management minutes bookSNNP Regional Health Bureau
  27. 27. D Documenting/Reporting for Monitoring Oversight Scoring from Self-Assessment FindingsTotal Number of 13 / 4 or 5 or 6 7 or 8 9 or 10 or 12 or Conditions Met  14 fewer 11 more 1 2 3 4 5SNNP Regional Health Bureau
  28. 28. D Documenting/Reporting for Monitoring Oversight Corrective Action Plan based on Self-Assessment Findings MEANS forEle- DELIVERABLE(S) RESPONSIBLE PROJECTED CURRENT ACTION(S) TO MONITORINGment that can be PARTY for COMPLETN SITUATION TAKE PROGRESS to # DOCUMENTED DELIVERABLE(S) DATE DELIVERABLE(S) No procedure or Availability of Develop Planning, M&E guidelines Procedures, document at each By end of 8 for analysis procedures or guidelines level (of health care Process Owner May 2013 guidelines and partners of all administration) required data SNNP Regional Health Bureau
  29. 29. E Documenting/Reporting Work Activities/Results for Evaluation Assessment Unit has written policies/procedures and documented evidence for having Condition evaluated for work effectiveness and Met? Evidence Available? efficiency the projections that had If “Yes”, Specify been set in Annual Budgets and Check  Implementation Plans: Assignment of personnel based on BPR1. Budgeting and deploying inputs  document; allocated funds (budgeting) Process maturity evaluation of evaluation for-2. Undertaking work processes  mat; BPR & BSC implementation manuals; planning, monitoring, and evaluation processes3. Completing planned activities X --4. Generating projected outputs X --5. Meeting pre-set objectives X --6. Achieving desired outcomes  Operationalization of health research process7. Contributing to desired impact  -- SNNP Regional Health Bureau
  30. 30. E Documenting/Reporting for Evaluation Assessment Condition Unit has documented evidence for Met? Evidence Available? allocation of budget and If “Yes”, Specify other resources for: Check 8. Collection of all required work Resource mobilization for HMIS; woreda-based  planning for research data and information9. Quality assessments of work X -- data10. Quality assessments of work X -- performance11. Analyses of work data X --SNNP Regional Health Bureau
  31. 31. E Documenting/Reporting for Evaluation Assessment Condition Unit has documented evidence for Met? Evidence Available? allocation of budget and If “Yes”, Specify other resources for: Check 12. Reporting on work data to Monthly, quarterly, semi-annual and annual  reports concerned stakeholders13. Using the findings from reports for the improvement Annual performance used for evidence-based  planning and written feedback of planning and implementation Availability of research staff to do research and14.Conducting research on ways technology transfer sub-process and to promote work  disseminate research results; rapid effectiveness and efficiency assessment conducted on data utilization for decision-makingSNNP Regional Health Bureau
  32. 32. E Documenting/Reporting for Evaluation Assessment Scoring from Self-Assessment FindingsTotal Number of 8/ 4 or 5 or 6 7 or 8 9 or 10 or 12 or Conditions Met  14 fewer 11 more 1 2 3 4 5SNNP Regional Health Bureau
  33. 33. E Documenting/Reporting for Evaluation Assessment Corrective Action Plan based on Self-Assessment Findings MEANS forEle- DELIVERABLE(S) RESPONSIBLE PROJECTED CURRENT ACTION(S) TO MONITORINGment that can be PARTY for COMPLETN SITUATION TAKE PROGRESS to # DOCUMENTED DELIVERABLE(S) DATE DELIVERABLE(S) Amount of budget mobilized Financial reports Inadequate Amount of allocation for Revise Response letters budget from partners SNNP RHB quality resource allocated allocations Reports on assessments, MEASURE June 1 performance Resource quality Evaluation 2013 Develop assessments reviews, and mapping resource and performance Partners analysis of document mapping reviews developed work data conducted Reports on work data analysis SNNP Regional Health Bureau
  34. 34. E Documenting/Reporting for Evaluation Assessment Corrective Action Plan based on Self-Assessment Findings MEANS forEle- DELIVERABLE(S) RESPONSIBLE PROJECTED CURRENT ACTION(S) TO MONITORINGment that can be PARTY for COMPLETN SITUATION TAKE PROGRESS to # DOCUMENTED DELIVERABLE(S) DATE DELIVERABLE(S) Terms of reference Skill gap in Capacity End of developing Number of trained Selection of building training January evaluation M&E Officers and trainees for basic MEASURE 2013 other focal frame work for concepts of Evaluation (training) 2 persons Attendance lists evaluation evaluation for activities, out- M&E Officers SNNP RHB May 2013 M&E framework Training documents puts, & pre-set and other focal M&E developed persons framework objectives Pre- & post-test evaluations SNNP Regional Health Bureau
  35. 35. NEXT PRESENTATION
  36. 36. SCORE ME Self-Assessment of M&E Capacity Sidama Zonal Health Department 01 December 2012
  37. 37. Organization’s Functional Capacities for Self-Assessment in M&E PerformanceA Strategic PlanningB Budgeting and Managing FinancesC Implementation PlanningD Documenting/Reporting Work Activities/Results for Monitoring OversightE Documenting/Reporting Work Activities/Results for Evaluation AssessmentF Managing Facilities and Physical AssetsG Managing and Developing StaffH Promoting Continuous Quality ImprovementSidama Zonal Health Department
  38. 38. D Documenting/Reporting Work Activities/Results for Monitoring Oversight Condition Elements of Capacity Building / Met? Evidence Available?Organizational Development Performance If “Yes”, Specify in M&E by Organization / Unit Check Unit has written policies/procedures and documented evidence for monitoring on a quarterly or more frequent basis:  Strategic plans1. Work inputs   Woreda-based plans  Standard guidelines for different2. Work activities  programs (TB, HIV, RH, TB/HIV collaborative activities)  Performance reports3. Work processes   Business process re-engineering  Balanced score card   Standard checklists4. Work outputs  E-HMIS Summary reportsSidama Zonal Health Department
  39. 39. D Documenting/Reporting for Monitoring Oversight ConditionUnit has written policies/procedures and Met? Evidence Available? documented evidence for If “Yes”, Specify responsiveness to ensuring: Check 5. Collection of all required work  data6. Timely collection and  HMIS tools and guidelines transmission of all required work   Standard reporting schedules  HMIS minutes book data  Performance Review Team minutes book7. High quality of all required work  Analysis of routine performance X data reports from respective woredas, as well as within the Organization (unit)8. Analysis of all required work data Sidama Zonal Health Department
  40. 40. D Documenting/Reporting for Monitoring Oversight ConditionUnit has written policies/procedures and Met? Evidence Available? documented evidence for If “Yes”, Specify responsiveness to ensuring: Check 9. Unit’s leadership/Board   Management minutes book  Management standards10. Unit’s management   Regular monthly/quarterly11. Unit’s funders and supporters  reporting, including review meetings with concerned bodies12. Unit’s target population  Contractual agreements with  partners representatives  Standard checklists13. Unit’s leadership/Board   Management minutes book  Management standards14. Unit’s management Sidama Zonal Health Department
  41. 41. D Documenting/Reporting for Monitoring Oversight Scoring from Self-Assessment Findings Total Number of 13 / 4 or 5 or 6 7 or 8 9 or 10 or 12 or Conditions Met  14 fewer 11 more 1 2 3 4 5Sidama Zonal Health Department
  42. 42. D Documenting/Reporting for Monitoring Oversight Corrective Action Plan based on Self-Assessment Findings MEANS forEle- DELIVERABLE(S) RESPONSIBLE PROJECTED CURRENT MONITORINGment ACTION(S) TO TAKE that can be PARTY for COMPLETN SITUATION PROGRESS to # DOCUMENTED DELIVERABLE(S) DATE DELIVERABLE(S)  Criteria based  Capacity selection of building on  Number of trainees and data manage- staff trained  MEASURE  Jan 2013 trainers ment on evaluation Evaluation (capacity  Training  Appropriate framework  SNNP RHB building) manuals Low technology for  Number of  Sidama ZHD  Jan 2013  Pre- and post- data manage- offline HMIS  All partners (techno- 7 quality ment (software data tests working in logy) data  Available data analysis management health  April technology for at zonal as centers programs in 2013 offline data well as woreda  Established Sidama (integra- management levels) One Report Zone tion)  Established  Data System One Report integration System Sidama Zonal Health Department
  43. 43. E Documenting/Reporting Work Activities/Results for Evaluation Assessment Unit has written policies/procedures and Condition documented evidence for having evaluated Met? Evidence Available? for work effectiveness and efficiency If “Yes”, Specify the projections that had been set in Annual Check  Budgets and Implementation Plans:1. Budgeting and deploying inputs X --2. Undertaking work processes  Performance Report3. Completing planned activities X --4. Generating projected outputs X --5. Meeting pre-set objectives X --6. Achieving desired outcomes X --7. Contributing to desired impact X -- Sidama Zonal Health Department
  44. 44. E Documenting/Reporting for Evaluation Assessment Condition Unit has documented evidence for Met? Evidence Available? allocation of budget and If “Yes”, Specify other resources for: Check 8. Collection of all required work   Performance reports data and information9. Quality assessments of work X -- data10. Quality assessments of work X -- performance  Analysis of data from performance reports11. Analyses of work data   Analysis of data from project evaluationsSidama Zonal Health Department
  45. 45. E Documenting/Reporting for Evaluation Assessment Condition Unit has documented evidence for Met? Evidence Available? allocation of budget and If “Yes”, Specify other resources for: Check   Dissemination to community representatives12. Reporting on work data to   Dissemination to partners concerned stakeholders  Dissemination/communication within the Organization13. Using the findings from  Reviewing performance reports by management teams reports for the improvement   Reviewing performance reports by of planning and core-process/supporting process implementation teams14.Conducting research on ways to promote work X -- effectiveness and efficiencySidama Zonal Health Department
  46. 46. E Documenting/Reporting for Evaluation Assessment Scoring from Self-Assessment FindingsTotal Number of 5/ 4 or 5 or 6 7 or 8 9 or 10 or 12 or Conditions Met  14 fewer 11 more 1 2 3 4 5Sidama Zonal Health Department
  47. 47. E Documenting/Reporting for Evaluation Assessment Corrective Action Plan based on Self-Assessment Findings MEANS forEle- DELIVERABLE(S) RESPONSIBLE PROJECTED CURRENT ACTION(S) TO MONITORINGment that can be PARTY for COMPLETN SITUATION TAKE PROGRESS to # DOCUMENTED DELIVERABLE(S) DATE DELIVERABLE(S)  Feedback  MEASURE following Evaluation budget  SNNP RHB Resource  Resource  Amount of approval  Sidama ZHD  Jan 1 (financial) (financial) budget  Amount of  All partners 2013 scarcity scarcity allocated budget really working in spent for health evaluation programs in purposes Sidama Zone Sidama Zonal Health Department
  48. 48. E Documenting/Reporting for Evaluation Assessment Corrective Action Plan based on Self-Assessment Findings MEANS forEle- DELIVERABLE(S) RESPONSIBLE PROJECTED CURRENT MONITORINGment ACTION(S) TO TAKE that can be PARTY for COMPLETN SITUATION PROGRESS to # DOCUMENTED DELIVERABLE(S) DATE DELIVERABLE(S)  Criteria based  MEASURE selection of  Institutionalizing Evaluation  Documented trainees and an evaluation  SNNP RHB Failure culture within the evaluation trainers  Sidama to plan plans  Training Organization ZHD  Number of manuals  Jan 3 for  Capacity  All partners staff trained  Pre- and post- 2013 evalu- building in working in on tests planning and health ations evaluation  Cross- implementation programs in frameworks checking the of evaluation Sidama plan Zone documented Sidama Zonal Health Department
  49. 49. Questions or Comments?
  50. 50. Next Steps
  51. 51. Thank You!
  52. 52. MEASURE Evaluation is funded by the U.S. Agency forInternational Development (USAID) and implemented by theCarolina Population Center at the University of North Carolinaat Chapel Hill in partnership with Futures Group International,ICF International, John Snow, Inc., Management Sciences forHealth, and Tulane University. Views expressed in thispresentation do not necessarily reflect the views of USAID or theU.S. government.MEASURE Evaluation is the USAID Global Health Bureausprimary vehicle for supporting improvements in monitoring andevaluation in population, health and nutrition worldwide.

×