Project monitoring and evaluation by Samuel Obino Mokaya


Published on

Project monitoring and evaluation.

Published in: Technology, Business

Project monitoring and evaluation by Samuel Obino Mokaya

  1. 1. PROJECT MANAGEMENT TRAINING SERIES Project Monitoring & Evaluation By Samuel Obino Mokaya Jomo Kenyatta University of Agriculture & Technology P. O. Box 62000-00200, Nairobi, Kenya Tel: 0722845562 / 0734615008 E-mail: [email_address] / o’ September 2009
  2. 2. Introduction
  3. 5. <ul><li>OUTPUTS </li></ul><ul><li>Studies completed </li></ul><ul><li>People trained </li></ul><ul><li>OUTCOMES </li></ul><ul><li>Income increased </li></ul><ul><li>Jobs created </li></ul><ul><li>INPUTS </li></ul><ul><li>Experts </li></ul><ul><li>Equipment </li></ul><ul><li>Funds </li></ul><ul><li>IMPACT </li></ul><ul><li>Health improved </li></ul><ul><li>Longevity increased </li></ul>The M and E Results Chain <ul><li>The most commonly used terms in monitoring and evaluation are: Feedback and Lesson learned </li></ul><ul><li>Feedback: A process through information and knowledge is disseminated and used to assess overall progress towards results or confirm the achievement of results </li></ul><ul><li>Lesson learned: An instructive example based on experience that is applicable to a situation – learning from experience. </li></ul>
  4. 6. Why M and E
  5. 8. Project Monitoring <ul><li>Monitoring is the continuous day-to-day assessment of project implementation in relation to agreed schedules, and of the use of inputs, infrastructure, and services by project beneficiaries. </li></ul><ul><li>Provides managers & other stakeholders with continuous feedback on implementation. </li></ul><ul><li>Identifies actual or potential successes & problems as early as possible to facilitate timely adjustments to project operation. </li></ul><ul><li>It is performed while a project is being implemented, with the aim of improving the project design & functioning while in action. </li></ul>
  6. 9. <ul><li>Designed to provide constant feedback on the progress of a project, the problems it is facing, and the efficiency with which it is being implemented. </li></ul><ul><li>Monitoring information is collected at specific times - daily, monthly or quarterly. </li></ul><ul><li>At some point this information needs to be collated, brought together and analyzed so that it can answer questions such as: </li></ul><ul><ul><li>How well are we doing? </li></ul></ul><ul><ul><li>Are we doing the right things? </li></ul></ul><ul><ul><li>What difference are we making? </li></ul></ul><ul><ul><li>Does the approach need to be modified, and if so how? </li></ul></ul>
  7. 10. <ul><li>It can indicate as early as possible any shortcoming with regard to delivery of inputs, execution of activities, or production of outputs in order that corrective action can be undertaken in time. </li></ul><ul><li>In summary monitoring addresses issues such as: </li></ul><ul><ul><ul><li>Procurement </li></ul></ul></ul><ul><ul><ul><li>Delivery of inputs </li></ul></ul></ul><ul><ul><ul><li>Schedules of activities </li></ul></ul></ul><ul><ul><ul><li>Progress made in Production of outputs </li></ul></ul></ul>
  8. 11. Principles for Monitoring
  9. 12. <ul><li>In terms of scope, monitoring involves: </li></ul><ul><ul><li>Progress towards outcomes. </li></ul></ul><ul><ul><li>Factors contributing to or impending achievement of the outcome. </li></ul></ul><ul><ul><li>The organization’s contribution to the outcome through outputs. </li></ul></ul><ul><ul><li>The partnership strategy. </li></ul></ul>
  10. 13. Monitoring Tools <ul><li>Project managers have a challenge of determining the correct mix of monitoring tools & approaches for each project. </li></ul><ul><li>They include; reporting & analysis, validation and participation: </li></ul>Reporting & analysis Validation Participation <ul><li>Annual project reports </li></ul><ul><li>Progress&/or quarterly reports </li></ul><ul><li>Project delivery reports </li></ul><ul><li>Substantive project documentation </li></ul><ul><li>Field visits </li></ul><ul><li>Spot-check visits </li></ul><ul><li>External assessments / monitoring </li></ul><ul><li>Client surveys </li></ul><ul><li>Evaluations </li></ul><ul><li>Outcome groups </li></ul><ul><li>Steering committee </li></ul><ul><li>Stakeholder meetings </li></ul><ul><li>Focus group meetings </li></ul><ul><li>Annual review </li></ul>
  11. 14. Project Evaluation <ul><li>Evaluation is the periodic assessment of a project's relevance, performance, efficiency, & impact (both expected and unexpected) in relation to stated objectives: </li></ul><ul><li>Studies the outcome of a project i.e. changes in income, housing quality, benefits distribution, cost-effectiveness, etc. </li></ul><ul><ul><ul><li>with the aim of informing the design of future projects. </li></ul></ul></ul><ul><li>In summary, evaluation is carried out to: </li></ul><ul><ul><li>Determine the EFFECTIVESS : Did the project achieve its planned purpose? </li></ul></ul><ul><ul><li>Determine SIGNIFICANCE : Did the project make a substantial contribution to development? </li></ul></ul><ul><ul><li>Measure EFFICIENCY : Did we achieve a satisfactory cost/benefit ratio; could we have accomplished our purpose at a lower cost? </li></ul></ul><ul><ul><li>Learn LESSON : Which can be applied to similar activities elsewhere. </li></ul></ul>
  12. 15. Types of Evaluation
  13. 16. Evaluation Process
  14. 17. Designing Terms of Reference
  15. 18. <ul><li>Monitoring and Evaluation systems can be an effective way to: </li></ul><ul><ul><li>Provide constant feedback on the extent to which the projects are achieving their goals. </li></ul></ul><ul><ul><li>Identify potential problems at an early stage & propose possible solutions. </li></ul></ul><ul><ul><li>Monitor the accessibility of the project to all sectors of the target beneficiaries. </li></ul></ul><ul><ul><li>Monitor the efficiency with which different components of the project are being implemented and suggest improvements. </li></ul></ul><ul><ul><li>Evaluate the extent to which the project is able to achieve its general objectives. </li></ul></ul><ul><ul><li>Provide guidelines for the planning of future projects. </li></ul></ul>
  16. 19. Monitoring Vs Evaluation Monitoring Evaluation <ul><li>Keeps track of daily activities </li></ul><ul><li>Accepts rules, policies and procedures </li></ul><ul><li>Work towards targets </li></ul><ul><li>Stresses the conversion of inputs to outputs </li></ul><ul><li>Concentrates on planned project elements </li></ul><ul><li>Reports progress </li></ul><ul><li>Takes a long range view </li></ul><ul><li>Questions pertinence of policies, rules and procedures </li></ul><ul><li>Measures progress and asks whether targets are adequate </li></ul><ul><li>Emphasizes achievement of purpose </li></ul><ul><li>Assesses planned elements & looks for unplanned changes, searches for causes & challenges assumptions </li></ul><ul><li>Records lessons learnt </li></ul>
  17. 20. Components of Good M & E Design
  18. 21. M & E Indicators <ul><ul><li>Measures of performance </li></ul></ul><ul><ul><li>Signs that something tangible has happened </li></ul></ul><ul><ul><li>Standards of measurement </li></ul></ul><ul><ul><li>They tell us “when we get there” </li></ul></ul><ul><li>Note that the goal of Good Project Management Practice (GPMP) is to deliver every component of the project to the required quality, within the agreed budget and on time. </li></ul><ul><li>Input indicators </li></ul><ul><li>Describe the material, financial, technological and human resources invested in a product, service, or program. Examples might include staff time, supplies and equipment, funds, or in-kind contributions to an intervention </li></ul>
  19. 22. <ul><li>Process indicators </li></ul><ul><li>Describe activities carried out to achieve the desired results or objectives of an intervention; they show what is done and how well it is done. Examples of process indicators include training sessions or educational programs prepared and presented to transfer knowledge or skills. </li></ul><ul><li>Output indicators </li></ul><ul><li>Describe the immediate results of an intervention </li></ul><ul><li>Jobs created </li></ul><ul><li>No of people trained </li></ul><ul><li>No of condoms dispensed </li></ul><ul><li>No of toilets constructed </li></ul>
  20. 23. Methods of Data Collection <ul><li>Interviews with beneficiaries and other stakeholders </li></ul><ul><li>Direct observation of situations / events as they happen </li></ul><ul><li>Key informant interviews: interviewing leaders or specialists in a subject </li></ul><ul><li>Questionnaires: written questions used to get written responses </li></ul><ul><li>Focus group discussions: a group of about 6 to 12 people interviewed together on a specific subject </li></ul><ul><li>Community meetings: a gathering of a fairly large number of community members (beneficiaries) to whom questions, problems and situation are put so as to elicit their responses </li></ul><ul><li>Fieldwork reports: structured report forms that are used for recording observations by the field workers </li></ul>
  21. 24. Designing an M and E Tool <ul><li>Entails formulation of a workable system design and planning operational and resource implications: </li></ul><ul><ul><li>What are the norms against which to compare M & E results to project performance and outcomes? </li></ul></ul><ul><ul><li>What information has to be obtained in order to make decisions on the project? </li></ul></ul><ul><ul><li>How is information to be collected? </li></ul></ul><ul><ul><li>How should the data be analyzed? </li></ul></ul><ul><ul><li>To whom should the M & E findings be reported? </li></ul></ul><ul><ul><li>Who will be responsible for carrying out the M & E? </li></ul></ul><ul><ul><li>How much should be invested in M and E? </li></ul></ul><ul><li>Most common M & E instruments are questionnaire, interviews, observation and document analysis. </li></ul>
  22. 25.   Method Description/ Purpose Advantages Disadvantages/ Challenges Literature search Gather background information on methods and results of evaluation methods used by others. Economic and efficient way of obtaining information. Difficult to assess validity and reliability of secondary data. Questionnaires / surveys Oral interviews or written questionnaires of a representative sample of respondents. Most appropriate when need to quickly and/or easily get lots of information from people in a non-threatening way. * Produce reliable information. * Can be completed anonymously. * Easy to compare and analyze. * Can be administered easily to a large number of people. * Collect a lot of data in an organized manner. * Many sample questionnaires already exist. * Demanding and could be costly. * Might not get careful feedback. * Wording can bias client's responses. * Data is analyzed for groups and are impersonal. * Surveys may need sampling expert. * Provide numbers but do not get the full story. * Open-ended data may be difficult to analyze. Interviews To fully understand someone's impressions or experiences, or learn more about their answers to questionnaires. Individual or group interviews could be organized to assess perceptions, views and satisfaction of beneficiaries. * Give full range and depth of information and yield rich data, details and new insights. * Can be flexible with the client. * Permit face-to-face contact with respondents and provide opportunity to explore topics in depth. * Allow interviewer to probe, explain or help clarify questions, increasing the likelihood of useful responses. * Allow interviewer to be flexible in administering interview to particular individuals or circumstances. * Can be hard to analyze and compare. * Interviewer can bias responses. * Can be expensive and time-consuming. * Need well-qualified and highly trained interviewers. * Interviewee may distort information through recall errors, selective perceptions and desire to please interviewer. * Flexibility can result in inconsistencies across interviews. * Volume of information too large and may be difficult to reduce data. Documentation review Impression of how program operates without interrupting the program by review of applications, finances, memos, minutes, etc. * Give comprehensive and historical information * Doesn't interrupt program or client's routine in program * Information already exists. * Few biases about information. * Often takes a lot of time * Information may be incomplete. Quality of documentation might be poor. * Need to be clear about purpose. * Not a flexible means to get data. Data restricted to what already exists.
  23. 26. Observation Involves inspection, field visits and observation to understand processes, infrastructure/services and their utilization. Gathers accurate information about how a program actually operates, particularly about processes. * Well-suited for understanding processes, views, operations of a program while they are actually occurring. * Can adapt to events as they occur and exist in natural, unstructured and flexible setting. * Provides direct information about behavior of individuals and groups. * Permits evaluator to enter into and understand situation/ context. * Provides good opportunities for identifying unanticipated outcomes. * Dependent on observer ’ s understanding and interpretation. * Has limited potential for generalization. * Can be difficult to interpret exhibited behaviors. * Can be complex to categorize observations. * Can influence behavior of program participants. * Can be expensive and time-consuming. * Needs well-qualified, highly trained observers and/or content experts. * Investigator has little control over situation. Focus groups A focus group brings together a representative group of 8 to 10 people, who are asked a series of questions related to the task at hand. Used for analysis of specific, complex problems, in order to identify attitudes and priorities in sample groups. Explore a topic in depth through group discussion, e.g., about reactions to an experience or suggestion, understanding common complaints, etc. * Efficient and reasonable in terms of cost. * Stimulate the generation of new ideas. * Quickly and reliably gets common impressions * Can be an efficient way to get a wide range and depth of information in a short time. * Can convey key information about programs. * Useful in project design and in assessing the impact of a project on a given set of stakeholders. * Can be hard to analyze responses.* Need good facilitators. * Difficult to schedule 8-10 people together. Case studies In-depth review of one or a small number of selected cases. To fully understand or depict beneficiaries ’ experiences in a program, and conduct comprehensive examination through cross comparison of cases. * Well-suited for understanding processes and for formulating hypotheses to be tested later. * Fully depicts client's experience in program input, process and results. * Powerful means to portray program to outsiders. * Usually time consuming to collect, organize and describe. * Represents depth of information, rather than breadth. Key informant interviews Interviews with persons who are knowledgeable about the community targeted by the project. A key informant is a person (or group) who has unique skills or professional background related to the issue/intervention being evaluated, is knowledgeable about the project participants and/or has access to other information of interest to the evaluator. * Flexible, in-depth approach. * Easy to implement. * Provides information concerning causes, reasons and/or best approaches from an &quot;insider&quot; point of view. * Advice/feedback increases credibility of study. * May have side benefit to solidify relationships between evaluators, beneficiaries and other stakeholders. * Risk of biased presentation/ interpretation from informants/interviewer. * Time required to select and get commitment may be substantial. * Relationship between evaluator and informants may influence type of data obtained. * Informants may interject own biases and impressions. Direct measurement Registration of quantifiable or classifiable data by means of an analytical instrument. * Precise. * Reliable and often requiring few resources. Registers only facts, not explanations.
  24. 27. <ul><li>Executive Summary </li></ul><ul><li>Brief terms of reference </li></ul><ul><li>Main findings </li></ul><ul><li>Conclusions and recommendations </li></ul><ul><li>Lessons learnt </li></ul><ul><li>Introduction </li></ul><ul><li>Project description </li></ul><ul><li>Purpose of evaluation </li></ul><ul><li>Who requested for the evaluation </li></ul><ul><li>Terms of Reference </li></ul><ul><li>Outline of main tasks </li></ul><ul><li>Duration of assignment </li></ul><ul><li>Submission of draft and final reports </li></ul>M and E Report
  25. 28. <ul><li>The M and E Team </li></ul><ul><li>Names of team members </li></ul><ul><li>Qualifications and experiences </li></ul><ul><li>Roles to be played </li></ul><ul><li>Methodology Used </li></ul><ul><li>How M and E was carried out – methods and justification </li></ul><ul><li>Period to be taken for each methodology chosen </li></ul><ul><li>Limitations of the methodology used </li></ul><ul><li>How problems were solved and authenticity of the report </li></ul><ul><li>Unresolved issues and their implications </li></ul><ul><li>Main Findings </li></ul><ul><li>List Findings </li></ul><ul><li>Collate and process data </li></ul><ul><li>Analyze and interpret findings </li></ul>
  26. 29. <ul><li>Conclusions and Recommendations </li></ul><ul><li>Major conclusions and justifications </li></ul><ul><li>Recommendations to improve future projects </li></ul><ul><li>Lessons Learnt </li></ul><ul><li>What worked well that should be repeated in future projects </li></ul><ul><li>What did not work well that should be avoided or minimized in future projects </li></ul><ul><li>Learning experiences to make future projects more successful </li></ul><ul><li>Annexes </li></ul><ul><li>Detailed terms of references </li></ul><ul><li>Brief resumes for team members </li></ul><ul><li>Photos of sample products / outputs </li></ul>
  27. 30. Conclusion <ul><li>Both monitoring & evaluation are crucial components in effective project management </li></ul><ul><li>Project management is about information gathered & required at different stages of the project </li></ul><ul><li>Expertise & clear understanding of the project is crucial </li></ul><ul><li>The focus is a process & system that yields useful & reliable information for decision making. </li></ul><ul><li>THANK YOU </li></ul><ul><li>Samuel Obino Mokaya </li></ul><ul><li>Jomo Kenyatta University of Agriculture & Technology/, KENYA </li></ul><ul><li>Tel: 0722845562 / 0734615008 </li></ul><ul><li>E-mail: [email_address] / o’ </li></ul>