SlideShare a Scribd company logo
1 of 42
… leading development-focused research in Africa
Monitoring
and Evaluation
Fundamentals
BY: MANSURAT RAJI
FOR:PRESTON ASSOCIATES FOR INTERNATIONAL DEVELOPMENT
DATE: 10th MAY, 2019
Pretest Link
https://forms.gle/VgZ73yJj8d5h1tA67
Outline
01
02
03
04
05
Terminologies
Learning Objectives
Overview of M & E
Monitoring and Evaluation Plan
Conclusion
Outline
01
02
03
04
05
Terminologies
Learning Objectives
Overview of M & E
Mentoring and Evaluation Plan
Conclusion
Learning Objectives
At the end of this presentation, participants will be able to:
Identify the basic purposes and scope of M&E
Describe the functions of an M&E plan
Identify and understand the main components of an M&E plan
Outline
01
02
03
04
05
Terminologies
Learning Objectives
Overview of M & E
Monitoring and Evaluation Plan
Conclusion
Terminology
Monitoring: Monitoring is the routine process of data collection and measurement of
progress toward program objectives.
Evaluation: A process that attempts to determine as systematically and objectively as
possible the relevance, effectiveness, and impact of activities in light of their objectives.
Indicator: An indicator is a variable that measures one aspect of a program, project, or a
specific population, health, or environmental outcome.
Goal: A broad statement of a desired, long-term outcome of a program. Goals express
general program intentions and help guide a program's development.
Terminology (continued)
Objective: Significant development results that contribute to the achievement of goals
and provide a general framework for more detailed planning for specific programs.
Inputs: The financial, human, and material resources used in a program/intervention
Processes: The multiple activities, both planning and implementation, carried out to
achieve the program's objectives.
Outcomes: The changes measured at the population level in the program's target
population, some or all of which may be the result of a given program or
intervention.
Impact: The long-term, cumulative effect of programs/interventions over time on what
they ultimately aim to change,
Outline
01
02
03
04
05
Terminologies
Learning Objectives
Overview of M & E
Monitoring and Evaluation Plan
Conclusion
Introduction - Monitoring and Evaluation
The prevalence of undernutrition among children in Kano state is 40%, have you ever
wondered how this calculation was derived?
The percentage of adolescents in Edo state who have access to comprehensive SRH
services rose from 12% to 33%, do you wonder how they know this?
These types of statistics and other similar information result from "monitoring and
evaluation“ efforts.
M&E is the process by which data are collected and analyzed in order to provide
information to policy makers and others for use in program planning and project
management.
What is Monitoring?
Monitoring of a program or intervention involves
the collection of routine data that measure
progress toward achieving program objectives.
It is used to track changes in program
performance over time.
Its purpose is to allow stakeholders to make
informed decisions regarding the effectiveness of
programs and the efficient use of resources.
What is Monitoring? (continued)
Monitoring is sometimes referred to as process evaluation because it focuses on the
implementation process and asks key questions:
How well has the program been implemented?
How much does implementation vary from site to site?
Did the program benefit the intended people? At what cost?
What is Monitoring? (continued)
Examples of program elements that can be monitored:
◦ Supply inventories
◦ Number of vaccine doses administered monthly
◦ Quality of service
◦ Service coverage
◦ Patient outcomes (changes in behavior, morbidity, etc.)
What is Monitoring? (continued)
Monitoring:
is an ongoing, continuous process;
requires the collection of data at multiple
points throughout the program cycle,
including at the beginning to provide a
baseline;
can be used to determine if activities need
adjustment during the intervention to
improve desired outcomes.
What is Evaluation?
Evaluation measures how well the program
activities have met expected objectives
and/or the extent to which changes in
outcomes can be attributed to the program
or intervention.
Evaluation is fundamentally an exercise to
help decision-makers understand how, and
to what extent, a program is responsible for
particular, measured results.
What is Evaluation? (continued)
Evaluations require:
 Data collection at the start of a program
(to provide a baseline) and again at the
end.
 A control or comparison group in order
to measure whether the changes in
outcomes can be attributed to the
program
 A well-planned study design
Monitoring or Evaluation?
The National Primary Health Care
Development Agency wants to know if the
NSHIP program being carried out in Nassarawa
state is increasing access to healthcare among
women.
FMoH wants to know how many people have
been reached through the SMOL program.
A country director is interested in finding out
if the postnatal care provided at PHCs meets
national standards of quality.
Why is M & E Important?
Monitoring and evaluation helps program implementers:
 Make informed decisions regarding program operations and service delivery based on
objective evidence
 Ensure the most effective and efficient use of resources
 Objectively assess the extent to which the program is having or has had the desired impact, in
what areas it is effective, and where corrections need to be considered
 Meet organizational reporting and other requirements, and convince donors that their
investments have been worthwhile or that alternative approaches should be considered
Why is M & E Important? (continued)
Examples of questions that M&E can answer:
 Was the program implemented as planned?
 Did the target population benefit from the program and at what cost?
 Which program activities were more effective and which less effective?
When Should M&E Take Place?
M&E is a continuous process that occurs throughout the life of a program.
 To be most effective, M&E should be planned at the design stage of a program, with
the time, money, and personnel that will be required calculated and allocated in
advance.
 Monitoring should be conducted at every stage of the program, with data collected,
analyzed, and used on a continuous basis.
 Evaluations are usually conducted at the end of programs. However, they should be
planned for at the start because they rely on data collected throughout the program,
with baseline data being important.
Outline
01
02
03
04
05
Terminologies
Learning Objectives
Overview of M & E
Monitoring and Evaluation Plan
Conclusion
Monitoring and Evaluation Plan
 The M & E plan details a program's objectives, the interventions developed to achieve these
objectives, and describes the procedures that will be used to determine whether or not the
objectives are met.
 It shows how the expected results of a program relate to its goals and objectives
 Describes the data needed and how these data will be collected and analyzed
 How this information will be used
 The resources that will be needed
 And how the program will be accountable to stakeholders.
Why are M & E Plans Important?
M&E plans:
 State how a program will measure its achievements and therefore provide
accountability
 Document consensus and provide transparency
 Guide the implementation of M&E activities in a standardized and coordinated way
 Preserve institutional memory
M & E Plan Components
Typically, the components of an M&E plan include:
 The introduction
 The program description and framework
 A detailed description of the plan indicators
 The data collection plan
 A plan for monitoring
 A plan for evaluation
 A plan for the utilization of the information gained
 A mechanism for updating the plan
M & E Plan Components: Introduction
The introduction to the M&E plan should include:
 Information about the purpose of the program, the specific M&E activities that are
needed, and why they are important.
 A development history that provides information about the motivations of the
internal and external stakeholders and the extent of their interest, commitment, and
participation.
M & E Plan Components: Program
Description
The program description should include:
A problem statement that identifies the specific problem to be addressed. This provides
information about the situation that needs to be changed, who it affects, its causes, its
magnitude, and its impact on society.
The program goal and objectives:
• The goal is a broad statement about a desired long-term outcome of the program.
For example, improvement in the reproductive health of adolescents or a reduction in
unwanted pregnancies in Edo state.
M & E Components: Program Description
(contd.)
 Objectives are statements of desired specific and measurable program results.
Examples of objectives would be to reduce the maternal mortality ratio to less than
70/100,000 live births by year 2020 or to increase contraceptive prevalence over the
life of the program.
Descriptions of the specific interventions to be implemented and their duration, geographic
scope, and target population.
The list of resources needed, including financial, human, and those related to the infrastructure
(office space, equipment, and supplies)
Program Description: “SMART” Objectives
The objectives listed in the program description should be "SMART," an acronym that
stands for:
Specific: Is the desired outcome clearly specified?
Measurable: Can the achievement of the objective be quantified and measured?
Appropriate: Is the objective appropriately related to the program's goal?
Realistic: Can the objective realistically be achieved with the available resources?
Timely: In what time period will the objective be achieved?
Here is a sample objective. Do you think it is SMART ( i.e., meets all of the criteria
above)?
Increase contraceptive prevalence by 15% in women 30-49 years of age
M & E Plan Components: Indicators
Indicators are clues, signs or markers that measure one aspect of a program and
show how close a program is to its desired path and outcomes. They are used to
provide benchmarks for demonstrating the achievements of a program.
Examples of indicators include:
 Number of service providers who have been trained on the delivery of age and
gender sensitive sexual and reproductive health services.
 Proportion of states with disaggregated data on Adolescent Sexual and
Reproductive Health.
M & E Plan Components: Data Sources and
Data Collection Plan
Data sources are sources of information used
to collect the data needed to calculate the
indicators.
The data collection plan should include
diagrams describing the systems used for data
collection, processing, analysis, and reporting.
Examples of data sources include:
Patient records, birth registers, censuses,
household surveys.
M&E Plan Components: Monitoring Plan
The monitoring plan describes:
◦ Specific program components that will be monitored, such as provider
performance or the utilization of resources
◦ How this monitoring will be conducted
◦ The indicators that will be used to measure results
Because monitoring is concerned with the status of ongoing activities, output indicators,
also known as process indicators, are used. For example, these indicators might be:
 How many children visit a child health clinic in one month?
 How many of these children are vaccinated during these visits?
M&E Plan Components: Evaluation Plan
The evaluation plan provides the specific research
design and methodological approaches to be used
to identify whether changes in outcomes can be
attributed to the program.
For instance, if a program wants to test whether
quality of patient care can be improved by training
providers, the evaluation plan would identify a
research design that could be used to measure the
impact of such intervention.
M&E Plan Components: Information
Dissemination and Use
This involves how the information gathered will
be stored, disseminated, and used should be
defined at the planning stage of the project and
described in the M&E plan.
This will help ensure that findings from M&E
efforts are not wasted because they are not
shared.
The various users of this information should be
clearly defined, and the reports should be written
with specific audiences in mind.
M&E Plan Components: Implementation
and Mechanism for Update
The capacities needed to implement the
efforts described in the M&E plan should be
included in the document.
A mechanism for reviewing and updating the
M&E plan should also be included. This is
because changes in the program can and will
affect the original plans for both monitoring
and evaluation.
Standards for M&E Plans
M&E plans should serve the information needs of the intended users in practical ways.
These users can range from those assessing national program performance at the
highest central levels to those allocating resources at the district or local level.
M&E plans should convey technically accurate information and should be realistic,
prudent and diplomatic.
The activities described in M&E plans should be conducted legally, ethically, and with
regard to those involved in and affected by them.
Outline
01
02
03
04
05
Terminologies
Learning Objectives
Overview of M & E
Monitoring and Evaluation Plan
Conclusion
Conclusion
Did you know?
One rule of thumb is that 5-10% of a project budget should be allocated for M&E.
Therefore, Monitoring and Evaluation (M&E) is an essential component of any
intervention, project, or program.
Bibliography
Frankel. N, Gage. A: M&E Fundamentals: A Self-Guided Mini-Course
https://www.measureevaluation.org/resources/publications/ms-07-20-en-epub
 M&E Fundamentals https://www.globalhealthlearning.org/course/m-e-fundamentals
UNAIDS - Monitoring and Evaluation Terms (glossary)
http://files.unaids.org/en/media/unaids/contentassets/documents/document/2010/11_ME_Gl
ossary_FinalWorkingDraft.pdf
UNFPA strategic plan, 2018-2021 https://www.unfpa.org/sites/default/files/admin-
resource/UNFPA_strategic_plan_2018-2021._Annex_1_-
_Integrated_results_and_resources_framework_-_FINAL_-_24Jul17.pdf
Post Test Link
https://forms.gle/TpeKr56irxHASo8G7
@Preston_int

More Related Content

What's hot

Monitoring and Evaluation Framework
Monitoring and Evaluation FrameworkMonitoring and Evaluation Framework
Monitoring and Evaluation FrameworkMichelle Joja
 
M& e slide share
M& e   slide shareM& e   slide share
M& e slide shareSelf
 
Monitoring & evaluation presentation[1]
Monitoring & evaluation presentation[1]Monitoring & evaluation presentation[1]
Monitoring & evaluation presentation[1]skzarif
 
7 M&E: Indicators
7 M&E: Indicators7 M&E: Indicators
7 M&E: IndicatorsTony
 
6 M&E - Monitoring and Evaluation of Aid Projects
6 M&E - Monitoring and Evaluation of Aid Projects6 M&E - Monitoring and Evaluation of Aid Projects
6 M&E - Monitoring and Evaluation of Aid ProjectsTony
 
Capacity Development For Monitoring And Evaluation
Capacity Development For Monitoring And EvaluationCapacity Development For Monitoring And Evaluation
Capacity Development For Monitoring And EvaluationKnowledge Management Center
 
Monitoring and evaluation presentatios
Monitoring and evaluation presentatios Monitoring and evaluation presentatios
Monitoring and evaluation presentatios athanzeer
 
The Results Chain: 
 A logical model to achieve results
The Results Chain: 
 A logical model to achieve resultsThe Results Chain: 
 A logical model to achieve results
The Results Chain: 
 A logical model to achieve results7Geese
 
Workshop: Monitoring, evaluation and impact assessment
Workshop: Monitoring, evaluation and impact assessmentWorkshop: Monitoring, evaluation and impact assessment
Workshop: Monitoring, evaluation and impact assessmentWorldFish
 
Monitoring and evaluation
Monitoring and evaluationMonitoring and evaluation
Monitoring and evaluationmigom doley
 
TTIPEC: Monitoring and Evaluation (Session 2)
TTIPEC: Monitoring and Evaluation (Session 2)TTIPEC: Monitoring and Evaluation (Session 2)
TTIPEC: Monitoring and Evaluation (Session 2)Research to Action
 
Monitoring & Evalution ...... Orientation PPT
Monitoring & Evalution ...... Orientation PPT Monitoring & Evalution ...... Orientation PPT
Monitoring & Evalution ...... Orientation PPT Gus Prakash
 
Introduction to monitoring and evaluation
Introduction to monitoring and evaluationIntroduction to monitoring and evaluation
Introduction to monitoring and evaluationMeshack Lomoywara
 
Participatory Monitoring and Evaluation
Participatory Monitoring and EvaluationParticipatory Monitoring and Evaluation
Participatory Monitoring and EvaluationLen Fontanilla
 
Monitoring and evaluation
Monitoring and evaluationMonitoring and evaluation
Monitoring and evaluationMd Rifat Anam
 

What's hot (20)

Monitoring and Evaluation Framework
Monitoring and Evaluation FrameworkMonitoring and Evaluation Framework
Monitoring and Evaluation Framework
 
M& e slide share
M& e   slide shareM& e   slide share
M& e slide share
 
Monitoring & evaluation presentation[1]
Monitoring & evaluation presentation[1]Monitoring & evaluation presentation[1]
Monitoring & evaluation presentation[1]
 
7 M&E: Indicators
7 M&E: Indicators7 M&E: Indicators
7 M&E: Indicators
 
6 M&E - Monitoring and Evaluation of Aid Projects
6 M&E - Monitoring and Evaluation of Aid Projects6 M&E - Monitoring and Evaluation of Aid Projects
6 M&E - Monitoring and Evaluation of Aid Projects
 
Capacity Development For Monitoring And Evaluation
Capacity Development For Monitoring And EvaluationCapacity Development For Monitoring And Evaluation
Capacity Development For Monitoring And Evaluation
 
Monitoring and evaluation presentatios
Monitoring and evaluation presentatios Monitoring and evaluation presentatios
Monitoring and evaluation presentatios
 
The Results Chain: 
 A logical model to achieve results
The Results Chain: 
 A logical model to achieve resultsThe Results Chain: 
 A logical model to achieve results
The Results Chain: 
 A logical model to achieve results
 
Workshop: Monitoring, evaluation and impact assessment
Workshop: Monitoring, evaluation and impact assessmentWorkshop: Monitoring, evaluation and impact assessment
Workshop: Monitoring, evaluation and impact assessment
 
Components of a monitoring and evaluation system
Components of a monitoring and evaluation system  Components of a monitoring and evaluation system
Components of a monitoring and evaluation system
 
Monitoring and evaluation
Monitoring and evaluationMonitoring and evaluation
Monitoring and evaluation
 
Project Monitoring & Evaluation
Project Monitoring & EvaluationProject Monitoring & Evaluation
Project Monitoring & Evaluation
 
Logical framework
Logical frameworkLogical framework
Logical framework
 
Monitoring and Evaluation
Monitoring and Evaluation Monitoring and Evaluation
Monitoring and Evaluation
 
TTIPEC: Monitoring and Evaluation (Session 2)
TTIPEC: Monitoring and Evaluation (Session 2)TTIPEC: Monitoring and Evaluation (Session 2)
TTIPEC: Monitoring and Evaluation (Session 2)
 
Monitoring & Evalution ...... Orientation PPT
Monitoring & Evalution ...... Orientation PPT Monitoring & Evalution ...... Orientation PPT
Monitoring & Evalution ...... Orientation PPT
 
Introduction to monitoring and evaluation
Introduction to monitoring and evaluationIntroduction to monitoring and evaluation
Introduction to monitoring and evaluation
 
M&e system
M&e systemM&e system
M&e system
 
Participatory Monitoring and Evaluation
Participatory Monitoring and EvaluationParticipatory Monitoring and Evaluation
Participatory Monitoring and Evaluation
 
Monitoring and evaluation
Monitoring and evaluationMonitoring and evaluation
Monitoring and evaluation
 

Similar to M & E Fundamentals.

Monitoring and Evaluation Lesson 2.pptx
Monitoring and Evaluation Lesson 2.pptxMonitoring and Evaluation Lesson 2.pptx
Monitoring and Evaluation Lesson 2.pptxVallentine Okumu
 
Monitoring and Evaluation of Health Services
Monitoring and Evaluation of Health ServicesMonitoring and Evaluation of Health Services
Monitoring and Evaluation of Health ServicesNayyar Kazmi
 
Almm monitoring and evaluation tools draft[1]acm
Almm monitoring and evaluation tools draft[1]acmAlmm monitoring and evaluation tools draft[1]acm
Almm monitoring and evaluation tools draft[1]acmAlberto Mico
 
monitoring and evaluation
monitoring and evaluationmonitoring and evaluation
monitoring and evaluationTarek Sallam
 
A Good Program Can Improve Educational Outcomes.pdf
A Good Program Can Improve Educational Outcomes.pdfA Good Program Can Improve Educational Outcomes.pdf
A Good Program Can Improve Educational Outcomes.pdfnoblex1
 
Program Planning Priciples of programme planning.pptx
Program Planning  Priciples of programme planning.pptxProgram Planning  Priciples of programme planning.pptx
Program Planning Priciples of programme planning.pptxUsmanMaqsood32
 
National health program evaluation
National health program evaluationNational health program evaluation
National health program evaluationrahul gajbhiye
 
Monitoring and evaluation by Olashore Emmanuel
Monitoring and evaluation by Olashore EmmanuelMonitoring and evaluation by Olashore Emmanuel
Monitoring and evaluation by Olashore EmmanuelEmmanuel Olashore
 
COMMUNITY EVALUATION 2023.pptx
COMMUNITY  EVALUATION 2023.pptxCOMMUNITY  EVALUATION 2023.pptx
COMMUNITY EVALUATION 2023.pptxgggadiel
 
Principles, Steps of programme planning, evaluation and monitoring of program...
Principles, Steps of programme planning, evaluation and monitoring of program...Principles, Steps of programme planning, evaluation and monitoring of program...
Principles, Steps of programme planning, evaluation and monitoring of program...GBPUA&T, Pantnagar
 
Agricultural extension program goals and objectives .ppt
Agricultural extension program goals and objectives .pptAgricultural extension program goals and objectives .ppt
Agricultural extension program goals and objectives .pptAliIbrahim517066
 
Final Project PresentationMentoring ProgramGroup Number AB.docx
Final Project PresentationMentoring ProgramGroup Number AB.docxFinal Project PresentationMentoring ProgramGroup Number AB.docx
Final Project PresentationMentoring ProgramGroup Number AB.docxlmelaine
 
USER GUIDE M&E 2014 LENNY HIDAYAT
USER GUIDE M&E 2014 LENNY HIDAYATUSER GUIDE M&E 2014 LENNY HIDAYAT
USER GUIDE M&E 2014 LENNY HIDAYATLenny Hidayat
 
The role of Monitoring and Evaluation in Improving Public Policies – Challeng...
The role of Monitoring and Evaluation in Improving Public Policies – Challeng...The role of Monitoring and Evaluation in Improving Public Policies – Challeng...
The role of Monitoring and Evaluation in Improving Public Policies – Challeng...UNDP Policy Centre
 
Process of Planning and EvaluationThe process of planning and ev.docx
Process of Planning and EvaluationThe process of planning and ev.docxProcess of Planning and EvaluationThe process of planning and ev.docx
Process of Planning and EvaluationThe process of planning and ev.docxstilliegeorgiana
 

Similar to M & E Fundamentals. (20)

Monitoring and Evaluation Lesson 2.pptx
Monitoring and Evaluation Lesson 2.pptxMonitoring and Evaluation Lesson 2.pptx
Monitoring and Evaluation Lesson 2.pptx
 
Monitoring and Evaluation of Health Services
Monitoring and Evaluation of Health ServicesMonitoring and Evaluation of Health Services
Monitoring and Evaluation of Health Services
 
M&E Concepts.pptx
M&E Concepts.pptxM&E Concepts.pptx
M&E Concepts.pptx
 
Almm monitoring and evaluation tools draft[1]acm
Almm monitoring and evaluation tools draft[1]acmAlmm monitoring and evaluation tools draft[1]acm
Almm monitoring and evaluation tools draft[1]acm
 
monitoring and evaluation
monitoring and evaluationmonitoring and evaluation
monitoring and evaluation
 
A Good Program Can Improve Educational Outcomes.pdf
A Good Program Can Improve Educational Outcomes.pdfA Good Program Can Improve Educational Outcomes.pdf
A Good Program Can Improve Educational Outcomes.pdf
 
Program Planning Priciples of programme planning.pptx
Program Planning  Priciples of programme planning.pptxProgram Planning  Priciples of programme planning.pptx
Program Planning Priciples of programme planning.pptx
 
National health program evaluation
National health program evaluationNational health program evaluation
National health program evaluation
 
Monitoring and evaluation by Olashore Emmanuel
Monitoring and evaluation by Olashore EmmanuelMonitoring and evaluation by Olashore Emmanuel
Monitoring and evaluation by Olashore Emmanuel
 
Monitoring and evaluation
Monitoring and evaluationMonitoring and evaluation
Monitoring and evaluation
 
Wcms 546505
Wcms 546505Wcms 546505
Wcms 546505
 
COMMUNITY EVALUATION 2023.pptx
COMMUNITY  EVALUATION 2023.pptxCOMMUNITY  EVALUATION 2023.pptx
COMMUNITY EVALUATION 2023.pptx
 
Principles, Steps of programme planning, evaluation and monitoring of program...
Principles, Steps of programme planning, evaluation and monitoring of program...Principles, Steps of programme planning, evaluation and monitoring of program...
Principles, Steps of programme planning, evaluation and monitoring of program...
 
Agricultural extension program goals and objectives .ppt
Agricultural extension program goals and objectives .pptAgricultural extension program goals and objectives .ppt
Agricultural extension program goals and objectives .ppt
 
Final Project PresentationMentoring ProgramGroup Number AB.docx
Final Project PresentationMentoring ProgramGroup Number AB.docxFinal Project PresentationMentoring ProgramGroup Number AB.docx
Final Project PresentationMentoring ProgramGroup Number AB.docx
 
Curriculum monitoring
Curriculum monitoringCurriculum monitoring
Curriculum monitoring
 
USER GUIDE M&E 2014 LENNY HIDAYAT
USER GUIDE M&E 2014 LENNY HIDAYATUSER GUIDE M&E 2014 LENNY HIDAYAT
USER GUIDE M&E 2014 LENNY HIDAYAT
 
The role of Monitoring and Evaluation in Improving Public Policies – Challeng...
The role of Monitoring and Evaluation in Improving Public Policies – Challeng...The role of Monitoring and Evaluation in Improving Public Policies – Challeng...
The role of Monitoring and Evaluation in Improving Public Policies – Challeng...
 
Monitoring and evaluation for hiv
Monitoring and evaluation for hivMonitoring and evaluation for hiv
Monitoring and evaluation for hiv
 
Process of Planning and EvaluationThe process of planning and ev.docx
Process of Planning and EvaluationThe process of planning and ev.docxProcess of Planning and EvaluationThe process of planning and ev.docx
Process of Planning and EvaluationThe process of planning and ev.docx
 

Recently uploaded

RadioAdProWritingCinderellabyButleri.pdf
RadioAdProWritingCinderellabyButleri.pdfRadioAdProWritingCinderellabyButleri.pdf
RadioAdProWritingCinderellabyButleri.pdfgstagge
 
GA4 Without Cookies [Measure Camp AMS]
GA4 Without Cookies [Measure Camp AMS]GA4 Without Cookies [Measure Camp AMS]
GA4 Without Cookies [Measure Camp AMS]📊 Markus Baersch
 
INTERNSHIP ON PURBASHA COMPOSITE TEX LTD
INTERNSHIP ON PURBASHA COMPOSITE TEX LTDINTERNSHIP ON PURBASHA COMPOSITE TEX LTD
INTERNSHIP ON PURBASHA COMPOSITE TEX LTDRafezzaman
 
Amazon TQM (2) Amazon TQM (2)Amazon TQM (2).pptx
Amazon TQM (2) Amazon TQM (2)Amazon TQM (2).pptxAmazon TQM (2) Amazon TQM (2)Amazon TQM (2).pptx
Amazon TQM (2) Amazon TQM (2)Amazon TQM (2).pptxAbdelrhman abooda
 
Indian Call Girls in Abu Dhabi O5286O24O8 Call Girls in Abu Dhabi By Independ...
Indian Call Girls in Abu Dhabi O5286O24O8 Call Girls in Abu Dhabi By Independ...Indian Call Girls in Abu Dhabi O5286O24O8 Call Girls in Abu Dhabi By Independ...
Indian Call Girls in Abu Dhabi O5286O24O8 Call Girls in Abu Dhabi By Independ...dajasot375
 
Customer Service Analytics - Make Sense of All Your Data.pptx
Customer Service Analytics - Make Sense of All Your Data.pptxCustomer Service Analytics - Make Sense of All Your Data.pptx
Customer Service Analytics - Make Sense of All Your Data.pptxEmmanuel Dauda
 
How we prevented account sharing with MFA
How we prevented account sharing with MFAHow we prevented account sharing with MFA
How we prevented account sharing with MFAAndrei Kaleshka
 
专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改
专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改
专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改yuu sss
 
Predicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdfPredicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdfBoston Institute of Analytics
 
毕业文凭制作#回国入职#diploma#degree澳洲中央昆士兰大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
毕业文凭制作#回国入职#diploma#degree澳洲中央昆士兰大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree毕业文凭制作#回国入职#diploma#degree澳洲中央昆士兰大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
毕业文凭制作#回国入职#diploma#degree澳洲中央昆士兰大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degreeyuu sss
 
B2 Creative Industry Response Evaluation.docx
B2 Creative Industry Response Evaluation.docxB2 Creative Industry Response Evaluation.docx
B2 Creative Industry Response Evaluation.docxStephen266013
 
EMERCE - 2024 - AMSTERDAM - CROSS-PLATFORM TRACKING WITH GOOGLE ANALYTICS.pptx
EMERCE - 2024 - AMSTERDAM - CROSS-PLATFORM  TRACKING WITH GOOGLE ANALYTICS.pptxEMERCE - 2024 - AMSTERDAM - CROSS-PLATFORM  TRACKING WITH GOOGLE ANALYTICS.pptx
EMERCE - 2024 - AMSTERDAM - CROSS-PLATFORM TRACKING WITH GOOGLE ANALYTICS.pptxthyngster
 
RS 9000 Call In girls Dwarka Mor (DELHI)⇛9711147426🔝Delhi
RS 9000 Call In girls Dwarka Mor (DELHI)⇛9711147426🔝DelhiRS 9000 Call In girls Dwarka Mor (DELHI)⇛9711147426🔝Delhi
RS 9000 Call In girls Dwarka Mor (DELHI)⇛9711147426🔝Delhijennyeacort
 
Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024
Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024
Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024thyngster
 
Building on a FAIRly Strong Foundation to Connect Academic Research to Transl...
Building on a FAIRly Strong Foundation to Connect Academic Research to Transl...Building on a FAIRly Strong Foundation to Connect Academic Research to Transl...
Building on a FAIRly Strong Foundation to Connect Academic Research to Transl...Jack DiGiovanna
 
科罗拉多大学波尔得分校毕业证学位证成绩单-可办理
科罗拉多大学波尔得分校毕业证学位证成绩单-可办理科罗拉多大学波尔得分校毕业证学位证成绩单-可办理
科罗拉多大学波尔得分校毕业证学位证成绩单-可办理e4aez8ss
 
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档208367051
 
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...Boston Institute of Analytics
 
办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一
办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一
办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一F La
 
办理(Vancouver毕业证书)加拿大温哥华岛大学毕业证成绩单原版一比一
办理(Vancouver毕业证书)加拿大温哥华岛大学毕业证成绩单原版一比一办理(Vancouver毕业证书)加拿大温哥华岛大学毕业证成绩单原版一比一
办理(Vancouver毕业证书)加拿大温哥华岛大学毕业证成绩单原版一比一F La
 

Recently uploaded (20)

RadioAdProWritingCinderellabyButleri.pdf
RadioAdProWritingCinderellabyButleri.pdfRadioAdProWritingCinderellabyButleri.pdf
RadioAdProWritingCinderellabyButleri.pdf
 
GA4 Without Cookies [Measure Camp AMS]
GA4 Without Cookies [Measure Camp AMS]GA4 Without Cookies [Measure Camp AMS]
GA4 Without Cookies [Measure Camp AMS]
 
INTERNSHIP ON PURBASHA COMPOSITE TEX LTD
INTERNSHIP ON PURBASHA COMPOSITE TEX LTDINTERNSHIP ON PURBASHA COMPOSITE TEX LTD
INTERNSHIP ON PURBASHA COMPOSITE TEX LTD
 
Amazon TQM (2) Amazon TQM (2)Amazon TQM (2).pptx
Amazon TQM (2) Amazon TQM (2)Amazon TQM (2).pptxAmazon TQM (2) Amazon TQM (2)Amazon TQM (2).pptx
Amazon TQM (2) Amazon TQM (2)Amazon TQM (2).pptx
 
Indian Call Girls in Abu Dhabi O5286O24O8 Call Girls in Abu Dhabi By Independ...
Indian Call Girls in Abu Dhabi O5286O24O8 Call Girls in Abu Dhabi By Independ...Indian Call Girls in Abu Dhabi O5286O24O8 Call Girls in Abu Dhabi By Independ...
Indian Call Girls in Abu Dhabi O5286O24O8 Call Girls in Abu Dhabi By Independ...
 
Customer Service Analytics - Make Sense of All Your Data.pptx
Customer Service Analytics - Make Sense of All Your Data.pptxCustomer Service Analytics - Make Sense of All Your Data.pptx
Customer Service Analytics - Make Sense of All Your Data.pptx
 
How we prevented account sharing with MFA
How we prevented account sharing with MFAHow we prevented account sharing with MFA
How we prevented account sharing with MFA
 
专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改
专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改
专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改
 
Predicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdfPredicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdf
 
毕业文凭制作#回国入职#diploma#degree澳洲中央昆士兰大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
毕业文凭制作#回国入职#diploma#degree澳洲中央昆士兰大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree毕业文凭制作#回国入职#diploma#degree澳洲中央昆士兰大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
毕业文凭制作#回国入职#diploma#degree澳洲中央昆士兰大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
 
B2 Creative Industry Response Evaluation.docx
B2 Creative Industry Response Evaluation.docxB2 Creative Industry Response Evaluation.docx
B2 Creative Industry Response Evaluation.docx
 
EMERCE - 2024 - AMSTERDAM - CROSS-PLATFORM TRACKING WITH GOOGLE ANALYTICS.pptx
EMERCE - 2024 - AMSTERDAM - CROSS-PLATFORM  TRACKING WITH GOOGLE ANALYTICS.pptxEMERCE - 2024 - AMSTERDAM - CROSS-PLATFORM  TRACKING WITH GOOGLE ANALYTICS.pptx
EMERCE - 2024 - AMSTERDAM - CROSS-PLATFORM TRACKING WITH GOOGLE ANALYTICS.pptx
 
RS 9000 Call In girls Dwarka Mor (DELHI)⇛9711147426🔝Delhi
RS 9000 Call In girls Dwarka Mor (DELHI)⇛9711147426🔝DelhiRS 9000 Call In girls Dwarka Mor (DELHI)⇛9711147426🔝Delhi
RS 9000 Call In girls Dwarka Mor (DELHI)⇛9711147426🔝Delhi
 
Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024
Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024
Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024
 
Building on a FAIRly Strong Foundation to Connect Academic Research to Transl...
Building on a FAIRly Strong Foundation to Connect Academic Research to Transl...Building on a FAIRly Strong Foundation to Connect Academic Research to Transl...
Building on a FAIRly Strong Foundation to Connect Academic Research to Transl...
 
科罗拉多大学波尔得分校毕业证学位证成绩单-可办理
科罗拉多大学波尔得分校毕业证学位证成绩单-可办理科罗拉多大学波尔得分校毕业证学位证成绩单-可办理
科罗拉多大学波尔得分校毕业证学位证成绩单-可办理
 
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档
 
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...
 
办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一
办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一
办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一
 
办理(Vancouver毕业证书)加拿大温哥华岛大学毕业证成绩单原版一比一
办理(Vancouver毕业证书)加拿大温哥华岛大学毕业证成绩单原版一比一办理(Vancouver毕业证书)加拿大温哥华岛大学毕业证成绩单原版一比一
办理(Vancouver毕业证书)加拿大温哥华岛大学毕业证成绩单原版一比一
 

M & E Fundamentals.

  • 1. … leading development-focused research in Africa
  • 2. Monitoring and Evaluation Fundamentals BY: MANSURAT RAJI FOR:PRESTON ASSOCIATES FOR INTERNATIONAL DEVELOPMENT DATE: 10th MAY, 2019
  • 4. Outline 01 02 03 04 05 Terminologies Learning Objectives Overview of M & E Monitoring and Evaluation Plan Conclusion
  • 5. Outline 01 02 03 04 05 Terminologies Learning Objectives Overview of M & E Mentoring and Evaluation Plan Conclusion
  • 6. Learning Objectives At the end of this presentation, participants will be able to: Identify the basic purposes and scope of M&E Describe the functions of an M&E plan Identify and understand the main components of an M&E plan
  • 7. Outline 01 02 03 04 05 Terminologies Learning Objectives Overview of M & E Monitoring and Evaluation Plan Conclusion
  • 8. Terminology Monitoring: Monitoring is the routine process of data collection and measurement of progress toward program objectives. Evaluation: A process that attempts to determine as systematically and objectively as possible the relevance, effectiveness, and impact of activities in light of their objectives. Indicator: An indicator is a variable that measures one aspect of a program, project, or a specific population, health, or environmental outcome. Goal: A broad statement of a desired, long-term outcome of a program. Goals express general program intentions and help guide a program's development.
  • 9. Terminology (continued) Objective: Significant development results that contribute to the achievement of goals and provide a general framework for more detailed planning for specific programs. Inputs: The financial, human, and material resources used in a program/intervention Processes: The multiple activities, both planning and implementation, carried out to achieve the program's objectives. Outcomes: The changes measured at the population level in the program's target population, some or all of which may be the result of a given program or intervention. Impact: The long-term, cumulative effect of programs/interventions over time on what they ultimately aim to change,
  • 10. Outline 01 02 03 04 05 Terminologies Learning Objectives Overview of M & E Monitoring and Evaluation Plan Conclusion
  • 11. Introduction - Monitoring and Evaluation The prevalence of undernutrition among children in Kano state is 40%, have you ever wondered how this calculation was derived? The percentage of adolescents in Edo state who have access to comprehensive SRH services rose from 12% to 33%, do you wonder how they know this? These types of statistics and other similar information result from "monitoring and evaluation“ efforts. M&E is the process by which data are collected and analyzed in order to provide information to policy makers and others for use in program planning and project management.
  • 12. What is Monitoring? Monitoring of a program or intervention involves the collection of routine data that measure progress toward achieving program objectives. It is used to track changes in program performance over time. Its purpose is to allow stakeholders to make informed decisions regarding the effectiveness of programs and the efficient use of resources.
  • 13. What is Monitoring? (continued) Monitoring is sometimes referred to as process evaluation because it focuses on the implementation process and asks key questions: How well has the program been implemented? How much does implementation vary from site to site? Did the program benefit the intended people? At what cost?
  • 14. What is Monitoring? (continued) Examples of program elements that can be monitored: ◦ Supply inventories ◦ Number of vaccine doses administered monthly ◦ Quality of service ◦ Service coverage ◦ Patient outcomes (changes in behavior, morbidity, etc.)
  • 15. What is Monitoring? (continued) Monitoring: is an ongoing, continuous process; requires the collection of data at multiple points throughout the program cycle, including at the beginning to provide a baseline; can be used to determine if activities need adjustment during the intervention to improve desired outcomes.
  • 16. What is Evaluation? Evaluation measures how well the program activities have met expected objectives and/or the extent to which changes in outcomes can be attributed to the program or intervention. Evaluation is fundamentally an exercise to help decision-makers understand how, and to what extent, a program is responsible for particular, measured results.
  • 17. What is Evaluation? (continued) Evaluations require:  Data collection at the start of a program (to provide a baseline) and again at the end.  A control or comparison group in order to measure whether the changes in outcomes can be attributed to the program  A well-planned study design
  • 18. Monitoring or Evaluation? The National Primary Health Care Development Agency wants to know if the NSHIP program being carried out in Nassarawa state is increasing access to healthcare among women. FMoH wants to know how many people have been reached through the SMOL program. A country director is interested in finding out if the postnatal care provided at PHCs meets national standards of quality.
  • 19. Why is M & E Important? Monitoring and evaluation helps program implementers:  Make informed decisions regarding program operations and service delivery based on objective evidence  Ensure the most effective and efficient use of resources  Objectively assess the extent to which the program is having or has had the desired impact, in what areas it is effective, and where corrections need to be considered  Meet organizational reporting and other requirements, and convince donors that their investments have been worthwhile or that alternative approaches should be considered
  • 20. Why is M & E Important? (continued) Examples of questions that M&E can answer:  Was the program implemented as planned?  Did the target population benefit from the program and at what cost?  Which program activities were more effective and which less effective?
  • 21. When Should M&E Take Place? M&E is a continuous process that occurs throughout the life of a program.  To be most effective, M&E should be planned at the design stage of a program, with the time, money, and personnel that will be required calculated and allocated in advance.  Monitoring should be conducted at every stage of the program, with data collected, analyzed, and used on a continuous basis.  Evaluations are usually conducted at the end of programs. However, they should be planned for at the start because they rely on data collected throughout the program, with baseline data being important.
  • 22. Outline 01 02 03 04 05 Terminologies Learning Objectives Overview of M & E Monitoring and Evaluation Plan Conclusion
  • 23. Monitoring and Evaluation Plan  The M & E plan details a program's objectives, the interventions developed to achieve these objectives, and describes the procedures that will be used to determine whether or not the objectives are met.  It shows how the expected results of a program relate to its goals and objectives  Describes the data needed and how these data will be collected and analyzed  How this information will be used  The resources that will be needed  And how the program will be accountable to stakeholders.
  • 24. Why are M & E Plans Important? M&E plans:  State how a program will measure its achievements and therefore provide accountability  Document consensus and provide transparency  Guide the implementation of M&E activities in a standardized and coordinated way  Preserve institutional memory
  • 25. M & E Plan Components Typically, the components of an M&E plan include:  The introduction  The program description and framework  A detailed description of the plan indicators  The data collection plan  A plan for monitoring  A plan for evaluation  A plan for the utilization of the information gained  A mechanism for updating the plan
  • 26. M & E Plan Components: Introduction The introduction to the M&E plan should include:  Information about the purpose of the program, the specific M&E activities that are needed, and why they are important.  A development history that provides information about the motivations of the internal and external stakeholders and the extent of their interest, commitment, and participation.
  • 27. M & E Plan Components: Program Description The program description should include: A problem statement that identifies the specific problem to be addressed. This provides information about the situation that needs to be changed, who it affects, its causes, its magnitude, and its impact on society. The program goal and objectives: • The goal is a broad statement about a desired long-term outcome of the program. For example, improvement in the reproductive health of adolescents or a reduction in unwanted pregnancies in Edo state.
  • 28. M & E Components: Program Description (contd.)  Objectives are statements of desired specific and measurable program results. Examples of objectives would be to reduce the maternal mortality ratio to less than 70/100,000 live births by year 2020 or to increase contraceptive prevalence over the life of the program. Descriptions of the specific interventions to be implemented and their duration, geographic scope, and target population. The list of resources needed, including financial, human, and those related to the infrastructure (office space, equipment, and supplies)
  • 29. Program Description: “SMART” Objectives The objectives listed in the program description should be "SMART," an acronym that stands for: Specific: Is the desired outcome clearly specified? Measurable: Can the achievement of the objective be quantified and measured? Appropriate: Is the objective appropriately related to the program's goal? Realistic: Can the objective realistically be achieved with the available resources? Timely: In what time period will the objective be achieved? Here is a sample objective. Do you think it is SMART ( i.e., meets all of the criteria above)? Increase contraceptive prevalence by 15% in women 30-49 years of age
  • 30. M & E Plan Components: Indicators Indicators are clues, signs or markers that measure one aspect of a program and show how close a program is to its desired path and outcomes. They are used to provide benchmarks for demonstrating the achievements of a program. Examples of indicators include:  Number of service providers who have been trained on the delivery of age and gender sensitive sexual and reproductive health services.  Proportion of states with disaggregated data on Adolescent Sexual and Reproductive Health.
  • 31. M & E Plan Components: Data Sources and Data Collection Plan Data sources are sources of information used to collect the data needed to calculate the indicators. The data collection plan should include diagrams describing the systems used for data collection, processing, analysis, and reporting. Examples of data sources include: Patient records, birth registers, censuses, household surveys.
  • 32. M&E Plan Components: Monitoring Plan The monitoring plan describes: ◦ Specific program components that will be monitored, such as provider performance or the utilization of resources ◦ How this monitoring will be conducted ◦ The indicators that will be used to measure results Because monitoring is concerned with the status of ongoing activities, output indicators, also known as process indicators, are used. For example, these indicators might be:  How many children visit a child health clinic in one month?  How many of these children are vaccinated during these visits?
  • 33. M&E Plan Components: Evaluation Plan The evaluation plan provides the specific research design and methodological approaches to be used to identify whether changes in outcomes can be attributed to the program. For instance, if a program wants to test whether quality of patient care can be improved by training providers, the evaluation plan would identify a research design that could be used to measure the impact of such intervention.
  • 34. M&E Plan Components: Information Dissemination and Use This involves how the information gathered will be stored, disseminated, and used should be defined at the planning stage of the project and described in the M&E plan. This will help ensure that findings from M&E efforts are not wasted because they are not shared. The various users of this information should be clearly defined, and the reports should be written with specific audiences in mind.
  • 35. M&E Plan Components: Implementation and Mechanism for Update The capacities needed to implement the efforts described in the M&E plan should be included in the document. A mechanism for reviewing and updating the M&E plan should also be included. This is because changes in the program can and will affect the original plans for both monitoring and evaluation.
  • 36. Standards for M&E Plans M&E plans should serve the information needs of the intended users in practical ways. These users can range from those assessing national program performance at the highest central levels to those allocating resources at the district or local level. M&E plans should convey technically accurate information and should be realistic, prudent and diplomatic. The activities described in M&E plans should be conducted legally, ethically, and with regard to those involved in and affected by them.
  • 37. Outline 01 02 03 04 05 Terminologies Learning Objectives Overview of M & E Monitoring and Evaluation Plan Conclusion
  • 38. Conclusion Did you know? One rule of thumb is that 5-10% of a project budget should be allocated for M&E. Therefore, Monitoring and Evaluation (M&E) is an essential component of any intervention, project, or program.
  • 39. Bibliography Frankel. N, Gage. A: M&E Fundamentals: A Self-Guided Mini-Course https://www.measureevaluation.org/resources/publications/ms-07-20-en-epub  M&E Fundamentals https://www.globalhealthlearning.org/course/m-e-fundamentals UNAIDS - Monitoring and Evaluation Terms (glossary) http://files.unaids.org/en/media/unaids/contentassets/documents/document/2010/11_ME_Gl ossary_FinalWorkingDraft.pdf UNFPA strategic plan, 2018-2021 https://www.unfpa.org/sites/default/files/admin- resource/UNFPA_strategic_plan_2018-2021._Annex_1_- _Integrated_results_and_resources_framework_-_FINAL_-_24Jul17.pdf
  • 40.