This presentation is the continuation of the first part, which was all about the basics of program evaluation. This ppt contains slides describing the impact evaluation in details and also the logical framework is also explained with practical examples. N.B: Please go through it, using slide view to use the animation effects.

1. Program Evaluation - 2
Dr Sourav Goswami
MGIMS, Sevagram
1st
July 2017

2. Framework
Definition
Logic Model for Program Evaluation
How to do an impact evaluation?
Choosing evaluation methods
References

3. What is Program Evaluation?
“A systematic method for collecting,
analyzing and using information to answer
questions about projects, policies and
programs, particularly about their
effectiveness and efficiency” - CDC

4. Logic Model for Prog Evaluation
INPUT
ACTIVITIES/
PROCESS OUTPUT
OUTCOME
Program
Investmen
t
What we
invest
What
We do
Participation
Who we reach
under what
condition
Short
term
Intermediate Long
term
What results

5. Logic Model for Prog Evaluation
SITUATION
ASSUMPTIONS EXTERNAL FACTORS

6. Uses of Logic Models
1. Program design and planning
2. Program implementation
3. Program evaluation and strategic reporting

7. Understanding LOGIC MODEL
Define SITUATION
1. What is the problem?
2. What causes the problem?
3. Who is affected by this problem ( i.e “the clients”)?
4. Who cares about whether or not this problem is
solved (i.e “the key stakeholders”) ?
5. What does the existing research and experience tell
us about how to solve this problem?
In program design and planning
stages, “Community Needs
Assessment” generally tells the
stakeholders about the SITUATION:
1. Target population
2. Extent of problem
3. Location
4. Key stakeholders
5. Existing research and experience

8. Understanding LOGIC MODEL
 Understanding the situation (EXAMPLE):
1. What is the problem?
- Parents are reported to the child welfare system for
suspected child abuse and/or neglect.
2. What causes the problem?
Not enough knowledge, skills or support
3. Who is affected by this problem ( i.e “the clients”)?
Racially/ ethnically diverse families with school age children
4. Who cares about whether or not this problem is solved (i.e “the
key stakeholders”) ?
Child welfare system, Funders, Families
5. What does the existing research and experience tell us about how
to solve this problem?
Positive Parenting Program & ongoing use of resources

9. Now, we know the SITUATION !!
So, lets put a simple question,
“ What would happen if this problem no longer existed?”
Or in other words,
What is the change that the program intended to
have in the macro level?

11. Identify the Long term Outcome
SITUATION:
Parents are
reported to
the Child
Welfare
System for
Suspected
child abuse
and/or
neglect.
What
would
happen if this probl
no longer exist ??
What
would
happen if this probl
no longer exist ??
Long Term
Outcome
Parents
consistently
parent safely
Children are
Free from
Abuse and/
Neglect

12. Identify the Intermediate Outcome
What
would
happen in order to
Long term Outcome
to happen ??
What
would
happen in order to
Long term Outcome
to happen ??
Long Term
Outcome
Parents
consistently
parent safely
Children are
Free from
Abuse and/
Neglect
Intermediate
Outcome
Parents are
using Positive
Parenting
Behavior
Parents
Access
Community
resources

13. Identify the Short Term Outcome
What
ould
appen for
nterme-diate
Outcome to
appen ??
What
ould
appen for
nterme-diate
utcome to
appen ??
Short Term
Outcome
Parents
Increase
Their
Knowledge
About
Positive
parenting
Parents
Increase
Their
Knowledge
About
Community
resources

14. I used “BACKWARD MAPPING” process to
demonstrate these 3 outcome i.e. I started with Long
term Outcome and move backward
We can also do a “FORWARD MAPPING” , where
we start with OUTPUTS or Short Term Outcome and
move forward.
# The important thing is to demonstrate the
“logical relationship” between the categories with “if
_______ then______ statement”

15. Logic Model for Parent Training Program
Output
Participation:
Parents are
Involved in
Community
Work, attend
Parenting
Classes
Activities/
Process
Input
Services:
Deliver
8 weeks
Of
Positive
Parenting
Classes
Funding
Identi-
fication
Of
Positive
Parenting
curriclum
Staff training
Convenient
Location for Class
Referral system for
parents

16. Logic Model for Parent Training Program
SITUATION
Parents are
reported to
the Child
Welfare System
for Suspected
child abuse
and/or neglect.
External factors
Larger circumstances that are likely
To affect the program, but over which
The program has little control
Assumptions
Conditions/resources that
Are needed for the success
Of the program

17. CAUTION !
!!
CAUTION !
!!
Logic Models aren't the real world
BUT
All programs have a “ Theory of Change”
(whether it is stated or not !!)

18. Types of Evaluation
Process (Formative) Evaluation
Outcome ( Summative ) Evaluation
Impact Evaluation
Economic Evaluation

19. What is an Impact Evaluation (IM)
IE measures POSITIVE/NEGATIVE effects
IE measures INTENDED AND UN-INTENDED
effects that are there ( or mentioned) in the
goals of the evaluation
IE always includes an analysis of WHAT
CAUSED the observed effects?

20. How do we plan an Impact Evaluation?
 Lets say, we like to evaluate a program that reduces infant mortality in
Melghat area of Maharashtra
 Now, it calls for 5 important questions that need to be answered before
we begin IE. Those are:
1. Is IE the best way to answer the question about the program?
2. What are the key evaluation questions?
3. What is the “ Theory of Change” ?
4. How to procure and build the “evaluation team”?
5. Who will be involved in making decisions and how will the
evaluation process be managed?

21. STEP1: Is ‘Impact Evaluation’ required?
FIRST answer the 3 questions:
1. Is there a clear idea that how the findings from IE will
be used?
2. Who will use it?
3. Given the resources and the time available, will the
findings be credible and relevant?
SECOND: Timing for conducting IE

22. STEP1: Is ‘Impact Evaluation’ required?
 FIRST answer the 3 questions:
1. Is there a clear idea that how the findings from IE will be used?
2. Who will use it?
3. Given the resources and the time available, will the findings be
credible and relevant?
 Now, we need to decide the PURPOSE OF THE
EVALUATION, i.e. whether it is Formative or Summative.
✔
✔
✔

23. STEP 2: Key Evaluation Questions
An IE generally answers 3 types of questions:
1. DESCRIPTIVE
2. CAUSAL
3. EVALUATIVE
WHAT HAPPENED?
What was the context?
What was done by the intervention?
What are the changes incurred?
WHAT PRODUCED/CONTRIBUTED
TO THESE CHANGES?
1.ATTRIBUTION:
# INTERVENTION > EFFECT
1.CONTRIBUTION
# INTERVENTION >
# OTHER CAUSE >
EFFECT
# OTHER CAUSE >
WHAT WAS THE OVERALL MERIT/ VALUE
OF THESE CHANGES?

24. STEP 3: Theory of Change
Every evaluation should begin with a “ Theory
of Change” that would explain, how program
evaluation will produce a series of results
contributed to changing the intended program.

25. STEP 4: Evaluation Team
Procurement of an external evaluation team
should follow the following procedures:
1.We need to find a team who has a knowledge
of the local culture and language.
2.Experience in conducting evaluation.
3.Good technical evaluation skills and
4.Sound project management skills

26. STEP 5: Decision making and
management
EVALUATON
MANAGEMENT
TEAM
EVALUATION
REFERENCE
GROUP
EVALUATION TEAM
Selects the Evaluation team
Technical guidance
Quality assurance
Budget
Field visit
Technical advice
Cultural advice

27. Choosing the evaluation methods
The methods used for each type of evaluation will
vary. Both qualitative and quantitative methods can
be used within the design of an evaluation.
Qualitative methods may be employed for formative
and process evaluations, e.g. focus groups, short-
answer or open-ended questionnaires.
Impact and outcome evaluations may be carried out
using a variety of quantitative methods.
 Using an experimental or quasi-experimental design to
demonstrate a change (or not) is the most powerful
programme evaluation for detecting changes in
outcome

28. Study types and their advantages and
disadvantages

29. Dissemination and feedback
Once an evaluation is complete it is important to provide
feedback to the stakeholders involved in the program.
Dissemination of the results will help gain further support
for the program if it is successful, and help others gain
support for the introduction of similar program.
Publicity from dissemination activities may also increase
the impact of the program.
If the program has not been successful it is important to
share this with others so that weaknesses or relevant issues
are considered in other similar interventions, including
whether or not to introduce such interventions.

30. To sum up..
Evaluation should be seen as an integral
component of any program.
An evaluation plan needs to be determined at
the beginning of a program development, so
that a plan for data collection for this purpose
is built into project implementation.
Information on the effectiveness of a program,
will help identify if there are any problems in
running a program.

31. References
 U.S. Department of Health and Human Services Centers for
Disease Control and Prevention. Office of the Director, Office
of Strategy and Innovation. Introduction to program
evaluation for public health programs: A self-study guide.
Atlanta, GA: Centers for Disease Control and Prevention Oct
2011.
 Gertler PJ, Martinez S, Premand P, Rawlings LB, Vermeersch
CMJ. Impact Evaluation in Practice. The World Bank.
Washington DC; 2011.
 Issel LM. Health Program Planning and Evaluation: A practical,
systematic approach for community Health.John & Bartlett
Learning. Burlington MA; 2014.
 Drummond MF, Sculpher MJ, Claxton K, Stoddart GL,
Torrance GW. Methods for the Economic Evaluation of Health
Care Programmes.4th
edition. Oxford University Press. United
Kingdom;2015

32. Thank you !!