SlideShare a Scribd company logo
EVIDENCE AND
PROSPECTS FOR A
HEALTHY NIGERIA
March 31,
2016
IMPACT EVALUATION
David Evans
World Bank
March 31,
2016
WHAT IS IMPACT
EVALUATION?
“An impact evaluation assesses
changes in the well-being of
individuals, households,
communities or firms that can
be attributed to a particular
project, program or policy.”
-World Bank
WHAT IS IMPACT
EVALUATION?
“An impact evaluation assesses
changes in the well-being of
individuals, households,
communities or firms that can
be attributed to a particular
project, program or policy.”
-World Bank
OBJECTIVE
Evaluate the causal impact of a program or an
intervention on some outcome
Examples
 How much do exposure to a television soap opera affect HIV/AIDS
awareness and testing?
 How much do monetary incentives reduce turnover among midwives?
What about non-monetary incentives?
 How much do a Quality Improvement Plan and coaching increase the
quality of care at primary health care facilities?
 How much does providing improve housing for midwives reduce
turnover in rural areas?
WHY EVALUATE?
1. Evaluation helps to learn whether programs
are actually achieving their objectives.
2. Evaluation helps to improve program
effectiveness.
3. Evaluation helps to garner resources for
scale-up.
WHAT DO WE NEED?
A COUNTERFACTUAL
What would have happened in the absence
of the program?
COUNTERFACTUAL CRITERIA
Treated & comparison groups
1. Have identical average characteristics (observed &
unobserved)
2. The only difference is the treatment
3. Therefore the only reason for any difference in
outcomes is the treatment
Key question: What would participant look like if she
hadn’t received the program?
PERFECT EXPERIMENT
Identify target beneficiaries
Clone them!
Identical on the outside (observable)
Identical on the inside (unobservable)
Kami Tami
We’re both five-
year-old puppets
We both love to
take up new
health
interventions!
Images © Sesame Workshop
PERFECT EXPERIMENT
Give the intervention to one set of
clones
Kami
Tami
PERFECT EXPERIMENT
Observe some time later
Because the groups are identical (inside &
out), the difference is due to the bednets!
Kami
Tami
FINDING A GREAT CONTROL GROUP
What would the participant look like if she
weren’t in the program?
Room For Improvement Control Groups
Before – After
Participants – Non Participants
RFI: BEFORE-AFTER
Before bednets
6 malaria episodes in 6
months
After bednets
2 malaria episodes in 6
months
What else might be going on besides the bednets?
• Seasonal differences
• Rising incomes: Households invest in other
measures
RFI: BEFORE-AFTER
Important to monitor before-after
 Monitoring systems tell us if things are moving in
the right direction
Insufficient to show impact of program
Too many factors changing over time
 Example of cash transfers in Nicaragua!
Counterfactual: What would have happened
in the absence of the project, with everything
RFI: PARTICIPANTS VS NON-
PARTICIPANTS
Compare recipients of a program
to
 People who were not eligible for the program
 People who chose not to enroll in the program
Home births Clinic births
Example: Complications in
childbirth Impact of clinic
births?
What else might explain
the difference?
Observable differences
Income
Education
Unobservable differences
Heard rumor about hospitals
Neighbor available to care for other children
Kami
RFI: Participants vs Non-Participants
Grover
© Muppet photos copyright Sesame
No way to know how much of difference is because of clinic
RFI: Participants vs Non-
Participants
Home births Clinic births
Example: Complications in
childbirth
Impact of clinic
births
Other factors!
SELECTION BIAS
People who choose to join the program are
different!
If we cannot account completely for those
differences in our data…
We usually cannot
 How do you capture attitudes toward health systems? Initiative?
…then our comparison will not show the true
impact of the program
WHAT SHOULD WE
DO?
Gold standard:
Randomized experimental design
RANDOMIZED EXPERIMENTAL
DESIGN
Randomly assign potential beneficiaries to be
in the treatment or comparison group
Treatment and comparison have the same
characteristics (observed and unobserved), on
average
Any difference in outcomes is due to
treatment
Randomization with two doesn’t work!
But differences average out in a big sample
On average, same number of Kamis and Grovers
 Observable AND unobservable
Result: Measure true impact of program 
RANDOMIZED
EXPERIMENTAL DESIGN
We don’t even look
similar!
Compariso
nTreatment
Compariso
n
Treatment
RANDOM ASSIGNMENT
Random sample
Gather data from random
sample of population
No guarantee of unbiased
impact measure
Random
assignment
Randomly assign program
Unbiased impact measure!
Treatment Control Treatment Control
CAN WE RANDOMIZE?
Randomization does not mean denying people the
benefits of the project
Usually there are existing constraints within project
implementation that allow randomization
Randomization is the fairest way to allocate treatment
 Tanzania CCT: Randomized across needy villages
 Nigeria Quality Improvement: Lottery among eligible facilities
RANDOMIZATION OPPORTUNITIES
STAGGERED ROLL-OUT OF
PROGRAM
Roll-out to 200
clinics
Roll-out to 200
more clinics
Roll-out to 400
more clinics
Jan
2013
July
2013
Jan
2014
• Randomize the order in which clinics receive
program
• Compare Jan 2013 group to Jan 2014 group at
end of first year• Example: Mexico parent health training –
staggered roll-out among vulnerable
Example: Program for children in Kenya
 Orphans – Must have program now!
 Randomized among less vulnerable children
RANDOMIZATION OPPORTUNITIES
SOME GROUPS MUST GET THE
PROGRAM!
Highly
vulnerable
Moderately
vulnerable Not
vulnerable
RANDOMIZATION OPPORTUNITIES
VARY TREATMENT
intensity nature
Malaria
information
campaign
100 villages
Malaria
information
campaign +
SMS
reminders
100 villages
Randomizeacross
communities
Radio
campaign
100 villages
Newspaper
campaign
100 villages
Randomizeacross
communities
Additional impact of SMS
reminders?
Which approach has greater
impact?
UNIT OF RANDOMIZATION
At what level should I randomize?
 Individual
 Household
 Clinic
 Community
Considerations
 Political feasibility of randomization at individual level
 Spillovers within groups
 Implementation capacity: One clinic administering
different treatments
UNIT OF RANDOMIZATION
Bigger unit = Bigger study
(Because of intra-community correlation)
Individual randomization:
630 participants
(315 treatment, 315 control)
Clinic-level randomization:
150 clinics
(75 treatment, 75 control)
Number of units you randomize matters more than total
3,000 participants!
WHAT IF RANDOMIZATION IS
IMPOSSIBLE?
Think again: It often is possible on some
level, and it’s the best way to get a clear
measure of impact
Some situations, not possible
 Evaluate the effect of a national health policy
 Interventions in the past
 Life saving vaccination (volunteers for control
group?)
Alternative methods available, compelling
in some circumstances
I volunteer!
A COUPLE OF LAST BIG
POINTS
WE SHOULD DO AN
EVALUATION IF A PROGRAM
IS…
1. Innovative: This approach hasn’t been used
before
2. Replicable: The program may be scaled up
3. Strategically relevant: The program could
involve significant resources or affect many people
4. Untested: We don’t know how well it works
5. Influential: The results will be used to make a
policy decision
Adapted from Impact Evaluation in
WHAT MAKES A GREAT
IMPACT EVALUATION
QUESTION?
1. Cause-effect
• YES: “What is the impact of ______ on ______?”
• NOT “Who is taking up our antenatal care
program?”
2. Prospective (future-looking)
 YES: “What is the impact of this program we are
about to roll out?”
 NO: “What was the impact of a program we rolled
out 5 years ago?”
KEY CONCLUSIONS
Impact evaluation tells us if our programs are
working
Randomization of treatment leads to unbiased
estimate of impact
Other methods rely on more assumptions
Lots of opportunities for randomization
 No withholding of benefits
 Staggered roll-out
 Varied treatment
Thank you!

More Related Content

What's hot

Logical framework
Logical frameworkLogical framework
Logical framework
Md. Ayatullah Khan
 
Presentation Training on Result Based Management (RBM) for M&E Staff
Presentation Training on Result Based Management (RBM) for M&E StaffPresentation Training on Result Based Management (RBM) for M&E Staff
Presentation Training on Result Based Management (RBM) for M&E Staff
Fida Karim 🇵🇰
 
Project Monitoring and Evaluation
Project Monitoring and Evaluation Project Monitoring and Evaluation
Project Monitoring and Evaluation
Central University of Karnataka Kalaburagi
 
Project Monitoring & Evaluation
Project Monitoring & EvaluationProject Monitoring & Evaluation
Project Monitoring & Evaluation
Srinivasan Rengasamy
 
Monitoring and Evaluating
Monitoring and EvaluatingMonitoring and Evaluating
Monitoring and Evaluating
GlobalGiving
 
Monitoring and Evaluation for Project management.
Monitoring and Evaluation for Project management.Monitoring and Evaluation for Project management.
Monitoring and Evaluation for Project management.
Muthuraj K
 
Theory of Change
Theory of ChangeTheory of Change
Theory of Change
clearsateam
 
Introduction to the Logical Framework Approach
Introduction to the Logical Framework ApproachIntroduction to the Logical Framework Approach
Introduction to the Logical Framework Approach
Damien Sweeney
 
Logical framework approach DR.MADHUR VERMA PGIMS ROHTAK
Logical framework approach DR.MADHUR VERMA PGIMS ROHTAKLogical framework approach DR.MADHUR VERMA PGIMS ROHTAK
Logical framework approach DR.MADHUR VERMA PGIMS ROHTAK
MADHUR VERMA
 
Impact assessment , monitoring and evaluation
Impact assessment , monitoring and evaluationImpact assessment , monitoring and evaluation
Impact assessment , monitoring and evaluation
Sakthivel R
 
Monitoring and evaluation (2)
Monitoring and evaluation (2)Monitoring and evaluation (2)
Monitoring and evaluation (2)Dr.RAJEEV KASHYAP
 
Participatory Research
Participatory ResearchParticipatory Research
Participatory Research
Rey Tagum
 
Logical framework
Logical  frameworkLogical  framework
Logical framework
Vaidehi Malvi
 
M & e indicators
M & e indicatorsM & e indicators
M & e indicators
Krishna Mishra
 
Impact evaluation in-depth: More on methods and example of impact evaluation ...
Impact evaluation in-depth: More on methods and example of impact evaluation ...Impact evaluation in-depth: More on methods and example of impact evaluation ...
Impact evaluation in-depth: More on methods and example of impact evaluation ...
CIFOR-ICRAF
 
Basics of Monitoring & Evaluation
Basics of Monitoring & EvaluationBasics of Monitoring & Evaluation
Basics of Monitoring & Evaluation
Fida Karim 🇵🇰
 
Difference between monitoring and evaluation
Difference between monitoring and evaluationDifference between monitoring and evaluation
Difference between monitoring and evaluationDoreen Ty
 
Monitoring and evaluation frameworks logical framework
Monitoring and evaluation frameworks logical frameworkMonitoring and evaluation frameworks logical framework
Monitoring and evaluation frameworks logical framework
Preston Healthcare Consulting
 
THEORY OF CHANGE
THEORY OF CHANGE THEORY OF CHANGE
THEORY OF CHANGE
ICCO Cooperation
 

What's hot (20)

Logical framework
Logical frameworkLogical framework
Logical framework
 
Presentation Training on Result Based Management (RBM) for M&E Staff
Presentation Training on Result Based Management (RBM) for M&E StaffPresentation Training on Result Based Management (RBM) for M&E Staff
Presentation Training on Result Based Management (RBM) for M&E Staff
 
Project Monitoring and Evaluation
Project Monitoring and Evaluation Project Monitoring and Evaluation
Project Monitoring and Evaluation
 
Project Monitoring & Evaluation
Project Monitoring & EvaluationProject Monitoring & Evaluation
Project Monitoring & Evaluation
 
Monitoring and Evaluating
Monitoring and EvaluatingMonitoring and Evaluating
Monitoring and Evaluating
 
Monitoring and Evaluation for Project management.
Monitoring and Evaluation for Project management.Monitoring and Evaluation for Project management.
Monitoring and Evaluation for Project management.
 
Theory of Change
Theory of ChangeTheory of Change
Theory of Change
 
Introduction to the Logical Framework Approach
Introduction to the Logical Framework ApproachIntroduction to the Logical Framework Approach
Introduction to the Logical Framework Approach
 
Logical framework approach DR.MADHUR VERMA PGIMS ROHTAK
Logical framework approach DR.MADHUR VERMA PGIMS ROHTAKLogical framework approach DR.MADHUR VERMA PGIMS ROHTAK
Logical framework approach DR.MADHUR VERMA PGIMS ROHTAK
 
Impact assessment , monitoring and evaluation
Impact assessment , monitoring and evaluationImpact assessment , monitoring and evaluation
Impact assessment , monitoring and evaluation
 
Monitoring and evaluation (2)
Monitoring and evaluation (2)Monitoring and evaluation (2)
Monitoring and evaluation (2)
 
Participatory Research
Participatory ResearchParticipatory Research
Participatory Research
 
Logical framework
Logical  frameworkLogical  framework
Logical framework
 
M & e indicators
M & e indicatorsM & e indicators
M & e indicators
 
Logical framework
Logical frameworkLogical framework
Logical framework
 
Impact evaluation in-depth: More on methods and example of impact evaluation ...
Impact evaluation in-depth: More on methods and example of impact evaluation ...Impact evaluation in-depth: More on methods and example of impact evaluation ...
Impact evaluation in-depth: More on methods and example of impact evaluation ...
 
Basics of Monitoring & Evaluation
Basics of Monitoring & EvaluationBasics of Monitoring & Evaluation
Basics of Monitoring & Evaluation
 
Difference between monitoring and evaluation
Difference between monitoring and evaluationDifference between monitoring and evaluation
Difference between monitoring and evaluation
 
Monitoring and evaluation frameworks logical framework
Monitoring and evaluation frameworks logical frameworkMonitoring and evaluation frameworks logical framework
Monitoring and evaluation frameworks logical framework
 
THEORY OF CHANGE
THEORY OF CHANGE THEORY OF CHANGE
THEORY OF CHANGE
 

Similar to What is impact evaluation?

What is Evaluation
What is EvaluationWhat is Evaluation
What is Evaluation
clearsateam
 
Presentation by Rachel Steinacher, on IPA and RCTs
Presentation by Rachel Steinacher, on IPA and RCTsPresentation by Rachel Steinacher, on IPA and RCTs
Presentation by Rachel Steinacher, on IPA and RCTs
Twaweza
 
Experimental Evaluation Methods
Experimental Evaluation MethodsExperimental Evaluation Methods
Experimental Evaluation Methods
clearsateam
 
IFPRI - The Problem of Impact Evaluation
IFPRI - The Problem of Impact EvaluationIFPRI - The Problem of Impact Evaluation
IFPRI - The Problem of Impact Evaluation
International Food Policy Research Institute- South Asia Office
 
Key Issues in Impact Evaluation: A MEET and GEMNet-Health Virtual Event
Key Issues in Impact Evaluation: A MEET and GEMNet-Health Virtual EventKey Issues in Impact Evaluation: A MEET and GEMNet-Health Virtual Event
Key Issues in Impact Evaluation: A MEET and GEMNet-Health Virtual Event
MEASURE Evaluation
 
Assessing a Healthcare.docx
Assessing a Healthcare.docxAssessing a Healthcare.docx
Assessing a Healthcare.docx
4934bk
 
Prog EVAL Framework.docx
Prog EVAL Framework.docxProg EVAL Framework.docx
Prog EVAL Framework.docx
Reynaldo Calo
 
The art of the possible will
The art of the possible   willThe art of the possible   will
The art of the possible will
howardcooper
 
Gilligan quantitative impact eval methods
Gilligan quantitative impact eval methodsGilligan quantitative impact eval methods
Gilligan quantitative impact eval methodsgenderassets
 
Capstone EBP fall prevention programming
Capstone EBP fall prevention programming Capstone EBP fall prevention programming
Capstone EBP fall prevention programming
AmandaDiveleyMHAPTA
 
Understanding, measuring and communicating your impact
Understanding, measuring and communicating your impactUnderstanding, measuring and communicating your impact
Understanding, measuring and communicating your impact
CharityComms
 
Implementatioin research
Implementatioin researchImplementatioin research
Implementatioin research
Digvijay Trivedi
 
3. How to Randomize
3. How to Randomize3. How to Randomize
3. How to Randomize
vinhthedang
 
World AIDS Day 2016: Economic evaluation for HIV in South Africa
World AIDS Day 2016:  Economic evaluation for HIV in South AfricaWorld AIDS Day 2016:  Economic evaluation for HIV in South Africa
World AIDS Day 2016: Economic evaluation for HIV in South Africa
Sydney Rosen
 
Evidence for Public Health Decision Making
Evidence for Public Health Decision MakingEvidence for Public Health Decision Making
Evidence for Public Health Decision Making
Vineetha K
 
Assessment 7 Course Textbook Edberg, M. (2015). Essentials .docx
Assessment 7  Course Textbook  Edberg, M. (2015). Essentials .docxAssessment 7  Course Textbook  Edberg, M. (2015). Essentials .docx
Assessment 7 Course Textbook Edberg, M. (2015). Essentials .docx
davezstarr61655
 
Maidan Summit 2012 Dr. Jyotsna Puri, 3ie
Maidan Summit 2012  Dr. Jyotsna Puri, 3ieMaidan Summit 2012  Dr. Jyotsna Puri, 3ie
Maidan Summit 2012 Dr. Jyotsna Puri, 3ie
Maidan.in
 
IFPRI- Impact Surveys 1
IFPRI- Impact Surveys 1IFPRI- Impact Surveys 1
Health service or illness service? Rebranding healthcare.
Health service or illness service? Rebranding healthcare. Health service or illness service? Rebranding healthcare.
Health service or illness service? Rebranding healthcare.
Emmanuel Mosoti Machani
 

Similar to What is impact evaluation? (20)

What is Evaluation
What is EvaluationWhat is Evaluation
What is Evaluation
 
Presentation by Rachel Steinacher, on IPA and RCTs
Presentation by Rachel Steinacher, on IPA and RCTsPresentation by Rachel Steinacher, on IPA and RCTs
Presentation by Rachel Steinacher, on IPA and RCTs
 
Experimental Evaluation Methods
Experimental Evaluation MethodsExperimental Evaluation Methods
Experimental Evaluation Methods
 
Ahmed ifpri impact evaluation methods_15 nov 2011
Ahmed ifpri impact evaluation methods_15 nov 2011Ahmed ifpri impact evaluation methods_15 nov 2011
Ahmed ifpri impact evaluation methods_15 nov 2011
 
IFPRI - The Problem of Impact Evaluation
IFPRI - The Problem of Impact EvaluationIFPRI - The Problem of Impact Evaluation
IFPRI - The Problem of Impact Evaluation
 
Key Issues in Impact Evaluation: A MEET and GEMNet-Health Virtual Event
Key Issues in Impact Evaluation: A MEET and GEMNet-Health Virtual EventKey Issues in Impact Evaluation: A MEET and GEMNet-Health Virtual Event
Key Issues in Impact Evaluation: A MEET and GEMNet-Health Virtual Event
 
Assessing a Healthcare.docx
Assessing a Healthcare.docxAssessing a Healthcare.docx
Assessing a Healthcare.docx
 
Prog EVAL Framework.docx
Prog EVAL Framework.docxProg EVAL Framework.docx
Prog EVAL Framework.docx
 
The art of the possible will
The art of the possible   willThe art of the possible   will
The art of the possible will
 
Gilligan quantitative impact eval methods
Gilligan quantitative impact eval methodsGilligan quantitative impact eval methods
Gilligan quantitative impact eval methods
 
Capstone EBP fall prevention programming
Capstone EBP fall prevention programming Capstone EBP fall prevention programming
Capstone EBP fall prevention programming
 
Understanding, measuring and communicating your impact
Understanding, measuring and communicating your impactUnderstanding, measuring and communicating your impact
Understanding, measuring and communicating your impact
 
Implementatioin research
Implementatioin researchImplementatioin research
Implementatioin research
 
3. How to Randomize
3. How to Randomize3. How to Randomize
3. How to Randomize
 
World AIDS Day 2016: Economic evaluation for HIV in South Africa
World AIDS Day 2016:  Economic evaluation for HIV in South AfricaWorld AIDS Day 2016:  Economic evaluation for HIV in South Africa
World AIDS Day 2016: Economic evaluation for HIV in South Africa
 
Evidence for Public Health Decision Making
Evidence for Public Health Decision MakingEvidence for Public Health Decision Making
Evidence for Public Health Decision Making
 
Assessment 7 Course Textbook Edberg, M. (2015). Essentials .docx
Assessment 7  Course Textbook  Edberg, M. (2015). Essentials .docxAssessment 7  Course Textbook  Edberg, M. (2015). Essentials .docx
Assessment 7 Course Textbook Edberg, M. (2015). Essentials .docx
 
Maidan Summit 2012 Dr. Jyotsna Puri, 3ie
Maidan Summit 2012  Dr. Jyotsna Puri, 3ieMaidan Summit 2012  Dr. Jyotsna Puri, 3ie
Maidan Summit 2012 Dr. Jyotsna Puri, 3ie
 
IFPRI- Impact Surveys 1
IFPRI- Impact Surveys 1IFPRI- Impact Surveys 1
IFPRI- Impact Surveys 1
 
Health service or illness service? Rebranding healthcare.
Health service or illness service? Rebranding healthcare. Health service or illness service? Rebranding healthcare.
Health service or illness service? Rebranding healthcare.
 

More from David Evans

Foundational Learning: What It Is, Why It Matters, and How To Get It
Foundational Learning: What It Is, Why It Matters, and How To Get ItFoundational Learning: What It Is, Why It Matters, and How To Get It
Foundational Learning: What It Is, Why It Matters, and How To Get It
David Evans
 
Do the Poor Waste Transfer Income? Evidence from Around the World
Do the Poor Waste Transfer Income? Evidence from Around the WorldDo the Poor Waste Transfer Income? Evidence from Around the World
Do the Poor Waste Transfer Income? Evidence from Around the World
David Evans
 
Como melhorar a educação na África: Evidencias mais recentes
Como melhorar a educação na África: Evidencias mais recentesComo melhorar a educação na África: Evidencias mais recentes
Como melhorar a educação na África: Evidencias mais recentes
David Evans
 
Teachers and Technology in the Time of COVID-19
Teachers and Technology in the Time of COVID-19Teachers and Technology in the Time of COVID-19
Teachers and Technology in the Time of COVID-19
David Evans
 
Successful Teachers, Successful Students: Recruiting and Supporting Society's...
Successful Teachers, Successful Students: Recruiting and Supporting Society's...Successful Teachers, Successful Students: Recruiting and Supporting Society's...
Successful Teachers, Successful Students: Recruiting and Supporting Society's...
David Evans
 
¿Cómo obtener lo mejor de las evaluaciones de impacto en educación?
¿Cómo obtener lo mejor de las evaluaciones de impacto en educación?¿Cómo obtener lo mejor de las evaluaciones de impacto en educación?
¿Cómo obtener lo mejor de las evaluaciones de impacto en educación?
David Evans
 
Working Conditions of Teacher in Low- and Middle-Income Countries
Working Conditions of Teacher in Low- and Middle-Income CountriesWorking Conditions of Teacher in Low- and Middle-Income Countries
Working Conditions of Teacher in Low- and Middle-Income Countries
David Evans
 
How to Get the Most Out of Education Impact Evaluations
How to Get the Most Out of Education Impact EvaluationsHow to Get the Most Out of Education Impact Evaluations
How to Get the Most Out of Education Impact Evaluations
David Evans
 
How to Influence Policymakers
How to Influence PolicymakersHow to Influence Policymakers
How to Influence Policymakers
David Evans
 
Quelles sont les évidences sur les programmes de formation professionnelle?
Quelles sont les évidences sur les programmes de formation professionnelle?Quelles sont les évidences sur les programmes de formation professionnelle?
Quelles sont les évidences sur les programmes de formation professionnelle?
David Evans
 
Career Day presentation on Economic Development
Career Day presentation on Economic DevelopmentCareer Day presentation on Economic Development
Career Day presentation on Economic Development
David Evans
 
Career Day presentation - Economic Development
Career Day presentation - Economic DevelopmentCareer Day presentation - Economic Development
Career Day presentation - Economic Development
David Evans
 
Dentro da Formação Continuada de Professores: Como melhorá-la?
Dentro da Formação Continuada de Professores:Como melhorá-la?Dentro da Formação Continuada de Professores:Como melhorá-la?
Dentro da Formação Continuada de Professores: Como melhorá-la?
David Evans
 
Inside Teacher Training: What Works to Make It Better?
Inside Teacher Training: What Works to Make It Better?Inside Teacher Training: What Works to Make It Better?
Inside Teacher Training: What Works to Make It Better?
David Evans
 
Reporting on Education: What You Need to Know
Reporting on Education: What You Need to KnowReporting on Education: What You Need to Know
Reporting on Education: What You Need to Know
David Evans
 
Inside in-service teacher training (RISE)
Inside in-service teacher training (RISE)Inside in-service teacher training (RISE)
Inside in-service teacher training (RISE)
David Evans
 
How much does it cost to get that impact? Measuring cost effectiveness
How much does it cost to get that impact? Measuring cost effectivenessHow much does it cost to get that impact? Measuring cost effectiveness
How much does it cost to get that impact? Measuring cost effectiveness
David Evans
 
The Challenge and Success of Impact Evaluation for Tanzania Community-Based C...
The Challenge and Success of Impact Evaluation for Tanzania Community-Based C...The Challenge and Success of Impact Evaluation for Tanzania Community-Based C...
The Challenge and Success of Impact Evaluation for Tanzania Community-Based C...
David Evans
 
How to build a better education review
How to build a better education reviewHow to build a better education review
How to build a better education review
David Evans
 

More from David Evans (19)

Foundational Learning: What It Is, Why It Matters, and How To Get It
Foundational Learning: What It Is, Why It Matters, and How To Get ItFoundational Learning: What It Is, Why It Matters, and How To Get It
Foundational Learning: What It Is, Why It Matters, and How To Get It
 
Do the Poor Waste Transfer Income? Evidence from Around the World
Do the Poor Waste Transfer Income? Evidence from Around the WorldDo the Poor Waste Transfer Income? Evidence from Around the World
Do the Poor Waste Transfer Income? Evidence from Around the World
 
Como melhorar a educação na África: Evidencias mais recentes
Como melhorar a educação na África: Evidencias mais recentesComo melhorar a educação na África: Evidencias mais recentes
Como melhorar a educação na África: Evidencias mais recentes
 
Teachers and Technology in the Time of COVID-19
Teachers and Technology in the Time of COVID-19Teachers and Technology in the Time of COVID-19
Teachers and Technology in the Time of COVID-19
 
Successful Teachers, Successful Students: Recruiting and Supporting Society's...
Successful Teachers, Successful Students: Recruiting and Supporting Society's...Successful Teachers, Successful Students: Recruiting and Supporting Society's...
Successful Teachers, Successful Students: Recruiting and Supporting Society's...
 
¿Cómo obtener lo mejor de las evaluaciones de impacto en educación?
¿Cómo obtener lo mejor de las evaluaciones de impacto en educación?¿Cómo obtener lo mejor de las evaluaciones de impacto en educación?
¿Cómo obtener lo mejor de las evaluaciones de impacto en educación?
 
Working Conditions of Teacher in Low- and Middle-Income Countries
Working Conditions of Teacher in Low- and Middle-Income CountriesWorking Conditions of Teacher in Low- and Middle-Income Countries
Working Conditions of Teacher in Low- and Middle-Income Countries
 
How to Get the Most Out of Education Impact Evaluations
How to Get the Most Out of Education Impact EvaluationsHow to Get the Most Out of Education Impact Evaluations
How to Get the Most Out of Education Impact Evaluations
 
How to Influence Policymakers
How to Influence PolicymakersHow to Influence Policymakers
How to Influence Policymakers
 
Quelles sont les évidences sur les programmes de formation professionnelle?
Quelles sont les évidences sur les programmes de formation professionnelle?Quelles sont les évidences sur les programmes de formation professionnelle?
Quelles sont les évidences sur les programmes de formation professionnelle?
 
Career Day presentation on Economic Development
Career Day presentation on Economic DevelopmentCareer Day presentation on Economic Development
Career Day presentation on Economic Development
 
Career Day presentation - Economic Development
Career Day presentation - Economic DevelopmentCareer Day presentation - Economic Development
Career Day presentation - Economic Development
 
Dentro da Formação Continuada de Professores: Como melhorá-la?
Dentro da Formação Continuada de Professores:Como melhorá-la?Dentro da Formação Continuada de Professores:Como melhorá-la?
Dentro da Formação Continuada de Professores: Como melhorá-la?
 
Inside Teacher Training: What Works to Make It Better?
Inside Teacher Training: What Works to Make It Better?Inside Teacher Training: What Works to Make It Better?
Inside Teacher Training: What Works to Make It Better?
 
Reporting on Education: What You Need to Know
Reporting on Education: What You Need to KnowReporting on Education: What You Need to Know
Reporting on Education: What You Need to Know
 
Inside in-service teacher training (RISE)
Inside in-service teacher training (RISE)Inside in-service teacher training (RISE)
Inside in-service teacher training (RISE)
 
How much does it cost to get that impact? Measuring cost effectiveness
How much does it cost to get that impact? Measuring cost effectivenessHow much does it cost to get that impact? Measuring cost effectiveness
How much does it cost to get that impact? Measuring cost effectiveness
 
The Challenge and Success of Impact Evaluation for Tanzania Community-Based C...
The Challenge and Success of Impact Evaluation for Tanzania Community-Based C...The Challenge and Success of Impact Evaluation for Tanzania Community-Based C...
The Challenge and Success of Impact Evaluation for Tanzania Community-Based C...
 
How to build a better education review
How to build a better education reviewHow to build a better education review
How to build a better education review
 

Recently uploaded

How to Give Better Lectures: Some Tips for Doctors
How to Give Better Lectures: Some Tips for DoctorsHow to Give Better Lectures: Some Tips for Doctors
How to Give Better Lectures: Some Tips for Doctors
LanceCatedral
 
ARTHROLOGY PPT NCISM SYLLABUS AYURVEDA STUDENTS
ARTHROLOGY PPT NCISM SYLLABUS AYURVEDA STUDENTSARTHROLOGY PPT NCISM SYLLABUS AYURVEDA STUDENTS
ARTHROLOGY PPT NCISM SYLLABUS AYURVEDA STUDENTS
Dr. Vinay Pareek
 
Tom Selleck Health: A Comprehensive Look at the Iconic Actor’s Wellness Journey
Tom Selleck Health: A Comprehensive Look at the Iconic Actor’s Wellness JourneyTom Selleck Health: A Comprehensive Look at the Iconic Actor’s Wellness Journey
Tom Selleck Health: A Comprehensive Look at the Iconic Actor’s Wellness Journey
greendigital
 
Couples presenting to the infertility clinic- Do they really have infertility...
Couples presenting to the infertility clinic- Do they really have infertility...Couples presenting to the infertility clinic- Do they really have infertility...
Couples presenting to the infertility clinic- Do they really have infertility...
Sujoy Dasgupta
 
New Directions in Targeted Therapeutic Approaches for Older Adults With Mantl...
New Directions in Targeted Therapeutic Approaches for Older Adults With Mantl...New Directions in Targeted Therapeutic Approaches for Older Adults With Mantl...
New Directions in Targeted Therapeutic Approaches for Older Adults With Mantl...
i3 Health
 
Evaluation of antidepressant activity of clitoris ternatea in animals
Evaluation of antidepressant activity of clitoris ternatea in animalsEvaluation of antidepressant activity of clitoris ternatea in animals
Evaluation of antidepressant activity of clitoris ternatea in animals
Shweta
 
Physiology of Chemical Sensation of smell.pdf
Physiology of Chemical Sensation of smell.pdfPhysiology of Chemical Sensation of smell.pdf
Physiology of Chemical Sensation of smell.pdf
MedicoseAcademics
 
New Drug Discovery and Development .....
New Drug Discovery and Development .....New Drug Discovery and Development .....
New Drug Discovery and Development .....
NEHA GUPTA
 
POST OPERATIVE OLIGURIA and its management
POST OPERATIVE OLIGURIA and its managementPOST OPERATIVE OLIGURIA and its management
POST OPERATIVE OLIGURIA and its management
touseefaziz1
 
Non-respiratory Functions of the Lungs.pdf
Non-respiratory Functions of the Lungs.pdfNon-respiratory Functions of the Lungs.pdf
Non-respiratory Functions of the Lungs.pdf
MedicoseAcademics
 
Physiology of Special Chemical Sensation of Taste
Physiology of Special Chemical Sensation of TastePhysiology of Special Chemical Sensation of Taste
Physiology of Special Chemical Sensation of Taste
MedicoseAcademics
 
Hemodialysis: Chapter 3, Dialysis Water Unit - Dr.Gawad
Hemodialysis: Chapter 3, Dialysis Water Unit - Dr.GawadHemodialysis: Chapter 3, Dialysis Water Unit - Dr.Gawad
Hemodialysis: Chapter 3, Dialysis Water Unit - Dr.Gawad
NephroTube - Dr.Gawad
 
Pulmonary Thromboembolism - etilogy, types, medical- Surgical and nursing man...
Pulmonary Thromboembolism - etilogy, types, medical- Surgical and nursing man...Pulmonary Thromboembolism - etilogy, types, medical- Surgical and nursing man...
Pulmonary Thromboembolism - etilogy, types, medical- Surgical and nursing man...
VarunMahajani
 
ANATOMY AND PHYSIOLOGY OF URINARY SYSTEM.pptx
ANATOMY AND PHYSIOLOGY OF URINARY SYSTEM.pptxANATOMY AND PHYSIOLOGY OF URINARY SYSTEM.pptx
ANATOMY AND PHYSIOLOGY OF URINARY SYSTEM.pptx
Swetaba Besh
 
Cervical & Brachial Plexus By Dr. RIG.pptx
Cervical & Brachial Plexus By Dr. RIG.pptxCervical & Brachial Plexus By Dr. RIG.pptx
Cervical & Brachial Plexus By Dr. RIG.pptx
Dr. Rabia Inam Gandapore
 
Maxilla, Mandible & Hyoid Bone & Clinical Correlations by Dr. RIG.pptx
Maxilla, Mandible & Hyoid Bone & Clinical Correlations by Dr. RIG.pptxMaxilla, Mandible & Hyoid Bone & Clinical Correlations by Dr. RIG.pptx
Maxilla, Mandible & Hyoid Bone & Clinical Correlations by Dr. RIG.pptx
Dr. Rabia Inam Gandapore
 
Novas diretrizes da OMS para os cuidados perinatais de mais qualidade
Novas diretrizes da OMS para os cuidados perinatais de mais qualidadeNovas diretrizes da OMS para os cuidados perinatais de mais qualidade
Novas diretrizes da OMS para os cuidados perinatais de mais qualidade
Prof. Marcus Renato de Carvalho
 
TEST BANK for Operations Management, 14th Edition by William J. Stevenson, Ve...
TEST BANK for Operations Management, 14th Edition by William J. Stevenson, Ve...TEST BANK for Operations Management, 14th Edition by William J. Stevenson, Ve...
TEST BANK for Operations Management, 14th Edition by William J. Stevenson, Ve...
kevinkariuki227
 
Ocular injury ppt Upendra pal optometrist upums saifai etawah
Ocular injury  ppt  Upendra pal  optometrist upums saifai etawahOcular injury  ppt  Upendra pal  optometrist upums saifai etawah
Ocular injury ppt Upendra pal optometrist upums saifai etawah
pal078100
 
HOT NEW PRODUCT! BIG SALES FAST SHIPPING NOW FROM CHINA!! EU KU DB BK substit...
HOT NEW PRODUCT! BIG SALES FAST SHIPPING NOW FROM CHINA!! EU KU DB BK substit...HOT NEW PRODUCT! BIG SALES FAST SHIPPING NOW FROM CHINA!! EU KU DB BK substit...
HOT NEW PRODUCT! BIG SALES FAST SHIPPING NOW FROM CHINA!! EU KU DB BK substit...
GL Anaacs
 

Recently uploaded (20)

How to Give Better Lectures: Some Tips for Doctors
How to Give Better Lectures: Some Tips for DoctorsHow to Give Better Lectures: Some Tips for Doctors
How to Give Better Lectures: Some Tips for Doctors
 
ARTHROLOGY PPT NCISM SYLLABUS AYURVEDA STUDENTS
ARTHROLOGY PPT NCISM SYLLABUS AYURVEDA STUDENTSARTHROLOGY PPT NCISM SYLLABUS AYURVEDA STUDENTS
ARTHROLOGY PPT NCISM SYLLABUS AYURVEDA STUDENTS
 
Tom Selleck Health: A Comprehensive Look at the Iconic Actor’s Wellness Journey
Tom Selleck Health: A Comprehensive Look at the Iconic Actor’s Wellness JourneyTom Selleck Health: A Comprehensive Look at the Iconic Actor’s Wellness Journey
Tom Selleck Health: A Comprehensive Look at the Iconic Actor’s Wellness Journey
 
Couples presenting to the infertility clinic- Do they really have infertility...
Couples presenting to the infertility clinic- Do they really have infertility...Couples presenting to the infertility clinic- Do they really have infertility...
Couples presenting to the infertility clinic- Do they really have infertility...
 
New Directions in Targeted Therapeutic Approaches for Older Adults With Mantl...
New Directions in Targeted Therapeutic Approaches for Older Adults With Mantl...New Directions in Targeted Therapeutic Approaches for Older Adults With Mantl...
New Directions in Targeted Therapeutic Approaches for Older Adults With Mantl...
 
Evaluation of antidepressant activity of clitoris ternatea in animals
Evaluation of antidepressant activity of clitoris ternatea in animalsEvaluation of antidepressant activity of clitoris ternatea in animals
Evaluation of antidepressant activity of clitoris ternatea in animals
 
Physiology of Chemical Sensation of smell.pdf
Physiology of Chemical Sensation of smell.pdfPhysiology of Chemical Sensation of smell.pdf
Physiology of Chemical Sensation of smell.pdf
 
New Drug Discovery and Development .....
New Drug Discovery and Development .....New Drug Discovery and Development .....
New Drug Discovery and Development .....
 
POST OPERATIVE OLIGURIA and its management
POST OPERATIVE OLIGURIA and its managementPOST OPERATIVE OLIGURIA and its management
POST OPERATIVE OLIGURIA and its management
 
Non-respiratory Functions of the Lungs.pdf
Non-respiratory Functions of the Lungs.pdfNon-respiratory Functions of the Lungs.pdf
Non-respiratory Functions of the Lungs.pdf
 
Physiology of Special Chemical Sensation of Taste
Physiology of Special Chemical Sensation of TastePhysiology of Special Chemical Sensation of Taste
Physiology of Special Chemical Sensation of Taste
 
Hemodialysis: Chapter 3, Dialysis Water Unit - Dr.Gawad
Hemodialysis: Chapter 3, Dialysis Water Unit - Dr.GawadHemodialysis: Chapter 3, Dialysis Water Unit - Dr.Gawad
Hemodialysis: Chapter 3, Dialysis Water Unit - Dr.Gawad
 
Pulmonary Thromboembolism - etilogy, types, medical- Surgical and nursing man...
Pulmonary Thromboembolism - etilogy, types, medical- Surgical and nursing man...Pulmonary Thromboembolism - etilogy, types, medical- Surgical and nursing man...
Pulmonary Thromboembolism - etilogy, types, medical- Surgical and nursing man...
 
ANATOMY AND PHYSIOLOGY OF URINARY SYSTEM.pptx
ANATOMY AND PHYSIOLOGY OF URINARY SYSTEM.pptxANATOMY AND PHYSIOLOGY OF URINARY SYSTEM.pptx
ANATOMY AND PHYSIOLOGY OF URINARY SYSTEM.pptx
 
Cervical & Brachial Plexus By Dr. RIG.pptx
Cervical & Brachial Plexus By Dr. RIG.pptxCervical & Brachial Plexus By Dr. RIG.pptx
Cervical & Brachial Plexus By Dr. RIG.pptx
 
Maxilla, Mandible & Hyoid Bone & Clinical Correlations by Dr. RIG.pptx
Maxilla, Mandible & Hyoid Bone & Clinical Correlations by Dr. RIG.pptxMaxilla, Mandible & Hyoid Bone & Clinical Correlations by Dr. RIG.pptx
Maxilla, Mandible & Hyoid Bone & Clinical Correlations by Dr. RIG.pptx
 
Novas diretrizes da OMS para os cuidados perinatais de mais qualidade
Novas diretrizes da OMS para os cuidados perinatais de mais qualidadeNovas diretrizes da OMS para os cuidados perinatais de mais qualidade
Novas diretrizes da OMS para os cuidados perinatais de mais qualidade
 
TEST BANK for Operations Management, 14th Edition by William J. Stevenson, Ve...
TEST BANK for Operations Management, 14th Edition by William J. Stevenson, Ve...TEST BANK for Operations Management, 14th Edition by William J. Stevenson, Ve...
TEST BANK for Operations Management, 14th Edition by William J. Stevenson, Ve...
 
Ocular injury ppt Upendra pal optometrist upums saifai etawah
Ocular injury  ppt  Upendra pal  optometrist upums saifai etawahOcular injury  ppt  Upendra pal  optometrist upums saifai etawah
Ocular injury ppt Upendra pal optometrist upums saifai etawah
 
HOT NEW PRODUCT! BIG SALES FAST SHIPPING NOW FROM CHINA!! EU KU DB BK substit...
HOT NEW PRODUCT! BIG SALES FAST SHIPPING NOW FROM CHINA!! EU KU DB BK substit...HOT NEW PRODUCT! BIG SALES FAST SHIPPING NOW FROM CHINA!! EU KU DB BK substit...
HOT NEW PRODUCT! BIG SALES FAST SHIPPING NOW FROM CHINA!! EU KU DB BK substit...
 

What is impact evaluation?

  • 1. EVIDENCE AND PROSPECTS FOR A HEALTHY NIGERIA March 31, 2016
  • 3. WHAT IS IMPACT EVALUATION? “An impact evaluation assesses changes in the well-being of individuals, households, communities or firms that can be attributed to a particular project, program or policy.” -World Bank
  • 4. WHAT IS IMPACT EVALUATION? “An impact evaluation assesses changes in the well-being of individuals, households, communities or firms that can be attributed to a particular project, program or policy.” -World Bank
  • 5. OBJECTIVE Evaluate the causal impact of a program or an intervention on some outcome Examples  How much do exposure to a television soap opera affect HIV/AIDS awareness and testing?  How much do monetary incentives reduce turnover among midwives? What about non-monetary incentives?  How much do a Quality Improvement Plan and coaching increase the quality of care at primary health care facilities?  How much does providing improve housing for midwives reduce turnover in rural areas?
  • 6. WHY EVALUATE? 1. Evaluation helps to learn whether programs are actually achieving their objectives. 2. Evaluation helps to improve program effectiveness. 3. Evaluation helps to garner resources for scale-up.
  • 7. WHAT DO WE NEED? A COUNTERFACTUAL What would have happened in the absence of the program?
  • 8. COUNTERFACTUAL CRITERIA Treated & comparison groups 1. Have identical average characteristics (observed & unobserved) 2. The only difference is the treatment 3. Therefore the only reason for any difference in outcomes is the treatment Key question: What would participant look like if she hadn’t received the program?
  • 9. PERFECT EXPERIMENT Identify target beneficiaries Clone them! Identical on the outside (observable) Identical on the inside (unobservable) Kami Tami We’re both five- year-old puppets We both love to take up new health interventions! Images © Sesame Workshop
  • 10. PERFECT EXPERIMENT Give the intervention to one set of clones Kami Tami
  • 11. PERFECT EXPERIMENT Observe some time later Because the groups are identical (inside & out), the difference is due to the bednets! Kami Tami
  • 12. FINDING A GREAT CONTROL GROUP What would the participant look like if she weren’t in the program? Room For Improvement Control Groups Before – After Participants – Non Participants
  • 13. RFI: BEFORE-AFTER Before bednets 6 malaria episodes in 6 months After bednets 2 malaria episodes in 6 months What else might be going on besides the bednets? • Seasonal differences • Rising incomes: Households invest in other measures
  • 14. RFI: BEFORE-AFTER Important to monitor before-after  Monitoring systems tell us if things are moving in the right direction Insufficient to show impact of program Too many factors changing over time  Example of cash transfers in Nicaragua! Counterfactual: What would have happened in the absence of the project, with everything
  • 15. RFI: PARTICIPANTS VS NON- PARTICIPANTS Compare recipients of a program to  People who were not eligible for the program  People who chose not to enroll in the program Home births Clinic births Example: Complications in childbirth Impact of clinic births? What else might explain the difference?
  • 16. Observable differences Income Education Unobservable differences Heard rumor about hospitals Neighbor available to care for other children Kami RFI: Participants vs Non-Participants Grover © Muppet photos copyright Sesame
  • 17. No way to know how much of difference is because of clinic RFI: Participants vs Non- Participants Home births Clinic births Example: Complications in childbirth Impact of clinic births Other factors!
  • 18. SELECTION BIAS People who choose to join the program are different! If we cannot account completely for those differences in our data… We usually cannot  How do you capture attitudes toward health systems? Initiative? …then our comparison will not show the true impact of the program
  • 19. WHAT SHOULD WE DO? Gold standard: Randomized experimental design
  • 20. RANDOMIZED EXPERIMENTAL DESIGN Randomly assign potential beneficiaries to be in the treatment or comparison group Treatment and comparison have the same characteristics (observed and unobserved), on average Any difference in outcomes is due to treatment
  • 21. Randomization with two doesn’t work! But differences average out in a big sample On average, same number of Kamis and Grovers  Observable AND unobservable Result: Measure true impact of program  RANDOMIZED EXPERIMENTAL DESIGN We don’t even look similar! Compariso nTreatment Compariso n Treatment
  • 22. RANDOM ASSIGNMENT Random sample Gather data from random sample of population No guarantee of unbiased impact measure Random assignment Randomly assign program Unbiased impact measure! Treatment Control Treatment Control
  • 23. CAN WE RANDOMIZE? Randomization does not mean denying people the benefits of the project Usually there are existing constraints within project implementation that allow randomization Randomization is the fairest way to allocate treatment  Tanzania CCT: Randomized across needy villages  Nigeria Quality Improvement: Lottery among eligible facilities
  • 24. RANDOMIZATION OPPORTUNITIES STAGGERED ROLL-OUT OF PROGRAM Roll-out to 200 clinics Roll-out to 200 more clinics Roll-out to 400 more clinics Jan 2013 July 2013 Jan 2014 • Randomize the order in which clinics receive program • Compare Jan 2013 group to Jan 2014 group at end of first year• Example: Mexico parent health training – staggered roll-out among vulnerable
  • 25. Example: Program for children in Kenya  Orphans – Must have program now!  Randomized among less vulnerable children RANDOMIZATION OPPORTUNITIES SOME GROUPS MUST GET THE PROGRAM! Highly vulnerable Moderately vulnerable Not vulnerable
  • 26. RANDOMIZATION OPPORTUNITIES VARY TREATMENT intensity nature Malaria information campaign 100 villages Malaria information campaign + SMS reminders 100 villages Randomizeacross communities Radio campaign 100 villages Newspaper campaign 100 villages Randomizeacross communities Additional impact of SMS reminders? Which approach has greater impact?
  • 27. UNIT OF RANDOMIZATION At what level should I randomize?  Individual  Household  Clinic  Community Considerations  Political feasibility of randomization at individual level  Spillovers within groups  Implementation capacity: One clinic administering different treatments
  • 28. UNIT OF RANDOMIZATION Bigger unit = Bigger study (Because of intra-community correlation) Individual randomization: 630 participants (315 treatment, 315 control) Clinic-level randomization: 150 clinics (75 treatment, 75 control) Number of units you randomize matters more than total 3,000 participants!
  • 29. WHAT IF RANDOMIZATION IS IMPOSSIBLE? Think again: It often is possible on some level, and it’s the best way to get a clear measure of impact Some situations, not possible  Evaluate the effect of a national health policy  Interventions in the past  Life saving vaccination (volunteers for control group?) Alternative methods available, compelling in some circumstances I volunteer!
  • 30. A COUPLE OF LAST BIG POINTS
  • 31. WE SHOULD DO AN EVALUATION IF A PROGRAM IS… 1. Innovative: This approach hasn’t been used before 2. Replicable: The program may be scaled up 3. Strategically relevant: The program could involve significant resources or affect many people 4. Untested: We don’t know how well it works 5. Influential: The results will be used to make a policy decision Adapted from Impact Evaluation in
  • 32. WHAT MAKES A GREAT IMPACT EVALUATION QUESTION? 1. Cause-effect • YES: “What is the impact of ______ on ______?” • NOT “Who is taking up our antenatal care program?” 2. Prospective (future-looking)  YES: “What is the impact of this program we are about to roll out?”  NO: “What was the impact of a program we rolled out 5 years ago?”
  • 33. KEY CONCLUSIONS Impact evaluation tells us if our programs are working Randomization of treatment leads to unbiased estimate of impact Other methods rely on more assumptions Lots of opportunities for randomization  No withholding of benefits  Staggered roll-out  Varied treatment

Editor's Notes

  1. RFI stand for “Room for Improvement”
  2. In Nicaragua, a cash transfer program was followed by a significant reduction of income. But then there was also a massive drop in coffee prices. It turns out, the cash transfer recipients had less of a reduction.
  3. Delta = .2., rho = .2. alpha = .05.