MODULE 10
Measuring progress in health
10
MONITORING & EVALUATION (M&E),
HEALTH IN ALL POLICIES (HiAP),
HEALTH IMPACT ASSESSMENT (HIA),
AND HEALTH LENS ANALYSIS

LEARNING OBJECTIVES
10
List reasons for monitoring and evaluation
Recognize different types of monitoring
and evaluation related to health
Distinguish between inputs, outputs,
outcomes and impact
1
2
3
Explain the purpose and key steps involved in health
impact assessment and health lens analysis
4
5 Identify sources of health data and policy advice

MONITORING AND EVALUATION IN THE
REVIEW STAGE OF THE POLICY CYCLE
10
REVIEW
Report
Evaluate
Monitor
AGENDA SETTING
Identify problem
Research
Set agenda
FORMULATION
Develop options and strategies
Negotiate
Formulate policy
IMPLEMENTATION
Enforce policy
Implement policy
Source: modified from http://www.geostrategis.com/images/policycycle.jpg
POLICY

MONITORING HEALTH
DETERMINANTS
Source: https://www.cdc.gov/mmwr/preview/mmwrhtml/rr5013a1.htm
The ongoing, systematic collection, analysis,
interpretation, and dissemination of data
regarding a health-related event for use in
public health action.
10

EVALUATION
10
Analysis of the effectiveness of
a specific intervention or set of interventions
in achieving an intended outcome or set of
outcomes.

HiAP is not an endpoint in itself, but a
continuous approach to the promotion of health,
health equity, and health systems.
Policy-makers usually require:
–Evidence of progress that HiAP has improved health
and well-being and their determinants
in the general population or in targeted population
groups; and
–Justification for the investment in establishing,
fostering and sustaining HiAP.
Acknowledgements: slide adapted from Zeinab Khadr’s Powerpoint for HiAP training
in the Eastern Mediterranean Region, February 2017.
10
WHY MONITORING AND EVALUATION IN
HiAP?

10
Acknowledgements: slide adapted from Zeinab Khadr’s Powerpoint for HiAP training
in the Eastern Mediterranean Region, February 2017.
Acknowledgements: slide adapted from Zeinab Khadr’s Powerpoint for HiAP training
in the Eastern Mediterranean Region, February 2017.
Once a policy has been changed, subsequent
monitoring is necessary to evaluate the
outcomes of the new policy, and thus
monitoring should be an iterative and cyclical
process that operates continuously.
–Built into the planning stage rather
than an ‘afterthought’.
WHY MONITORING AND EVALUATION IN
HiAP?

Holds partners and agencies accountable and also
makes them aware of the health gains made
Tests the validity and precision of health impact
predictions made in assessments
Adds to the body of knowledge/evidence
base on interventions
Provides early warning of unforeseen results/impacts
BENEFITS OF MONITORING
AND EVALUATION (M&E)
10
Empowers communities when they are involved
in monitoring and evaluation

10
Provides a way to inform stakeholders and the community
Strengthens the vision of health in
the community and stakeholders
Provides a better understanding of the value of HiAP
Lessons learned to improve HiAP approaches
BENEFITS OF MONITORING
AND EVALUATION (M&E)

STAGES OF THE HEALTH
MONITORING CYCLE
10
Source: WHO (2013) Handbook on Health Inequality Monitoring with a Special Focus
on Low- and Middle-Income Countries. Geneva, WHO, p. 2.
THERE ARE FIVE STAGES OF THE CYCLE OF HEALTH MONITORING
Select relevant
indicator
Obtain
data
Make
changes
Report
results
Analyse
data

DIRECT AND INDIRECT
INDICATORS
FOR MONITORING HEALTH
10
INDICATORS OFTEN CORRESPOND TO DIFFERENT STAGES
OF A POLICY INTERVENTION AND CAN BE CATEGORIZED AS:
“Inputs” – the resources used to initiate a policy;
“Outputs” – processes, products and services that
immediately result from a policy or project;
“Outcomes” – short- to medium-term results
of a policy or project; and
“Impacts” – long-term effects produced by a policy or project.

WHAT DO WE WANT TO KNOW?
10
HEALTH IN ALL POLICIES IMPACT CHAIN
Output Outcome 1 Outcome 2 Impact
WHAT
• Interventions on
SDH “Healthy Public
Policy”
HOW
• Strategic
partnerships
• Governance of
systems of decision-
making
Well-being,
morbidity, and
mortality (level and
distribution)
Outcome 3
Attributable
risk proportion
Exposure
risk factor
Determinants

10 WHAT DO WE WANT TO KNOW?
Process evaluation
1
How do Health in All Policies mechanisms (e.g. health lens,
health impact assessment, policy dialogues) influence the
process of policy-making and government business?
Source: slide from the South Australian Health in All Policies Summer School Training Manual,
Government of South Australia, 2013.

Process evaluation
1
Example questions:
• Is the health lens operating according to the model?
• Did participants develop a shared understanding of the aims and
process? How was this achieved (or not)?
• Were appropriate decision-makers involved in the process?
• Were agencies’ goals and expectations met? Was the process of
mutual benefits to partners?
• What are the barriers and enablers?
Source: slide from the South Australian Health in All Policies Summer School Training Manual,
Government of South Australia, 2013.
WHAT DO WE WANT TO KNOW?
10

10 WHAT DO WE WANT TO KNOW?
Examine Outputs
2
Is Health in All Policies effective in building health
and equity considerations into policy and
government business?
Example questions:
• Is there evidence of the process informing
policy or programmes? Have decision-making processes
changed?
• Have participants gained new knowledge/skills/attitudes?
• Is there transfer of learning to other areas of work?
Source: slide adapted from the South Australian Health in All Policies Summer School Training Manual,
Government of South Australia, 2013.

10 WHAT DO WE WANT TO KNOW?
What impact do the consequent policies and government
business have on the determinants
of health and their distribution?
Outcome and impact evaluation
3
Source: slide adapted from the South Australian Health in All Policies Summer School Training Manual,
Government of South Australia, 2013.

WHAT DO WE WANT TO KNOW?
10
Here we face some serious challenges e.g.
• Complexity of determinants and policy interventions
• Attributing change to the intervention
• Often long time frames
• Measurement tools for social health and well-being
• Measuring prevention (what didn’t happen).
Outcome and impact evaluation
3
Source: slide adapted from the South Australian Health in All Policies Summer School Training Manual,
Government of South Australia, 2013.

Establishing the health impacts or outcomes for HiAP is methodologically
challenging.
Development of a programme logic framework or model that outlines a
coherent chain of relationships to show how HiAP activities can lead to long-
term goals and population health outcomes.
Not seeking to establish causality through statistical correlation
(not yet).
What impact do the consequent policies and government
business have on the determinants of health and
their distribution?
10 CHARACTERIZING M&E FOR HiAP

SOUTH AUSTRALIAN CASE
10
SOUTH AUSTRALIAN CONTEXT: SUPPORTIVE HISTORY AND CHANGING POLICY PRIORITIES
2016
Version
Strategies
Theory of change
Address social
determinants of
health (SDoH)
Focus on SDoH
outside the Health
sector
Intersectoral action
is required to bring
about change in
the SDoH
Requires high
level political
commitment,
dedicated resources
and
skilled personnel
to drive change
HiAP must address
core business of
partner agencies
Develop relational
systems that
connect individuals,
agencies and
sectors
Undertake joint
problem/opportunity
identification and
decision-making
Utilise governance
Systems that
connect HiAP work
with senior decision-
makers
Mediated by:
Organisational
Institutional
culture, capacity and
priorities
Power
relationships
Political will
Resources
I
M
P
L
E
M
E
N
T
A
T
I
O
N
Adapted from: Baum F., Lawless A., Delany T., MacDougall C., Williams C., Broderick D., Wildgoose D., Harris E., McDermott D., Kickbusch I, Popay J. and Marmot M. (2014)
Policy
entrepreneurs
Intermediaries
(champions)
Relationship
building
and maintenance
Public Health
Partnerships Branch
since early 2014
Rapid reviews,
desktop analyses
& partnership
agreements
Central mandate
for action
Other - but fewer -
HiAP initiatives
Accountability
and reporting
Activities
Impacts on
policy environment
Understandings
about SDoH
Learning
Co-benefits
Capacity
Networks
Outputs
HiAP resources
Training
Outcomes for South
Australian Population
Investment in social
determinants that
contributes to future
health and equity
Policy
supports
health,
wellbeing
and equity
SA a better
place to
live with
increased
population
health and
equity
Improved
performance against
sectoral targets
Build & secure an authorising environment for intersectional partnerships to promote health
within the context of reduced support for HP and for HiAP as a dedicated initiative

BRINGING IT ALL TOGETHER
10
Intersectoral policy-making and systems / process aspects
of HiAP (process, impact, outcome evaluation)
CHARACTERIZING TYPES OF IMPACT
ASSESSMENT APPROACHES
Strengthening public health evidence; observe changes
in determinants of health or attributed burden of disease
Strengthen institutions - public health and socio-political
1
2
3

BRINGING IT ALL TOGETHER
10
METHODOLOGICAL APPROACHES
Organizational learning and critical action research
Realistic evaluation
Programme logic as a basis for attribution of health and equity outcomes
Theories on policy agenda setting and implementation
Complexity as a frame for understanding policy-making processes
Health, human impact assessment

STRENGTHEN PUBLIC HEALTH
EVIDENCE: SUPPORTING EVALUATIVE
CAPACITIES
10
Types of assessments to predict health and
health equity impacts (predictive evaluations /
policy analysis)
• Health impact assessment
• Cost-benefit analysis
• Integrated impact assessment
• Environmental impact assessment
• Human health risk assessment
Effectiveness data / intervention research / what
interventions work?

WHAT IS HEALTH IMPACT
ASSESSMENT (HIA)?
10
A HIA is a combination of procedures,
methods and tools that assesses the potential
effects of a policy or project on the health of a
population and the distribution of those effects
within the population. HIAs also identify
appropriate actions to manage those effects.

HIA IS BASED ON FOUR VALUES
10
Democracy
Equity
Sustainable development
Ethical use of evidence

APPLYING HIA
10
Source: http://www.who.int/hia/tools/en/
Policy and
programme
development
phase for
prospective
assessments
Policy
implementation
phase
Screening
Scoping
Appraisal
Reporting
Monitoring
Quickly establishes health relevance of the
policy or project. Is HIA required?
Identifies key health issues and public
concerns, establishes ToR, sets boundaries
Conclusions and recommendations to
remove/mitigate negative impacts on
health or to enhance positive impacts
Action, where appropriate, to monitor
actual impacts on health to enhance
existing evidence base
Rapid or in-depth assessments of health
impacts using available evidence - who
will be affected, baseline, prediction,
significance, mitigation

The HiAP Health Lens Analysis process builds on traditional Health
Impact Assessment methodology to allow the process to deliver both
rigour and flexibility that accommodates the operational culture and
policy imperatives of the partner agency.
Health lens is a model to analyze a policy problem in order to derive
benefit for population health while at the same time being able to
support and achieve the goals of other sectors.
Evaluation, an essential component of the HiAP process, is built into
each individual health lens.
WHAT IS HEALTH LENS
ANALYSIS (HLA)?
10

APPLYING HLA
10
The South Australian Health Lens Analysis model
Health Lens Analysis

10
Health in All Policies is an overarching
conceptual framework for systematic
engagement with sectors outside of health.
The Health Lens Analysis model draws on several
components of HIA and like HIA, uses general public health
methods of investigation and analysis.
HIA is applied to existing programmes or policy.
HEALTH IMPACT ASSESSMENT AND
HiAP HEALTH LENS ANALYSIS MODEL

10
HiAP is applied to programmes or policy
yet to be developed.
HiAP does not rigidly apply the steps of HIA and their
application is highly context specific.
HEALTH IMPACT ASSESSMENT AND
HiAP HEALTH LENS ANALYSIS MODEL

STRENGTHEN PUBLIC HEALTH AND
OTHER SOCIO-POLITICAL
INSTITUTIONS
10
Strengthen public health institutions:
• Reports on sustainable development
• Reporting on health determinants, including HiAP activities
• Public health legislation
• Capacities for health impact evaluation
• Education and training for HiAP
• New tools for monitoring and evaluation
1

PUBLIC HEALTH REPORTING
10

STRENGTHEN PUBLIC HEALTH AND
OTHER SOCIO-POLITICAL
INSTITUTIONS
10
Strengthen social and political institutions:
• Reporting and laws in parliament includes criteria for health
• Human rights impact assessments
• Media education
• Citizen health forums
• Education
• Interlinked data on health determinants
2

End of
Module 10
Please continue
to Module 11