CHE Economic evaluation seminar by Dr Hazel Squires, Research Fellow, University of Sheffield 11 February 2016

1. A methodological framework
for developing the structure
of Public Health economic
models
Dr Hazel Squires, ScHARR, University of Sheffield
Prof. Jim Chilcott, Prof. Ron Akehurst,
Dr. Jennifer Burr, Prof. Mike Kelly

2. Introduction
• Why we need a conceptual modelling framework
specific to Economic Modelling in Public Health
(PHE)
• Methods for developing the PHE conceptual
modelling framework
• An outline of the PHE conceptual modelling
framework
• Conclusions

3. Background
• Healthcare agencies such as the National
Institute for Health and Care Excellence (NICE)
need to decide how to spend their money.
• Mathematical models are used to provide a
rational, coherent & transparent framework of
the cost-effectiveness of healthcare
interventions.
• Inappropriately simple models and lack of
justification may lead to poor validity and
credibility, resulting in suboptimal allocation of
resources.

4. Current methods for dealing
with structural uncertainty
1) Retrospective: following model implementation
by expressing the impact of uncertainties upon
the model results.
• Scenario analysis
• Model averaging
2) Prospective: considering the process through
which decisions are made around the
conceptualisation, structuring and
implementation of the model.
• Model development practice in health economics is
extremely varied (Chilcott, 2010).
Chilcott J, Tappenden P, Rawdin A, Johnson M, Kaltenthaler E, Paisley S et al. Avoiding and identifying errors in health
technology assessment models: qualitative study and methodological review. Health Technol Assess 2010; 14(25).

5. What is a Conceptual
Modelling (CM) Framework?
• ‘A set of steps that help to guide modellers
through the development of a model structure,
from developing and describing an
understanding of the decision problem to the
abstraction and non-software specific
description of the quantitative model, using a
transparent approach which enables each stage
to be shared and questioned.’
Based upon Kaltenthaler E., Tappenden P., Paisley S. and Squires H. Identifying and reviewing evidence to inform the
conceptualisation and population of cost-effectiveness models, No. 14, 2011 and Robinson S. Conceptual
modelling for simulation Part I: definition and requirements. Journal of the Operational Research Society 2008;
59:278-290.

6. Key potential benefits of a
CM Framework
• Aids the development of modelling objectives;
• Provides tools for communication with stakeholders;
• Guides model development and experimentation;
• Improves model validation (developing the right model);
• Improves model verification (developing the model right);
• Allows model reuse;
• Helps to characterise structural uncertainties and identify
primary research needs.

7. Economic evaluation of Public
Health compared with clinical
interventions
• A key objective of Public Health is reduction of health
inequities;
• PH interventions tend to:
- Be multi-component with complex causal chains;
- Operate within dynamically complex systems;
- Be dependent upon human behaviour;
- Be dependent upon the social determinants of health,
requiring consideration of non-health costs & outcomes;
• It is much less clear what a ‘good’ outcome is;
• The culture & politics of the system is important in
choosing & assessing interventions

8. Methods for developing PHE
conceptual modelling framework
1) Literature review of key challenges in PHE modelling
(Squires et al., 2016)
2) Review of conceptual modelling frameworks in broader
literature.
3) Qualitative research to understand modellers’ experiences
with developing the structure of PH economic models &
their views about the benefits/ barriers of using a CM
framework.
4) Pilot of the draft CM framework within a case study of
diabetes screening & prevention.
5) Evaluation via qualitative analysis & theory-based
evaluation.
Squires H, Chilcott J, Akehurst R, Burr J, Kelly M. A systematic literature review of the key challenges for developing the
structure of public health economic models. Int. J. of Public Health. Epub ahead of print.
http://www.ncbi.nlm.nih.gov/pubmed/26747470

9. The CM framework:
Key principles
(1) A systems approach to Public Health modelling should
be taken (feedback loops & unintended consequences
are important);
(2) Developing a thorough documented understanding of
the problem is imperative prior to and alongside
developing and justifying the model structure;
(3) Strong communication with stakeholders and members
of the team throughout model development is essential;
(4) A systematic consideration of the determinants of health
is central to identifying all key impacts of the
interventions within Public Health economic modelling.

10. A systems approach

11. The CM framework:
Key principles
(1) A systems approach to Public Health modelling should
be taken (feedback loops & unintended consequences
are important);
(2) Developing a thorough documented understanding of
the problem is imperative prior to and alongside
developing and justifying the model structure;
(3) Strong communication with stakeholders and members
of the team throughout model development is essential;
(4) A systematic consideration of the determinants of health
is central to identifying all key impacts of the
interventions within Public Health economic modelling.

12. The CM framework:
Key principles
(1) A systems approach to Public Health modelling should
be taken (feedback loops & unintended consequences
are important);
(2) Developing a thorough documented understanding of
the problem is imperative prior to and alongside
developing and justifying the model structure;
(3) Strong communication with stakeholders and members
of the team throughout model development is essential;
(4) A systematic consideration of the determinants of health
is central to identifying all key impacts of the
interventions within Public Health economic modelling.

13. The CM framework:
Key principles
(1) A systems approach to Public Health modelling should
be taken (feedback loops & unintended consequences
are important);
(2) Developing a thorough documented understanding of
the problem is imperative prior to and alongside
developing and justifying the model structure;
(3) Strong communication with stakeholders and members
of the team throughout model development is essential;
(4) A systematic consideration of the determinants of health
is central to identifying all key impacts of the
interventions within Public Health economic modelling.

14. The determinants of health
Dahlgren G., Whitehead M. Policies and strategies to promote social equity in health. 1991. Institute
for Future Studies, Stockholm.

15. CM framework overview
Aligning the framework with
the decision making process
Identifying relevant
stakeholders
Understanding the problem
i) Developing a conceptual model of the problem
ii) Establishing resource pathways
Developing and justifying the model structure
i) Reviewing existing economic evaluations
ii) Choosing specific model interventions
iii) Determining the model boundary
iv) Determining the level of detail
v) Choosing the model type
vi) Developing a qualitative description of the quantitative model

16. CM framework overview
Aligning the framework with
the decision making process
Identifying relevant
stakeholders
Understanding the problem
i) Developing a conceptual model of the problem
ii) Establishing resource pathways
Developing and justifying the model structure
i) Reviewing existing economic evaluations
ii) Choosing specific model interventions
iii) Determining the model boundary
iv) Determining the level of detail
v) Choosing the model type
vi) Developing a qualitative description of the quantitative model

17. CM framework overview
Aligning the framework with
the decision making process
Identifying relevant
stakeholders
Understanding the problem
i) Developing a conceptual model of the problem
ii) Establishing resource pathways
Developing and justifying the model structure
i) Reviewing existing economic evaluations
ii) Choosing specific model interventions
iii) Determining the model boundary
iv) Determining the level of detail
v) Choosing the model type
vi) Developing a qualitative description of the quantitative model

18. Stakeholders
Stakeholders are identified based upon the
classification from Soft Systems Methodology
(SSM):
• Customers which might include patient
representatives and lay members;
• Actors which might include clinical experts and
epidemiologic experts for all relevant diseases
and methods experts;
• System owners which might include policy
experts (in addition to some of the people
identified as actors).

19. CM framework overview
Aligning the framework with
the decision making process
Identifying relevant
stakeholders
Understanding the problem
i) Developing a conceptual model of the problem
ii) Establishing resource pathways
Developing and justifying the model structure
i) Reviewing existing economic evaluations
ii) Choosing specific model interventions
iii) Determining the model boundary
iv) Determining the level of detail
v) Choosing the model type
vi) Developing a qualitative description of the quantitative model

20. Example conceptual
model of the problem
Contraception project example:
Why is this a problem?
_ _
Maximise health Minimise costs to NHS & PSS Minimise costs to other sectors
_ _ _ _ _ _ _ _ _
Poor health outcomes of the mother
Low birth weight babies + Poor long term socioeconomic outcomes
+ + +
Miscarriage/ stillbirth Abortion Unwanted teenage birth
+ + +
What is the problem? +
Unintended teenage pregnancies STIs
_ + +
Poor contraceptive use by sexually active teenagers
(interaction between a male & female)
+ +
Disadvantaged background
Competing risks
Dependent on history & timing of STIs

21. Questions to help develop
diagram: The disease &
determinants of health
• Have any relevant disease natural histories been
captured?
• Are the following determinants of health important in
determining outcomes & in what way:
• Age, sex & other inherent characteristics of the population of
interest?
• Individual lifestyle factors?
• Social and community networks?
• Living & working conditions & access to essential goods &
services?
• General socioeconomic, cultural & environmental conditions?

22. Questions to help develop
diagram: Dynamic complexity
• Are there any other (positive or negative)
consequences of each concept?
• Are there any other possible causal links
between the factors? (to establish whether there
are any feedback loops)
• Could there be any other factors which explain
both of these outcomes, for links which may not
be causal, but correlated?
• Are there interactions between different sets of
people?
• Is timing/ ordering of events important?

23. Questions to help develop
diagram: Intervention outcomes
• What is a good outcome?
• What would happen in the absence of the
interventions versus as a result of the interventions
– would behaviour be prevented or delayed?
• What evidence exists to describe the outcomes of
the intervention/ comparator over time? Are
behavioural outcomes important?
• Are there any determinants of health reported by
the effectiveness studies which are not included
within the causal diagram? Can such a relationship
be described?

24. Questions to help develop
diagram: Dynamic complexity
of interventions
• Might a third party act to reduce or increase the
impact of interventions?
• Are there any substantial impacts of social and/or
community networks upon intervention
effectiveness? Will these impacts be captured over
the long term within the effectiveness evidence?
• Are there any substantial impacts of the
interventions upon other lifestyle factors?
• Might the interventions have other impacts not
already considered?

25. Sources of evidence
Starting with high
yield sources
Individual stakeholder
assumptions & beliefs
Stakeholder
discussion
Modeller assumptions
& beliefs
Project scope
Literature
sources
Existing diagrams/
previous work
Conceptual model of the problem

26. CM framework overview
Aligning the framework with
the decision making process
Identifying relevant
stakeholders
Understanding the problem
i) Developing a conceptual model of the problem
ii) Establishing resource pathways
Developing and justifying the model structure
i) Reviewing existing economic evaluations
ii) Choosing specific model interventions
iii) Determining the model boundary
iv) Determining the level of detail
v) Choosing the model type
vi) Developing a qualitative description of the quantitative model

27. CM framework overview
Aligning the framework with
the decision making process
Identifying relevant
stakeholders
Understanding the problem
i) Developing a conceptual model of the problem
ii) Establishing resource pathways
Developing and justifying the model structure
i) Reviewing existing economic evaluations
ii) Choosing specific model interventions
iii) Determining the model boundary
iv) Determining the level of detail
v) Choosing the model type
vi) Developing a qualitative description of the quantitative model

28. Develop understanding of the
problem
Assess whether there is
an existing model which
could be employed
Identify strengths &
limitations of different
model structures
Identify strengths &
limitations of different
model types
Identify key variables which
generally affect model results
(incl. any not already
identified) & key variables
included within the causal
diagram which do not
Identify the sort
of data available
Identify factors with not many
causal links & assess whether
they would have a substantial
impact upon the difference
between outcomes of
interventions & comparators
Identify types of
outcomes reported
Identify long term
evidence & mechanisms
Describe effectiveness of
interventions (to help
choose which to model
& for parameterisation)
Model boundary Model detail Model type
Discuss potential model perspectives,
outcomes, interventions &
populations with stakeholders
Review existing health
economic models
Review effectiveness of
relevant interventions
Review evidence of
relationships between factors

29. Does the factor have many causal links?
Yes No
Is the factor likely to have a substantial
impact upon the difference between costs &
effects of the interventions? This may be
based upon (though not limited to):
(1) the review of economic evaluations;
(2) the description of resource pathways;
(3) clinical papers describing the causal links;
(4) existing models in similar areas which
describe the impact of the factor;
(5) methodological choices eg. discounting;
(6) expert advice.
Yes No
INCLUDE
Is the factor associated with the interventions,
populations & outcomes being modelled?
EXCLUDE
INCLUDE
EXCLUDE
Yes No
Yes
Is the impact of the factor predominantly
captured by other included factors?
Yes
EXCLUDE
No
Would stakeholders prefer to
include the factor for model
credibility AND is it relatively easy
to incorporate in terms of
modelling skill & data availability?
INCLUDE
No
Will the model only assess one intervention AND
is the intervention likely to be cost-effective AND
does the factor only further increase benefits
Yes
No
Determining
the model
boundary

30. A systems approach

31. Determining level of detail
Key types of model assumptions/ simplifications:
1. The relationship between the included factors over time;
2. The extrapolation of study outcomes;
3. The level of detail used to describe each included factor;
4. How interventions will be implemented in practice.
Questions to help the modeller for each of these are
proposed within the framework.

32. Choosing model type (1)
A B C D
Cohort/ aggregate level/ counts Individual level
Expected value,
continuous state,
deterministic
Markovian, discrete
state, stochastic
Markovian, discrete
state
Non-Markovian,
discrete state
1 No
interaction
Untimed Decision tree
rollback
Simulation decision
tree
Individual sampling model: Simulated
patient-level decision tree
2 Timed Markov model
(deterministic)
Simulation Markov
model
Individual sampling model: Simulated
patient-level Markov model
3 Interaction
between
entity and
environment
Discrete
time
System dynamics
(finite difference
equations)
Discrete time
Markov chain
model
Discrete-time
individual event
history model
Discrete individual
simulation
4 Continuous
time
System dynamics
(ordinary
differential
equations)
Continuous time
Markov chain
model
Continuous time
individual event
history model
Discrete event
simulation
5 Strong interactions
between entities. Spatial
aspects important.
X X X Agent-based
simulation
Revised version of taxonomy by Brennan A, Chick SE, Davies R. A taxonomy of model structures
for economic evaluation of health technologies. Health Econ 2006; 15(12):1295-1310.

33. Choosing
model type
(2)
Determine the most appropriate model type for the characteristics of the problem.
Is this feasible within the time and resource constraints of the decision making process given:
(i) the data available?
AND
(ii) the accessibility of any existing relevant good quality economic evaluations for use as
a starting point?
AND
(iii) the expertise of the modeller?
Are you intending to use the
model again for other
projects?
Can you answer the decision makers’ question
with a few provisos and uncertainties with a
simpler model type?
Yes No
Explore with the
decision maker the
most useful purpose of
the modelling given the
project constraints
Develop the simpler model
type, documenting the
provisos, uncertainties &
implications of the
simplifications
NoYes
Do you think a
simpler model type
would lead to the
same conclusions?
Develop
the
model
Yes No
Develop the more
complex model
Develop the simpler model,
documenting the provisos,
uncertainties & implications
of the simplifications
Yes No

34. Conclusions
• Very little research has been undertaken
around conceptual modelling in health
economic evaluation.
• We have developed a conceptual modelling
framework for use within Public Health
economic models (paper under review).
• This has yet to be tested on a range of case
studies.
• If anybody wants to know more about the
framework or potentially use it, let me know!
Email: h.squires@sheffield.ac.uk
PhD thesis: http://etheses.whiterose.ac.uk/5316/