from the Wellbeing policy tools course held by the What Works Centre for Wellbeing and the Centre for Economic Performance on 4 Oct 2017

1. Martin Knapp
Personal Social Services Research Unit, London
School of Economics & Political Science
& NIHR School for Social Care Research
Economic evaluation:
overview
Economic evaluation:
overview
What Works Centre for Wellbeing
London, 4 October 2017

2. 2
Economic evaluation: basic building blocksEconomic evaluation: basic building blocks
Outcomes?
Before-after
change?
Comparative
to what?
Trade-offs?
Costs?

3. Economic evaluation: overview (A-E)
In context: other measures (F-G)
Economic evaluation: overview (A-E)
In context: other measures (F-G)
A. Why economic evaluation?
B. Cost measurement
C. Outcome measurement
D. Study design and methods
E. Making trade-offs
Later (…after Paul…):
F. Example using a randomised trial
G. Example using modelling
Image from Flat Icon http://www.flaticon.com/

4. A Why economic
evaluation?

5. Why?
oBecause resources are scarce.
oSo we – society - cannot meet every need, or agree to
every request, or accommodate every preference.
oAnd so we – society - must choose how to get the best
out of our available resources.
Consequently …
o… any new service or ‘intervention’ will be looked at
very carefully: Is it effective? Is it affordable? Does it
save money? And is it cost-effective?
Decision-makers need economic evidenceDecision-makers need economic evidence
Of course, there are other
relevant criteria too, such
as:
•Respectful of individual
rights, dignity, culture.
•Equitable (fair)
•Protects vulnerable
groups
•Encourages social or
community cohesion
•Long-term sustainability

6. 6
o Commissioning of services
o Provision of services (in-house management)
o Marketing by manufacturers
o Comparison between localities or providers
o Market management by public bodies
o Policy development - national or local
o Lobbying by interest groups
o Guideline development – e.g. NICE
o Regulation of service quality
Uses of economic evaluative evidenceUses of economic evaluative evidence
Different needs and uses could mean that different types
of economic evaluative are required

7. B
Cost
measurement

8. 8
o Decide which costs are relevant
o Collect data on service utilisation
patterns and similar activity indicators
o Attach suitable unit costs to those
indicators
Cost measurement: three stagesCost measurement: three stages

9. Genes
Family
Income
Emply’t
Resilience
Trauma
Phys env
Events
Chance
Influences on individual needs & assetsInfluences on individual needs & assets
People
with
needs &
assets

10. Health care
Social care
Housing
Education
Crim justice
Benefits
Employment
Social netw
Income
Mortality
Genes
Family
Income
Emply’t
Resilience
Trauma
Phys env
Events
Chance
Needs & assets in turn influence costsNeeds & assets in turn influence costs
People
with
needs &
assets

11. Health care
Social care
Housing
Education
Crim justice
NHS
LAs
DCLG
DfE
MoJ
Benefits
Employment
DWP
Firms
Social netw
Income
CVOs
AllMortality
Indiv
Genes
Family
Income
Emply’t
Resilience
Trauma
Phys env
Events
Chance
… impacting on potentially many budgets… impacting on potentially many budgets
People
with
needs &
assets

12. Total cost excluding benefits averaged £5,960 per
child per year, at 2000/01 prices (benefits = £4307)
Romeo, Knapp, Scott (2009) British J Psychiatry 2009
Example: costs of young children with
persistent antisocial behaviour
Example: costs of young children with
persistent antisocial behaviour

13. o All 10-year olds in a London borough, 1970 (n=1689). Led
by Michael Rutter at that time
o Teacher ratings, child questionnaires
o Intensively studied 50% of children with psychological
problems and random 8% of others
o At age 10:
 No problems at school, no clinical diagnosis (65)
 Antisocial behaviour at school, only (61)
 Conduct disorder (16)
 Emotional problems at school, only (32)
 Emotional disorder (8)
o Followed up at age 26-28 …
Research question: What services were used and what costs
incurred between aged 10 and 28?
Example: Longer-term economic impacts of
child behavioural problems
Example: Longer-term economic impacts of
child behavioural problems

14. 0
20000
40000
60000
No problems Conduct
problems
Conduct
disorder
Criminal
justice
Benefits
Relationships
Social care
Health
Education
Scott, Knapp, Henderson, Maughan British Medical Journal 2001
Costs (£)
from ages
10 to 28
Service costs ages 10-28 by childhood
antisocial behaviour at age 10
Service costs ages 10-28 by childhood
antisocial behaviour at age 10

15. Health & social care
system perspective
oHome care
oInpatient services
oOutpatient, A&E
oCommunity health
oGP time
oSocial work inputs
oResidential care settings,
etc.
Public sector perspective
oHealth & social care
oEducation services
oCriminal justice
oWelfare benefits, etc.
Examples of different
study perspectives
Cost breadth depends on study perspectiveCost breadth depends on study perspective
Societal perspective
oAll of the above … plus
oUser & family out-of-
pocket payments
oLost productivity
oCost of unpaid care

16. How many service ‘units’ does an individual utilise? E.g.
how many day centre attendances, GP consultations?
Where does this information come from?
oUser recall: e.g. how many attendances in the past
month? Face-to-face, telephone, postal, web-based
oProxy recall: information from family members or service
staff
oCase files for individual service users
oStaff files (consultation / visit records)
oManagement information systems
oBilling systems
Utilisation patternsUtilisation patterns

17. o Prices or user charges if we think market forces reflect
social opportunity costs
o Expenditure by service providers, divided by volume of
provision or number of users
o Opportunity costs - the value of alternatives or
opportunities missed (the benefit forgone by losing its best
alternative use) …
o … especially important for non-marketed inputs such as
(unpaid) carer time or volunteer activities
o Previously calculated ‘off-the-shelf’ unit costs – the annual
PSSRU volume for health & social care is ‘priceless’
http://www.pssru.ac.uk/project-pages/unit-
costs/2016/index.php
Unit costs – different optionsUnit costs – different options

18. C
Outcome
measurement

19. Ideally, they should:
a.directly link to the service aims
b.be influenced by involving service users
c.capture impacts on everyone affected
d.be quantitative … using robust measures
(good psychometric properties)
e.supported by qualitative evidence reflecting
user experience
f.assess change over time
g.assess change in comparison to an alternative
scenario
h.allow comparison with other studies or
settings
Outcomes – what are they?Outcomes – what are they?

20. Time
Measuring outcomes - 2Measuring outcomes - 2
Wellbeing

21.  Symptoms of illness
 Extent of disability
 Needs (met, unmet)
 Social functioning
 Self-care abilities
 Employment &
activities
 Behavioural
characteristics
 Quality of life (illness-
specific)
 Normalised lifestyle
 Choice & control
 Family well-being
 Carer ‘impact’
 Societal perceptions
Outcomes measured in terms of ‘effects’ (e.g.
in health services research)
Outcomes measured in terms of ‘effects’ (e.g.
in health services research)
Generic indicators:
•Health-related quality of
life (‘health’)
•Quality-adjusted life years
(QALYs)
•Disability-adjusted life
years (DALYs)

22. 22
These effectiveness measures are usually robust in
psychometric terms, and often intuitive.
But … for most of the important decisions faced by
policy-makers or other decision-makers, comparisons
need to be made with a common, generic numeraire.
Options for evaluation:
• Money
• Utility (QALYs)
• Wellbeing
Moving towards generic outcome measuresMoving towards generic outcome measures

23. • Include expenditure saved … but we need to go much
further
• Stated preference – just ask people; but do people
answer honestly or can they do so accurately?
• Revealed preference – observe how people make
decisions already, and infer the value they attach
• Compensation settlements through litigation – unreliable
• Market value of (some?) outcomes; e.g. productivity gains
from higher employment rate
• Social return on investment (SROI) activities try to find
monetary values for wider range of outcomes
Remember: Money is worth (in wellbeing terms) different
amounts to different people (e.g. varies with income)
Outcomes measured in terms of moneyOutcomes measured in terms of money

24.  Utility - a generic measure combining quality and
quantity of life; widely used in health services
research
 Different dimensions of health-related QOL are
combined using societal weights
 The QALY (quality-adjusted life year) is example of
a utility measure – combines years of (extra) life with
the quality of that life
 QALY range: 0 (death) to 1 (perfect health)
 Evaluation question: how many additional QALYs are
generated by treatment (relative to a comparator)
 Most commonly used QALY-generating tool is EQ5D
Outcomes measured in terms of utilityOutcomes measured in terms of utility

25. 25
Wellbeing as the common numeraire to reflect outcomes
Practical challenges:
•Which measure(s) to use? Need to be self-report
•How are measures obtained in practice?
•Can long-term projections of wellbeing be made?
•Aggregation across individuals – possible need for
equity weights and/or adjustments in analyses (e.g.
subgroups)
•Which decision rules given current absence of
thresholds / reference points to guide decisions?
Outcomes measured in terms of wellbeingOutcomes measured in terms of wellbeing

26. D
Study
designs and
methods

27. • Simple before-after calculations (with no ‘parallel’
comparison group)
• Randomised trial – allocate people to interventions
by chance
• Quasi-experimental design – allocate people to
interventions in some other way
• Observational study – look at people in the groups to
which they are ‘allocated’ by routine services
• Mathematical modelling – simulate some parts of the
evaluation using extant data
Evaluation designs – to capture before &
after, but also to allow comparisons
Evaluation designs – to capture before &
after, but also to allow comparisons
Each study design has advantages & disadvantages; they
cost different amounts; they take different time durations

28. E
Making
trade-offs

29. If the policy/practice question is:
‘Does this intervention work?’
Then the economic question is:
‘Is it worth it?’
Often the decision-maker faces difficult
(perhaps controversial?) trade-offs
So … we must define what we mean by ‘work’
and by ‘worth’ – hence we must define
outcomes and costs.
The core economic questionThe core economic question

30. If an intervention is more effective and also more costly, then
calculate the cost per unit gain in outcome
(effectiveness). So … Is it worth it?
Trade-offs: Is it worth it?Trade-offs: Is it worth it?

31. C2 - C1
New service
less effective
and more
costly
0 E2 - E1
New service
less effective
but less costly
New service more
effective but also
more costly
New service
more effective
and also less
costly
C = costs
E = effects
1 = old service
2 = new service
Possible results from cost-
effectiveness analysis
Possible results from cost-
effectiveness analysis
A
B
Y
Z

32. If an intervention is more effective and also more costly, then
calculate the cost per unit gain in outcome
(effectiveness). Is it worth it?
• Show decision-makers the cost-effectiveness findings; ask
them to choose their preferred option. (Health economists
often show acceptability curves (CEACs) to highlight data
uncertainty & willingness-to-pay impacts.)
• Ask decision-makers to be explicit about willingness to
pay.
• Set a threshold, rigidly or as a guide. E.g. NICE in England
& Wales uses cost per QALY to compare across disorders /
diseases: current guide (whose relevance is however
disputed) is £20,000 per QALY.
Trade-offs: Is it worth it?Trade-offs: Is it worth it?

33. C2 - C1
0 E2 - E1
C = costs
E = effects
1 = old service
2 = new service
Using C-E thresholdsUsing C-E thresholds
A
B
Y
Z
The red line represents
one particular cost-
effectiveness threshold:
-Above the line is not
cost-effective
-Below the line is cost-
effective
The blue line represents a
different cost-
effectiveness threshold:
-X was previously cost-
effective, but is not any
longer
X

34. Various types of economic evaluation …
and many different labels
Various types of economic evaluation …
and many different labels
Cost-effectiveness analysis
Cost-benefit analysis
Cost-utility analysis
Cost-consequences analysis
Cost-minimisation analysis
Cost-offset analysis
Social return on investment
Arguably, the label is not important so
long as it is absolutely clear what is
being measured and how

35. Thank you
m.knapp@lse.ac.uk