The document summarizes a literature review examining decision-making methodologies used in health technology appraisal processes. It describes common decision criteria identified in the literature, including efficacy, safety, disease burden, quality of evidence, ethics and cost-effectiveness. Decision-making methods reviewed integrate these criteria through approaches like assigning weights, ranking criteria or using decision rules. The literature search included publications from organizations in several countries.

1. OHTAC decision determinants:
systematic review
Ann-Sylvia Brooker, MSc, PhD
Joanna Bielecki, MISt
Murray Krahn, MD, MSc
April 14, 2015
Saskatoon, SK

2. Research Objective and
Research Questions:
The objective of the literature review is to examine the decision-
making methodologies applicable to the health technology
appraisal process (decision-making process).
Research Questions :
• ***What decision criteria are used during the decision-making
process?
• What methods are used to evaluate the decision criteria (e.g.
assigned weights, ranked, rated)?
• What decision-making methods are used to integrate these
criteria in order to develop recommendations regarding
funding decisions concerning health technologies (e.g. are
decision rules used?).

3. Inclusion Criteria
• Information from national and international health
technology organizations and insurance agencies
making policy recommendations or funding coverage
decisions concerning the technology.
• Published research describing the decision-making
methods of HTA organizations.
• What is
• What should be
• Also – criteria for resource allocation

4. Exclusion Criteria
• Primary research evaluating the effectiveness of
specific aspects of the decision-making process.
• Theoretical/methodological papers that discussed
only one decision criteria (e.g. HTA and ethics) 
These articles were forwarded for consideration to
the relevant OHTAC sub committees .
• Information regarding HTA agency operations.
• Information regarding sources of evidence.
• Literature that described the details of evaluating a
technology.

5. Literature Search
• Search in published academic
literature; between February 2007 and
March 2013.
• Reference list of published articles.
• HTA agency website search.
• An email was sent to every member of
INA-HTA re: decision criteria or
decision-making processes.

6. Results
• 1479 abstracts were scrutinized from
academic literature.
• 18 members of INAHTA responded to
the email. The response rate was 32%
(18/56). 4/18 (22%) did not have
relevant documentation in English or
French. Another 5/18 (28%) responded
that their agency had an advisory role
only. However, 4 organizations had
relevant documentation.
•

7. Results
• 26 documents are included in this
review.
• Documents from US, Canada, Alberta,
Ghana, Germany, Australia, South
Africa, Singapore, Chile, UK, USA,
Netherlands, Brazil, Scotland, New
Zealand, England. (English speaking)

8. Author/Year Title of
Publication
Decision Criteria Evaluation of Criteria Decision-Making
Method
Duclos et al., 2012 Developing
evidence-based
immunization
recommendations
and GRADE
-Epidemiologic features of
the Disease
-Clinical characteristics
of the targeted disease
-Vaccine and
immunization
characteristics
-Economic considerations
-Health-system
considerations
-Social impacts
-Legal considerations
-Ethical considerations
For effectiveness: GRADE
(however, risk of bias
checklist from Cochrane)
For economic & cost-
effective ness: WHO
For adverse effects:
Cochrane
The committee makes
recommendations by
consensus.
Goetghebeur et al.,
2012
Bridging Health
Technology
Assessment
(HTA) and Efficient
Health Care Decision
Making with Multi-
criteria Decision
Analysis (MCDA):
Applying the EVIDEM
Framework to
Medicines Appraisal
-Disease Impact
-Context of Intervention
-Intervention outcomes
-Type of benefit
-Economics
-Quality of Evidence
An MCDA estimate for the
technology was calculated by
combining scores and weights
for each criterion.
The weighting exercise is
not designed to be
prescriptive. The
framework is designed to
support health care
decision making by
stimulating reflection and
exchange and making the
thinking process more
explicit.

9. Author/Year Title of Publication Decision Criteria Evaluation of Criteria Decision-Making
Method
Husereau et al.,
2010
Priority setting for
health technology
assessment at
CADTH
-Alternatives
-Budget impact
-Clinical impact
-Disease burden
-Economic impact
-Available Evidence
Weights are given as follows:
-Alternatives (.081)
-Budget impact (.143)
-Clinical impact (.258)
-Disease burden (.216)
-Economic impact (.167)
-Available Evidence (.135)
Background report
prepared by 2
researchers provides
scores to these criteria.
But, final decision is
made by a committee.
Jehu-Appiah et al.,
2008
Balancing equity
and efficiency in
health priorities in
Ghana: the use of
multi-criteria
decision analysis
-number of potential
beneficiairies
-severity of disease
-cost-effectiveness
-poverty reduction
-vulnerable population
But also social/ethical
acceptability
Intervention complexity
Ranking of criteria as follows
(in descending order of
importance):
1. Vulnerable populations
2. Cost-effectiveness
3. Severity of disease
4. Number of potential
beneficiaries
5. Poverty reduction
Additional “non-
quantifiable” criteria:
-ethical/social acceptability
-intervention complexity
They suggest a
framework that
combines quantitative
and non-quantitative
analytical criteria (e.g.
ethical/social
responsibility and
intervention
complexity) and that
final approval should
occur after a number of
elaborations by
different advisory
panels

10. Author/Year Title of
Publication
Decision Criteria Evaluation of Criteria Decision-Making
Method
Kreis et al.,
2011
From evidence
assessments to
coverage
decisions? The
case example
of glinides in
Germany
-needs
-costs
-safety and
effectiveness
-services are:
adequate, expedient,
cost-effective and, do
not exceed what is
necessary.
A drug can be excluded
based on inexpediency.
First, if the on the basis
of relevant studies, a
drug is inferior to a
comparable therapy
option. Or, second,
studies demonstrating a
benefit are lacking.
A specific decision
making method
were not found in
this article.
Lopert, 2009 Evidence-Based
Decision-Making
within
Australia’s
Pharmaceutical
Benefits Scheme
-Efficacy
-Safety
-Quality
-Cost-effectiveness
-Clinical need
-Uncertainty in cost-
effectiveness
-Total cost to PBS
-Ability to constrict a
restriction
-Potential for adverse
outcomes
-Affordability of drug
-Rule of Rescue
No fixed weight for
these factors. The
factors will be of
greater or lesser
importance in different
situations.
An expert
committee
deliberates on the
evidence and
provides its
recommendation to
the Minister. The
Minister may veto a
positive
recommendation

11. HTA agency Country Decision
Criteria
Evaluation of
Criteria
Decision-making method
PHARMAC New Zealand (a) the health
needs
(b) the
particular
health needs of
Maori and
Pacific peoples;
(c) the
availability and
suitability of
existing
medicines,
and related
products and
related things;
(d) clinical
benefits/risks;
(e) cost-
effectiveness
(f) the
budgetary
(g) the direct
cost to health
service users;
(h) the
Government’s
priorities for
health funding,
(i) other criteria
as PHARMAC
thinks fit.
Pharmac gives
weight to each
criterion as
PHARMAC
considers
appropriate.
PHARMAC makes decision after reviewing
evidence and consulting with public, groups,
that may be affected by its proposals, and with
expert advisory committees.

12. HTA agency Country Decision Criteria Evaluation of
Criteria
Decision-making method
NICE England -ICER
-certainty of the
ICER estimate
-certainty of the
health related
quality of life
measure
-benefits beyond
those captured in
the QALY measure
-whether
technology is a
“life-extending
treatment at the
end of life”
-aspects that
relate to non-
health objectives
of the NHS (e.g.
whether a
substantial
proportion of
savings/ benefits
occur outside the
NHS and PSS.)
The appraisal
committee uses
different decision
rules depending
on whether the
ICER is (1) less
that £20,000 per
QALY gained (2) in
the range of
£20,000 to
£30,000 per QALY
gained, and (3)
above a most
plausible ICER of
£30,000 per QALY
gained.
The Appraisal Committee does not
use a precise maximum acceptable
ICER above which a technology would
automatically be defined as not cost
effective or below which it would.
But a stronger case must be made for
an ICER (2) in the range of £20,000 to
£30,000 per QALY gained and even
more for an (3) ICER above a most
plausible ICER of £30,000 per QALY
gained.

13. Summary of Decision Criteria

14. Decision Criteria Terms
Efficacy- Potential
benefit of the
intervention
(mortality, morbidity,
PRO)
*****
Health benefit, potential health gain in terms of
mortality (saving life, life expectancy gains, average life-
year benefit per patient, prolongation of disease-free
survival); morbidity (health benefit, enhanced health
outcome, relative advantage, incremental health gain);
patient-reported outcomes (quality of life, number of
QALYs gained per patient, disability adjusted life years,
relative value to patient). Overall gain in quality of care.
Health benefits relative to current standard therapy.
Safety of the
intervention
***
Side (adverse) effects, unintended consequences, safety
and tolerability, risks, risk management, harm, risk of
event, risk of toxicity compared with standard therapy.
External impact of
intervention
*
Impact on patient’s family, possible harms to others,
infectious disease involved, population effect (positive or
negative), herd immunity, public health interest, social
impact, social benefit, prevention of ill health,
prevention.
Need (clinical)
***
Treatment alternatives, comparative intervention
limitations (unmet needs), availability of alternative
treatments, availability of effective alternative
treatments, availability of preventative measures, clinical
need, emergencies and need.
Disease determinants Factors responsible for the persistence of the burden.
Disease burden-
clinical
**
Prevalence of disease, incidence of disease, number of
patients, severity of disease, impact of disease/condition
on quality of life, number of potential beneficiaries,
indirect beneficiaries,

15. Decision criteria - TermsDecision Criteria Terms
Quality of
evidence (re:
effectiveness
research)
***
Availability of evidence, strength of evidence, consistency
of findings, quality of data, choice of end points, validity
of data, certainty, precision of effect, selection of
studies, proof, scientific evidence, time of assessment in
technology development, therapy mechanism of action.
Relevance of
evidence/genera
lizability/
effectiveness in
real practice
*
Relevance of evidence, representativeness of patients
(studies vs. real world), representativeness of technology
user (e.g. skill of surgeon or health care practitioner in
studies vs. real world), representativeness of context
(e.g. acute vs long term care; country differences),
response rate, patient compliance, level of
generalization, effectiveness in real practice, evidence of
effectiveness.

16. Decision criteria - TermsDecision Criteria Terms
Ethics and moral
issues
***
Consistency with societal values, moral
consequences of HTA, ethical implications, rule of
rescue.
Vulnerable and
needy
populations
**
Vulnerable populations (e.g. age, gender,
geography, ethnicity, indigenous populations), life
extending treatment for end of life, social groups
with high risk and/or increased vulnerability, age of
targeted group, population equity, positive poverty
reduction.
Human dignity Human integrity and dignity, basic human rights,
meets patient’s basic needs.
Patient autonomy
and patient
preference **
Patient autonomy, patient preference. (e.g.
patient-centered healthcare? Is there patient &
public involvement?)
Equity, fairness
and justice
*
Equity, fairness, health equity, equality,
distributive justice, formal justice, procedural
justice, social justice, addressing health status
inequalities at population level, geographical
equity, equity of access, timeliness of access.
Utility Utility, utilitarianism.
Solidarity Solidarity, collectivism, cohesion.
Cultural aspects Cultural and religious convictions.

17. Decision Criteria Terms
Disease burden-cost
**
Cost to treat disease, cost to prevent disease, national cost of
the disease/condition to the health care system.
Opportunity costs Opportunity costs to the population.
Efficiency / value for
money for patient.
*****
Maximizing impact on health for a given level of resource
compared to available alternatives for this patient group (e.g.
cost-effectiveness, cost-utility, cost per QALY, cost-effectiveness
utility curves, cost consequence analysis.). Could include
comparisons of interventions with different objectives (e.g.
psycho-therapy vs. pain meds).
Quality of evidence
(re: efficiency & cost
estimates)
*
Uncertainty in QALYs, possible benefit/harms not included in the
QALY (i.e. non-health benefits, social benefits)
Cost per patient. Cost per patient, unit cost.
Financial/budget
impact-costs of
intervention *
Budget impact, affordability, operating and start-up costs,
national medical costs per year, financial impact on government.
Financial impact-
savings of
intervention
Cost-savings, national savings in terms of costs of absences per
year, savings in terms of medical costs.
Costs (benefits) of
externalities
Costs of externalities such as: impact on patient’s family,
possible harms to others, infectious disease involved, population
effect (positive or negative), public health interest, social
impact, social benefit, prevention of ill health, prevention.

18. Decision criteria - Terms
Decision Criteria Terms
Priorities:
national, local
level etc.
National priorities, local priorities (does it meet a
local health need? public expectations?),
international priorities, strategic direction.
Stakeholder
interests and
pressures;
political aspects.
Advocacy, pressure from patient groups, pressure
from physician groups, producer interests,
recommendations made by other countries, clinical
expert opinions; political pressure, political impact.

19. Decision Criteria Terms
Feasibility (at the
organizational
level and at the
system level)
***
System requirements, physical environment, system
capacity, local capacity, ability to implement,
implementation, organization’s structure,
organizational burden, logistics, process, well-
organized, feasibility of delivery; all enablers and
barriers to diffusion within the health system
infrastructure (operational, capital, human resources,
legislative, regulatory) including ease of integration
into local community, system integration, acceptability.
Flexibility of
implementation *
Flexibility, reversibility, revisability, ability to evaluate,
provision to revision.
Ensuring adequate
quality and
sustainability of
intervention
Appropriate use of intervention, appropriateness,
appropriate setting/level of service, sustainability,
longevity.