Spotlight Webinar: ROBINS-I

Follow us @nccmt Suivez-nous @ccnmo
Funded by the Public Health Agency of Canada | Affiliated with McMaster University
Production of this presentation has been made possible through a financial contribution from the Public Health Agency of Canada. The
views expressed here do not necessarily reflect the views of the Public Health Agency of Canada..
ROBINS-I
Presenters:
Jonathan Sterne, BA, MSc, PhD
Julian Higgins, BA, PhD
Judy Brown, PhD
Duvaraga Sivajohanathan, MPH
September 27, 2017 1:00 – 2:30 PM ET

2
Housekeeping
Use Chat to post comments and/or
questions during the webinar
• ‘Send’ questions to All (not
privately to ‘Host’)
Connection issues
• Recommend using a wired
Internet connection (vs.
wireless),
• WebEx 24/7 help line
• 1-866-229-3239
Participant Side
Panel in WebEx
Chat

3
After Today
The PowerPoint presentation (in English and
French) and English audio recording will be made
available.
These resources will be available at:
http://www.nccmt.ca/previous-webinars

4
How many people are watching
today’s session with you?
Poll Question #1
A. Just me
B. 1-3
C. 4-5
D. 6-10
E. >10

Your profession?
Put a √ on your answer (or RSVP via email)
/
Epidemiologist Management (director,
supervisor, etc.)
Allied health
professionals (nurse,
dietician, dental
hygenist, etc.)
Librarian Physician / Dentist Other
5

ROBINS-I
http://www.nccmt.ca/resources/search/281
Episode 35
6

NCC
Infectious
Diseases
Winnipeg, MB
NCC
Methods
and Tools
Hamilton, ON
NCC Healthy
Public Policy
Montreal, QC
NCC
Determinants
of Health
Antigonish, NS
NCC
Aboriginal
Health
Prince George, BC
NCC
Environmental
Health
Vancouver, BC
7

Registry of Methods and Tools
Online Learning
Opportunities
WorkshopsMultimedia
Public Health+
Networking and
Outreach
NCCMT Products and Services
8

9
Poll Question #2
How familiar are you with the
method or tool we are discussing
today?
A. I am not familiar with the method or tool
B. I have heard of the method or tool
C. I have used the method or tool

10
Presenters
Jonathan Sterne, BA, MSc, PhD
Professor of Medical Statistics and
Epidemiology and Head of School,
University of Bristol
Julian Higgins, BA, PhD
Professor of Evidence Synthesis,
University of Bristol

ROBINS-I
Jonathan Sterne and Julian Higgins
Population Health Sciences, Bristol Medical School, University of Bristol
With special thanks to Miguel Hernán, Matthew Page, Barney Reeves, Jelena
Savović and other ROBINS-I collaborators

Need for a tool for
non-randomized studies
• Systematic reviews on the effects of interventions may need to
include non-randomized studies (NRSI)
• long-term or rare outcomes (especially adverse effects)
• interventions at population or organization level
• lack of randomized trials
• Reviews need to critique included studies, but existing tools for
NRSI were
• missing the point
• missing important issues
• not closely aligned with the ‘domain-based’ approach now
widely accepted

Risk of bias
• It is important to determine the extent to which results of the
included studies can be believed
• We do this by assessing risk of bias, which is not the same as...
• random error due
to sampling
variation
• reflected in the
confidence
interval
• bias can occur in
well-conducted
studies
• not all
methodological
flaws introduce bias
QualityImprecision Reporting
• good methods
may have been
used but not well
reported

The new tool
• Cochrane identified the need for a new tool and commissioned
development of one in 2011
• Initial development of ROBINS-I was funded by the Cochrane
Methods Innovation Fund
• Ongoing work on ROBINS-I is funded by the UK Medical Research
Council Methodology Panel (MR/M025209/1)

ROBINS-I: development chronology
Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1
20132011 2012 2014 2015
Cochrane
MIF
application
Meeting in Paris agreed to
establish working groups
on individual bias domains
Face to face meeting of all
collaborators agreed main
features of the new tool
Revision
following
initial piloting
Launched at
Hyderabad, posted at
www.riskofbias.info
Initial scoping meeting at the
Madrid Colloquium
Online survey of
review groups Initial version of the tool
presented at Quebec
Colloquium
Piloting and
cognitive
interviews
Training/piloting
event with key
Cochrane
personnel in Paris
Working groups
established and briefing
document circulated
Changes to
improve
understanding
and usability
Further funding
from MRC
Paper
submitted
2016

ROBINS-I: contributors
• Core group:
• Jonathan Sterne, Barney Reeves, Jelena Savović, Lucy Turner, Julian Higgins
• Collaborators:
• David Moher, Yoon Loke, Elizabeth Waters, Craig Ramsay, Peter Tugwell,
George Wells, Vivian Welch
• Additional working group members:
• Doug Altman, Mohammed Ansari, Nancy Berkman, Isabelle Boutron,
Belinda Burford, James Carpenter, An-Wen Chan, David Henry, Miguel
Hernán, Asbjørn Hróbjartsson, Peter Jüni, Jamie Kirkham, Terri Piggott,
Deborah Regidor, Hannah Rothstein, Lakho Sandhu, Lina Santaguida, Bev
Shea, Ian Shrier, Jeff Valentine, Meera Viswanathan
• And: Jan Vandenbroucke, Jon Deeks, Toby Lasserson, Rachel Churchill,
Alexandra McAleenan, Roy Elbers, Matthew Page, Rebecca Armstrong, Sasha
Shepperd, Hugh Waddington, Su Golder ...

Scope
• The tool concerns the risk of bias (RoB) in the results of a NRSI
that compares the health effects of two or more interventions
• quantitative studies
• estimating effectiveness (harm or benefit) of an intervention
• did not use randomization to allocate units (individuals or
clusters) to comparison groups
Before-after studies
Cohort studies Time series studies
Case-control studies
Non-randomized
experimental studies

Key methodological considerations
• Risk of bias is specific to a particular result from the study
• Assessment is rooted in the notion of a “target randomized trial”
• Important distinction between effects of interest
• effect of assignment to intervention vs starting and adhering
to intervention

Assessing risk of bias in
relation to a target trial
• RoB assessment facilitated by considering NRSI as an attempt to
mimic a high quality hypothetical, pragmatic, randomized trial of
interventions of interest
• “target trial”
• need not be feasible or ethical
The NRSI Target RCT
Research
question
Risk of bias Applicability

Overview of the tool
• Preliminary considerations at protocol stage
• Identify key confounding domains & co-interventions
• For each study:
• Target (idealized) randomized trial to match the study
• PICO; effect estimate of interest (see later)
• For each bias domain (result-level assessment)
• Signalling questions
• Free text descriptions
• Risk of bias judgements
• Overall risk of bias judgement (result-level assessment)
• feed into GRADE

Domain Related terms
Bias due to confounding Selection bias as it is sometimes used in relation to clinical
trials (and currently in widespread use within Cochrane);
Allocation bias; Case-mix bias; Channelling bias
Bias in selection of participants
into the study
Selection bias as it is usually used in relation to
observational studies and sometimes used in relation to
clinical trials; Inception bias; Lead-time bias; Immortal
time bias
Bias in classification of
interventions
Misclassification bias; Information bias; Recall bias;
Measurement bias; Observer bias
Bias due to deviations from
intended interventions
Performance bias; Time-varying confounding
Bias due to missing data Attrition bias; Selection bias as it is sometimes used in
relation to observational studies
Bias in measurement of
outcomes
Detection bias; Recall bias; Information bias;
Misclassification bias; Observer bias; Measurement bias
Bias in selection of the reported
result
Outcome reporting bias; Analysis reporting bias
Bias domains

Allocation bias; Case-mix bias; Channelling bias.
into the study
time bias.
interventions
outcomes
result
Bias domains
Pre- or at-intervention features, for which
considerations of bias in NRSI are mainly distinct
from those in RCTs

Allocation bias; Case-mix bias; Channelling bias.
into the study
time bias.
interventions
outcomes
result
Bias domains
25
Pre- or at-intervention features, for which
considerations of bias in NRSI are mainly distinct
from those in RCTs
Post-intervention features, for which many
considerations of bias in NRSI are similar to those in
RCTs

An epidemiological perspective
Confounding
Misclassification
bias
Selection bias

Confounding
Misclassification
bias
Selection bias
Pre-intervention
Post-intervention
Post-intervention
At-intervention
Pre-intervention
Post-intervention

Confounding
Misclassification
bias
Selection bias
Pre-intervention
Post-intervention
Post-intervention
At-intervention
Pre-intervention
Post-intervention
...confounding
...deviations from
intended intervention
1
4
...missing data
...selection of
participants...
2
5
...classification of
interventions
...measurement of
the outcome
3
6

Confounding
Misclassification
bias
Selection bias
Pre-intervention
Post-intervention
Post-intervention
At-intervention
Pre-intervention
Post-intervention
...confounding
...deviations from
intended intervention
1
4
...missing data
...selection of
participants...
2
5
...classification of
interventions
...measurement of
the outcome
3
6
Selective
reporting bias
...selection of the
reported result
7

Signalling questions
• e.g.
• “Did the authors use an
appropriate analysis method
that controlled for all the
important confounding
domains?”
• “Were outcome data available
for all, or nearly all,
participants?”
Yes
Probably yes
Probably no
No
No information

Risk of bias judgements
Response option Interpretation
Low risk of bias The study is comparable to a well-performed randomized trial
with regard to this bias domain.
Moderate risk of bias The study is sound for a non-randomized study with regard to
this bias domain but cannot be considered comparable to a
well-performed randomized trial.
Serious risk of bias The study has some important problems in this domain of
bias.
Critical risk of bias The study is too problematic in this domain of bias to provide
any useful evidence.
No information No information on which to base a judgement about risk of
bias for this domain.

34
Bias due to confounding
Bias in selection of participants into the study
Bias in classification of interventions
Bias due to deviations from intended
interventions
Bias due to missing data
Bias in measurement of outcomes
Bias in selection of the reported result
1. Seven domains

36
2. Signalling questions
1. Seven domains

37
3. Free text descriptions
1. Seven domains

38
4. Risk of bias judgements
1. Seven domains

39
(5. Predict direction of bias)
1. Seven domains

40
6. Overall risk of bias judgement
1. Seven domains

Overall risk of bias judgement
Low risk of bias The study is judged to be at low risk of bias for all domains (for
the result).
Moderate risk of
bias
The study is judged to be at low or moderate risk of bias for all
domains (for the result).
Serious risk of bias The study is judged to be at serious risk of bias in at least one
domain, but not at critical risk of bias in any domain.
Critical risk of bias The study is judged to be at critical risk of bias in at least one
domain (for the result).
No information There is no clear indication that the study is at serious or critical
risk of bias and there is a lack of information in one or more key
domains of bias (a judgement is required for this).

42
6. Overall risk of bias judgement
1. Seven domains
0. Preliminary considerationsTarget randomized trial
Effect of interest
Important confounders
Important co-interventions

Nurses’ Health Study
• The study compared CVD outcomes in women who reported
current hormone use with women who reported no hormone use
• based on 2-yearly surveys from 1976 to 1986
• Outcomes assessed by self-report, asking families and examining
medical records

Target trial and effect of interest
Participants: Post-menopausal women not currently on
hormone replacement therapy
Intervention: Hormone replacement therapy (oestrogen +/-
progestogen)
Control: No hormone replacement therapy
• We will focus on the effect of starting and adhering to
intervention

Result to be assessed
We will focus on one outcome and
one specific result
Outcome: Major coronary disease
(composite of fatal and non-fatal MI,
CHD, CABG or angioplasty)
Result for inclusion in meta-analysis:
RR 0.56 (95% CI 0.40, 0.80) adjusted
for age and all risk factors (first result
in the abstract)

Menopausal
Symptoms
See GP for medical help Smoke more, eat sweet
high fat foods to cope
GP notes normal
blood pressure & not
overweight
GP notes high
blood pressure /
overweight
TAKE HRT
LESS CHD MORE CHD
DO NOT TAKE HRT
Confounding

Confounding
• A confounding domain C is a pre-
intervention prognostic factor
that predicts whether an
individual receives one or the
other intervention of interest C D
I

Confounding
other intervention of interest C D
I
• Large, well-conducted RCTs permit causal inferences because
they guarantee that treatment assignment is unrelated to
measured or unmeasured prognostic factors
×

C D
I
I C D
I C
D
• We should also avoid
conditioning on common effects
of T and D
other intervention of interest
• We should avoid controlling for
(conditioning on) factors on the
causal pathway from T to the
outcome D
Confounding

Confounding domains
• An a priori list of confounding domains might look like this:
• History of CVD (MI, CHD, stroke, angina, CABG or angioplasty)
• Hypertension
• Adiposity (e.g. BMI)
• Diabetes
• Age
• Socio-economic status
• Smoking
• Reproductive factors (oophorectomy, hysterectomy, past use
of contraceptives (COCs, progesterone))

Nurses’ Health Study
• CHD rates compared as a hazard ratio, adjusting for
• age (5-year categories)
• cigarette smoking (3 categories)
• hypertension (Y/N)
• high cholesterol level (Y/N)
• parental MI (Y/N)
• BMI (5 categories)
• past use of oral contraceptives (Y/N)
• time period
• Risk of bias here is serious, since important confounding domains
were not controlled for, and others were unlikely to be well
characterized by the measured variables

Selection of participants into the study
• Selection bias occurs when exclusion of some eligible participants,
or follow up time of some participants, leads to the association
between intervention and outcome differing what would have
been observed in the target trial
• This phenomenon is distinct from confounding
• We use “selection bias” to refer only to biases that are internal to the
study, and not to issues of generalizability
• Includes “Inception bias”, “Lead-time bias” and “Immortal time bias”
• It is also importantly (and confusingly) different from use of the term
in clinical trials (and Cochrane), which refers to confounding
• Bias will occur if selection into the study is affected by intervention and
by outcome
• Selection out of the study is also a concern
• covered under the missing data domain

Time
Outcome
Existing (prevalent) user of intervention
Outcome
No intervention (comparator)
• This can occur when prevalent users, rather than new (incident)
users of intervention are included in analyses (prevalent user bias,
lead time bias, immortal time bias)
Selection bias due to choice of
follow up time

Time
Outcome
Outcome
follow up time

Time
Outcome
Lead time
Start of use of intervention
Outcome
follow up time

Time
Outcome
Outcome
Unseen event
Lead time
Start of use of intervention
Outcome
follow up time

Selection of participants into the study
• Risk of bias is serious, since prevalent users of HRT were
included, so nurses who experienced coronary disease shortly
after initiation of HRT may have been excluded.

(Aside)
The ITT HRs of CHD for initiators versus noninitiators were 1.42 (0.92–2.20) for
the first 2 years, and 0.96 (0.78 –1.18) for the entire follow-up. The ITT HRs
were 0.84 (0.61–1.14) in women within 10 years of menopause, and 1.12 (0.84
–1.48) in the others. These ITT estimates are similar to those from the Women’s
Health Initiative. Because the ITT approach causes severe treatment
misclassification, we also estimated adherence-adjusted effects by inverse
probability weighting. The HRs were 1.61 (0.97–2.66) for the first 2 years, and
0.98 (0.66 –1.49) for the entire follow-up.

Deviations from intended
intervention
• Deviations from intended intervention are not important when
interest is on the effect of assignment to intervention
• e.g. some people don’t respond to invitations to be screened
• ...providing these deviations reflect routine care
• rather than behaviour that reflects expectations of a difference
between intervention and comparator
• But deviations such as poor adherence, poor implementation and
co-interventions may lead to bias when interest is in the effect
starting and adhering to intervention
• We therefore have different tools for these two effects of interest

A growing family of tools
• Siblings (in development):
• ROBINS-I for cohort studies
• ROBINS-I for case-control studies
• ROBINS-I for before-after studies
• ROBINS-I for controlled before-after studies
• ROBINS-I for instrumental variables analyses
• Cousin (in development):
• ROBINS-E for non-randomized (observational) studies of the causal effects
of exposure on outcome
• Extended family:
• Cochrane risk of bias tool for randomized trials
• QUADAS-2 tool for quality of test accuracy studies
• ROBIS tool for risk of bias in systematic reviews

Concluding remarks
• ROBINS-I is based on extensive and careful consideration of
domains of bias in results of non-randomized studies
• Explicit comparison with a “target” randomized trial
• For pre-intervention and at-intervention bias domains, bias
assessments mainly distinct from randomized trials
• For post-intervention bias domains, bias assessments for NRSI
have many similarities with randomized trials

Presenters
64
Judy Brown, PhD
Health Research Methodologist,
McMaster University
Duvaraga Sivajohanathan, MPH
Health Research Methodologist,
McMaster University

PEBC use of the ROBINS-I
We like the ROBINS-I!
• Explains the biases well
• Separates questions for cohort vs. case-control studies
• Forces the user to think about and evaluate effect of confounders
• No final score –uses levels of risk of bias so it can be incorporated into GRADE if
need be
• But –it’s very large –which is good if you’re not familiar with the tool or critical
appraisal
• So we cut a pasted a little checklist for ourselves so when one is appraising the 5th
cohort –the questions are there and the summary, but not all the in depth
explanations

66
Your Comments/Questions
• Use Chat to post comments
and/or questions
• ‘Send’ questions to All (not
privately to ‘Host’)
Chat
Participant Side
Panel in WebEx

67
Poll Question #3
Could this method or tool be useful
in practice?
A. Very useful
B. Somewhat useful
C. Not at all useful
D. Don’t know

68
Your Feedback is Important
Please take a few minutes to share your thoughts
on today’s webinar.
Your comments and suggestions help to improve
the resources we offer and plan future webinars.
The short survey is available at:
https://nccmt.co1.qualtrics.com/jfe/form/SV_0SR6
Ks5SMQIpiV7

69
Poll Question #4
What are your next steps? (Check all
that apply)
A. Access the method/tool referenced in the
presentation
B. Read the NCCMT summary about the
method/tool described today
C. Consider using the method/tool in
practice
D. Tell a colleague about the method/tool

Share your story!
• Are you using EIDM in your practice? We want
to hear about it!
• Email us: nccmt@mcmaster.ca
• Need support for EIDM? Contact us for help!
• Email us: nccmt@mcmaster.ca
• We typically respond within 24 business hours
70

71
Join us for our next webinar
Spotlight on Methods and Tools:
Evaluability Assessments in Public
Health
Date: Friday November 10, 2017
Time: 1:00 – 2:30pm ET
Evaluability assessments are completed before an evaluation and
are designed to maximize the chances that a subsequent evaluation
will result in useful information. How can the evaluability
assessment method help you? Join us to find out!
Register at: https://health-evidence.webex.com/health-
evidence/onstage/g.php?MTID=ee163884c6bd2b167569d20d9843
891a8

Webinar Series from NCCMT
www.nccmt.ca/webinar-series
• Spotlight on Methods and Tools
• Topic-Specific Methods and Tools
• Online Journal Club
• Peer-to-peer Webinars
72

Funded by the Public Health Agency of Canada | Affiliated with McMaster University
Production of this presentation has been made possible through a financial contribution from the Public Health Agency of Canada. The
views expressed here do not necessarily reflect the views of the Public Health Agency of Canada..
For more information about the
National Collaborating Centre
for Methods and Tools:
NCCMT website www.nccmt.ca
Contact: nccmt@mcmaster.ca

Spotlight Webinar: ROBINS-I

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Spotlight Webinar: ROBINS-I

Similar to Spotlight Webinar: ROBINS-I (20)

More from The National Collaborating Centre for Methods and Tools

More from The National Collaborating Centre for Methods and Tools (20)

Recently uploaded

Recently uploaded (20)

Spotlight Webinar: ROBINS-I