1. Mind the Gap:
Risk of Bias in Pediatric Trials
StaR Child Health Summit
Winnipeg, Manitoba
May 7, 2012
Michele Hamm, MSc
Alberta Research Centre for Health Evidence

2. Conflict of Interest Disclosure
Michele Hamm
Mind the Gap: Risk of Bias in Pediatric Trials
• Received grant/research support from: KT Canada

3. Introduction
 There is a growing body of literature documenting
the limitations and methodological flaws of
pediatric research, specifically risk of bias.
 The introduction of bias into a trial can lead to the
overestimation of treatment benefits or
underestimation of treatment harms.

4. Risk of Bias
 Risk of bias relates to internal validity
 Design features associated with effect estimate
magnitude:
 Sequence generation
 Allocation concealment
 Blinding - participants/personnel, outcome assessors
 Incomplete outcome data
 Selective outcome reporting
 “Other” sources of bias

5. Progress of the RoB SDG
2nd StaR Child Health Summit: Empirical
evidence showing that pediatric RCTs are at
high/unclear risk of bias.
3rd StaR Child Health Summit:
1) Mixed methods study: Why are biased trials
being conducted?
2) Knowledge translation: Development and pilot
testing of a wiki-based educational resource

6. Mixed Methods Study of
Pediatric Trialists
 Objective: To determine the barriers and
facilitators faced by pediatric trialists in the design
and conduct of methodologically rigorous
trials, using both quantitative and qualitative data.
 Explanatory mixed methods design
 Survey
 Semi-structured interviews

7. Survey Methods
 Internet-based survey (SurveyMonkey/REDCap)
 Surveyed corresponding authors of pediatric trials
published in 2008 and 2009
 Entire sample of Canadian researchers (n=90)
 Random sample of international researchers (n=600)
 Surveyed from MICYRN membership (n=163)
 Questions to determine:
1) knowledge and awareness of bias
2) perceived barriers and facilitators in conducting trials
3) utility of potential KT strategies for future
interventions

8. Survey Results
 23.0% response rate (186/807)
 44.6% trained in medicine; 35.5% had formal
research training
 Median number of trials involved in:
 As PI: 3 (IQR 1-5)
 As part of study team: 5 (IQR 2-10)
 Most common subspecialties represented:
 Public health (8.6%)
 Developmental, psychosocial, and learning problems
(7.5%)
 Mental health or psychiatry (7.0%)

9. Survey Results
Barriers:
 Lack of sufficient funding (70.3%);
 Overwhelming volume of literature (63.1%);
 Logistics make it difficult to minimize bias (52.9%)
Facilitators:
 Interest in staying current with literature (93.0%);
 Opportunities to discuss methods with knowledgeable
colleagues (92.8%);
 Rigorous methods encouraged by colleagues (80.4%)

10. Interview Methods
 Semi-structured interviews building upon
quantitative survey data
 MICYRN survey respondents invited to
participate in an interview
 Target sample size of 12 pediatric trialists
 Questions to determine:
 Relationships between participants’
beliefs, behaviours, and attitudes about conducting
research on children and appropriate design and
conduct of methodologically sound trials

11. Interview Results
 13 interviews conducted
 Necessary to augment original sample:
 3 respondents from survey, 3 from MICYRN, 7
referrals
 7 trained in medicine, 4 trained in both medicine
and research, and 2 trained in research
 Subspecialties represented:
anesthesiology, clinical epidemiology, critical
care, emergency medicine, infectious
disease, neonatology, neurology, oncology, psych
ology, rheumatology

12. Interview Results
Individual Factors - Barriers
 Knowledge/training regarding research methods
 Lack of formal training – on the job learning
 Emphasize certain aspects, overlook others

13. Interview Results
Individual Factors – Barriers
“Probably we don’t look at, we don’t know all the bias that
can happen in a trial because we don’t check, we don’t
believe there’s bias. We may miss some, we may forget
some, and then do not report the bias because we don’t
know it exists.”
“Because there’s almost zero research training in the clinical
curriculum for most clinicians these days. Like there’s
almost nothing in the med school program, there’s almost
nothing in the rehab program – there really needs to be
somebody on the protocol who’s got a little bit more
training.”

14. Interview Results
Individual Factors – Facilitators
 Sense of ownership
 Opportunity to generate enthusiasm, gain support, and
educate colleagues

15. Interview Results
Individual Factors – Facilitators
“So you really have to take the time to engage people and be the
one that’s proactive, engaging them. Because they’re busy, they
might not even know what your study is unless you’re the
change agent that really goes out there and talks to them about it
and gets them motivated about why you think it’s important.”
“Listen to what [your colleagues] need to execute the study so that
when you develop your protocol, you’ve built that into the
approach. Or, if you couldn’t, you’ve at least had that dialogue
with them about how scientifically you can’t be as flexible as
might be ideal… so that they at least understand the rationale.”

16. Interview Results
Institutional Factors – Barriers
 Negative research culture
 Lack of recognition of distinction between clinical
care and clinical research
 Variable across institutions

17. Interview Results
Institutional Factors – Barriers
“I think that other people view [research] as kind of a thorn in their
side. It’s something they play along with if they have to and the
division head tells them they have to.”
“I think a lot of investigators really have a hard time separating
what decision they would make clinically from what decision
they would make as part of a trial… because the feeling is I
want to be convenient to the family, and I really know this stuff
because I’m an expert in this clinical area, and I don’t think they
realize that there’s a pretty clear demarcation between what you
do in clinical care and what you do as research.”

18. Interview Results
Institutional Factors – Facilitators
 Cohesive study teams
 Clinicians, methodologists, research staff
 Networks within subspecialties
 Reliable internal review processes

19. Interview Results
Institutional Factors – Facilitators
“We have the help of the research institute and you can have a
person for any kind of question or any kind of design that can
help, and we have access to those kinds of resources.”
“I think the fact that [research network] is there enables you to
think of multi-centre RCTs, whereas if it wasn’t there, you’d
kind of have to go and find things from scratch. But by existing,
it brings people together with shared interests and I think that
that is a huge asset when it comes to even the thought of
designing a multi-centre RCT. It’s like you want to design one
for [research network].”

20. Interview Results
Policy Factors – Barriers
 Funding
 Ethics review process

21. Interview Results
Policy Factors – Barriers
“We all tend to want to make the budget as small as we can to increase
our chances to actually get it funded and the reality is that some
trials really require the full-time effort of somebody who’s got a lot
of experience, and therefore comes with a price tag. And it can be
hard to make the argument to ensure that you’ve got funding, right?
So I think that’s where you start cutting other corners, and you
don’t have the data quality, and at the end of the day, you maybe
don’t have the rigorous, homerun kind of trial that you had
envisioned.”
“Most of the problem is to ask for revisions and they are not consistent
one between the others. So you can have a question in one and the
other one… wants a different answer.”

22. Interview Results
Policy Factors – Facilitators
 None

23. Discussion
 Internal validity is not a primary concern –
overshadowed by pragmatics and generalizability
 Internal validity is a prerequisite for external validity
 Lack of formal training and a negative research
culture contribute to acceptance of sub-optimal
trials
 Importance of mentorship and clinician-scientists

24. Future Directions
 Objectives: To design and evaluate a tailored KT
intervention to improve methodological rigor in
child health trials.
 Researcher involvement sought throughout
 Potential interventions: online modules, checklists

25. Future Directions
 Consideration of traditional KT interventions
 Adaptation for changing researcher behaviour, rather
than clinician behaviour
 Social media component
 Wiki platform
 Incorporates successful elements of KT interventions:
interactivity, presentation of materials in multiple
formats, endorsement by opinion leaders
 Informed by Diffusion of Innovations, Social
Influences Theory, and Theory of Planned Behaviour

26. Future Directions

27. Future Directions

28. Future Directions

29. Future Directions

30. Future Directions

31. Future Directions

32. Future Directions
Pilot testing:
 Trialists, systematic reviewers, trainees
 Preliminary feedback: content, format, usability
 Participants will review wiki, take part in a short
interview
 Revisions to wiki based on input

33. Future Directions
 RoB wiki to be used as a model for other StaR
Child Health SDGs
 Evaluation of effectiveness
 RoB in pediatric trials
 Knowledge translation intervention

34. Acknowledgements
Supervisory Committee: RoB SDG Members:
Lisa Hartling Jamie Brehaut
Terry Klassen An-Wen Chan
Shannon Scott Jeremy Grimshaw
David Moher Martin Meremikwu
Ginny Moyer
Prathap Tharyan