Risk of Bias_StaR Child Health Summit_07May12

Uploaded on


  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Mind the Gap:Risk of Bias in Pediatric TrialsStaR Child Health SummitWinnipeg, ManitobaMay 7, 2012Michele Hamm, MScAlberta Research Centre for Health Evidence
  • 2. Conflict of Interest Disclosure Michele Hamm Mind the Gap: Risk of Bias in Pediatric Trials• Received grant/research support from: KT Canada
  • 3. Introduction There is a growing body of literature documenting the limitations and methodological flaws of pediatric research, specifically risk of bias. The introduction of bias into a trial can lead to the overestimation of treatment benefits or underestimation of treatment harms.
  • 4. Risk of Bias Risk of bias relates to internal validity Design features associated with effect estimate magnitude:  Sequence generation  Allocation concealment  Blinding - participants/personnel, outcome assessors  Incomplete outcome data  Selective outcome reporting  “Other” sources of bias
  • 5. Progress of the RoB SDG2nd StaR Child Health Summit: Empirical evidence showing that pediatric RCTs are at high/unclear risk of bias.3rd StaR Child Health Summit:1) Mixed methods study: Why are biased trials being conducted?2) Knowledge translation: Development and pilot testing of a wiki-based educational resource
  • 6. Mixed Methods Study ofPediatric Trialists Objective: To determine the barriers and facilitators faced by pediatric trialists in the design and conduct of methodologically rigorous trials, using both quantitative and qualitative data. Explanatory mixed methods design  Survey  Semi-structured interviews
  • 7. Survey Methods Internet-based survey (SurveyMonkey/REDCap) Surveyed corresponding authors of pediatric trials published in 2008 and 2009  Entire sample of Canadian researchers (n=90)  Random sample of international researchers (n=600) Surveyed from MICYRN membership (n=163) Questions to determine: 1) knowledge and awareness of bias 2) perceived barriers and facilitators in conducting trials 3) utility of potential KT strategies for future interventions
  • 8. Survey Results 23.0% response rate (186/807) 44.6% trained in medicine; 35.5% had formal research training Median number of trials involved in:  As PI: 3 (IQR 1-5)  As part of study team: 5 (IQR 2-10) Most common subspecialties represented:  Public health (8.6%)  Developmental, psychosocial, and learning problems (7.5%)  Mental health or psychiatry (7.0%)
  • 9. Survey ResultsBarriers: Lack of sufficient funding (70.3%); Overwhelming volume of literature (63.1%); Logistics make it difficult to minimize bias (52.9%)Facilitators: Interest in staying current with literature (93.0%); Opportunities to discuss methods with knowledgeable colleagues (92.8%); Rigorous methods encouraged by colleagues (80.4%)
  • 10. Interview Methods Semi-structured interviews building upon quantitative survey data MICYRN survey respondents invited to participate in an interview Target sample size of 12 pediatric trialists Questions to determine:  Relationships between participants’ beliefs, behaviours, and attitudes about conducting research on children and appropriate design and conduct of methodologically sound trials
  • 11. Interview Results 13 interviews conducted Necessary to augment original sample:  3 respondents from survey, 3 from MICYRN, 7 referrals 7 trained in medicine, 4 trained in both medicine and research, and 2 trained in research Subspecialties represented: anesthesiology, clinical epidemiology, critical care, emergency medicine, infectious disease, neonatology, neurology, oncology, psych ology, rheumatology
  • 12. Interview ResultsIndividual Factors - Barriers Knowledge/training regarding research methods  Lack of formal training – on the job learning  Emphasize certain aspects, overlook others
  • 13. Interview ResultsIndividual Factors – Barriers“Probably we don’t look at, we don’t know all the bias that can happen in a trial because we don’t check, we don’t believe there’s bias. We may miss some, we may forget some, and then do not report the bias because we don’t know it exists.”“Because there’s almost zero research training in the clinical curriculum for most clinicians these days. Like there’s almost nothing in the med school program, there’s almost nothing in the rehab program – there really needs to be somebody on the protocol who’s got a little bit more training.”
  • 14. Interview ResultsIndividual Factors – Facilitators Sense of ownership  Opportunity to generate enthusiasm, gain support, and educate colleagues
  • 15. Interview ResultsIndividual Factors – Facilitators“So you really have to take the time to engage people and be the one that’s proactive, engaging them. Because they’re busy, they might not even know what your study is unless you’re the change agent that really goes out there and talks to them about it and gets them motivated about why you think it’s important.”“Listen to what [your colleagues] need to execute the study so that when you develop your protocol, you’ve built that into the approach. Or, if you couldn’t, you’ve at least had that dialogue with them about how scientifically you can’t be as flexible as might be ideal… so that they at least understand the rationale.”
  • 16. Interview ResultsInstitutional Factors – Barriers Negative research culture Lack of recognition of distinction between clinical care and clinical research Variable across institutions
  • 17. Interview ResultsInstitutional Factors – Barriers“I think that other people view [research] as kind of a thorn in their side. It’s something they play along with if they have to and the division head tells them they have to.”“I think a lot of investigators really have a hard time separating what decision they would make clinically from what decision they would make as part of a trial… because the feeling is I want to be convenient to the family, and I really know this stuff because I’m an expert in this clinical area, and I don’t think they realize that there’s a pretty clear demarcation between what you do in clinical care and what you do as research.”
  • 18. Interview ResultsInstitutional Factors – Facilitators Cohesive study teams  Clinicians, methodologists, research staff  Networks within subspecialties Reliable internal review processes
  • 19. Interview ResultsInstitutional Factors – Facilitators“We have the help of the research institute and you can have a person for any kind of question or any kind of design that can help, and we have access to those kinds of resources.”“I think the fact that [research network] is there enables you to think of multi-centre RCTs, whereas if it wasn’t there, you’d kind of have to go and find things from scratch. But by existing, it brings people together with shared interests and I think that that is a huge asset when it comes to even the thought of designing a multi-centre RCT. It’s like you want to design one for [research network].”
  • 20. Interview ResultsPolicy Factors – Barriers Funding Ethics review process
  • 21. Interview ResultsPolicy Factors – Barriers“We all tend to want to make the budget as small as we can to increase our chances to actually get it funded and the reality is that some trials really require the full-time effort of somebody who’s got a lot of experience, and therefore comes with a price tag. And it can be hard to make the argument to ensure that you’ve got funding, right? So I think that’s where you start cutting other corners, and you don’t have the data quality, and at the end of the day, you maybe don’t have the rigorous, homerun kind of trial that you had envisioned.”“Most of the problem is to ask for revisions and they are not consistent one between the others. So you can have a question in one and the other one… wants a different answer.”
  • 22. Interview ResultsPolicy Factors – Facilitators None
  • 23. Discussion Internal validity is not a primary concern – overshadowed by pragmatics and generalizability  Internal validity is a prerequisite for external validity Lack of formal training and a negative research culture contribute to acceptance of sub-optimal trials Importance of mentorship and clinician-scientists
  • 24. Future Directions Objectives: To design and evaluate a tailored KT intervention to improve methodological rigor in child health trials. Researcher involvement sought throughout Potential interventions: online modules, checklists
  • 25. Future Directions Consideration of traditional KT interventions  Adaptation for changing researcher behaviour, rather than clinician behaviour  Social media component Wiki platform  Incorporates successful elements of KT interventions: interactivity, presentation of materials in multiple formats, endorsement by opinion leaders  Informed by Diffusion of Innovations, Social Influences Theory, and Theory of Planned Behaviour
  • 26. Future Directions
  • 27. Future Directions
  • 28. Future Directions
  • 29. Future Directions
  • 30. Future Directions
  • 31. Future Directions
  • 32. Future DirectionsPilot testing: Trialists, systematic reviewers, trainees Preliminary feedback: content, format, usability Participants will review wiki, take part in a short interview Revisions to wiki based on input
  • 33. Future Directions RoB wiki to be used as a model for other StaR Child Health SDGs Evaluation of effectiveness  RoB in pediatric trials  Knowledge translation intervention
  • 34. AcknowledgementsSupervisory Committee: RoB SDG Members:Lisa Hartling Jamie BrehautTerry Klassen An-Wen ChanShannon Scott Jeremy GrimshawDavid Moher Martin Meremikwu Ginny Moyer Prathap Tharyan