Recommended
More Related Content
What's hot
What's hot (20)
Similar to Systematic reviews: Why, How, So what?
Similar to Systematic reviews: Why, How, So what? (20)
More from David Johns
More from David Johns (7)
Recently uploaded
Recently uploaded (20)
Systematic reviews: Why, How, So what?
- 1. David Johns PhD, MPH, RD Specialty Registrar in Public Health (ST4) @DavidJohnsRD
- 4. “A review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyse data from the studies that are included in the review... (Cochrane Collaboration, 2014) analyse data
- 7. ▧ Population ▧ Intervention ▧ Comparator ▧ Outcome ▧ Study design
- 8. ▧ Population ▧ Intervention ▧ Comparator ▧ Outcome ▧ Study design
- 9. ▧ Population ▧ Intervention ▧ Comparator ▧ Outcome ▧ Study design
- 10. ▧ Population ▧ Intervention ▧ Comparator ▧ Outcome ▧ Study design
- 11. ▧ Population ▧ Intervention ▧ Comparator ▧ Outcome ▧ Study design
- 12. 1.Defining the review question(s) and developing criteria for including studies 2. Searching for studies. We just did this bit! Databases Search terms (MeSH) Grey literature Dates Language 3. Selecting studies & collecting data. Incl/Excl tools Get help! Min 2-3 people Record 4. Risk of bias A bit like critical appraisal See the Cochrane Risk of Bias Tool 5. Analyzing data and undertaking meta-analyses We’ll get to this in a mo… 6. Addressing reporting bias Not everything gets published 7. Presenting results Not really got a helpful nugget of info here. WAIT! Check out PRISMA guidelines. 8. Interpret results & draw conclusions Know it’s limitations
- 13. 3. The rise of editorial review
- 14. Spot the difference ‘TRADITIONAL’ REVIEWS One or more None Broad w no hypothesis No search strategy Not stated Not stated – subjective Often narrative Sometimes influenced by authors beliefs Not possible Not possible SYSTEMATIC REVIEWS Two or more Often published on PROSPERO Specific incl PICOS. Clear hypothesis Detailed & comprehensive Listed Specific inclusion/exclusion criteria Narrative, qualitative or quantitative Drawn from complete evidence base Yes as accurately documented Possible – Cochrane annually Authors: Protocol: Research Q: Search: Sources: Selection criteria: Synthesis: Conclusions: Reproducibility: Update:
- 15. 4. Meta-analysis
- 16. 0.2 0.5 1.0 2.0 5 Scale for statistic/outcome being displayed (Odds Ratio/Relative Risk/Mean difference) Line of null effect or no difference
- 17. 0.2 0.5 1.0 2.0 5 Favours treatment Favours control 95% confidence interval Point estimate. i.e. the study result Size of the box relates to the size of the study
- 18. 0.2 0.5 1.0 2.0 5 Favours treatment Favours control The important bit! Point estimate & confidence interval of combined effect In this example the combined effect is statistically significant
- 19. 0.2 0.5 1.0 2.0 5 Favours treatment Favours control The important bit! Point estimate & confidence interval of combined effect In this example the combined effect is not statistically significant
- 20. Study Treatment n/N Control n/N Risk Ratio ( 95%CI) Subtotal 1813 1814 0.78 (0.65-0.94) Author Names and Year Total Events: x (Treatment), x (Control) Heterogeneity: Chi2 = 19.21, df = 12 (p=0.08); I2 = 38% Test for overall effect Z = 3.57 (P = 0.0035) Focus on I2 for now. If its < 50% but above that and we need to think if our interventions are consistent
- 22. 5. Limitations Rubbish IN - Rubbish OUT You can’t compare the effects of apples and carburettors p.s. I’m really just talking about heterogeneity here Meta-analysis of observational study is still only association! They can be out-of-date. Make sure you look for what’s been published since! V Not to scale! There are good ones and bad ones. Quality still counts
- 23. Practice makes perfect! Well, statistically better at least!
- 24. An invisible unicorn has been grazing in my office for a month… Prove me wrong. http://uk.cochrane.org/news/invisible-unicorn-has-been-grazing-my-office- month%E2%80%A6-prove-me-wrong No evidence of effect is not evidence of no effect
- 25. Thanks! What did I forget? Please complete the evaluation and next month our FY2 doctor will lead discussion of a systematic review
- 26. Extras – Publication bias & Funnel plots Studysize Effect size Average effect size
Editor's Notes
- There are many different study designs, but a Systematic Review is unique. Basically, it’s a study of studies about an intervention. The benefit of the systematic review is that it is a one-stop shop summery of the evidence about a research question. In the Pyramid of Evidence Based Medicine, a Systematic Review of Randomized Control Trials is located at the top; because so many studies are used, it greatly reduces bias.
- Unbiased and comprehensive summary and interpretation of current evidence Increase power (bigger sample sizes), Improve precision (more confidence)
- The PICO process is a technique used in evidence based practice to frame and answer a clinical or health care related question
- The PICO process is a technique used in evidence based practice to frame and answer a clinical or health care related question
- The PICO process is a technique used in evidence based practice to frame and answer a clinical or health care related question
- The PICO process is a technique used in evidence based practice to frame and answer a clinical or health care related question Don’t try compare oranges and apples
- The PICO process is a technique used in evidence based practice to frame and answer a clinical or health care related question
- Once a well-defined research question has been established, it is important to outline where you will search for the evidence. Systematic searches should aim to search as many different sources as possible. This can be broken down into… PsychINFO – key database for mental health literature MEDLINE – large medical database EMBASE – large medical database SCOPUS – includes many scientific disciplines Cochrane Library – high-quality evidence Web of Science – includes many scientific disciplines CINAHL – includes biomedicine, healthcare, nursing and allied health articles 3. When searching online databases, the terms and their synonyms for each of the components of the PICO model must be written out, including abbreviations. It is also important to use alternate spellings and word endings. This can be done using a number of strategies within the database Medical Subject Headings (or MeSH terms) are terms predefined by the database using human indexers in concordance with thorough protocols 4. Selection bias; Performance bias; Detection bias; Attrition bias; Reporting bias; Other bias.
- My irritation/annoyance at the moment is the rise of editorials being portrayed in the media and by clinicians as new research. They are useful and provide a discussion point for the research community to debate and dissect topics BUT they are not systematic reviews; they do not even include methods. We cannot exclude the possibility of bias including confirmation bias (where you read the stuff that proves your point).
- To make a valid decision about using an intervention, ideally we should not rely on the results obtained from single studies. This is because results can vary from one study to another for various reasons, including confounding factors, and the different study samples used. By combining individual studies, and thus using more data, the precision and accuracy of the estimates in the individual studies can be improved upon. Additionally, if the individual studies were underpowered, combining them in a meta-analysis can increase the overall statistical power to detect an effect.
- The horizontal line and whether it crosses the “line of null effect” is particularly important to take note of for each study. If you remember, the incredibly basic definition of the 95% confidence interval is: “The range of values within which you can be 95% certain the true value lies.” If the horizontal line crosses the line of null effect what that is effectively saying is that the null value lies within your confidence interval and hence could be the true value. If I were breaking this down to its most simplest explanation: “any study line which crosses the line of null effect does not illustrate a statistically significant result.”
- Note the diamond – what does it represent?
- Heterogeneity: If these studies are all testing the same intervention, why don’t they get the same results? Are the differences caused by chance, or is there something else involved? If it is chance, then we have nothing to worry about. If the differences are not the result of chance, then we need to be cautious in how we interpret the results. To make it easy to assess the consistency of the papers analysed, a statistic called I2 is used (‘i-sqaured’). The rule of thumb is that you want the I2 to be less than 50%. Anything higher than that and the papers could be inconsistent due to some reason other than chance (which is bad!). For our example, thankfully, the I2 is 38%- not perfect but still within our target range. You will notice there are other statistics there like Chi2 and z. For the purposes of this tutorial, the I2 is the most useful in interpreting a forest plot.
- The average takes into account study size when combining results. Bigger studies give more precise results and thus are seen as more important when calculating the combined effect.
- Apple’s and carburetors – this is a quote from a Cochrane systematic review course I attended – it is random and extreme but that’s the point. To try rationalize it (which you probably shouldn't), you can’t look at F&V interventions and Air pollution interventions in the same meta-analysis even if they share a common outcome like number of CVD events. Be critical: Did the authors look for the right type of papers? Do you think all the important, relevant studies were included? Did they assess quality?
- See example worksheets with questions…
- There is a unicorn on this slide
- Funnel plots allow us to visually see where we might be missing studies as you would expect small studies to have large variability on both sides of the line of no effect. There are a range of statistical methods that can assess the risk of publication bias