Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

NCCMT Spotlight Webinar: MetaQAT

161 views

Published on

Interested in a tool to appraise all types of public health evidence?

Do you appraise public health evidence? Are you interested in a single tool that can appraise many different types of evidence and study designs? Join us for a webinar to learn about the MetaQAT.

This tool was developed by Public Health Ontario.

How can the MetaQAT help you?

Many different types of evidence and study designs may be used to answer a single public health question. The MetaQAT provides a single process that can be used to appraise these different types of evidence. This supports the use of evidence to formulate recommendations and develop public health actions.

The MetaQAT consists of a four domain appraisal framework that includes relevance, reliability, validity, and applicability. A suite of design-specific companion tools are included to provide further guidance to assess validity of common designs.

This webinar includes an overview of the MetaQAT by its developer, Dr. Laura Rosella, followed by a presentation from Dr. Catherine Bornbaum, who used the MetaQAT in conducting a systematic review.

To see the summary statement of this tool developed by NCCMT, click here: http://www.nccmt.ca/resources/search/243

The National Collaborating Centre for Methods and Tools is funded by the Public Health Agency of Canada and affiliated with McMaster University. The views expressed herein do not necessarily represent the views of the Public Health Agency of Canada.

NCCMT is one of six National Collaborating Centres (NCCs) for Public Health. The Centres promote and improve the use of scientific research and other knowledge to strengthen public health practices and policies in Canada.

Published in: Healthcare
  • Be the first to comment

  • Be the first to like this

NCCMT Spotlight Webinar: MetaQAT

  1. 1. Follow us @nccmt Suivez-nous @ccnmo Funded by the Public Health Agency of Canada | Affiliated with McMaster University Production of this presentation has been made possible through a financial contribution from the Public Health Agency of Canada. The views expressed here do not necessarily reflect the views of the Public Health Agency of Canada.. MetaQAT Presenters: Laura Rosella, PhD, MHsc Catherine Bornbaum, PhD March 21, 2017 1:00 – 2:30 PM ET
  2. 2. Follow us @nccmt Suivez-nous @ccnmo 2 Housekeeping Use Chat to post comments and/or questions during the webinar • ‘Send’ questions to All (not privately to ‘Host’) Connection issues • Recommend using a wired Internet connection (vs. wireless), • WebEx 24/7 help line • 1-866-229-3239 Participant Side Panel in WebEx Chat
  3. 3. Follow us @nccmt Suivez-nous @ccnmo 3 After Today The PowerPoint presentation (in English and French) and English audio recording will be made available. These resources are available at: PowerPoint: https://www.slideshare.net/NCCMT/presentations Audio Recording: https://www.youtube.com/user/nccmt/videos
  4. 4. Follow us @nccmt Suivez-nous @ccnmo 4 How many people are watching today’s session with you? Poll Question #1 A. Just me B. 1-3 C. 4-5 D. 6-10 E. >10
  5. 5. Follow us @nccmt Suivez-nous @ccnmo Your profession? Put a √ on your answer (or RSVP via email) / Epidemiologist Management (director, supervisor, etc.) Allied health professionals (nurse, dietician, dental hygenist, etc.) Librarian Physician / Dentist Other 5
  6. 6. Follow us @nccmt Suivez-nous @ccnmo MetaQAT http://www.nccmt.ca/resources/search/243 Episode 32 6
  7. 7. NCC Infectious Diseases Winnipeg, MB NCC Methods and Tools Hamilton, ON NCC Healthy Public Policy Montreal, QC NCC Determinants of Health Antigonish, NS NCC Aboriginal Health Prince George, BC NCC Environmental Health Vancouver, BC 7
  8. 8. Registry of Methods and Tools Online Learning Opportunities WorkshopsMultimedia Public Health+ Networking and Outreach NCCMT Products and Services 8
  9. 9. Follow us @nccmt Suivez-nous @ccnmo 9 Poll Question #2 How familiar are you with the method or tool we are discussing today? A. I am not familiar with the method or tool B. I have heard of the method or tool C. I have used the method or tool
  10. 10. Follow us @nccmt Suivez-nous @ccnmo 10 Presenter Laura Rosella, PhD, MHsc Canada Research Chair in Population Health Analytics Assistant Professor, Dalla Lana School of Public Health, University of Toronto Scientist, Public Health Ontario Adjunct Scientist, Institute for Clinical Evaluative Sciences
  11. 11. Follow us @nccmt Suivez-nous @ccnmo 11 Presenter Catherine Bornbaum, PhD Research Associate, Dalla Lana School of Public Health, University of Toronto Adjunct Research Professor, Western University
  12. 12. A public health approach to appraising evidence: Meta-tool for Quality Appraisal (MetaQAT) Laura Rosella, PhD Dalla Lana School of Public Health Public Health Ontario March 21, 2017
  13. 13. Project team • Laura Rosella, Scientist • Beata Pach, Manager, Library Services • Carolyn Bowman, Senior Research Coordinator • Sarah Morgan, Library Technician • Tiffany Fitzpatrick, Epidemiologist • Vivek Goel, Scientist 13
  14. 14. Objectives 1. Rationale and development of the PHO MetaQAT tool 2. Orientation of the tool and how to use it 14
  15. 15. 15 Idea/request for review topic Formulation of specific review question Literature search Application of inclusion & exclusion criteria Quality appraisal of included studies Data extraction and analysis Synthesis or conclusions Will start directly here for review of one study Some iteration may be required Can be done at the same time i.e. when documenting appraisal you can also add desired data extraction fields
  16. 16. “Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context.” • Judiciously assessing quality of evidence is challenging • Requires expertise (preferably methods and content) • If we are attempting to inform an evidence base that will inform public health decisions – it needs to be a judicious and transparent process • Transparency is critical – not only will it enable us to better support our recommendations but also challenge our own assumptions to ensure the science is as strong as possible • The “why” is often as important as the outcome Quality (critical) appraisal Source: Burls A. What is critical appraisal? 2nd ed. [Internet]. Newmarket, UK: Hayward Group; 2009 [cited 2015 Mar 2]. Available from http://www.bandolier.org.uk/painres/download/whatis/What_is_critical_appraisal.pdf 16
  17. 17. Appraisal should be about understanding • Both strengths and weaknesses • We need to make decisions, important to use evidence to fullest extent possible • Understand how evidence can be used appropriately 17
  18. 18. Subjective is not a bad word • Judgments are made based on appraisal principles • Using the same principles, similar issues will be identified • No one right answer, that doesn’t mean anything goes! • Transparency is key: another person should be able to understand why you made your assessment 18
  19. 19. Transparency • Importance of documentation – if others know what you appraised and why, they can ‘appraise your appraisal’ • Transparency is key: another person should be able to understand why you made your assessment • Think of it as a way that one can audit your analysis of the evidence – akin to auditing code following a statistical analysis 19
  20. 20. Internal Validity • Elwood: measure of how easily a difference in an outcome between two groups can be attributed to the effects of an exposure or intervention • L. Green: “the essence of rigor” External Validity • The process of generalizing the findings of the study to the population from which the sample was drawn (or even beyond that population to a more universal statement) • Without internal validity a study cannot have external validity • L. Green: “ the essence of relevance” 20 Internal and external validity
  21. 21. Internal validity can be influenced by... • Study design • Measurement properties of the variables • Study recruitment/response rates • Selection pressures • Sampling strategy • Setting • Investigators/funding • Among others factors… • Preferred term: RISK OF BIAS (RoB) 21
  22. 22. 22
  23. 23. 23 www.riskofbias.info
  24. 24. 24
  25. 25. Risk of Bias says nothing about the intended use of the evidence • Application of the evidence is important in public health • Context matters – where the evidence was generated, where it is going to be applied • Importance of different factors when appraising evidence should be guided by the intended application 25
  26. 26. Contemporary view of “evidence-based” • Evidence-informed • The idea that evidence (in the scientific sense) informs practice but that other factors are important • Weight of evidence versus strength of evidence • EBM traditionally focuses on strength of evidence (i.e. RCT) and discounts indirect evidence from other sources • Larry Green: “ If We Want More Evidence Based Practice, We Need More Practice Based Evidence” 26
  27. 27. Evidence Informed Public Health (EIPH) • “public health endeavor in which there is an informed, explicit, and judicious use of evidence that has been derived from any of a variety of science and social science research and evaluation methods” • Acknowledges the many factors, beyond simply the evidence, influence decision-making • “EIPH is a complex, multi-disciplinary process that occurs within dynamic and ever-changing communities and encompasses different sectors of society” http://www.nccmt.ca/uploads/media/media/0001/01/b331668f85bc6357f262944f0aca38c14c89c5a4.pdf 27
  28. 28. Beware the evidence pyramid
  29. 29. A Hierarchy of Strength of Evidence for Treatment/ Intervention Decisions (Quantitative Research) 29 Relative Strength of Evidence (with 1 being the strongest) Type of Evidence 1 • systematic review of randomized controlled trials 2 • systematic review of observational studies addressing population health important outcomes 3 • single randomized trial 4 • single observational study addressing population important outcomes 5 • physiologic and epidemiological study (e.g., study of infection prevalence, prevalence of cardiovascular risk factors) http://www.nccmt.ca/uploads/media/media/0001/01/b331668f85bc6357f262944f0aca38c14c89c5a4.pdf
  30. 30. What’s better? • A large and rigorous prospective cohort study or a small poorly conducted randomized control trial? • A systematic review of dozens of well-conducted case control studies or one large randomized control trial? • Large and rigorous randomized controlled trial done in Japan or a time-series analysis in Calgary? (assuming you live in Toronto) 30 Critical appraisal and the evidence hierarchy
  31. 31. What’s better? • A large and rigorous prospective cohort study or a small poorly conducted randomized control trial? • A systematic review of dozens of well-conducted case control studies or one large randomized control trial? • Large and rigorous randomized controlled trial done in Japan or a time-series analysis in Calgary? (assuming you live in Toronto) • A well conducted study that’s ‘lower’ on the evidence hierarchy may actually be better...every study needs to be interpreted and assessed in its own right 31 Critical appraisal and the evidence hierarchy
  32. 32. 32 Points to remember: 1. Overall quality appraisal ≠ internal validity only 2. Risk of bias assessment speaks mainly to internal validity 3. The study’s quality is only one consideration in the weight of evidence
  33. 33. Skills and knowledge required for quality appraisal • Knowledge of topic area • Knowledge and experience with the specific methods used • Knowledge of intended application • Experience with appraisal • Excellent documentation skills - record why decisions were made • Note: These traits can be covered by a team 33
  34. 34. TOOLS, TOOLS, EVERYWHERE 34
  35. 35. A systematic review of 121 published critical appraisal tools • 87% of critical appraisal tools were specific to a research design with most tools having been developed for experimental studies • 49% of the critical appraisal tools summarized the quality appraisal into a numeric summary score • Few critical appraisal tools had documented evidence of validity of their items, or reliability of use • “We found no gold standard critical appraisal tool for any type of study design” 35 Source: Katrak P, Bialocerkowski AE, Massy-Westropp N, Kumar S, Grimmer KA. A systematic review of the content of critical appraisal tools. BMC Med Res Methodol. 2004 [cited 2014 Aug 7];4:22. Available from: http://www.biomedcentral.com/content/pdf/1471-2288-4-22.pdf
  36. 36. Suggested benefits of appraisal tools frameworks • Checklists that ensure that we don’t overlook important considerations (i.e. Aide-mémoire) • CAN limit what we do look for depending what’s in the tool • Standardize the approaching to reviewing the evidence • This can be easily challenged because many of the judgements are subjective – so I'd rather know why you judged something a particular way versus what your judgement was 36
  37. 37. A comparison of tools Voss and Rehfuess QUALITY APPRAISAL IN SYSTEMATIC REVIEWS OF PUBLIC HEALTH INTERVENTIONS: AN EMPIRICAL STUDY ON THE IMPACT OF CHOICE OF TOOL ON META-ANALYSIS Recommendations: 1) Testing of a broader set of QATs on a more up-to-date systematic review of a public health intervention covering a wide range of epidemiological study designs 2) Research into the development of a reliable QAT with broad applicability across study designs. 37 Source: Voss PH, Rehfuess EA. Quality appraisal in systematic reviews of public health interventions: an empirical study on the impact of choice of tool on meta-analysis. J Epidemiol Community Health. 2013;67(1):98-104.
  38. 38. Features of a Gold Standard QAT Voss and Rehfuess 1) Instead a set of companion QATs comprising common components and study design-specific components could initially be developed for standard designs and then expanded to address more complex designs. 2) An overall summary score may lead to information loss and oversimplification. Instead, the authors propose a careful quantification of domains (rather than individual questions) in a weighted checklist. 3) Should address both internal and external validity with a clear distinction between these. 4) Much more effort should be invested into the development and testing of appropriate and specific questions and answer categories; a manual can help with their correct interpretation. 5) Subject-specific adaptation of selected questions makes quality appraisal more precise and reliable than a one size fits all QAT. 38 Source: Voss PH, Rehfuess EA. Quality appraisal in systematic reviews of public health interventions: an empirical study on the impact of choice of tool on meta-analysis. J Epidemiol Community Health. 2013;67(1):98-104.
  39. 39. METAQAT PROCESS 39
  40. 40. MetaQAT project stages 40 Stage 1: Gather information Stage 2: Put information together Stage 3a: Draft the meta-tool Stage 3b: Refine draft meta-tool Stage 4: Pilot testing Stage 5: Validation
  41. 41. Stage 1: Gather information • Searched for existing tools • Databases • Grey literature • Guidance and background documents on critical appraisal in public health (NICE, ECDC, NCCMT, Cochrane) • Websites • Universities with public health programs • NCCMT repository of critical appraisal tools • Consult with experts • Consult with PHO staff 41
  42. 42. Summary on tools to guide appraisal • Many tools exist (most not validated) • Most from evidence-based medicine perspective • Internal validity (or risk of bias) focus • Many use numeric scales or scores • Not valid, not recommended by Cochrane Collaboration • Many tools appropriate for specific contexts (topic, project, study design) • Something more flexible was required 42
  43. 43. What kind of tool? • One appropriate for our context • Various types of reviews • Rapid reviews, evidence briefs, systematic reviews, etc. • Various types of content • Chronic disease, infectious disease, environmental health, health promotion, injury prevention, etc. • Strong focus on the applicability of the evidence • Not just the quality • Not reinventing the wheel • Build on existing content 43
  44. 44. What tool for PHO? • Many tools appropriate for specific contexts (topic, project, study design) • Resources available to find tools • NCCMT tool repository • University SA library • PHO needs: • Compatible with all of the types of evidence we use • Compatible with all of the types of projects we do • Consistent with current best practice • No numeric scores; acceptable to field • Promotes a transparent process • Outcome of stage 1: not a critical appraisal tool (too limited) 44
  45. 45. Stage 2: Putting the information together • Group tools by study design • Compare the questions between tools • Similarities across tools within each design • Compare these similarities across study designs • Core process is similar across tools • Coverage of process matches OPHLA guide • Generic, applies to all evidence types • Generic framework emerges • All questions can be grouped into four categories • Still need study specific detail for high level of rigour • Not a critical appraisal tool, a quality assessment process 45
  46. 46. Stage 3a: Develop draft meta-tool • Develop framework and select companion tools • Companion tools – pragmatic approach • Widespread use in public health literature • Preference to tools with documented validation • Compromise – reporting tools widely used • Ease of use • Revise based on scientific advice 46
  47. 47. Stage 3b: Refine draft meta-tool • Feedback sessions • Staff selected for feedback sessions • Sample article assessed, interpretations of questions • Revisions made for clarity, reorganization • Revised version presented to third group • Revisions made • Tool approved for pilot testing and further development and evaluation 47
  48. 48. Stage 4: Pilot testing • Introduced to teams for trial use in specific projects • Variety of topics and types of projects • Training methods developed • Introduction to tool as project reached appraisal stage • Evaluation • Fit of tool for projects and staff, requirements for full implementation 48
  49. 49. Selected pilot testing outcomes INTEGRATION OF DIVERSE GROUPS OF EVIDENCE • Applied to projects with a heterogeneous mix of evidence • The flexibility of MetaQAT allowed for the integrated assessment of evidence across methods and designs • Facilitated screening decisions • Broadened scope of appraisal promoted appropriate consideration of context versus internal validity alone • Maintains consistent process of appraisal across a group of evidence that includes grey literature and surveillance literature 49
  50. 50. Stage 5: Validation • Two groups of appraisers • One group used MetaQAT, other group appraised strengths and weaknesses of several studies • Written appraisals analysed • Content analysis • Both groups identified similar issues • MetaQAT group identified public health relevance 50
  51. 51. Meta-tool • The “meta” concept is not a single tool/framework (traditional sense), it is a process guide which orients the user to the appropriate use of several appraisal tools, and places them within a larger framework to guide their use • A tool of tools: Appraisal framework that incorporates existing tools for design-specific risk of bias assessment • Documentation – records the appraisal process for transparency 51
  52. 52. MetaQAT structure 52 Relevancy Companion tools AMSTAR CASP TREND CONSORT Mixed Methods AGREE II Navigation Guide PHO Guide for Grey Literature Reliability Validity Applicability Appraisal Framework Rosella, L. Bowman, C., Pach, B., Morgan, S., Fitzpatrick, T., Goel, V. The development and validation of a meta-tool for quality appraisal of public health evidence: Meta Quality Appraisal Tool (MetaQAT). Public Health. 2016;136:57-65.
  53. 53. MetaQAT structure Domain Idea Relevancy • Determine if the item being appraised is sufficiently related to your topic or research question. • In a structured systematic review, relevancy can also be considered in the inclusion and exclusion criteria. Reliability • Evaluate the completeness of reporting; complete reporting is necessary in order to conduct a thorough examination of quality. • Lack of detail regarding the conduct of the study or report may be indicative of lower quality information. Validity • The likelihood and magnitude of error or bias in a study. • “Risk of Bias” or “Internal Validity” Applicability • Consider how the evidence might be applied to public health practice. • Also can consider more broadly what can be learned from the evidence and how you can apply that knowledge to public health decision making. 53 Rosella, L. Bowman, C., Pach, B., Morgan, S., Fitzpatrick, T., Goel, V. The development and validation of a meta-tool for quality appraisal of public health evidence: Meta Quality Appraisal Tool (MetaQAT). Public Health. The form uses the term “study” to refer to the work being appraised and is meant to include all study types, including research published as grey literature, as syntheses, and any other type of evidence being considered.
  54. 54. MetaQAT appraisal form 54 • Domain • Main question • Prompting questions • Guide thinking about the main question • Stimulate thought, not limit • Written answer • Optional check boxes
  55. 55. MetaQAT appraisal form • Written answer format • Written summary of key points • Tick box options available BUT • Written summary is the answer to the question (many formats available) 55
  56. 56. MetaQAT appraisal form • Branch point in process – to design-specific companion tools IF DESIRED • In validity domain, question b) Is the research methodology free from bias? • Design-specific detail via companion tools IF DESIRED 56 • Can also create context-specific prompts
  57. 57. Summarizing the appraisal of an item • Textual summary of key strengths and weaknesses by domain and question • This information will provide a good understanding of the quality of the item • E.g. which is more helpful? 8/10 or “the study group was small but characteristics were very similar to our local population” • We are working on guidance to help with summary/synthesis language • “The use of scales for assessing quality or risk of bias is explicitly discouraged in Cochrane reviews” 57 Source: Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions. Version 5.1.0 [Internet]. London: Cochrane Collaboration; 2011 [cited 2015 Feb 10]. Section 8.3.3, Quality scales and Cochrane reviews. Available from: http://handbook.cochrane.org
  58. 58. MetaQAT user guide • The MetaQAT document includes relevant background information and user guidance – please read! (url to follow) • For more details: Rosella L, Bowman C, Pach B, Morgan S, Fitzpatrick T, Goel V. The development and validation of a meta-tool for quality appraisal of public health evidence: Meta Quality Appraisal Tool (MetaQAT). Public Health. 2016;136:57-65. Available from: http://www.publichealthjrnl.com/article/S0033-3506(15)00437- 0/abstract 58
  59. 59. Hard work is hard • The process of systematically appraising evidence and documenting this process is difficult and can be time consuming but these challenges may exist regardless of the tool/framework 59
  60. 60. 60 https://www.publichealthontario.ca/en/ServicesAndTools/Pages/Critical-Appraisal-Tool.aspx Screen shots to follow
  61. 61. 61
  62. 62. 62
  63. 63. 63
  64. 64. References • Rosella L, Bowman C, Pach B, Morgan S, Fitzpatrick T, Goel V. The development and validation of a meta-tool for quality appraisal of public health evidence: Meta Quality Appraisal Tool (MetaQAT). Public Health. 2016;136:57-65. Available from: http://www.publichealthjrnl.com/article/S0033-3506(15)00437-0/abstract • Voss PH, Rehfuess EA. Quality appraisal in systematic reviews of public health interventions: an empirical study on the impact of choice of tool on meta-analysis. J Epidemiol Community Health. 2013;67:98-104. • Heller RF, Verma A, Gemmell I, Harrison R, Hart J, Edwards R. Critical appraisal for public health: A new checklist. Public Health. 2008;122:92-98. • Jüni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999;282(11)1054-1060. • Katrak P, Bialocerkowski AE, Nassy-Westropp N, Kumar VSS, Grimmer KA. A systematic review of the content of critical appraisal tools. BMC Med Res Methodol. 2004 [cited 2013 Dec 16];4(22): . Available from: http://www.biomedcentral.com/1471-2288/4/22 64
  65. 65. Information management • Excel version of form is available as template 65
  66. 66. 66
  67. 67. 67
  68. 68. 68
  69. 69. 69 NCCMT Spotlight on Methods and Tools: Using the MetaQAT for Quality Appraisal in a Systematic Review Catherine Bornbaum, PhD Managing Director, Population Health Analytics Research Program Research Associate, Dalla Lana School of Public Health, University of Toronto Adjunct Research Professor, Health & Rehabilitation Sciences, Western University
  70. 70. Background • Failure to optimize the use of research evidence may result in reduced quality of care1, inefficient use of resources2,3, and poorer health outcomes for individuals and communities4 • Given the challenges with effectively translating knowledge between researchers, practitioners, and decision-makers5, knowledge translation experts advocated for an intermediary person to mitigate challenges in knowledge sharing6,7 • Knowledge brokers work collaboratively with key stakeholders to facilitate the transfer and exchange of evidence and context • Key roles: • Knowledge manager, linkage agent, capacity builder 70
  71. 71. To understand how knowledge brokers operate in health-related practice, and assess if use of a knowledge broker could facilitate effective knowledge translation initiatives. Objectives: 1. To improve conceptualization of the knowledge brokering role in health-related settings; and • Test theory of knowledge brokering in practice (confirmatory thematic analysis8) 2. To determine whether knowledge brokers contribute to effective knowledge translation in health related settings • Explore outcome data for evidence of changes in knowledge, skill, policy and practice9 • Critically appraise evidence using the Meta Quality Appraisal Tool (MetaQAT) 71 Goal
  72. 72. Inclusion and Exclusion Criteria • Inclusion Criteria • Reports on an actual application of knowledge brokering in practice • Health-related setting (e.g., public health, clinical setting, rehabilitation) • Document available in English • Exclusion Criteria • Reports on the theoretical assumptions about the practice of knowledge brokering only • Study protocols 72
  73. 73. Method Search strategy Data Sources • Databases • MEDLINE; EMBASE; PsychINFO; CINAHL; SCOPUS; SocINDEX; Health Business Elite • Grey literature • Knowledge translation networks: Canadian Foundation for Healthcare Improvement, National Collaborating Centre for Methods and Tools • Health-focused organizations: CIHR, Canadian Public Health Information, Health Evidence, Ontario Public Health Unit, World Health Organization • Hand searching • E.g., Implementation Science, BMC Health Service Research, etc. • Reference lists 73
  74. 74. Method 74 IdentificationScreeningEligibilityIncluded Records identified through databases (n = 7,022) Records identified through other sources (n = 340) Records identified through databases (n = 502) Records identified through other sources (n = 71) January 2014 November 2014 Records after duplicates removed (n = 6,363) Records screened (n = 227) Full-text articles assessed for eligibility (n = 46) Full text articles excluded (n = 181) Records excluded (n = 6,136) Articles included in synthesis (n = 26) Unique projects included in synthesis (n = 20) Articles included in synthesis (n = 3) Unique projects included in synthesis (n = 3) Total articles (n = 29) Total unique projects (n = 23) Records after duplicates removed (n = 573) Records screened (n = 13) Full-text articles assessed for eligibility (n = 12) Records excluded (n = 560) Full text articles excluded (n = 1)
  75. 75. MetaQAT USER INSIGHTS • Adapted MetaQAT forms to study context prior to conducting critical appraisal • Modified “hint questions” where appropriate • Adapted for Excel Method Critical appraisal • Why we chose the MetaQAT to conduct critical appraisal • Permitted analysis of diverse research designs (e.g., qualitative, quantitative, grey literature) and health-related settings • Four step critical appraisal framework to assess: relevancy, reliability, validity and applicability • Research design-specific companion tools for quantitative, qualitative and mixed method research design appraisal 75
  76. 76. Conducting Critical Appraisal Using an Excel-based template to organize findings 76 Additional companion tools required? Assessments of Relevancy, Validity, Reliability and Applicability
  77. 77. Conducting Critical Appraisal Completing companion tools 77 Where applicable, complete the companion tool before completing the MetaQAT
  78. 78. 78 Presenting MetaQAT Results PowerPoint format Project Relevance Reliability Validity Applicability A16,17 Relevant setting and population Insufficient description of population, intervention, outcomes Potential bias in design, sampling, outcomes Insufficient description of context/outcomes B18 Relevant setting and population Insufficient description of population and intervention Potential bias in sampling, outcomes (unsystematic reflection) Difficult to generalize C19 Relevant setting and population No description of analytic methods Potential bias in sampling, outcomes (no KB perspective) Insufficient description of context/outcomes D20-22 Relevant setting and population No significant concerns No significant concerns No significant concerns E23 Relevant setting and population Insufficient description of population, intervention, outcomes Potential bias in design, sampling, data sources Insufficient data to support applicability F24,25 Relevant setting and population Insufficient description of population and intervention Potential bias in design, data sources, exclusion criteria Insufficient description of KB role/outcomes G26-28 Relevant setting and population No significant concerns Chance cannot be ruled out (no CIs presented) No significant concerns
  79. 79. 79 Presenting MetaQAT Results Manuscript Format
  80. 80. Interested in our findings? 80 Review available from: https://implementationscience.biomedcentral.com/articles/10.1186/s13012-015-0351-9
  81. 81. • When conducting a meta-synthesis, quality appraisal tool needs to be broad enough to account for various design-specific concerns • Although designed for use in public health contexts, MetaQAT is applicable to diverse health-related contexts • E.g., clinical, rehabilitation, health policy settings • The “Applicability” appraisal was highly useful for assessing utility of knowledge brokering strategies – a key component of our review • Overall, MetaQAT was the only tool capable of meeting our broad needs relative to methodological and contextual factors 81 Lessons Learned
  82. 82. Public Health Ontario Library Services: https://www.publichealthontario.ca/en/ServicesAndTools/Pages /Critical-Appraisal-Tool.aspx 82 Ready to give it a try?
  83. 83. Acknowledgements • We wish to acknowledge the Canadian Institutes of Health Research for their support of this research program (KAL-129895) • We wish to thank Public Health Ontario Research Librarians, Allison McArthur and Domna Kapetanos for their assistance with the design and conduct of the search strategy 83
  84. 84. Thank you! Want to connect? Feel free to reach out: Catherine Bornbaum, PhD catherine.bornbaum@utoronto.ca @CathBornbaum 84
  85. 85. References 1. Jernberg T, Johanson P, Held C, Svennblad B, Lindbäck J, Wallentin L et al.: Association between adoption of evidence-based treatment and survival for patients with ST-elevation myocardial infarction. Journal of the American Medical Association 2011, 305: 1677-1684. 2. David D, Davis M, Jadad A, Perrier L, Rath D, Ryan D et al. (Eds):The case for knowledge translation: Shortening the journey from evidence to effect. In British Medical Journal 2003, 327: 33-35. 3. Madon T, Hofman K, Kupfer L, Glass R: Public health: Implementation science. Science 2007, 318: 1728-1729. 4. Chalmers I (Eds):If evidence-informed policy works in practice, does it matter if it doesn't work in theory? In Evidence & Policy: A journal of Research, Debate and Practice 2005, 1: 227-242. 5. Graham I, Logan J, Harrison M, Straus S, Tetroe J, Caswell W et al. (Eds):Lost in knowledge translation: Time for a map? In Journal of Continuing Education in the Health Professions 2006, 26: 13-24. 6. Dobbins M, DeCorby K, Twiddy T (Eds):A knowledge transfer strategy for public health decision makers. In Worldviews on Evidence-Based Nursing 2004, 1: 120-128. 7. Ward V, House A, Hamer S: Knowledge brokering: The missing link in the evidence to action chain? Evidence & Policy 2009, 5: 267-279. 8. Popay J, Roberts H, Sowden A, Petticrew M, Arai L, Rodgers M, et al.: Guidance on the conduct of narrative synthesis in systematic reviews: A product from the ESRC methods programme. 2006. 9. Kujbida G, Stratton J. Effective knowledge translation tactics for increasing the use of health status and surveillance data. 2014. Retrieved from: www.peelregion.ca/health/library/pdf/effective-kt.pdf 85
  86. 86. Follow us @nccmt Suivez-nous @ccnmo 86 Your Comments/Questions • Use Chat to post comments and/or questions • ‘Send’ questions to All (not privately to ‘Host’) Chat Participant Side Panel in WebEx
  87. 87. Follow us @nccmt Suivez-nous @ccnmo 87 Poll Question #3 Could this method or tool be useful in practice? A. Very useful B. Somewhat useful C. Not at all useful D. Don’t know
  88. 88. Follow us @nccmt Suivez-nous @ccnmo 88 Your Feedback is Important Please take a few minutes to share your thoughts on today’s webinar. Your comments and suggestions help to improve the resources we offer and plan future webinars. The short survey is available at: https://nccmt.co1.qualtrics.com/SE/?SID=SV_23n CMSgc5sz9QJD
  89. 89. Follow us @nccmt Suivez-nous @ccnmo 89 Poll Question #4 What are your next steps? (Check all that apply) A. Access the method/tool referenced in the presentation B. Read the NCCMT summary about the method/tool described today C. Consider using the method/tool in practice D. Tell a colleague about the method/tool
  90. 90. Follow us @nccmt Suivez-nous @ccnmo 90 Join us for our next webinar Spotlight on Methods and Tools: Knowledge Translation and Implementation Video Series Date: Wednesday April 5, 2017 Time: 1:00 – 2:30pm EST Interested in a tool to support planning program development? Do you plan program implementation? Are you interested in an overview of the key concepts regarding implementation of evidence-based practices? Join us for a webinar to learn about the Knowledge Translation and Implementation Video Series. Register at: https://health-evidence.webex.com/health- evidence/onstage/g.php?MTID=e4a0468604363b8e349ce95 b2277da0f6
  91. 91. Follow us @nccmt Suivez-nous @ccnmo Funded by the Public Health Agency of Canada | Affiliated with McMaster University Production of this presentation has been made possible through a financial contribution from the Public Health Agency of Canada. The views expressed here do not necessarily reflect the views of the Public Health Agency of Canada.. For more information about the National Collaborating Centre for Methods and Tools: NCCMT website www.nccmt.ca Contact: nccmt@mcmaster.ca

×