Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Quality Standards ERS 2017


Published on

Quality Standards ERS 2017

Published in: Health & Medicine
  • Be the first to comment

  • Be the first to like this

Quality Standards ERS 2017

  1. 1. ERS 2017 Milan, 9th September 2017 Quality Standards Working Group Meeting
  2. 2. Agenda • Working group progress update o Discussion of the two main papers • Priorities for future research, actions, publications, promotion and implementation o Review potential studies – Are they still relevant, feasible, valid and a priority o Other research ideas o Setting priorities • Additional items
  3. 3. Attendees • N Roche (chair)
  4. 4. Active study update • REG_P005: o Phase I: Assess the quality of the asthma comparative effectiveness literature to identify evidence for consideration by future guideline developers o Phase II: Develop (implementable) tools to facilitate quality appraisal of published real-world evidence
  5. 5. Assess the quality of the asthma comparative effectiveness literature to identify evidence for consideration by future guideline developers Paper Focus Journal Lead author Status 1 Overview of the taskforce work Allergy Nicolas First draft due to be submitted to EAACI and REG 2 Development of the tool and link to (and differentiation from) other work in this area ERJ? Jon First draft almost complete 3 Results of literature review Clinical and experimental allergy? Nicolas On hold – determine whether required following publication of paper 1.
  6. 6. REG_P005: Paper 1 Objective Provide an overview of how this Task Force developed a quality assessment tool and applied it to some selected PICOT questions for which RCTs provide only limited evidence. Working title Quality standards in respiratory real-life effectiveness research: a Respiratory Effectiveness Group – European Academy of Allergy and Clinical Immunology Task Force report Rationale • In observational research as well as in RCTs, ensuring high-quality methodology is crucial to avoid biases that would compromise the reliability of results. • While quality standards for RCTs are well-defined and extensively disseminated, e.g., by the CONSORT statements, this is less so in the field of observational comparative effectiveness research. • Several tools assess the quality of evidence provided by observational research, they have not been specifically designed for effectiveness research evaluation
  7. 7. Methodology • Development and testing of the quality assessment tool • Systematic review of tools proposed in the literature. • Results discussed during Taskforce meetings and used to build a first version of the REG/EAACI tool. • Tested by the taskforce • Refined and formatted • Tested again on a larger basis using all papers retrieved by the literature search • Selection of topics • A first set of 21 PICOT questions was identified during a Taskforce meeting, selected based on their relevance for asthma management, limited ability of available RCTs to provide robust answers and potential of observational research to address them. • Nine were prioritised, and then ranked and feasibility discussed. • Four final questions were selected. • Assessment of the literature • Identify relevant articles corresponding to the PICOT questions • Papers considered of sufficient quality were sent to at least two reviewers • Summary tables were built, mentioning the main conclusion of the articles, the type of study, the GRADE assessment, the possible impact on clinical practice and whether similar evidence is available from RCTs. REG_P005: Paper 1
  8. 8. Results REG_P005: Paper 1 PRIMARY CRITERIA 1. Background 1.1. Clearly stated research question 2. Design 2.1 Population defined 2.2. Comparison groups defined and justified 3. Measures 3.1. (If relevant), exposure (e.g. treatment) is clearly defined 3.2. Primary outcomes defined 4. Analysis 4.1. Potential confounders are addressed 4.2. Study groups are compared at baseline 5. Results 5.1. Results are clearly presented for all primary and secondary endpoints as well as confounders 6. Discussion/Interpretation 6.1. Results consistent with known information or if not, an explanation is provided 6.2 The clinical relevance of the results is discussed 7. Conflict of interests 7.1. Potential conflicts of interest, including study funding, are stated SECONDARY CRITERIA 1. Background 1.1. The research is based on a review of the background literature (ideal standard is a systematic review) 2. Design 2.1. Evidence of a priori design, e.g. protocol registration in a dedicated website 2.2 Population justified 2.3 The datasource (or database), as described, contains adequate exposures (if relevant) and outcome variables to answer the research question 2.4 Setting justified 3. Measures 3.1 Sample size / Power pre-specified 4. Analysis NO SECONDARY CRITERIA 5. Results 5.1. Flow chart explaining all exclusions and individuals screened or selected at each stage of defining the final sample 5.2. The authors describe the statistical uncertainty of their findings (e.g. p-values, confidence intervals) 5.3. The extent of missing data is reported 6. Discussion/Interpretation 6.1. Possible biases and/or confounding factors described 7. Conflict of interests NO SECONDARY CRITERIA *For guideline development, secondary criteria only appear if all primary criteria are met
  9. 9. Results REG_P005: Paper 1 • PICOT questions dealt with: • The influence of adherence/persistence on asthma outcomes • The influence of smoking on asthma outcomes and treatment effectiveness • The impact of the device on outcomes • The relation between particle size of maintenance therapy and asthma outcomes • 46 papers were selected • The main limitation of currently available database studies was the lack of detailed clinical data regarding: • control as assessed by validated questionnaires, • severity as assessed by symptoms, and lung function, • symptoms suggestive of comorbidities such as rhinitis or gastro-oesophageal reflux disease. • As a consequence, authors used proxies. • In most retained studies, these factors were used to match and/or adjust analyses, and so the level of evidence could be raised to moderate. • The level could not be raised to high, since residual confounding by hidden factors could not be ruled out. For instance, data on behaviours are frequently lacking, while smoking history or BMI are absent from several databases.
  10. 10. Conclusions REG_P005: Paper 1 • The final tool was considered easy to use by members of the REG and EAACI networks who were involved in the testing phase (n=66) • Fair levels of agreement were obtained in the implementation phase • ≥73% for primary items and ≥69% for secondary ones. • The high number of PICOT questions (n=21) identified, outlines the evidence gaps that are still present regarding asthma treatment. • This contrasts with the high number of RCTs assessing asthma care, illustrating that classical RCTs are not sufficient to answer all questions regarding real-life effectiveness of therapeutic strategies. • The number of published articles available for each of the PICOT questions was surprisingly low, illustrates that observational effectiveness research in asthma is still infrequently performed. • 46% of retrieved papers were considered of sufficient quality to be analysed further in terms of their results and how these could inform current knowledge and recommendations.
  11. 11. REG_P005: Paper 2 Objective To outline the methodology and process that lead to the development of the Real Life Evidence AssessmeNt Tool (RELEVANT). The tool was created to assist the Task Force in discriminating flawed CER studies from those of sufficiently high quality to warrant consideration by asthma guideline bodies Working title The Real Life Evidence AssessmeNt Tool (RELEVANT): development of a novel quality assurance asset
  12. 12. Methodology • Development and testing of the quality assessment tool was across 6 key phases • Phase I: Review of the literature • Phase II: Initial tool creation • Phase III: Taskforce review and feedback • Phase IV: Pilot application of the quality assessment tool • Phase V: Extended pilot • Phase VI: Development of an online tool REG_P005: Paper 2
  13. 13. Results REG_P005: Paper 2 • Initial tool creation (Phase I-III) • A 43-item checklist was created from primary reference papers, and was reduced to 25 quality criteria by the task force members’ discussions. • Tool appraisal and item reduction (Phase IV & V) • Initial pilot: • Evaluation of inter-rater agreement of the initial tool indicated wide variation (33-100%) in concordance across individual criteria • The qualitative feedback suggested this variation was largely driven by differences in semantic interpretation of some of the criteria • This was addressed by rephrasing, removal of any redundancies and eliminating subjective words • Extended pilot: • The inter-rater agreement of the revised tool found concordance in excess of 70% for 94% of primary criteria and 93% of secondary criteria • Qualitative feedback from the pilot raters identified some remaining criteria in the tool that could be further simplified to reduce potential inter-rater variability.
  14. 14. Results REG_P005: Paper 2 • RELEVANT (Phase VI) • Guides systematic appraisal of the quality of published observational CER papers across seven domains: Background, Design, Measures, Analysis, Results, Discussion/Interpretation, Conflicts of Interest • Raters must indicate fulfillment (Yes / No / NA) of 11 quality criteria across these seven domains • RELEVANT quality” is defined as fulfillment of all 11 primary criteria. • Failure to meet any one primary criterion reflects a potential “fatal flaw” in a study’s design or, if failure reflects absence of the necessary detail, a lack of necessary transparency in reporting • If all primary criteria are fulfilled, assessment of ten additional, secondary, parameters is prompted to enable further characterization of the relative strengths and weaknesses of the paper
  15. 15. Conclusions REG_P005: Paper 2 • RELEVANT is a user-friendly quality appraisal tool comprising 21 quality criteria (11 primary; 10 secondary) across seven core quality domains. • It is the first of its kind to support quality appraisal of published research and to have been developed through iterative feedback derived from pilot implementation and inter-rater agreement evaluation.
  16. 16. Next steps • Publication of papers • The quality standards in respiratory real-life effectiveness research tool: o Finalise tool o Make it available – where? o Promote it
  17. 17. Other research ideas?
  18. 18. Setting priorities • Are these projects: o Relevant? o Feasible? o Valid? o A priority? • How do we set priorities in quality standards research? • How to we ensure these priorities are pursued? • What are the two most important projects to push forwards?
  19. 19. REG Summit 2018: “Working together to make an impact” • Who do we partner with for greatest impact • How do we increase acceptability/validity of real-life research
  20. 20. Any other business?