Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

REG-EAACI Taskforce Report

627 views

Published on

An overview of the work and initial results of the REG-EAACI Taskforce assessing the quality of literature in the field of real-world respiratory medicine.

Published in: Health & Medicine
  • Hello! Who wants to chat with me? Nu photos with me here http://bit.ly/helenswee
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

REG-EAACI Taskforce Report

  1. 1. REG-EAACI TASKFORCE REPORT REG SUMMIT 2016, LYON, FRANCE, 16 APRIL SESSION: ‘INFLUENCING’ Guideline Development: the REG/EAACI Taskforce Reports TIME: 11.45AM-12.45PM Presenters: Nicolas Roche (Hôpital de l’Hôtel Dieu, Paris) & Jon Campbell (Skaggs School of Pharmacy, Denver, Colorado) On behalf of: Nikos Papadopoulos, Leif Bjermer, Guy Brusselle, Alison Chisholm, Jerry Krishnan, Zoe Mitchel, David Price, Mike Thomas, Eric van Ganse, Maarten van den Berge helped by Sarah Acaster and Katy Gallop —
  2. 2. Taskforce Members Leads: Nicolas Roche & Jon Campbell • David Price • Mike Thomas • Eric van Ganse • George Christoff • Guy Brusselle • Jennifer Quint • Jerry Krishnan • Leif Bjermer • Nikos Papadopoulos • Maarten Van Den Berge
  3. 3. Background • RCTs are not sufficient to provide holistic evidence • Real-life studies are subject to many sources of biases
  4. 4. Where do observational studies fit in? • (Almost) Everybody agrees on: o Pitfalls of RCTs o Need for real-life data • Guideline developers are often reluctant to include real-life data • Quality issues • Need to help readers • Quality assessment o Tools required o Remaining difficulties in quality assessment (need to help reviewers) o Need to improve reporting
  5. 5. SETTING AND DEFINING STANDARDS CREATING A LEVEL PLAYING FIELD TO HELP RAISE (AND GUIDE ASSESSMENT OF) THE QUALITY OF REAL-LIFE RESEARCH ATS 2013 —
  6. 6. Goal • Create a level playing field and solid foundations for future research that will: o Standardize the field o Enable benchmarking o Assist in assessing the quality of real-life data (including their potential value to clinical practice guidelines)
  7. 7. Conceptual framework of therapeutic research, linking the various types of studies based on ecology of care and population characteristics. Typical positions of the most common study designs have been positioned but can be moved in any direction depending on their specificities Roche Net al. Lancet Respir Med. 2013;1:e29-30. Framework for integrating evidence
  8. 8. GRADE classifications: observational studies vs RCTs Source of evidence Initial quality rating Factors decreasing quality Factors increasing quality Final rating RCTs High Risk of bias Inconsistency Indirectness Imprecision Publication bias Large effect Dose- response Influential residual confounders High Moderate Low Very Low Observational studies Low Guyatt et al. PATS 2012 (2007 ATS/ERS Workshop) + -
  9. 9. Stakeholders • Researchers • Database designers / promotors • Guidelines developers • Policy makers • Reviewers (journals, projects) • Readers • …
  10. 10. Aims & Objectives • Conduct a systematic critical review of the real-life asthma literature published between 2004/13 o Restricted to comparative effectiveness research • Describe the quality of currently available real-life research in asthma • Highlight studies worthy of possible integration into asthma-related guidelines and policy decisions • Recommend quality targets for the future o And topics to address in future observational CER studies
  11. 11. Strategy • Agree on / build a quality assessment tool • Define a search strategy, perform a review of the literature • Apply the quality assessment tool on retrieved articles • Synthesise, discuss and conclude on quality, and potential influence of current comparative effectiveness literature on future guideline and/or need for additional studies
  12. 12. Taskforce Timelines 2014-2015 • Task 1: literature search to identify asthma real-life research articles (chair: N Roche) • Task 2: construction of a dedicated quality assessment tool (chair : J Campbell) 2015-2016 • Task 3: quality assessment of identified asthma real-life research articles • Task 4: report • Task 5: disseminate
  13. 13. 2014-2015 Task 1 (literature search): • Formal targeted literature search • Poll among Taskforce members and REG members • Limit retrieved papers to top priority (4) PICOT questions Task 2 (quality assessment tool): • Combination of available tools • Discussion and final version of the quality assessment tool
  14. 14. 2015-2016 Task 3 (quality assessment): • Quality assessment of articles identified from the literature search • Conducted by members of the Taskforce and REG-EAACI network reviewers • Results of quality assessment used to determine which papers could be used to complement results of RCTs and inform guidelines. Task 4 (dissemination): • Results to be presented at EAACI meeting • Publication (submission ~Q3 2016; journals: Allergy and CTA): o A review article (detailed results of quality assessment) o A position paper (information of guidelines)
  15. 15. QUALITY ASSESSMENT TOOL DEVELOPMENT JON CAMPBELL: Skaggs School of Pharmacy, Denver, Colorado —
  16. 16. Development Phases • Phase I: literature review • Phase II: Initial tool creation • Phase III: Taskforce review and pilot • Phase IV: Larger Pilot & tool finalization/minor modifications • Phase V: tool finalization for use – development of an online tool
  17. 17. Phase I: Literature Review Many Study Assessment Tools Exist • STROBE Statement: Checklist of items that should be included in reports of observational studies • Quality standards for real-world research: List of quality criteria for observational database comparative studies (Roche et al) • Report of the ISPOR Task Force on retrospective databases: A checklist for retrospective database studies • GRACE Checklist: Quality of observational cohort studies for decision making support • ENCePP Checklist: checklist for study protocols for pharmacoepidemiology • Standards in the conduct of registry studies for patient-centered outcomes research (PCORI): review of existing guidelines and literature to develop methodological standards
  18. 18. Phase I: Literature Review • Purpose of most existing tools: o Primarily to provide standardization of best practice and reporting of observational / comparative effectiveness research studies. • Purpose of REG-EAACI Taskforce tool: o Decision aid to assess whether or not a study provides evidence that could inform future guidelines (yes or no). If yes, the tool’s criteria can aid in describing any particular strengths or limitations of the evidence.
  19. 19. Phase I: Literature Review • A synthesis of existing tools was visually presented in tabular form. o Overlap was assessed across tools • Taskforce decided to focus on merging two existing tools o Roche and colleagues o ISPOR task force – ISPOR task force tool included many of the existing tools into its development
  20. 20. Quality criteria for observational database comparative studies Roche et al Arch Conference Ann Am Thorac Soc Feb 2014
  21. 21. ISPOR Task Force. Berger et al Value in Health 2014. 4 relevance & 28 credibility (yes/no) questions (weaknesses and fatal flaws identified)
  22. 22. Phase 1: Roche (REG) and Berger (ISPOR) • Roche has 24 relevance and credibility questions • ISPOR starts with 4 relevance questions and then moves to 28 credibility questions o Much overlap between two on domains and items o Both are yes/no items; ISPOR has a “can’t answer” option (NA, not reported, not enough info, not enough training to answer). o ISPOR has the concept of fatal flaws and weaknesses
  23. 23. Phase II: Initial Tool Creation • Synthesis of pre-existing quality recommendations to develop a first draft Taskforce Quality Assessment Tool • Recognition that asthma specificity is not necessary and provides greater tool utility
  24. 24. TABLE Checklist combining ISPOR/Roche et al. assessments Red= Roche (Emphasizes derived from Roche et al.) Green= ISPOR (Emphasizes derived from ISPOR) Assessment using Roche/ISPOR Yes or No Yes = 1 point No = 0 Background/ Relevance 4 Items 1. Clear underlying hypotheses and specific research questions Yes/No 3. Relevant population Yes/No 4. Relevant interventions and outcomes are included Yes/No 5. Applicable context (setting/practice pattern) Yes/No Maximum Raw Score = 4 pts Adjusted Score = 4/4 (100%) #Yes’s/#Items Design 8 Items 1. Observational comparative effectiveness database study with a priori hypotheses and goals? Yes/No 2. (Independent steering committee involved in) a priori definition of study methodology? Yes/No 3. Evidence of a priori protocol, review of analyses, statistical analysis plan, and interpretation of results? Yes/No 4. Comparison groups concurrent or justified? Yes/No 5. Was a study design used to minimize or account for confounding? Yes/No 6. Comparison groups selected to be sufficiently similar to each other (e.g. either by restriction or recruitment based on the same indications for treatment? Yes/No 7. Sources criteria and methods for selecting participants appropriate to address study questions/hypotheses? Yes/No 8. Registration in a public repository with commitment to publish results Yes/No Maximum Raw Score = 8 pts Adjusted Score = 8/8 (100%) #Yes’s/#Items Data/Database 3 Items 1. High quality databases that are sufficient to support the study Yes/No 2. Was exposure defined and measured in a valid way? Yes/No 3. Primary outcomes defined and measured in a valid way? Yes/No Maximum Raw Score = 3 pts Adjusted Score = 3/3 (100%) #Yes’s/#Items OUTCOMES 6 Items 1. A. Clearly defined primary and secondary outcomes chosen a priori Yes/No 2. B. The use of proxy and composite measures is justified and explained Yes/No 3. C. Validity of proxy measures has been checked Yes/No 4. Length of observation: Sufficient f/u duration to reliably assess outcomes of interest and long-term Tx effects? Yes/No 5. Patients: Well described inclusion and exclusion criteria, reflecting target patients’ characteristics in the real world. Yes/No 6. Sample size: calculated based on clear a priori hypotheses regarding the occurrence of outcomes of interest and target effect of studied Tx versus comparator? Yes/No Maximum Raw Score = 6 pts Adjusted Score = 6/6 (100%) #Yes’s/#Items
  25. 25. TABLE Checklist combining ISPOR/Roche et al. assessments (continued) Analyses 4 Items (Categorized under OUTCOMES umbrella in Roche et al.) 1. Study groups are compared at baseline using univariate analyses and analyses of subgroups or interaction effects reported. Yes/No 2. Avoided biases related to baseline differences using matching and/or adjustments Yes/No Continued below Analyses continued Yes = 1 pt No = 0 3. Sensitivity analyses are performed to check the robustness of results and the effects of key assumptions on definitions or outcomes. Yes/No 4. (From ISPOR Analyses #1) Thorough assessment of potential measured and un-measured confounders Yes/No Maximum Raw Score = 4 pts Adjusted Score = 4/4 (100%) #Yes’s/#Items Results/Reporting 10 Items 1. Flow chart explaining all exclusions and individuals screened or selected at each stage of defining the final sample Yes/No 2. Sensitivity analyses of several databases go in the same direction as primary analyses. Yes/No 3. Detailed description of patients’ characteristics/descriptive statistics (Demographics, characteristic of disease of interest, Co-morbidities and concomitant treatments) Yes/No 4. Extensive presentation of results/authors describe the key components of their statistical approaches Yes/No 5. Extensive presentation of unmatched and matched populations (if matching) using univariate and multivariate, unadjusted and adjusted analyses Yes/No 6. If patients’ are lost to follow-up then there characteristics are compared with patients remaining? + Was follow-up similar or accounted for (ISPOR Data #4) between groups? Yes/No 7. Were confounder-adjusted estimates of Tx effects reported? Yes/No 8. Did the authors describe the statistical uncertainty of their findings? Yes/No 9. Was the extent of missing data reported? Yes/No 10. Were the absolute and relative measures of Tx effects reported? Yes/No Maximum Raw Score = 10 pts Adjusted Score = 10/10 (100%) #Yes’s/#Items Discussion/Interpretation 6 Items 1. Discussion of differences with results of efficacy randomized controlled trials Yes/No 2. Results consistent with prior known information or if not was an adequate explanation provided? Yes/No 3. Are the observed Tx effects considered clinically meaningful? Yes/No 4. Summary and interpretation of findings, focusing 1st on whether they confirm or contradict a priori hypotheses Yes/No 5. Discussion of possible biases and confounding factors, especially related to the observational nature of the study Yes/No 6. Suggestions for future research to challenge, strengthen, or extend the study results Yes/No Maximum Raw Score = 6 pts Adjusted Score = 6/6 (100%) #Yes’s/#Items Conflict of Interest 2 Items 1. Were there any potential conflicts of interest Yes/No 2. If there were potential conflicts of interest, were steps taken to address this? Yes/No Maximum Raw Score = 2 pts Adjusted Score = 2/2 (100%) #Yes’s/#Items
  26. 26. Design (8 items) Outcomes (6 items) Results/Reporting (11 items) Conflict of Interest (2 items) Figure: Wheel of Domains for combined Roche/ISPOR checklist (modeled after PRECIS wheel)
  27. 27. Phase II: Initial Tool Creation • The initial tool included: o 13 primary quality criteria – All primary items must be satisfied to recommend a study that could inform future guidelines o 14 secondary criteria o Across 8 domains: – Background/relevance, design, data, measures, analyses, results/reporting, discussion/interpretation, conflict of interest • Tool refined, based on: o Task Force Members votes for items to keep/remove/merge
  28. 28. Phase III: Within taskforce pilot • Methods: o 6 papers distributed to 9 taskforce members o Two reviewer groups (1 x group of 5 and 1 x group of 4) o Each group member scored / appraised 3 papers o Calculation of an “agreement rate” (per item & per domain – split by primary and supporting “secondary” domains) o Global assessment = paper of sufficient quality to inform guideline development (Y/N) required fulfilment of all primary criteria and agreement by ALL raters • Results: o Overall agreement at ~50% level; agreement appeared to be random • Action: o Revise tool based on taskforce feedback to – Simplify it, and – Remove areas of potential ambiguity/subjectivity. o Conduct an extended pilot, using a wider rater group, to see whether the tool revisions had improved agreement rates.
  29. 29. Phase IV: Larger Pilot & tool finalization/minor modifications • Methods: o 22 x raters involved in total o 3 x rater groups: – A, B, C (2 x groups of 7; 1 x group of 8) – Members of groups A and B received 2 papers on intervention adherence / persistence – Members of group C received 2 papers on ICS particle size / formulation for quality assessment using the revised tool o Agreement was evaluated across each field for each rater group, separately and in total
  30. 30. AVERAGE AGREEMENT ACROSS FIELDS Adherence (Elkout & Barnes) HRU & persistence (Tan & Lee) Particle size & formulation summary (Barnes & Terzano) OVERALL SUMMARY (weighted by # group contributors) PRIMARY DOMAINS 1. Background 1.1. Clearly stated research question 79% 100% 86% 89% 2. Design 2.1 Population defined and justified 64% 94% 71% 77% 2.2. Comparision groups defined and justified 93% 71% 79% 79% 2.3. Setting defined and justified 93% 100% 93% 95% 3. Measures 3.1. (If relevant), exposure is clearly defined 93% 71% 76% 78% 3.2. Primary outcomes clearly defined and measured 71% 89% 93% 85% 4. Analylsis 4.1. Potential confounders are considered and adjusted for in the analysis, and reported 64% 81% 71% 73% 4.2. Study groups are compared at baseline 79% 79% 79% 77% 5. Results 5.1. Results are clearly presented for all primary and secondary endpoints as well as confounders 79% 94% 71% 82% 6. Discussion / Interpretation 6.1. Results consistent with known information or if not, an explanation is provided 86% 100% 86% 91% 6.2 The clincial relevance of the results is discussed 85% 88% 93% 88% 7. Conflict of interests 7.1. Potential conflicts of interest, including study funding, are stated 79% 100% 93% 91% A B C A, B,C
  31. 31. Fulfilment of 100% of primary criteria or …? RATER GROUP A B C PAPER TOIC Adherence HRU & persistence Particle size & formulation PAPERS Elkout et al Voshaar et al Tan et al Lee et al Barnes et al Terzano et al NUMBER OF RATERS 8 7 7 # primary criteria fulfillment %primary criteria fulfillment Quality criteria attainment Average attainment (across A, B, C) 12 of 12 correct 100% 64% 64% 57% 62% ≥11 of 12 correct 92% 57% 71% 57% 62% ≥10 of 12 correct 83%) 57% 71% 57% 62% ≥9 of 12 correct 75% 71% 79% 64% 71% ≥8 of 12 correct 67% (67% correct) 79% 93% 79% 83% More lenient quality criteria fulfilemnt (e.g. 10-of-12 of 11-of-12) had limited impact on overall agreement – decisions to require 100% fulfilment (pending further refinement)
  32. 32. Phase IV: Larger Pilot & tool finalization/minor modifications • Remaining disagreement: Reviewer feedback suggested some of the persisting disagreement was the result of: o Multiple “sub-questions” within some of the appraisal criteria, e.g. “Potential conflicts of interest, including study funding, are stated”, which contains a question about author conflicts of interest and also the specific study funding. Such questions lead to greater potential for disagreement between raters. o Poorly written papers (even though those selected for evaluation were of above-average quality) • Action: Each tool criteria was reviewed by the taskforce members once more and any remaining ambiguities removed. This was undertaken at the Amsterdam Taskforce Meeting.
  33. 33. Phase V: tool finalization for use, development of an online tool • Once the Excel version of the tool had been approved by the taskforce, it was converted into a Google form. • An online tool offered: o Smoother user/rate experience, o Minimized opportunities for data mis-entry, AND o Allowed automated collation into an online spreadsheet and delivered an overview of responses to each question.
  34. 34. THE TOOL – desktop version (I)
  35. 35. THE TOOL – desktop version (II)
  36. 36. The TOOL: Web version RATER ASSESSMENT (for taskforce purposes / participant profiling only)
  37. 37. The TOOL: Web version PRIMARY CRITERIA (I)
  38. 38. The TOOL: Web version PRIMARY CRITERIA (II)
  39. 39. The TOOL: Web version PRIMARY CRITERIA (III)
  40. 40. The TOOL: Web version
  41. 41. SECONDARY CRITERIA
  42. 42. Acknowledgements • EAACI Task force members • REG Staff o Alison Chisholm o Zoe Mitchell • Sarah Acaster and Katy Gallop • Robert Perry
  43. 43. LITERATURE REVIEW NICOLAS ROCHE: Hôpital de l’Hôtel Dieu, Paris —
  44. 44. PICOT questions • TF members + informal literature search: n=21 • Preselection: n=9 o Education o Adherence/persistence o Smoking asthmatics o Devices o Molecules (ICS) o Biotherapies (omalizumab) o Formulations (Extra-fine) o Strategies (ICS vs FDC…) o Antibiotics for exacerbations (macrolides vs others) o ACOS vs asthma vs COPD, ICS vs LABDs vs both
  45. 45. P People of all ages prescribed regular maintenance ICS I Adherence to recommended therapy C Comparison of outcomes between groups of different levels of adherence (e.g. 0- 25%, 25-50%, 50-75%, over 75%) O Exacerbations, admissions, symptoms, QOL T 12 months P People of all ages prescribed regular maintenance ICS I Effectiveness of different inhaler devices/delivery systems in maintaining asthma control C Comparison of outcomes between groups using the same molecule through different inhaler systems (pMDI, Breath activated MDI, DPI) O Exacerbations, admissions, symptoms, QOL T 12 months P Smokers and ex-smokers with asthma receiving ICS and having exacerbations I Effectiveness of different treatment regimens C Comparison of outcomes between those going from low dose ICS to: HDICS, LDICS + LABA, HDICS + LABA. All regimens +/- LTRA O Exacerbations T 12 months P Adults presenting for care with asthma exacerbation in ambulatory setting (depending on dataset, ED setting and discharged to home would be great as well) I Azithromycin or other macrolide C Two comparators: Other class of antibiotic (e.g., fluoroquinolone; beta lactam); no antibiotic. O Relapse (all-cause unscheduled office visit, presenting to ED, or hospitalization) T 30 days 24 0 7 3
  46. 46. P People of all ages prescribed regular maintenance ICS I Adherence to recommended therapy C Comparison of outcomes between groups of different levels of adherence (e.g. 0- 25%, 25-50%, 50-75%, over 75%) O Exacerbations, admissions, symptoms, QOL T 12 months P People of all ages prescribed regular maintenance ICS I Effectiveness of different inhaler devices/delivery systems in maintaining asthma control C Comparison of outcomes between groups using the same molecule through different inhaler systems (pMDI, Breath activated MDI, DPI) O Exacerbations, admissions, symptoms, QOL T 12 months P Smokers and ex-smokers with asthma receiving ICS and having exacerbations I Effectiveness of different treatment regimens C Comparison of outcomes between those going from low dose ICS to: HDICS, LDICS + LABA, HDICS + LABA. All regimens +/- LTRA O Exacerbations T 12 months P People of all ages prescribed regular maintenance ICS I Effectiveness of small vs standard-size ICS particles in maintaining asthma control C Comparison of outcomes between groups using the same or different molecules administered as extrafine of fine particles O Exacerbations, admissions, symptoms, QOL T 12 months 24 12 7 3
  47. 47. Search and selection flow MEDLINE (hits) Jan 2004-Dec 14: 1,347 Jan 2015-Dec 15: 52 Total: 1,399 EMBASE (hits) Jan 2004-Dec 14: 2,086 Jan 2015-Dec 15: 275 Total: 2,361 Meeting inclusion criteria Jan 2004-Dec 14: 42 Jan 2015-Dec 15: 4 Total: 46 PICOT 1 ADHERENCE Jan 2004-Dec 14: 23 Jan 2015-Dec 15: 1 24 Exclusions Jan 2004-Dec 14: 2,823 Jan 2015-Dec 15: 280 • Not an asthma study (e.g. COPD, Allergic rhinitis) • Not an observational study, including: – Literature review – Clinical trial – Case study – Cross-sectional survey • Not aligned to one of the 4 PICOT questions Search period: January 2004 to December 2015 Total Papers Jan 2004-Dec 14: 3,433 Jan 2015-Dec 15: 327 Total: 3,760 Duplicates removed Jan 2004-Dec 14: 568 Jan 2015-Dec 15: 43 Total: 611 Total Abstracts Reviewed Jan 2004-Dec 14: 2,865 Jan 2015-Dec 15: 284 Total: 3,149 PICOT 2 DEVICE TYPE Jan 2004-Dec 14: 7 Jan 2015-Dec 15: 0 7 PICOT 3 SMOKING ASTHMA Jan 2004-Dec 14: 2 Jan 2015-Dec 15: 1 3 PICOT 4 PARTICLE SIZE Jan 2004-Dec 14: 10 Jan 2015-Dec 15: 2 12
  48. 48. Rating and final selection strategy Permutation Rater 1 Rater 2 Rater 3 Assessment YES All primary criteria fulfilled YES All primary criteria fulfilled – POSITIVE Paper is of sufficient quality to be considered by future guideline bodies 2 NO ≥1 primary criteria not fulfilled YES All primary criteria fulfilled YES All primary criteria fulfilled POSITIVE Paper is of sufficient quality to be considered by future guideline bodies 4 NO ≥1 primary criteria not fulfilled YES All primary criteria fulfilled NO ≥1 primary criteria not fulfilled NEGATIVE Paper is NOT of sufficient quality to be considered by future guideline bodies 6 NO ≥1 primary criteria not fulfilled NO ≥1 primary criteria not fulfilled – NEGATIVE Paper is NOT of sufficient quality to be considered by future guideline bodies
  49. 49. Who participated? (I) 0% 20% 40% 60% 80% Percentageofraters
  50. 50. Who participated? (II) 12% 62% 2% 6% 12% 2% 6% Commercial Researcher Faculty Hospital Practitioner Medical Writer Non-Profit Researcher Private Practice Allergologist Student / Fellow
  51. 51. Who participated? (III) 0% 5% 10% 15% 20% 25% 30% 35% 40% 0-10 11-24 25-49 50-100 ≥100 Percentageofraters Number of peer review papers
  52. 52. Summary of ratings 8 4 2 8 22 15 1 1 4 21 1 2 0 0 3 0 5 10 15 20 25 Adherence Devices Smoking Asthma Particle Size All Papers NUMBEROFPAPERS PICOT QUESTION FOCUS Sufficiently High Quality Insufficient Quality TBC
  53. 53. Summary of ratings 33% 57% 67% 67% 48% 63% 14% 33% 33% 46% 4% 29% 0% 0% 7% 0% 10% 20% 30% 40% 50% 60% 70% 80% Adherence Devices Smoking Asthma Particle Size All Papers PERCENTAGEOFPAPERS PICOT QUESTION FOCUS Sufficiently High Quality Insufficient Quality TBC n=24 n=7 n=3 n=12
  54. 54. Summary of failed criteria 0 2% 17% 4% 6% 17% 20% 14% 1% 4% 15% 0% 5% 10% 15% 20% 25% 1.1 2.1 2.2 3.1 3.2 4.1 4.2 5.1 6.1 6.2 7.1 PercentageoftotalCriteria FailuresRecordedbyRaters Taskforce Quality Assessment Tool Primary Criteria
  55. 55. Summary of failed categories 0% 19% 10% 37% 14% 5% 15% 0% 5% 10% 15% 20% 25% 30% 35% 40% PercentageoftotalCriteriaFailures RecordedbyRaters Taskforce Quality Assessment Tool Primary Quality Categories
  56. 56. Reading grid • Quality assessment • Summary of methods o Studied population, intervention, outcomes, setting • Summary of results o Magnitude of differences / robustness • Possible remaining biases • Final level of evidence (GRADE) • Comparison with data from RCTs o Explanations for differences o Possible impact on guidelines
  57. 57. Key messages from papers of sufficient quality: PICOT 1 PICOT 1: “ADHERENCE TO ICS THERAPY” P People of all ages prescribed regular maintenance ICS I Adherence to recommended therapy C Comparison of outcomes between groups of different levels of adherence (e.g. 0-25%, 25-50%, 50-75%, over 75%) O Exacerbations, admissions, symptoms, QOL T 12 months
  58. 58. PICOT 1 / Adherence papers rated as being of sufficiently high quality to inform future guideline development by Taskforce Quality Raters 1 Sadatsafavi M, Lynd LD, Marra CA, FitzGerald JM. Dispensation of long-acting β agonists with or without inhaled corticosteroids, and risk of asthma-related hospitalisation: a population-based study. Thorax. 2014;69(4):328-34 2 Friedman HS, Navaratnam P, McLaughlin J. Adherence and asthma control with mometasone furoate versus fluticasone propionate in adolescents and young adults with mild asthma. J Asthma. 2010;47(9):994-1000. 3 Campbell JD, Allen-Ramey F, Sajjan SG, Maiese EM, Sullivan SD. Increasing pharmaceutical copayments: impact on asthma medication utilization and outcomes. Am J Manag Care. 2011;17(10):703-10. 4 Tan H, Sarawate C, Singer J, Elward K, Cohen RI, Smart BA, Busk MF, Lustig J, O'Brien JD, Schatz M. Impact of asthma controller medications on clinical, economic, and patient-reported outcomes. Mayo Clin Proc. 2009;84(8):675-84 5 Williams LK, Pladevall M, Xi H, Peterson EL, Joseph C, Lafata JE, Ownby DR, Johnson CC. Relationship between adherence to inhaled corticosteroids and poor outcomes among adults with asthma. J Allergy Clin Immunol. 2004;114(6):1288-93 6 Taegtmeyer AB, Steurer-Stey C, Price DB, Wildhaber JH, Spertini F, Leuppi JD. Predictors of asthma control in everyday clinical practice in Switzerland. Curr Med Res Opin. 2009;25(10):2549-55 7 Laforest L, Licaj I, Devouassoux G, Chatté G, Belhassen M, Van Ganse E, Chamba G. Relative exposure to controller therapy and asthma exacerbations: a validation study in community pharmacies. Pharmacoepidemiol Drug Saf. 2014;23(9):958-64. 8 Laforest L, Licaj I, Devouassoux G, Chatte G, Martin J, Van Ganse E. Asthma drug ratios and exacerbations: claims data from universal health coverage systems. Eur Respir J. 2014;43(5):1378-86 9 Björnsdóttir US, Sigurðardóttir ST, Jonsson JS, Jonsson M, Telg G, Thuresson M, Naya I, Gizurarson S. Impact of changes to reimbursement of fixed combinations of inhaled corticosteroids and long-acting β2-agonists in obstructive lung diseases: a population-based, observational study. Int J Clin Pract. 2014;68(7):812-9. Awaiting second review
  59. 59. Risk of asthma-related hospitalization and maintenance treatment • Regular treatment has to be ICS-based o No risk difference between ICS-LABA and ICS alone o Greater risk with LABA alone vs ICS-LABA & ICS alone • Regular > irregular treatment or none o Lower risk with regular treatment o Except with LABA • Database. Well matched • GRADE: low → moderate o No clinical data on severity • Clinical impact: yes • Similar evidence from RCTs: no Sadatsafavi Thorax 2014
  60. 60. Adherence to ICS • Adherence MF-DPI>FP • SABA use MF-DPI<FP • No difference in other clinical outcomes • Database. Well matched • GRADE: low → moderate o No clinical data on severity • Clinical impact: limited/none • Similar evidence from RCTs: no Freidman Thorax 2014
  61. 61. Impact of shifting part of drug costs to patients • Higher copayments o Less adherence o More adverse asthma outcomes (outpatient and ED visits), higher costs • Database • GRADE: low → moderate o No clinical data on severity • Clinical impact: yes • Similar evidence from RCTs: no Campbell Am J Manag Care 2011
  62. 62. ICS vs LTRA by adherence • Monotherapy o Adherent patients: ICS>LTRA o Non-adherent patients: ICS<LTRA • Combination: Best = ICS-LABA • Database + survey • GRADE: low → moderate o No clinical data on severity for the database part o Responders’ bias for the survey • Clinical impact: yes • Similar evidence from RCTs: partly (pragmatic) Tan Mayo Clin Proc 2009
  63. 63. Adherence and asthma exacerbations • Low adherence increases the risk of ED visits and oral steroid treatment • Database, well matched • GRADE: low → moderate o No clinical data on severity • Clinical impact: yes • Similar evidence from RCTs: no Williams JACI 2004
  64. 64. Adherence and asthma control • Lower ACQ improvement associated with low adherence o Other predictors: age, severity • Prospective cohort study, multivariate analysis • GRADE: low → moderate o Selection bias? • Clinical impact: yes • Similar evidence from RCTs: no Taegtmeyer Curr Med Res Opin 2009
  65. 65. Adherence and asthma outcomes • Low adherence associated with poorer control and more hospital contacts and oral steroid courses • Prospective study in pharmacies , multivariate analysis, validation of MPR data • GRADE: low → moderate o Selection bias & single pharmacy for each patient • Clinical impact: yes • Similar evidence from RCTs: no Laforest 2014 Pharmacoepidemiol Druf Saf
  66. 66. Adherence and asthma exacerbations • Low MPR increases the risk of oral steroid treatment and hospitalization • Database, unmatched, multivariate adjustment • GRADE: low → moderate o No clinical data on severity • Clinical impact: yes • Similar evidence from RCTs: no Laforest ERJ 2014
  67. 67. PICOT 1: Conclusions • Several concordant observational studies showing an association between low adherence and poor asthma outcomes o Each individually provides a moderate level of evidence o Globally high level (concordance)? • Adherence is influenced by copayment level o With an impact on outcomes • Maintenance therapy with LABA alone is dangerous
  68. 68. Key messages from papers of sufficient quality: PICOT 2 PICOT 2: “DEVICE TYPE” P People of all ages prescribed regular maintenance ICS I Effectiveness of different inhaler devices/delivery systems in maintaining asthma control C Comparison of outcomes between groups using the same molecule through different inhaler systems (pMDI, Breath activated MDI, DPI) O Exacerbations, admissions, symptoms, QOL T 12 months
  69. 69. PICOT 2 / Devices papers rated as being of sufficiently high quality to inform future guideline development by Taskforce Quality Raters 1 Price D, Chrystyn H, Kaplan A, Haughney J, Román-Rodríguez M, Burden A, Chisholm A, Hillyer EV, von Ziegenweidt J, Ali M, van der Molen T. Effectiveness of same versus mixed asthma inhaler devices: a retrospective observational study in primary care. Allergy Asthma Immunol Res. 2012;4(4):184-91 2 Price D, Roche N, Christian Virchow J, Burden A, Ali M, Chisholm A, Lee AJ, Hillyer EV, von Ziegenweidt J. Device type and real-world effectiveness of asthma combination therapy: an observational study. Respir Med. 2011;105(10):1457-66. doi: 10.1016/j.rmed.2011.04.010. Epub 2011 May 25. 3 Price D, Haughney J, Sims E, Ali M, von Ziegenweidt J, Hillyer EV, Lee AJ, Chisholm A, Barnes N. Effectiveness of inhaler types for real-world asthma management: retrospective observational study using the GPRD. J Asthma Allergy. 2011;4:37-47. 4 Thomas M, Price D, Chrystyn H, Lloyd A, Williams AE, von Ziegenweidt J. Inhaled corticosteroids for asthma: impact of practice level device switching on asthma control. BMC Pulm Med. 2009;9:1. doi: 10.1186/1471-2466-9-1.
  70. 70. Mixed vs single device(s) • Single device = better control and less severe exacerbations • Database study, adjusted analyses • GRADE: low → moderate o No clinical data on severity and pts knowledge/skills • Clinical impact: yes • Similar evidence from RCTs: no Price Allergy Asthma Immunol Res 2012
  71. 71. Impact of switching • DPI to pMDI or BAI or other DPI • BAI to pMDI or other BAI • Poorer outcomes following the switch • Database study, matched & adjusted analyses • GRADE: low → moderate o No clinical data on severity and pts knowledge/skills • Clinical impact: possible • Similar evidence from RCTs: no Thomas BMC Pulm Med 2009
  72. 72. DPI vs pMDI for SFC • pMDI better for several asthma outcomes • Database study, matched • GRADE: low → moderate o No clinical data on severity and pts knowledge/skills/adherence • Clinical impact: uncertain (reinforce adherence, account for preferences?) • Similar evidence from RCTs: no
  73. 73. DPI vs pMDI & BAI for ICS administration • BAI and DPI better than pMDI for several asthma outcomes • Database study, unmatched, adjusted • GRADE: low → moderate o No clinical data on severity and pts knowledge/skills/adherence • Clinical impact: uncertain (reinforce adherence, account for preferences? Contradictory data) • Similar evidence from RCTs: no Price J Asthma Allergy 2011
  74. 74. Summary • Using a single device might be better • Switching without a consultation is followed by poorer outcomes • pMDI better for FDC? • While BAI and DPI better for ICS?
  75. 75. Key messages from papers of sufficient quality: PICOT 3 PICOT 3: “SMOKING ASTHMATICS” P Smokers and ex-smokers with asthma receiving ICS and having exacerbations I Effectiveness of different treatment regimens C Comparison of outcomes between those going from low dose ICS to: HDICS, LDICS + LABA, HDICS + LABA. All regimens +/- LTRA O Exacerbations T 12 months
  76. 76. Impact of smoking on control • Lower control in smokers • Same benefit for all (current, ex, never smokers) • Prospective cohort study, unadjusted analyses • GRADE: low → low o High drop-out rate, o No adjustment nor matching o Few clinical data available besides control • Clinical impact: no • Similar evidence from RCTs: no Brusselle Respir Med 2012
  77. 77. Impact of smoking on control • Better outcomes with extrafine vs standard size particles, larger differences in current and ex smokers • Database matched study, adjusted analyses • GRADE: low → moderate o No clinical / spirometry data • Clinical impact: uncertain (exploratory) • Similar evidence from RCTs: no Roche AJRCCM 2015
  78. 78. Key messages from papers of sufficient quality: PICOT 4 PICOT 4: “SMALL AIRWAYS MANAGEMENT; ICS PARTICLE SIZE” P People of all ages prescribed regular maintenance ICS I Effectiveness of small vs standard-size ICS particles in maintaining asthma control C Comparison of outcomes between groups using the same or different molecules administered as extrafine of fine particles O Exacerbations, admissions, symptoms, QOL T 12 months
  79. 79. PICOT 2 / Smoking Asthma papers rated as being of sufficiently high quality to inform future guideline development by Taskforce Quality Raters 1 van Aalderen WM, Grigg J, Guilbert TW, Roche N, Israel E, Martin RJ, Colice G, Postma DS, Hillyer EV, Burden A, Thomas V, von Ziegenweidt J, Price D. Small-particle Inhaled Corticosteroid as First- line or Step-up Controller Therapy in Childhood Asthma. J Allergy Clin Immunol Pract. 2015;3(5):721-31 2 Martin RJ, Price D, Roche N, Israel E, van Aalderen WM, Grigg J, Postma DS, Guilbert TW, Hillyer EV, Burden A, von Ziegenweidt J, Colice G. Cost-effectiveness of initiating extrafine- or standard size-particle inhaled corticosteroid for asthma in two health-care systems: a retrospective matched cohort study. NPJ Prim Care Respir Med. 2014;24:14081 3 Colice G, Martin RJ, Israel E, Roche N, Barnes N, Burden A, Polos P, Dorinsky P, Hillyer EV, Lee AJ, Chisholm A, von Ziegenweidt J, Barion F, Price D. Asthma outcomes and costs of therapy with extrafine beclomethasone and fluticasone. J Allergy Clin Immunol. 2013;132(1):45-54 4 Price D, Thomas M, Haughney J, Lewis RA, Burden A, von Ziegenweidt J, Chisholm A, Hillyer EV, Corrigan CJ. Real-life comparison of beclometasone dipropionate as an extrafine- or larger-particle formulation for asthma. Respir Med. 2013;107(7):987-1000 5 Price D, Martin RJ, Barnes N, Dorinsky P, Israel E, Roche N, Chisholm A, Hillyer EV, Kemp L, Lee AJ, von Ziegenweidt J, Colice G. Prescribing practices and asthma control with hydrofluoroalkane- beclomethasone and fluticasone: a real-world observational study. J Allergy Clin Immunol. 2010;126(3):511-8.e1-10 6 Allegra L, Cremonesi G, Girbino G, Ingrassia E, Marsico S, Nicolini G, Terzano C; PRISMA (PRospectIve Study on asthMA control) Study Group. Real-life prospective study on asthma control in Italy: cross-sectional phase results. Respir Med. 2012;106(2):205-14 7 Barnes N, Price D, Colice G, Chisholm A, Dorinsky P, Hillyer EV, Burden A, Lee AJ, Martin RJ, Roche N, von Ziegenweidt J, Israel E. Asthma control with extrafine-particle hydrofluoroalkane- beclometasone vs. large-particle chlorofluorocarbon-beclometasone: a real-world observational study. Clin Exp Allergy. 2011;41(11):1521-32 8 Price D, Small I, Haughney J, Ryan D, Gruffydd-Jones K, Lavorini F, Harris T, Burden A, Brockman J, King C, Papi A. Clinical and cost effectiveness of switching asthma patients from fluticasone- salmeterol to extra-fine particle beclometasone-formoterol: a retrospective matched observational study of real-world patients. Prim Care Respir J. 2013;22(4):439-48
  80. 80. RiRL/REG retrospective matched cohort studies Treatments Population Database Results 1 BDP pMDI St vs EF Initiation Step-up Vs LABA Children 5-11 UK (CPRD) US (Optuminsight) EF>St EF=adding LABA 4 BDP pMDI St vs EF Initiation Switch 12-80 UK GPRD CPRD EF > St 5 pMDI St FP vs EF BDP Initiation Step-up 5-60 UK GPRD EF >/= St at lower doses 7 BDP pMDI St vs EF Initiation Step-up 5-60 UK GPRD EF > St
  81. 81. RiRL/REG retrospective matched cohort studies Treatments Population Database Results 2 C-E BDP/FP pMDI St vs EF Initiation 12-60/ 12-80 UK/US EF dominant 3 C-E pMDI St FP vs EF BDP Initiation Step-up 12-80 UK/US EF >/= St at lower doses and costs 8 C-E St FP-SAL vs EF BDP-FOR C-E 18-80 UK GPRD CPRD EF >/= St at lower doses EF dominant
  82. 82. RiRL/REG retrospective matched cohort studies • GRADE: low → moderate o No clinical data on severity and pts knowledge/skills/adherence o Considering all studies together: high? • Clinical impact: yes • Similar evidence from RCTs: no
  83. 83. Prospective cohort study • Unmatched, adjusted • Comparison between treatments = secondary / exploratory objective • GRADE: low → low o Selection bias o Secondary objective • Clinical impact: no • Similar evidence from RCTs: no Allegra Respir Med 2012
  84. 84. REG-EAACI TASKFORCE REPORT PANEL DISCUSSION REG SUMMIT 2016, LYON, FRANCE, 16 APRIL SESSION: ‘INFLUENCING’ Guideline Development: the REG/EAACI Taskforce Reports TIME: 12.45-13.45pm Chair / Moderator: Nikos Papadopoulos Centre for Paediatrics and Child Health, Institute of Human Development, University of Manchester, Manchester, UK; Department of Allergy, 2nd Pediatric Clinic, University of Athens, Athens, Greece —
  85. 85. • Novelty: never been done before • Expert taskforce • Wide range of participants: o Countries, Societies, Guideline groups • Focussed work o Important evidence gaps • Current literature • High-threshold set for quality • Bespoke tool o Developed for specific purpose it was used for o Tested for inter-rate agreement STRENGTHS
  86. 86. • Narrow literature focus • Small number of papers for some PICOT questions • Expertise of reviewers • Best-of-3 assessment approach? • Assessment of reporting rather than study quality (same thing from a guidelines’ perspective)? WEAKNESSES
  87. 87. Discussion • Considering o the discussions around the QA tool o the frequent difficulty in finding required information in reviewed papers • Integration of comparative effectiveness research in asthma guidelines o is at risk of remaining a subject of reluctance from guidelines and policy makers o until reporting markedly improves
  88. 88. PRACTICAL APPLICATIONS • How the literature findings and literature quality assessment tool can best be used in the future, e.g.: o By guideline groups: – To revise guidelines – To appraise evidence o Journal editors/reviewers: – To assess the quality of submission o Education: – To train / guide new researchers o Advocacy – To highlight limitations in the evidence and clear areas for improvement in the reporting of, if not the design of observational studies in the future
  89. 89. Importance of the results • The tool • The literature appraisal
  90. 90. Importance by target Target Importance Researchers Editorial boards Reviewers Guideline developers, policy makers Others
  91. 91. Importance by topic Topic Importance PICOT 1: adherence LABA alone dangerous LTRA better MF-DPI>FP Hight copayment dangerous PICOT 2: device Single Switch pMDI vs BAI vs DPI
  92. 92. Importance by topic Topic Importance PICOT 3: smoking Lower control in smokers More treatment effect with EF in smokers PICOT 4: extrafine particles EF>/=St More cost-effective
  93. 93. POTENTIAL TO EXTEND THE WORK • Disease areas • Study design (pragmatic trials) • Wider collaborations – REG/EAACI ±: o ATS o ± ERS o ± others

×