Published on

APPRAISAL OF GUIDELINES FOR RESEARCH & EVALUATION (AGREE) II Instrument: Assessment of the Quality of Clinical Practice Guidelines.
A training workshop presented to the healthcare professionals (i.e. physicians, nurses, pharmacists, technicians,...etc.) of King Khalid University Hospital, King Saud University in Riyadh, Saudi Arabia in August 2014.
I present my highest gratitude to all of my professors and colleagues who have and still inspire me.
Namely, Prof. Mahmoud El-Zalabany, Prof. Tarek Omar, Prof. Nabil Dowidar and Prof. Afaf Ibrahim (Alexandria Univ., EG), Prof. Abdelhamid Attia (Cairo Univ., EG), Prof. Lubna Al-Ansary, Dr. Hayfaa Wahbi, Dr. Rasmieh Alzeidan (King Saud Univ., KSA), Prof. David Sackett and Dr. Melissa Brouwers (McMaster Univ., CAN)

Published in: Healthcare
No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Sir William Osler, 1st Baronet (/ˈɒz.lə/, July 12, 1849 – December 29, 1919) was a Canadian physician and one of the four founding professors of Johns Hopkins Hospital. Osler created the first residency program for specialty training of physicians, and he was the first to bring medical students out of the lecture hall for bedside clinical training.[1] He has frequently been described as the "Father of Modern Medicine".[2] Osler was a person of many interests, who in addition to being a physician, was a bibliophile, historian, author, and renowned practical joker.

    1. 1. ‫ـيم‬‫ـ‬‫ـ‬‫ـ‬‫ـ‬‫ح‬‫الر‬‫ـن‬‫ـ‬‫ـ‬‫ـ‬‫ـ‬‫م‬‫الرح‬‫هللا‬ ‫ـم‬‫ـ‬‫ـ‬‫ـ‬‫ـ‬‫س‬‫ب‬ In the name of Allah. Most Gracious, Most Merciful
    2. 2. Instructor Dr. Yasser Sami Amer MS Pediatrics, MS Healthcare Informatics, CPHQ Supervisor, EBPU, QMD, KSUMC CPGs Advisor , KSUHs, AUHs Member, G-I-N Adaptation & Implementation Working Groups
    3. 3.  EBM? – CPG? 
    4. 4. EBM ?! – CPGs?!
    5. 5. Check Your Handouts @ SOFT COPIES @:- ????????????? I- PDF of [APPRAISAL OF GUIDELINES FOR RESEARCH & EVALUATION (AGREE) II Instrument] - full document (user's manual + instrument) 2- Excel file AGREE concordance calculator (electronic instrument) with THE COMMENT (JPEG file) 3- Link to the AGREE Enterprise website here (YOU are advised to create a free account before the training and browse the website) [ ] It contains the online AGREE instrument: My AGREE Plus 4- Samples of AGREE II appraisal reports (solved using the online tool/ my AGREE plus) Individual appraisals (3) (i) Lung cancer SIGN (ii) IVF number of embryos to transfer ASRM (iii) status epilepticus Group appraisal (1) (i) CVAD ( 2 files: scores and comments) HARD COPIES 1- Learning objectives & training agenda 2-Pre-training questionnaire (SUBMIT PLEASE) 2- PDF for training AGREE (27 pages) 3- AGREE II Instrument (Paper form) (11 pages) 4- AGREE II Instrument–at-a-glance document (one page) 5- Evaluation Form (SUBMIT PLEASE)
    6. 6. Learning objectives By the end of this session, participants should be able to know:- PART 1: The Appraisal for Guidelines REsearch and Evaluation II (AGREE II) Instrument: Assessment of the quality of any CPG 1. Brief overview on EBM, Research Methods and CPGs 2. What is the AGREE II Instrument?  Domains  questions/items  scoring 3. What are the uses of the AGREE Instrument? 4. How to use the AGREE? 5. AGREE II Instrument tools:  User’s manual @ & Instrument Paper form  Excel concordance calculator @  The online tool (My AGREE PLUS) @ PART 2: Working groups (hands on session) Working groups of 3 – 5 appraisers (or more) and will use the printed paper AGREE II Instrument to appraise the practice documents provided (Evidence-based Series #4-9: sections 1 – 3). Also a demonstration for the Excel calculator tool and the online tool (My AGREE Plus)
    7. 7. Uncertainty!.....Probability! Sir William Osler (1849 – 1919)
    8. 8. The Epsom Derby, 1821 (oil on canvas) by Theodore Gericault; Louvre, Paris, France Importance of Research People observe what they expect to observe, until shown otherwise
    9. 9. Importance of Research People observe what they expect to observe, until shown otherwise • gA&list=PLSC6ykdWzbjHk4fKrmB94yj11dqfkXF 0U 57 years later! Sallie Gardner (horse) at a Gallop Eadweard Muybridge in 1878 – 3 seconds silent film, USA
    10. 10. Importance of Research People observe what they expect to observe, until shown otherwise Sallie Gardner (horse) at a Gallop Eadweard Muybridge in 1878 – 3 seconds silent film, USA
    11. 11. 8/23/2014 CPGs Adaptation & Implementation in KSUMC: KAUH 2014 12
    12. 12. EBM is all about Clinical Decision Making
    13. 13. Level Evidence Grade Recomm. Strength of EVIDENCE PYRAMID
    14. 14. CPG-ICU-002 (VTE) 2013 8/23/2014 CPGs Adaptation & Implementation in KSUMC: KAUH 2014 15
    15. 15. HWCPG-ENT-001 (ABRS) 2013 8/23/2014 CPGs Adaptation & Implementation in KSUMC: KAUH 2014 16
    16. 16. HWCPG-PHARM-001 (Vanco) 8/23/2014 CPGs Adaptation & Implementation in KSUMC: KAUH 2014 17
    17. 17. EVIDENCE PYRAMID ? 18
    18. 18. Evidence Pyramid SR RCT Cohort Case control Case series Case report Expert opinion I II III IV A B C LevelsofEvidence GradesofRecommendations MA
    19. 19. Strength of EVIDENCE
    20. 20. Research Descriptive Single gp, no control gp Case report Case series Comparative > One gp, control gp. Observational (OBSERVE) Cohort Case-control Cross- sectional Experimental (INTERVENE) RCT Non-RCT
    21. 21. Types of study designs Comparative ! > One gp, control gp. TIME
    22. 22. Confounder (other factors) (another exposure that is a risk factor for the disease) OUTCOME (Effect = Disease) EXPOSURE (Cause)
    23. 23. Basic concepts of research designs
    24. 24. P value < 0.05 AND 95% CI
    25. 25. Types of Clinical Research Studies Clinical Research Studies Observational We OBSERVE Interventional = Experimental (Clinical Trials) We INTERVENE
    26. 26. Observational studies “….investigators OBSERVE what happens to participants…..!” “…….investigators assess health outcomes in groups of participants according to a protocol or research plan. Participants may receive interventions, which can include medical products, such as drugs or devices, or procedures as part of their routine medical care, but participants are not assigned to specific interventions by the investigator (as in a clinical trial). For example, investigators may observe a group of older adults to learn more about the effects of different lifestyles on cardiac health.” -
    27. 27. Observational studies: another definition Measures characteristics of the subject without attempting to manipulate/influence the subjects. CAN NOT prove causation, only association. -STAT INTRO
    28. 28. Researchers measure Exposures & Outcomes but do not measure any particular ‘intervention’ as part of the study Observational studies Prospective (forward in time) Retrospective (backward in time)
    29. 29. Observational studies: types • Cohort • Case-control • Cross-sectional
    30. 30. Cohort study An observational study in which a defined group of people (the cohort) is followed over time. The outcomes of people in subsets of this cohort are compared, to examine people who were exposed or not exposed (or exposed at different levels) to a particular intervention or other factor of interest. A prospective cohort study assembles participants and follows them into the future. A retrospective (or historical) cohort study identifies subjects from past records and follows them from the time of those records to the present. Because subjects are not allocated by the investigator to different interventions or other exposures, adjusted analysis is usually required to minimize the influence of other factors (confounders)
    31. 31. Interventional studies (clinical trials) • Participants receive specific interventions according to the research plan or protocol created by the investigators. These interventions may be medical products (e.g. drugs or devices); procedures; or changes to participants' behavior, for example, diet. • Compare (i) a new medical approach to a standard one that is already available or to a placebo that contains no active ingredients or to no intervention. (ii) compare interventions that are already available to each other. • The investigators try to determine the safety and efficacy of the intervention by measuring certain outcomes in the participants. • For example, investigators may give a drug or treatment to participants who have high blood pressure to see whether their blood pressure decreases.
    32. 32. Experimental studies: another definition Attempt to manipulate or influence the subjects in order to obtain data. Properly designed experiments can be used to prove causation. -STAT INTRO
    33. 33. Clinical trial (Interventional study) An experiment to compare the effects of two or more healthcare interventions. Clinical trial is an umbrella term for a variety of designs of healthcare trials, including: 1. Uncontrolled trials 2. Controlled trials 3. Randomised controlled trials
    34. 34. Randomized controlled/ clinical trial (RCT) • An experiment in which 2 or more interventions, possibly including a control intervention or no intervention, are compared by being randomly allocated to participants. • In most trials one intervention is assigned to each individual but sometimes assignment is to defined groups of individuals (for example, in a household) or interventions are assigned within individuals (for example, in different orders or to different parts of the body)
    35. 35. A review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyse data from the studies that are included in the review. Statistical methods (meta-analysis) may or may not be used to analyse and summarise the results of the included studies. Systematic review (synonym: systematic overview)
    36. 36. Meta-analysis The use of statistical techniques in a systematic review to integrate the results of included studies. Sometimes misused as a synonym for systematic reviews, where the review includes a meta-analysis.
    37. 37. Cochrane Review • Systematic summaries of evidence of the effects of healthcare interventions, intended to help people make practical decisions. • For a review to be called a ‘Cochrane Review’ it must be in CDSR or CMR. • Prepared using Review Manager (RevMan) software provided by the Collaboration, and adhere to a structured format described in the Cochrane Handbook for Systematic Reviews of Interventions. CDSR= Cochrane Database of SRs, CMR=Cochrane Methodology Register
    38. 38. EBM: Clinical questions (PICO or PICOT)
    39. 39. PICOT
    40. 40. Best study design to answer clinical questions
    41. 41. PICO example: formulation of searchable clinical questions
    42. 42. A 28-year-old male presents with recurrent furunculosis (skin boils) for past 8 months; these episodes have been treated with drainage and several courses of antibiotics but keep recurring. He asks if recurrences can be prevented. P Population/patient patients with recurrent furunculosis I Intervention/indicator prophylactic antibiotics C Comparator/control no treatment O Outcome reduction in recurrence rate of furunculosi
    43. 43. A physician thought he heard that after a myocardial infarction (MI), aspirin may not improve mortality. He asks the clinical pharmacist, while on rounds, “Does daily aspirin really improve mortality, or is there evidence to the contrary?” Prognosis P = postmyocardial infarction patient; I = an aspirin a day; C = no aspirin; O = improve mortality Cohort studies first.
    44. 44. CPGs
    45. 45. Definition: (old) “Systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances” (IOM 1990). Increasing international interest in the development and implementation of CPGs. Clinical Practice Guidelines (CPGs) 48
    46. 46. CPGs New Definition CPGs are “Statements that include Recommendations intended to optimize patient care that are informed by a Systematic Review of evidence and an assessment of the benefits and harms of alternative care options”. Committee on Standards for Developing Trustworthy CPGs (IOM-AHRQ 2011) EVIDENCE-BASED CPGs
    47. 47. Standards for high quality CPGs IOM 2011 – G-I-N 2012
    48. 48. IOM 2011: 8 Standards for Developing Trustworthy CPGs 1. Establishing transparency. 2. Management of conflict of interest (COI). 3. GDG composition. 4. CPG – SR intersection. 5. Establishing [E] foundations for and rating strength of [Rs]. (i.e. E – Rs link) 6. Articulation of [Rs]. 7. External review. 8. Updating.
    49. 49. Date of download: 8/19/2014 From: Guidelines International Network: Toward International Standards for Clinical Practice Guidelines Ann Intern Med. 2012;156(7):525-531. doi:10.7326/0003-4819-156-7-201204030-00009 Copyright © American College of Physicians. All rights reserved.
    50. 50. 1 2 3 4 5 6 7 8 9 10 11
    51. 51. 8/23/2014 55
    52. 52. Purpose To guide on how to use and apply the AGREE II for assessing CPGS Items Domains User’s Manual Website
    53. 53. Outlines • Overview of AGREE II including the items and domains. • Present the scoring method of the CPG. • How to use and apply the AGREE II for assessing CPG. • Share the overall scoring. • The AGREE Enterprise Website
    54. 54. What is the AGREE II Tool ? • Quality (Methodological rigor & transparency; confident in resulting Recommendations) 59 Using the AGREE II Instrument Assess Guide •CPGs Development •CPGs Reporting History 1st AGREE was published in 2003,then refined in 2009 AGREE II (New scoring “7 point scale” – Items modifications – New user’s manual) – 11 YEARS ! Can be applied to any CPG in any Disease area !
    55. 55. • Healthcare providers/ clinicians • CPG developers/ adapters • Policy makers • Educators Who can use the AGREE II ?
    56. 56. Considerations before a CPG Assessment • increase the reliability of the assessment 2 – 4 Appraisers • in full and obtain all related information and needed documents before undertaking the AGREE II assessment ( to make a well informed assessment) Read CPG first
    57. 57. Rating Scale • All AGREE II Items are rated on the following 7-point scale Score Meaning 7 (Strongly Agree) = If the quality of reporting is exceptional and full criteria and considerations in User’s manual are met. 1 (Strongly Disagree) = No information relevant to AGREE II item OR the concept is very poorly reported 2 – 6 = when the reporting of the item does not meet the full criteria or considerations, depending on the completeness & quality of reporting .
    58. 58. The AGREE II Includes…. •Core Items •(6 Domains) 23 •Overall Assessment Items2
    59. 59. 23 Items in 6 Domains USER’S MANUAL page 7 DOMAINS No. of Items 1 Scope & Purpose 3 2 Stakeholder Involvement 3 3 Rigour of Development 8 4 Clarity & Presentation 4 5 Applicability 3 6 Editorial Independence 2
    60. 60. Items and Domains: A Closer Look AGREE II: USER’S MANUAL Per each Domain (guidance for rating the 23 items) Pages 11 – 41 User’s Manual Description: Where to Look: How to Rate: Item content includes the following CRITERIA: Additional CONSIDERATIONS:
    61. 61. Domain 1
    62. 62. DOMAIN 1. SCOPE AND PURPOSE 1. Objective(s): Health impact & benefits of a CPG on target population Introduction, scope, purpose, rationale, background & objectives
    63. 63. Examples of CPG Objective(s)
    64. 64. SIGN Asthma CPG 2012
    65. 65. 2. Health Question(s):- Questions, Scope, Purpose, Rationale and Background 3. Target population:- Pt. population, target population, relevant pt.s, scope and purpose DOMAIN 1. SCOPE AND PURPOSE
    66. 66. Health/ Clinical/ Key Questions Patient (& disease characteristics) Intervention(s) Professionals (Target users) Outcomes (purpose of the CPGs) Healthcare settings (& context) 19th March 2013 EBCPGs: Dr. Yasser Sami Amer 71 CPG Scope: PIPOH Model
    67. 67. Health/ clinical Questions (PIPOH) Sample P: Patient/target population: Pediatric (age 1–18 years) and Adult (age 19 years or older) patients planned for undergoing surgical procedures. Note: These CPGs do not specifically address newborn (premature and full-term) infants. While the CPGs do not address all concerns for patients with renal or hepatic dysfunction, antimicrobial prophylaxis often does not need to be modified for these patients when given as a single preoperative dose before surgical incision. I: Intervention and practices considered and CPG Category: Assessment of Therapeutic Effectiveness and Prevention Primary antimicrobial prophylaxis (i.e., prevention of an initial infection) for surgical procedures, including antibiotic choice, dose, and dosage regimen. Antimicrobial Prophylaxis in Surgery
    68. 68. P: Professionals and Intended Users (target users/ stakeholders) and Clinical Specialty: Physicians, Nurses, Allied Health Personnel and Clinical Pharmacists in the departments of Surgery and all Surgical subspecialties (Colon and Rectal Surgery, Gastroenterology, Plastic Surgery, Urology, Thoracic Surgery, Vascular Surgery and Neurological Surgery), Obstetrics and Gynecology, Orthopedic Surgery, Ophthalmology and Pharmacology.
    69. 69. O: Major Outcomes Considered 1. Postoperative infection rates 2. Postoperative morbidity and mortality rates 3. Duration and cost of health care 4. Adverse effects H: Healthcare settings King Saud University Hospitals (KKUH/ KAUH) – Tertiary/ Governmental/ University Hospital – Departments and clinics (mentioned in the clinical specialty)
    70. 70. SIGN Asthma CPG 2012 supporting material
    71. 71. Domain 2
    72. 72. 4. Guideline group Methods, guideline panel list, acknowledgements, & appendices 5. Patient preference Scope, methods, guideline panel list, external review &target population perspectives 6. Target users Target user & intended user DOMAIN 2. STAKEHOLDER INVOLVEMENT
    73. 73. Domain 3
    74. 74. 7. Systematic methods for E search 8. Selection Criteria of E 9. Strengths & Limitations of E 10. Methods of Rs 11. Benefits , side effect and risks in Rs 12. Evidence Links (Gs of Rs – LoE) 13. External Review 14. Update Procedure DOMAIN 3. RIGOUR OF DEVELOPMENT Methods, literature search strategy & appendices + inclusion/ exclusion criteria + Evidence tables, clinical evidence, evidence description (results), evidence interpretation (discussion) Methods, CPG Development process  same sections + Rs, Key Evidence + acknowledgements + CPG update, date of CPG
    75. 75. Methodology of development • In same CPG document (in brief or detailed) • In a separate document or supplement (usu. Detailed) • Not documented
    76. 76. Within CPG document (ENT IDSA ABRS CPG)
    77. 77. Mentioned but not documented
    78. 78. KSUMC CPGs: Adaptation Process Methodology
    79. 79. Separate Methodology document (SIGN 50: A guideline developer’s handbook -2011)
    80. 80. NICE: The guideline manual (2012)
    81. 81. Domain 4
    82. 82. DOMAIN 4. CLARITY OF PRESENTATION 15. Rs are specific Rs & executive summary 16. Options for management + discussion, Treatment (options/alternatives) 17. Recommendation identifiable Key Rs; separate (e.g. QRG)
    83. 83. Domain 5
    84. 84. DOMAIN 5. APPLICABILITY 18. Facilitators & barriers CPG dissemination/ implementation, barriers, CPG utilization & Quality indicators 19. Tools + tools, resources, appendices 20. Resource implications Method, cost utility, cost effectiveness, acquisition costs & implications for budgets 21. Monitoring /audit criteria Rs, Quality indicators & audit criteria
    85. 85. Domain 6
    86. 86. DOMAIN 6. EDITORIAL INDEPENDENCE 22. Funding body Disclaimer & funding source 23. Competing interests Methods, Conflicts of interest (COI), CPG panel & appendices
    87. 87. Overall Assessment
    88. 88. OVERALL CPG ASSESSMENT 1. Rate the overall quality of this CPG 2. I would recommend this CPG for use 3. Notes YES Yes ,with modification No
    89. 89. Selected CPG Domains Overall Is CPG Recommended for use ?1 2 3 4 5 6 CPG 1 CPG 2 CPG 3 CPG 4 CPG 5 Finally AGREE II Domain Scores Color Coding (Prof. Lubna Al-Ansary) < 40 % Red >41-70 Yellow >71 Green
    90. 90. AGREE Domains Scores Table CPG 1: SIGN CPG 2: EPR3 CPG 3: GINA CPG 4: ICSI CPG 5: Singapore MOH Domain 1 60 % 74 % 45 % 61 % 42 % Domain 2 55 % 56 % 60 % 58 % 63 % Domain 3 92 % 83 % 79 % 38 % 43 % Domain 4 95 % 90 % 92 % 75 % 87 % Domain 5 70 % 82 % 80 % 33 % 58 % Domain 6 80 % 60 % 22 % 50 % 10 % Overall assessment 7 6 5 3 4 Recommende d for use Yes Yes (w M) Yes (w M) No No
    91. 91. Graphical Representation
    92. 92. AGREE II DOMAINS (ENT-ABRS CPG) IDSA CPG 2012 Canadian CPG 2011 D1: Scope & Purpose 86 % 80 % D2: Stakeholder Involvement 86 % 69 % D3: Rigour of Development 90 % 85 % D4: Clarity & Presentation 92 % 90 % D5: Applicability 67 % 64 % D6: Editorial Independence 96 % 76 % Overall assessment 7 5 Recommended for use Yes No This table uses the AGREE II Domain Score Colour Coding proposed by Dr. Lubna Alansary (< 40% red - > 41 – 70% yellow - > 71 % green)
    93. 93. AGREE II DOMAINS CPG1 (Singapore) CPG2 (WFSBP) CPG3 (Canada) D1: Scope & Purpose 90.7% 90.7% 96.3 % D2: Stakeholder Involvement 72.2% 65 % 81.5 % D3: Rigour of Development 39. 5% 84.5 % 90.3 % D4: Clarity & Presentation 98.1% 88.9 % 87.03 % D5: Applicability 34.7% 23.6% 40.3 % D6: Editorial Independence ZERO % 86.1% 55.6 % Overall assessment 4 5 6 Recommended for use NO NO YES This table uses the AGREE II Domain score colour coding proposed by Dr Lubna Al-Ansary (< 40% Red - >41-70% Yellow - >71% Green)
    94. 94. The paper tool (handout!)
    95. 95. Using the AGREE II Instrument 101
    96. 96. AGREE II rater concordance calculator (EXCEL)
    97. 97. AGREE II rater concordance calculator (McMaster University)
    98. 98. AGREE Enterprise website
    99. 99. The online tool “My AGREE PLUS”
    100. 100. My AGREE PLUS (OR Our AGREE!)
    101. 101. My AGREE II Report PDF
    102. 102. A tool for evaluation of clinical validity/ credibility of CPGs (in progress)
    103. 103. AGREE-REX: Recommendation EXcellence: Innovations to enhance the capacity of practice guidelines to improve health and health care systems Develop a useful, reliable, and valid knowledge resource to complement the AGREE II, which will guide the development, reporting, and evaluation of the clinical credibility of CPG recommendations.
    104. 104. Acknowledgments
    105. 105. Questions?
    106. 106. EVALUATION !
    107. 107. Dr. Yasser Sami Amer MSc Pediatrics, MSc HC Informatics, CPHQ EBCPGs Advisor & Trainer
    108. 108. AGREE II Workshop: Group photo: August 20th, 2014 King Khalid University Hospital, King Saud University, Riyadh, Saudi Arabia
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.