Evidence based practice_step_by_step__critical.26


Published on

Published in: Health & Medicine, Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Evidence based practice_step_by_step__critical.26

  1. 1. By Ellen Fineout-Overholt, PhD, RN, FNAP, FAAN, Bernadette Mazurek Melnyk, PhD, RN, CPNP/PMHNP, FNAP, FAAN, Susan B. Stillwell, DNP, RN, CNE, and Kathleen M. Williamson, PhD, RNCritical Appraisal of the Evidence: Part I An introduction to gathering, evaluating, and recording the evidence. This is the fifth article in a series from the Arizona State University College of Nursing and Health Innovation’s Center for the Advancement of Evidence -Based Practice. Evidence-based practice (EBP) is a problem-solving approach to the delivery of health care that integrates the best evidence from studies and patient care data with clinician expertise and patient preferences and values. When delivered in a context of caring and in a supportive organizational culture, the highest quality of care and best patient outcomes can be achieved. The purpose of this series is to give nurses the knowledge and skills they need to implement EBP consistently, one step at a time. Articles will appear every two months to allow you time to incorporate information as you work toward implementing EBP at your institution. Also, we’ve scheduled “Chat with the Authors” calls every few months to provide a direct line to the experts to help you resolve questions. Details about how to participate in the next call will be pub- lished with September’s Evidence-Based Practice, Step by Step.I n May’s evidence-based prac- database’s own indexing lan- library subscription or those tice (EBP) article, Rebecca R., guage, or controlled vocabulary, flagged as “free full text” by a our hypothetical staff nurse, matched the keywords or syn- database or journal’s Web site.and Carlos A., her hospital’s ex- onyms, those terms were also Others are available through in-pert EBP mentor, learned how to searched. At the end of the data- terlibrary loan, when anothersearch for the evidence to answer base searches, Rebecca and Car- hospital library shares its articlestheir clinical question (shown los chose to retain 18 of the 18 with Rebecca and Carlos’s hospi-here in PICOT format): “In hos­ studies found in PubMed; six of tal library.pitalized adults (P), how does a the 79 studies found in CINAHL; Carlos explains to Rebecca thatrapid response team (I) compared and the one study found in the the purpose of critical appraisalwith no rapid response team (C) Cochrane Database of System- isn’t solely to find the flaws in aaffect the number of cardiac ar­ atic Reviews, because they best study, but to determine its worthrests (O) and unplanned admis­ answered the clinical question. to practice. In this rapid criticalsions to the ICU (O) during a As a final step, at Lynne’s rec- appraisal (RCA), they will reviewthree­month period (T)?” With ommendation, Rebecca and Car- each study to determinethe help of Lynne Z., the hospi- los conducted a hand search of • its level of evidence.tal librarian, Rebecca and Car- the reference lists of each study • how well it was conducted.los searched three databases, they retained looking for any rele- • how useful it is to practice.PubMed, the Cumulative Index vant studies they hadn’t found in Once they determine whichof Nursing and Allied Health their original search; this process studies are “keepers,” RebeccaLiterature (CINAHL), and the is also called the ancestry method. and Carlos will move on to theCochrane Database of Systematic The hand search yielded one ad- final steps of critical appraisal:Reviews. They used keywords ditional study, for a total of 26. evaluation and synthesis (to befrom their clinical question, in- discussed in the next two install-cluding ICU, rapid response RAPID CRITICAL APPRAISAL ments of the series). These finalteam, cardiac arrest, and un­ The next time Rebecca and Car- steps will determine whetherplanned ICU admissions, as los meet, they discuss the next overall findings from the evi-well as the following synonyms: step in the EBP process—critically dence review can help cliniciansfailure to rescue, never events, appraising the 26 studies. They improve patient outcomes.medical emergency teams, rapid obtain copies of the studies by Rebecca is a bit apprehensiveresponse systems, and code printing those that are immedi- because it’s been a few years sinceblue. Whenever terms from a ately available as full text through she took a research class. Sheajn@wolterskluwer.com AJN ▼ July 2010 ▼ Vol. 110, No. 7 47
  2. 2. shares her anxiety with Chen M., new EBP team, Carlos provides and the Boston ­ niversity Medi- U a fellow staff nurse, who says R ­ ebecca and Chen with a glossary cal Center Alumni Medical Li- she never studied research in of terms so they can learn basic brary [http://medlib.bu.edu/ school but would like to learn; research terminology, such as sam­ bugms/content.cfm/content/ she asks if she can join Carlos ple, independent variable, and de­ ebmglossary.cfm#R].) and ­ ebecca’s EBP team. Chen’s R pendent variable. The glossary Determining the level of evi- spirit of inquiry encourages Re- also defines some of the study de- dence. The team begins to divide becca, and they talk about the signs the team is likely to come the 26 studies into categories ac- opportunity to learn that this across in doing their RCA, such cording to study design. To help project affords them. Together as systematic review, randomized in this, Carlos provides a list of they speak with the nurse man- controlled trial, and cohort, qual- several different study designs ager on their ­ edical–surgical m itative, and descriptive studies. (see Hierarchy of Evidence for unit, who agrees to let them use (For the definitions of these terms Intervention Studies). Rebecca, their allotted continuing educa- and others, see the glossaries pro- Carlos, and Chen work together tion time to work on this project, vided by the Center for the Ad- to determine each study’s design after they discuss their expecta- vancement of Evidence-Based by reviewing its abstract. They tions for the project and how its Practice at the Arizona State Uni- also create an “I don’t know” outcome may benefit the patients, versity College of Nursing and pile of studies that don’t appear the unit staff, and the hospital. Health Innovation [http://nursing to fit a specific design. When they Learning research terminol- andhealth.asu.edu/evidence-based- find studies that don’t actively ogy. At the first meeting of the practice/resources/glossary.htm] answer the clinical question but Hierarchy of Evidence for Intervention Studies Type of evidence Level of evidence Description Systematic review or I A synthesis of evidence from all relevant random­zed controlled trials. i meta-analysis Randomized con­ II An experiment in which subjects are randomized to a treatment group trolled trial or control group. Controlled trial with­ III An experiment in which subjects are nonrandomly assigned to a out randomization treatment group or control group. Case-control or IV Case-control study: a comparison of subjects with a condition (case) cohort study with those who don’t have the condition (control) to determine characteristics that might predict the condition. Cohort study: an observation of a group(s) (cohort[s]) to determine the development of an outcome(s) such as a disease. Systematic review of V A synthesis of evidence from qualitative or descrip­ive studies to t qualitative or descrip­ answer a clinical question. tive studies Qualitative or de- VI Qualitative study: gathers data on human behavior to understand why scriptive study and how decisions are made. Descriptive study: provides background informa­ion on the what, t where, and when of a topic of interest. Expert opinion or VII Authoritative opinion of expert committee. consensus Adapted with permission from Melnyk BM, Fineout-Overholt E, editors. Evidence-based practice in nursing and healthcare: a guide to best practice [forthcoming]. 2nd ed. Philadelphia: Wolters Kluwer Health/Lippincott Williams and Wilkins.48 AJN ▼ July 2010 ▼ Vol. 110, No. 7 ajnonline.com
  3. 3. Critical Appraisal Guide for Quantitative Studies 1. Why was the study done? • as there a clear explanation of the purpose of the study and, if so, what was it? W 2. What is the sample size? • ere there enough people in the study to establish that the findings did not occur by chance? W 3. Are the instruments of the major variables valid and reliable? • ow were variables defined? Were the instruments designed to measure a concept valid (did H they measure what the researchers said they measured)? Were they reliable (did they measure a concept the same way every time they were used)? 4. How were the data analyzed? • hat statistics were used to determine if the purpose of the study was achieved? W 5. Were there any untoward events during the study? • id people leave the study and, if so, was there something special about them? D 6. How do the results fit with previous research in the area? • id the researchers base their work on a thorough literature review? D 7. What does this research mean for clinical practice? • Is the study purpose an important clinical issue? Adapted with permission from Melnyk BM, Fineout-Overholt E, editors. Evidence-based practice in nursing and healthcare: a guide to best practice [forthcoming]. 2nd ed. Philadelphia: Wolters Kluwer Health/Lippincott Williams and Wilkins.may inform thinking, such as or a meta-analysis—is the most a ­ ppraisal process (to ­ ppear in ad­ escriptive research, expert opin- reliable and the best evidence to f ­ uture installments of this series).ions, or guidelines, they put them answer their clinical question. Creating a study evaluationaside. Carlos explains that they’ll Using a critical appraisal table. Carlos provides an onlinebe used later to support Rebecca’s guide. Carlos recommends that template for a table where Re-case for having a rapid response the team use a critical appraisal becca and Chen can put all theteam (RRT) in her hospital, sh­ checklist (see Critical Appraisal data they’ll need for the RCA.ould the evidence point in that Guide for Quantitative Studies) Here they’ll record each study’sdirection. to help evaluate the 15 studies. essential elements that answer the After the studies—including This checklist is relevant to all three questions and begin to ap-those in the “I don’t know” studies and contains questions praise the 15 studies. (To use thisgroup—are categorized, 15 of about the essential elements of template to create your own eval-the original 26 remain and will research (such as, pur­ ose of the p uation table, download the Eval­be ­ncluded in the RCA: three i study, sample size, and major uation Table Template at http://systematic reviews that include variables). links.lww.com/AJN/A10.)one meta-analysis (Level I evi- The questions in the critical ap­dence), one randomized con- praisal guide seem a little strange EXTRACTING THE DATAtrolled trial (Level II evidence), to Rebecca and Chen. As they re- Starting with level I evidencetwo cohort studies (Level IV evi- view the guide together, Carlos studies and moving down thedence), one retrospective pre- explains and clarifies each ques- hierarchy list, the EBP team takespost study with historic controls tion. He suggests that as they try each study and, one by one, finds(Level VI evidence), four preex- to figure out which are the essen- and enters its essential elementsperimental (pre-post) interven- tial elements of the studies, they into the first five columns oftion studies (no control group) focus on answering the first three the evaluation table (see Table(Level VI ­ vidence), and four EBP e questions: Why was the study 1; to see the entire table withimplementation projects (Level done? What is the sample size? all 15 studies, go to http://links.VI ­ vidence). Carlos reminds e Are the instruments of the major lww.com/AJN/A11). The teamRebecca and Chen that Level I variables valid and reliable? The discusses each element as theye­ vidence—a systematic review remaining questions will be ad- enter it, and tries to determine ifof randomized controlled trials dressed later on in the critical it meets the criteria of the critical ajn@wolterskluwer.com AJN ▼ July 2010 ▼ Vol. 110, No. 7 49
  4. 4. 50 Table 1. Evaluation Table, Phase I First Author (Year) Conceptual Design/Method Sample/Setting Major Variables Studied Measure- Data Findings Appraisal: Framework (and Their Definitions) ment Analysis Worth to Practice Chan PS, et al. None SR N = 18 studies IV: RRT Arch Intern Med Purpose: effect of RRT on DV1: HMR 2010;170(1):18-26. HMR and CR Setting: acute care hos- DV2: CR • Searched 5 databases pitals; 13 adult, 5 pedsAJN ▼ July 2010 from 1950-2008, and ▼ “grey literature” from Average no. beds: NR MD conferences • Included only studies Attrition: NR with a control group McGaughey J, et al. None SR (Cochrane review) N = 2 studies IV: RRTVol. 110, No. 7 Cochrane Database Purpose: effect of RRT DV1: HMR Syst Rev 2007;3: on HMR 24 adult hospitals CD005529. • Searched 6 databases from 1990-2006 Attrition: NR • Excluded all but 2 RCTs Winters BD, et al. None SR N = 8 studies IV: RRT Crit Care Med Purpose: effect of RRT on DV1: HMR 2007;35(5): HMR and CR Average no. beds: 500 DV2: CR 1238-43. • Searched 3 databases from 1990-2005 Attrition: NR • Included only studies with a control group Hillman K, et al. None RCT N = 23 hospitals IV: RRT protocol for HMR Note: Lancet 2005; Purpose: effect of RRT on Average no. beds: 340 6 months CR • Criteria for 365(9477): 2091-7. CR, HMR, and UICUA • Intervention group • 1 AP rates of activating (n = 12) • 1 ICU or ED RN UICUA RRT • Control group DV1: HMR (unexpected (n = 11) deaths, excluding DNRs) DV2: CR (excluding Setting: Australia DNRs) DV3: UICUA Attrition: none Shaded columns indicate where data will be entered in future installments of the series. AP = attending physician; CR = cardiopulmonary arrest or code rates; DNR = do not resuscitate; DV = dependent variable; ED = emergency department; HMR: hospital-wide mor- tality rates; ICU = intensive care unit; IV = independent variable; MD = medical doctor; NR = not reported; Peds = pediatric; RCT = randomized controlled trial; RN = registered nurse; RRT = rapid response team; SR = systematic review; UICUA = unplanned ICU admissions.ajnonline.com
  5. 5. appraisal guide. These elements— suggests they leave the column in. find­ngs, not to compare them isuch as purpose of the study, sam- He says they can further discuss with other like studies. Rebeccaple size, and major variables—are this point later on in the process realizes that she enjoys this kindtypical parts of a research report when they synthesize the studies’ of conversation, in which sheand should be presented in a pre­ findings. As Rebecca and Chen and Chen have a voice and candictable fashion in every study review each study, they enter its contribute to a deeper under-so that the reader understands citation in a separate reference list standing of how research impactswhat’s being reported. so that they won’t have to create practice. As Rebecca and Chen con- tinue to enter data into the table, they begin to see similarities and Usually the important information in a study differences across studies. They mention this to Carlos, who tells can be found in the abstract. them they’ve begun the process of synthesis! Both nurses are en- couraged by the fact that they’re learning this new skill. As the EBP team continues to this list at the end of the pro­­ cess. The MERIT trial is next in thereview the studies and fill in the The reference list will be shared stack of studies and it’s a goodevaluation table, they realize that with colleagues and placed at the trial to use to illustrate this phaseit’s taking about 10 to 15 minutes end of any RRT policy that re- of the RCA process. Set in Aus-per study to locate and enter the sults from this ­ ndeavor. e tralia, the MERIT trial1 examinedinformation. This may be because Carlos spends much of his whether the introduction of anwhen they look for a description time answering Rebecca’s and RRT (called a medical emergencyof the sample, for example, it’s Chen’s questions concerning how team or MET in the study) wouldimportant that they note how the to phrase the information they’re reduce the incidence of cardiacsample was obtained, how many entering in the table. He suggests arrest, unplanned admissions topatients are included, other char- that they keep it simple and con- the ICU, and death in the hospi-acteristics of the sample, as well sistent. For example, if a study tals studied. See Table 1 to followas any diagnoses or illnesses the indicated that it was implement- along as the EBP team finds andsample might have that could be ing an RRT and hoped to see a enters the trial data into the table.important to the study outcome. change in a certain outcome, the Design/Method. After RebeccaThey discuss with Carlos the like- nurses could enter “change in and Chen enter the citation infor-lihood that they’ll need a few ses- [the outcome] after RRT” as the mation and note the lack of a con­sions to enter all the data into the purpose of the study. For studies ceptual framework, they’re readytable. Carlos responds that the examining the effect of an RRT to fill in the “Design/Method”more studies they do, the less on an outcome, they could say as column. First they enter RCTtime it will take. He also says the purpose, “effect of RRT on for randomized controlled trial,that it takes less time to find the [the outcome].” Using the same which they find in both the studyinformation when study reports words to describe the same pur- title and introduction. But MERITare clearly written. He adds that pose, even though it may not have is called a “cluster-­ andomised rusually the important informa- been stated exactly that way in controlled trial,” and cluster is ation can be found in the abstract. the study, can help when they term they haven’t seen before. Rebecca and Chen ask if it compare studies later on. Carlos explains that it means thatwould be all right to take out Rebecca and Chen find it frus- hospitals, not individuals or pa-the “Conceptual Framework” trating that the study data are tients, were randomly assigned tocolumn, since none of the stud- not always presented in the same the RRT. He says that the likelyies they’re reviewing have con- way from study to study. They reason the researchers chose toceptual frameworks (which help ask Carlos why the authors or randomly assign hospitals is thatguide researchers as to how a journals wouldn’t present similar if they had randomly assignedstudy should proceed). Carlos information in a similar manner. i ­ndividual patients or units, oth-r­ eplies that it’s helpful to know Carlos explains that the purpose ers in the hospital might havethat a study has no framework of publishing these studies may heard about the RRT and poten-underpinning the research and have been to disseminate the tially influenced the outcome. ajn@wolterskluwer.com AJN ▼ July 2010 ▼ Vol. 110, No. 7 51
  6. 6. To randomly assign hospitals the RRTs were activated and pro- continue the work—as long as ( ­ instead of units or patients) to vided their protocol for calling the Carlos is there to help. the intervention and comparison RRTs. However, these elements In applying these principles groups is a cleaner research de- might be helpful to the EBP team for evaluating research studies sign. later on when they make decisions to your own search for the evi- dence to answer your PICOT question, ­ emember that this se- r ries can’t contain all the available Keep the data in the table consistent by using infor­ ation about research meth­ m od­ logy. Fortunately, there are o simple, inclusive terminology. many good resources available in books and online. For example, to find out more about sample size, which can affect the likeli- To keep the study purposes about implementing an RRT in hood that researchers’ results oc­ con­ istent among the studies in s their hospital. So that they can cur by chance (a random finding) the RCA, the EBP team uses inclu- come back to this information, rather than that the intervention sive terminology they developed they place it in the last column, brought about the expected out- after they noticed that different “Appraisal: Worth to Practice.” come, search the Web using terms trials had different ways of de- After reviewing the studies to that describe what you want to scribing the same objectives. Now make sure they’ve captured the know. If you type sample size they write that the purpose of the essential elements in the evalua- findings by chance in a search en- MERIT trial is to see if an RRT tion table, Rebecca and Chen still gine, you’ll find several Web sites can reduce CR, for cardiopulmo- feel unsure about whether the in- that can help you better under- nary arrest or code rates, HMR, formation is complete. Carlos stand this study essential. for hospital-wide mortality rates, r ­ eminds them that a system-wide Be sure to join the EBP team and UICUA for unplanned ICU practice change—such as the in the next installment of the se- admissions. They use those same change Rebecca is exploring, that ries, “Critical Appraisal of the terms consistently throughout the of implementing an RRT in her Evi­ ence: Part II,” when Rebecca d evaluation table. hospital—requires careful consid- and Chen will use the MERIT Sample/Setting. A total of 23 eration of the evidence and this is trial to illustrate the next steps hospitals in Australia with an only the first step. He cautions in the RCA process, complete average of 340 beds per hospi- them not to worry too much the rest of the evaluation table, tal is the study sample. Twelve about perfection and to put their and dig a little deeper into the hospitals had an RRT (the inter- efforts into understanding the studies in order to detect the vention group) and 11 hospitals i ­nformation in the studies. He re- “keepers.” ▼ didn’t (the control group). minds them that as they move on Major Variables Studied. The to the next steps in the critical Ellen Fineout-Overholt is clinical profes­ sor and director of the Center for the independent variable is the vari- appraisal process, and learn even Advancement of Evidence-Based Practice able that influences the outcome more about the studies and proj- at Arizona State University in Phoenix, (in this trial, it’s an RRT for six ects, they can refine any data in where Bernadette Mazurek Melnyk is dean and distinguished foundation months). The dependent vari­ the table. Rebecca and Chen feel professor of nursing, Susan B. Stillwell able is the outcome (in this case, uncomfortable with this uncer- is clinical associate professor and pro­ HMR, CR, and UICUA). In this tainty but decide to trust the pro- gram coordinator of the Nurse Educator Evidence-Based Practice Mentorship trial, the outcomes didn’t include cess. They continue extracting Program, and Kathleen M. Williamson do-not-resuscitate data. The RRT data and entering it into the table is associate director of the Center for the was made up of an attending phy­ even though they may not com- Advancement of Evidence-Based Practice. Contact author: Ellen Fineout-Overholt, sician and an ICU or ED nurse. pletely understand what they’re ellen.fineout-overholt@asu.edu. While the MERIT trial seems entering at present. They both to perfectly answer Rebecca’s r ­ ealize that this will be a learn- REFERENCE PICOT question, it contains ele- ing opportunity and, though the 1. Hillman K, et al. Introduction of ments that aren’t entirely relevant, le­ rning curve may be steep at a the medical emergency team (MET) system: a cluster-randomised con­ such as the fact that the research- times, they value the outcome of trolled trial. Lancet 2005;365(9477): ers collected information on how improving patient care enough to 2091-7.52 AJN ▼ July 2010 ▼ Vol. 110, No. 7 ajnonline.com