Chapter 018

1Copyright © 2013, 2009, 2005, 2001, 1997 by Saunders, an imprint of Elsevier Inc.
Chapter 18
Critical Appraisal of Nursing Studies

Critical Appraisal of Research
 An examination of all aspects of a research
report
 Systematic
 Unbiased
 Careful

(Cont’d)
 Purpose is to judge the study’s
 Strengths
 Weaknesses
 Meaning
 Significance

Evolution of Critical Appraisal of
Research in Nursing
 Originally intended to focus on limitations
 Followed both publication and oral presentation
 Rare after 1960s in nursing, but still in
existence in multidisciplinary publications
 Skills in critical appraisal now are introduced
at baccalaureate level of nursing education

Critical Appraisal of Research in
Nursing
 Critical appraisal of studies is essential
 In quantitative research, issues are credibility
and generalization
 Construct validity
 Internal validity
 Statistical conclusion validity
 External validity

Nursing (Cont’d)
 Important questions:
 Methodology: was research sound enough to
produce credible findings?
 Believability: do findings reflect reality?
 What are the study’s strengths?
 Nursing’s knowledge base: are findings consistent
with those of previous studies?

Nursing (Cont’d)
 Important questions (Cont’d) :
 Are there implications for
• Increasing nursing’s knowledge base?
• Generating theory (qualitative)?
• Application to practice?
 What are the limitations and flaws?
• How do they affect how the study is understood,
believed, applied?

When are Critical Appraisals of
Research Implemented in Nursing?
 As a nurse
 When reading journal articles (informal process)
 In undergraduate and graduate coursework
 As free-standing assignments
 In order to perform research synthesis
 As part of a thesis or dissertation—literature
review

When are Critical Appraisals of Research
Implemented in Nursing? (Cont’d)
 At research presentations
 After a study is presented
 In journals
 The commentary pages following new research
(some journals)
 Peer review
 For conferences (abstract review)
 For publication (manuscript review)
 For potential funding (proposal review)

Critical Appraisal of Studies by
Students
 Three levels
 Identifying elements or steps of study
 Determining study strengths and limitations
 Evaluating credibility and meaning of study
findings

Critical Appraisal of Research by
Nurses
 Performed in conjunction with reading
research articles in professional journals
(must be critical appraisal, not merely
acceptance of the printed word)
 Will enable practicing nurses to synthesize
most credible, significant, and appropriate
evidence for use in their practice

Critical Appraisal of Research by
Educators
 Part of curricular content update
 Role modeling for students
 Collaboration with other faculty in research
 Mentoring of students who are conducting
research (usually graduate students)

Critical Appraisal of Studies By
Researchers
 Self-critique related to research trajectory
 Critique of literature in one’s area, to update
one’s literature review

Presentations and Publications
 Useful to the author
 Generates ideas for subsequent research
 Commentaries following published papers
 Letters to the editor

Critical Appraisal of Abstracts for
Conference Presentations
 Performed by a committee
 Several criteria
 Is the presentation suitable for this conference?
 Is research still in progress, or has it been
completed?
 Are the study problem, purpose, methodology, and
results logical?

Critical Appraisal of Abstracts for
Conference Presentations (Cont’d)
 Several criteria (Cont’d)
 What are the study’s implications?
• For nursing’s knowledge base
• For theory development (qualitative)
• For practice
 Is the writing clear and concise?
 Are proper parts of the abstract present?

Articles For Publication
 Performed by nurse researchers and faculty
who have conducted research
 Is research well designed, and does it
contribute to the body of knowledge?
 Two or three reviewers; anonymous process

Proposals
 For approval of proposed research
 Academic
 Institutional
 To permit data collection in an institution
 For funding of proposed research
 Intramural (either university or hospital)
 Private
 Federal

NURSES’ Expertise in Critical
Appraisal
Educational Level Expected Level of Expertise in Critical Appraisal of
Research
Baccalaureate • Identify the steps of the quantitative research
process in a study.
• Identify the elements of a qualitative study.
Master's • Determine study strengths and weaknesses in
quantitative and qualitative studies.
• Evaluate the credibility and meaning of a study
and its contribution to nursing knowledge and
practice.
Doctorate or Post-
doctorate
• Synthesize multiple studies in systematic
reviews, meta-analyses, meta-syntheses, and
mixed-methods systematic reviews.

Critical Appraisal Process for
Quantitative Research
 Three steps: performed one at a time, or
simultaneously
 Step I: Identifying the steps of the research
process in studies
 Step II: Determining study strengths and
weaknesses
 Step III: Evaluating the credibility and meaning of
a study to nursing knowledge and practice

Step I: Identifying the Steps of
the Research Process in Studies
 Understanding terms and concepts in report
 Identifying study elements and grasping
nature, significance, and meaning of these
elements

Guidelines for Identifying the
Steps of the Research Process
 Review the abstract
 Read the entire study
 Review these questions
 Does title clearly identify the focus of the study by
including the major study variables and the
population?
 Does title indicate type of study conducted?

Guidelines for Identifying the Steps
of the Research Process (Cont’d)
 Review these questions (Cont’d)
 Was abstract clear?
 Was writing style of report clear and concise?
 Were parts of research report plainly identified?
 Were relevant terms defined?

Guidelines for Writing a Critical
Appraisal
 Introduction
 Describe qualifications of authors
 Discuss clarity of article title
 Discuss quality of abstract
 State the problem
 Significance of the problem
 Background of the problem
 Problem statement

Appraisal (Cont’d)
 State the purpose
 Examine the literature review
 Are relevant previous studies and theories
described?
 Are the references current?
 Are the studies described, critically appraised, and
synthesized?
 Is a summary provided of the current knowledge
(what is known and not known) about research
problem?

 Examine the study framework or theoretical
perspective
 Is the framework explicitly expressed or must
reviewer extract the framework from implicit
statements in introduction or literature review?
 Is the framework based on tentative, substantive,
or scientific theory?
 Does the framework identify, define, and describe
relationships among concepts of interest? Provide
examples.

 Examine the study framework or theoretical
perspective (Cont’d)
 Is a model of framework provided for clarity? If
model is not presented, develop one and describe
it
 Link study variables to relevant concepts in the
map
 How is framework related to nursing's body of
knowledge?

 List any research objectives, questions, or
hypotheses
 Identify and define (conceptually and
operationally) study variables or concepts
 Indicate the study’s variables:
 Independent variables
 Dependent variables
 Research variables or concepts

 Identify attribute or demographic variables
and other relevant terms
 Identify the research design
 Identify the specific design of the study. Draw a
model.
 Does the study include a treatment or
intervention? If so, is the treatment clearly
described with a protocol and consistently
implemented?

 Identify the research design (Cont’d)
 If study has more than one group, how were
subjects assigned to groups?
 Were the extraneous variables identified and
controlled for?
 Were pilot study findings used to design study? If
so, briefly discuss pilot and changes made in
study based on pilot.

 Describe sample and setting
 Identify inclusion or exclusion sample or eligibility
criteria
 Identify the specific type of probability or
nonprobability sampling method used to obtain the
sample. Did the researchers identify a sampling
frame for the study?

 Describe sample and setting (Cont’d)
 Identify the sample size. Discuss the refusal rate
and/or percentage, and include the rationale for
refusal if presented in the article. Discuss the
power analysis if this process was used to
determine sample size.
 Identify sample attrition (number and percentage)
for the study.

 Describe sample and setting (Cont’d)
 Identify characteristics of the sample.
 Discuss IRB approval. Describe the informed
consent process used in study.
 Identify the study setting and indicate if it is
appropriate for study purpose.

 Identify/describe each measurement strategy
used in study
 Identify each study variable that was measured
 Identify name and author of each measurement
strategy
 Identify type of each measurement strategy
 Identify validity of each scale and accuracy of
physiologic measures

 Identify/describe each measurement strategy
used in study (Cont’d)
 Identify level of measurement (nominal, ordinal,
interval, ratio) achieved by each measurement
method used in study.
 Describe reliability of each scale for previous
studies and this study. Identify the precision of
each physiological measure.

 Describe procedures for data collection
 Describe statistical techniques conducted to
analyze study data
 List statistical procedures conducted to describe
sample
 Was level of significance or alpha identified? If so,
indicate what it was (0.05, 0.01, 0.001, and so
forth)

 Describe the statistical techniques conducted
to analyze study data (Cont’d)
 Identify focus (description, relationships, or
differences) for each analysis technique
 List statistical analysis technique performed
 Provide specific results
 Identify probability (p) of statistical significance
achieved by the result

 Describe researcher’s interpretation of
findings
 Are findings related back to study framework? If
so, do findings support study framework?
 Which findings are consistent with those
expected?
 Which findings were not expected?
 Are the findings consistent with previous research
findings?

 What study limitations did researcher
identify?
 How did researcher generalize the findings?
 What were the implications of findings for
nursing practice?
 What suggestions for further study were
identified?
 Is description of the study sufficiently clear for
replication?

Step II: Determining Study
Strengths and Weaknesses
 Comparing study with ideal research report
 Formulate some idea of whether the
compared difference
 Is none or slight (a strength)
 Is substantial
 If substantial, is this poor writing or poor
research design?
 Are the logical links in the study
substandard? Do the parts match?

Guidelines for Determining study
Strengths and Weaknesses
 Read the following questions and make
judgments about elements or steps in study
 Provide a rationale for your decisions
 Document from relevant research sources

Research Problem and Purpose
 Is problem sufficiently delimited in scope that
it is researchable but not trivial?
 Is problem significant to nursing and clinical
practice?
 Does purpose narrow and clarify aim of
study?
 Was study feasible to conduct in terms of
funding; researchers’ expertise; availability of
subjects, facilities, and equipment; and
ethical considerations?

Review of Literature
 Is literature review organized to demonstrate
progressive development of evidence from
previous research?
 Is a theoretical knowledge base developed
for the problem and purpose?

Review of Literature (Cont’d)
 Is a clear, concise summary presented of
current empirical and theoretical knowledge
in area of the study?
 Does literature review summary identify what
is known and not known about research
problem and provide direction for formation of
the research purpose?

Study Framework
 Is framework presented with clarity? Is
model/conceptual map of framework
adequate to explain phenomenon of
concern?
 Is framework linked to the research purpose?
If not, would another framework fit more
logically?

Study Framework (Cont’d)
 Is framework related to body of knowledge in
nursing and clinical practice?
 Do framework elements really reflect study
variables?
 If a proposition or relationship from a theory is
to be tested, is the proposition clearly
identified and linked to study hypotheses?

Research Objectives, Questions,
or Hypotheses
 Are objectives, questions, or hypotheses
expressed clearly?
logically linked to the research purpose?

Research Objectives, Questions,
or Hypotheses (Cont’d)
 Are hypotheses stated to direct the conduct
of quasi-experimental and experimental
research?
logically linked to concepts and relationships
(propositions) in framework?

Variables
 Are variables reflective of concepts identified
in framework?
 Are variables clearly defined (conceptually
and operationally) and based on previous
research or theories?
 Is the conceptual definition of each variable
consistent with the operational definition?

Design
 Is design used in study the most appropriate
design to obtain needed data?
 Does design provide a means to examine all
objectives, questions, or hypotheses?

Design (Cont’d)
 Is the treatment clearly described? Is the
treatment appropriate for examining the study
purpose and hypotheses? Does study
framework explain the links between the
treatment (independent variable) and the
proposed outcomes (dependent variables)?

Design (Cont’d)
 Was a protocol developed to promote
consistent implementation of treatment to
ensure intervention fidelity? Did researcher
monitor implementation of treatment to
ensure consistency? If treatment was not
consistently implemented, what might be the
impact on the findings?

Design (Cont’d)
 Did researcher identify threats to design
validity and minimize them as much as
possible?
 Is design logically linked to sampling method
and statistical analyses?

Design (Cont’d)
 If more than one group is used, do groups
appear equivalent?
 If a treatment was implemented, were
subjects randomly assigned to treatment
group or were treatment and comparison
groups matched? Were the treatment and
comparison groups assignments appropriate
for purpose of the study?

Sample, Population, and Setting
 Is sampling method adequate to produce a
representative sample?
 What are the potential biases in the sampling
method? Are any subjects excluded from
study because of age, socioeconomic status,
or ethnicity without a sound rationale?
 Did sample include an understudied
population, such as young, elderly, or a
minority group?

(Cont’d)
 Were sampling criteria (inclusion and
exclusion) appropriate for type of study
conducted?
 Is sample size sufficient to avoid a type Ii
error? Was a power analysis conducted to
determine sample size? If a power analysis
was conducted, were the results of the
analysis clearly described and used to
determine final sample size? Was attrition
rate projected in determining final sample
size?

(Cont’d)
 Were the rights of human subjects protected?
If so, in what ways?
 Is setting typical of clinical settings?
 Was refusal rate a problem? If so, how might
this weakness influence findings?

 Was sample attrition a problem?
 Did researchers provide a rationale for
attrition of study participants?
 How did attrition influence final sample and
study results and findings?(Did a random
sample become a convenience sample due
to excessive attrition or refusal?)

Measurements
 Do measurement methods selected for study
adequately measure study variables?
 Are measurement methods sufficiently
sensitive to detect small differences between
subjects? Should additional measurement
methods have been used to improve quality
of study outcomes?

Measurements (Cont’d)
 Do the chosen measurement methods have
adequate validity and reliability? What
additional reliability or validity testing is
needed to improve quality of measurement
methods?

 Scales and questionnaires
 Are the instruments clearly described?
 Are techniques to complete and score the
instruments provided?
 Are validity and reliability of instruments
described?
 Did researcher reexamine validity and reliability of
instruments?
 If an instrument was developed for study, is
instrument development process described?

 Observation
 Is what is to be observed clearly identified and
defined?
 Is interrater reliability described?
 Are techniques for recording observations
described?

 Interviews
 Do interview questions address concerns
expressed in research problem?
 Are interview questions relevant for research
purpose and objectives, questions, hypotheses?
 Does design of questions tend to bias subjects’
responses?
 Does sequence of questions tend to bias subjects’
responses?

 Physiological measures
 Are physiological measures/instruments clearly
described? If appropriate, are brand names, of
instruments identified?
 Are accuracy, precision, and error of physiological
instruments discussed?

 Physiological measures (Cont’d)
 Are physiological measures appropriate for the
research purpose and objectives, questions, or
hypotheses?
 Are methods for recording data from physiological
measures clearly described? Is the data recording
consistent?

Data Collection
 Is data collection process clearly described?
 Are forms used to collect data organized to
facilitate computerizing the data?
 Is training of data collectors clearly described
and adequate?

Data Collection (Cont’d)
 Is data collection process conducted in a
consistent manner?
 Are data collection methods ethical?
 Do data collected address research
objectives, questions, or hypotheses?
 Did any adverse events occur during data
collection, and were these appropriately
managed?

Data Analysis
 Are data analysis procedures appropriate for
type of data collected and clearly described?
 Did researcher address missing data? How
was missing data managed?
 Do data analysis techniques address study
purpose? Research objectives, questions, or
hypotheses?
 Are results presented in an understandable
way by narrative, tables, or figures, or a
combination of methods?

Data Analysis (Cont’d)
 Are statistical analyses logically linked to
design?
 Was sample size sufficient to detect
significant differences if they had been
present?
 Was a power analysis conducted for non-
significant results?
 Are results interpreted appropriately?

Interpretation of Findings
 Are findings discussed in relation to each
objective, question, hypothesis?
 Are various explanations for significant and
non-significant findings examined?
 Are findings clinically significant?
 Are findings linked to study framework?

Interpretation of Findings (Cont’d)
 Are study findings an accurate reflection of
reality, and valid for use in clinical practice?
 Do conclusions fit results from data
analyses? Are conclusions based on
statistically significant and clinically important
results?
 Does study have limitations not identified by
researcher?

Interpretation of Findings (Cont’d)
 Did researcher generalize findings
appropriately?
 Were identified implications for practice
appropriate, based on study findings and
findings from previous research?
 Were quality suggestions made for future
research?

Step III: Evaluating a Study
 Determination of validity, credibility,
significance, and meaning of a study
 Accomplished by examining links
 To those within study process, relative to findings
 To those that depend on previous studies in same
area
 Did author take research in a direction
supported by previous research?

Step III: Evaluating a Study
(Cont’d)
 Are strengths of the study sufficient?
 How do weaknesses of study undermine
author’s discussion?
 Principal findings
 Conclusions
 Limitations
 Implications
 Suggestions for subsequent research

Guidelines for Evaluating a Study
 Questions to ask:
 Are there any other hypotheses that explain the
findings? Did author identify these?
 Do findings seem valid? Believable?
 To what populations can findings be generalized?
 What remains unclear? Did author identify these
points?

(Cont’d)
 Questions to ask (Cont’d) :
 What would have decreased/eliminated study
limitations?
 Based on this, what subsequent research is
indicated: what comes next?
 What is the state of the body of knowledge, now,
with these findings?

(Cont’d)
 Based on previous literature, did author use
previous work to
 Generate up-to-date problem statement?
 Derive a purpose that advances enquiry in this
area?
 Improve on previous studies’ designs?
 Select a sample that was more representative?
More inclusive?
 Select or devise better measurement strategies?

(Cont’d)
 Based on previous literature, did author use
previous work to
 Select appropriate statistical tests and apply them
correctly
 Analyze findings so that they build on those of
previous studies?
 Identify current knowledge in this area?
 State implications for practice, body of nursing
knowledge, or theory?

(Cont’d)
 Include a final summary of quality of the
research report
 Does study contribute to the body of knowledge,
and in what way?
 Current state of knowledge in this area: are
practice applications appropriate now?

Critical Appraisal Process:
Qualitative Studies
 Requires a different approach than steps
used for quantitative study
 Three prerequisites in applying rigorous
appraisal standards
 An appreciation for philosophical foundation of
qualitative research
 Basic knowledge of major qualitative approaches
 Empathy for participant’s perspective

Qualitative Studies (Cont’d)
 Problem Statement
 Is research problem explicit?
 Why was this study needed?
 Why did researcher care about this topic? Do we
know?

 Purpose and Research Questions
 Does purpose make sense, in light of the
problem?
 What are explicit or implied research questions?
 Are those questions related to problem and
purpose?
 Does qualitative inquiry make sense for these
questions?

 Literature Review
 Were both quantitative and qualitative studies
cited?
 Were other types of literature reviewed?
 Are references current?
 Does collective authorship of referenced articles
transcend nursing?
 Did author evaluate available literature?
 Does synthesized information from literature
review reveal a gap?

 Philosophical Foundation of the Study
 Method/submethod stated or implied?
 Is source for philosophical foundation primary or
secondary?
 Were methods of study congruent with this
philosophy and research tradition?

 Sampling and Sample
 How were study participants selected?
 Where were they recruited? Did sites fit sampling
needs of study?
 Inclusion and exclusion criteria
 What is the “n”?
 What were refusal rate and the attrition rate?

 Data Collection
 How did researchers collect data?
 More than one method of data collection?
 Any rationale for data collection choices?
 Span of time for data collection?
 One interview or more?
 Participant review of typed transcripts?
 What changes in data collection methods were
made in response to context and early information
obtained?

 Protection of Human Study Participants
 Informed consent
 Planned action, should the interviews reveal
disturbing content—counselor availability? Contact
numbers?
 Benefits and risks of participation ID’d by authors
 Any adjustments to recruitment, consent, data
collection, and analysis to prevent participant
distress?

 Data Management and Analysis
 How were data managed and analyzed?
 How was rigor of process maintained?
 Was data management/analysis method
consistent with purpose and data?
 Measures to minimize or allow for effects of
researcher bias, such as dual coding (not used in
all methods), expert review, participant validation,
bracketing

 Findings
 Did findings address purpose?
 Did researcher go Beyond mere reporting of
quotations and actually interpret?
 Were themes or essences identified?
 Data congruency with interpretations:
• Did quotations match themes/essences?
• Did researcher attempt to address variations in findings?

 Discussion
 Any new information produced?
 Findings discussed, in comparison with other
studies or literature?
 Implications of findings addressed:
• Body of nursing knowledge?
• Clinical application?
• Policy application?
• Theory generation?

 Logic and Form of Findings
 Were readers able to “hear the voice” of
participants and gain understanding of
phenomenon studied?
 Were elements of research report clearly
identifiable by reader?
 Were study findings, purpose, and method
congruent with article’s presentation?
 Was there logic, and organization, in the way
findings were presented?

 Evaluation Summary
 Philosophical congruence: Were development and
implementation of study congruent with
philosophical foundation of study?
 Methodological coherence: Did data collection,
analysis, and interpretation processes fit together
to form a coherent approach to address the
research problem?

 Evaluation Summary (Cont’d)
 Intuitive comprehension: Do findings provide a
credible reflection of reality and expand reader’s
comprehension of the study topic? If so, how can
findings be used in nursing practice?
 Intellectual contribution: What do findings
contribute to current body of knowledge?

Chapter 018

Recommended

Recommended

More Related Content

What's hot

What's hot (13)

Similar to Chapter 018

Similar to Chapter 018 (20)

More from stanbridge

More from stanbridge (20)

Chapter 018