The document discusses the development of a critical thinking self-reflection tool for nursing students. Faculty were dissatisfied with quantitative measures and wanted a qualitative tool to evaluate critical thinking. They created a tool consisting of statements aligned with components of critical thinking like interpretation, analysis, and evaluation. The tool guides students to reflect on clinical situations and think critically. It was implemented across nursing courses and found to reliably measure changes in critical thinking over time.

1. A Qualitative Tool for Critical
Thinking Skill Development
Janice Surina Cise, DNS, RN
Connie S. Wilson, EdD, RN
Martha J. Thie, EdD, RN
Quantitative measures were not meeting the needs of faculty for
evaluating critical thinking among Baccalaureate of Science
Nursing students.The authors discuss the critical thinking self-
reflection tool that was developed for teaching critical thinking and
qualitatively evaluating changes over time with established
interrater reliability and content validity.The tool is consistent with
the concepts of critical thinking in the curriculum and was
implemented systematically in all clinical nursing courses.
The widening responsibilities of
nurses, coupled with multifaceted
client health problems in increasingly
complex environments, demand that
nursing students, upon graduation,
have the requisite skills to think criti-
cally and make independent deci-
sions. Faculty are charged with facili-
tating critical thinking skills in their
students, providing feedback to the
students concerning their progress,
and evaluating the outcomes of stu-
dents’ educational efforts.
While critical thinking is a highly
valued educational outcome, there is
no agreed upon meaning; each school
adopts or creates its own definition.1
Consequently, many definitions exist.
Nurse educators often believe critical
thinking is related to competent nurs-
ing practice. Likewise, critical thinking
has been closely associated or even
considered synonymous with the
nursing process, clinical judgment,
and problem solving.2 Quite naturally,
terminology associated with these ac-
tivities has been applied to definitions
of critical thinking. In addition, many
nurse educators believe nursing stu-
dents develop critical thinking skills
as they progress through the program.
However, research does not support
the premise that a professional pro-
gram of study significantly increases
critical thinking.3
Accreditation commissions have
given little direction for defining or
measuring critical thinking as an edu-
cational outcome. The Commission
on Collegiate Nursing Education
(CCNE) does not prescribe specific
criteria, but [expects] “the nursing pro-
gram will evaluate the extent to which
any major component of their curricu-
NURSE EDUCATOR
Volume 29, Number 4 pp 147-151
© 2004; Lippincott Williams & Wilkins, Inc.
lum is achieved.”4 (p66) The National
League for Nursing Accreditation
Commission (NLNAC) has, until re-
cently, specified critical thinking as an
outcome in program effectiveness.5,6
The NLNAC required a critical think-
ing definition, its measurement, track-
ing of trended and aggregated data,
and the use of data in program deci-
sions.4,7-9 The selection of measurable
outcomes congruent with the curricu-
lum resides with the program faculty.
At the University of Indianapolis,
a critical thinking assessment team
was formed to identify how critical
thinking could be defined, facilitated,
and measured in the BSN program.
The faculty struggled with identifying
a definition of critical thinking that re-
flected the outcomes of the program
and provided a practical guide to
teaching and evaluating these skills
among BSN students.
After review of many definitions
and much deliberation, the faculty se-
lected the definition of critical think-
ing derived from the Delphi Study of
the American Philosophical Associa-
tion.10 Critical thinking is defined as a
“purposeful, self-regulatory judgment
which results in interpretation, analy-
sis, evaluation, and inference as well
as the explanation of the rationale
upon which the judgment is based.
Critical thinking is reasonable, ratio-
nal, reflective, autonomous thinking
that inspires an attitude of in-
quiry.”10(p3) The definition was chosen
because it was research-based and it
defined specific cognitive skills that
were measurable. Two instruments
that were derived from the Delphi Re-
port are the California Critical Think-
ing Skills Test (CCTST)11 and the Cali-
fornia Critical Thinking Dispositions
Inventory (CCTDI).12
The assessment team imple-
mented the CCTST and the CCTDI
quantitative measurements in 1997 as
a pretest during the student’s first clin-
ical nursing course (200 level) and
again as a posttest in the last semester
of the 400 level. The results were dis-
appointing. Aggregated data showed
no consistent improvement of posttest
scores when compared to pretest
scores. Several posttest scores actually
declined.
The assessment team consulted
the literature to better understand the
findings. Quantitative instruments such
as CCTST, CCTDI, the Arnett Critical
Thinking Test, and the Watson Glaser
Critical Thinking Test are considered
inconsistent and inconclusive at mea-
suring critical thinking in nursing stu-
dents.2,4,13 Furthermore, Stone et al4
concluded that the CCTDI does not re-
NURSE EDUCATOR Volume 29, Number 4 July/August 2004 147
Authors’ Affiliation: Associate Profes-
sors, University of Indianapolis, School of
Nursing, Indianapolis, Ind.
Corresponding Author: Dr Cise, Univer-
sity of Indianapolis, School of Nursing,
1400 E Hanna Ave, Indianapolis, IN 46227
(cise@uindy.edu).

2. late to critical thinking skills at all. The
CCTST was found to have low correla-
tion to course grades for courses with
high level problem solving.4 A higher
correlation was found with the CCTST
and the SAT general aptitude test than
nursing school grades.4 Thus, the as-
sessment team determined that the
quantitative measurements were not
meeting the need for evaluating the
change in critical thinking among the
BSN students. Frustration over the in-
consistencies in quantitative assess-
ment, lack of ability to see reliable
pretest and posttest improvement, and
inability to make curricular decisions
based on these assessment results has
steered the faculty toward qualitative
assessment methods.
Qualitative Approaches
Limitations with standardized instru-
ments have led to a recent focus on
qualitative measurements. Measures
to assess critical thinking qualitatively
include graduate and employee sur-
veys, portfolios, and performance ob-
servation, but none are specific instru-
ments for critical thinking.14 However,
the literature suggests content and
process-specific instruments that
make critical thinking explicit in the
curriculum may be a better measure
of critical thinking.4
Measurement instruments to qual-
itatively document critical thinking in-
clude narrative and reflective written
assignments.15-17 Finally, the literature
has clearly demonstrated that critical
thinking is fostered through self-re-
flection.18,19
Jasper studied nurses’ percep-
tions and concluded that reflective
writing “appears to provide a mecha-
nism through which nurses can de-
velop analytical and critical skills,
which then imparts on the conduct of
professional practice.”19(p461) In a qual-
itative study, Kok and Chabeli20 found
that journal writing promotes reflec-
tive thinking in clinical nursing
education. However, it must be ac-
knowledged that developing these in-
struments or assignments are time
consuming and may involve faculty
development along with additional
student preparation and instruction.
Instrument terminology would need
clarity in definition while validity and
reliability would need to be assessed.
Tool Development
Nursing faculty at the University of In-
dianapolis have long been interested
in creating learning strategies for the
development of critical thinking skills
in students. The assessment team set
out to identify a process to help stu-
dents learn to use critical thinking, es-
pecially in clinical situations that held
significance for the students.
Some BSN faculty routinely used
self-reflection to encourage students
to evaluate their clinical practice. For
example, following each patient care
day students were asked to reflect
on the following statements: “What
went well?” “If you could do it over
again, what would you do differ-
ently?” “What are your plans for im-
provement that will help you be
more successful in the future?” “What
help do you need to meet your
goals?”
While it fostered self-reflection, it
was clear that the assignment did not
meet faculty’s expectations as an in-
strument for developing critical think-
ing skills, nor was it a way to measure
critical thinking qualitatively. The as-
sessment team wanted an instrument
that would assist nursing students to
become more aware of their nursing
practice, develop critical thinking
skills, and be able to document the
process of their thinking.
The team searched the literature
and networked with other nurse edu-
cators. Paul and Elder,21 as well as
Thornhill and Wafer,22 offered practi-
cal suggestions. Paul and Elder’s
Guide On Improving Student Learn-
ing offers 30 ideas to foster critical
thinking. The team adopted the ideas
to “encourage students to think–quite
explicitly about their thinking”21(p4)
and “encourage students to think of
content as a form of thinking.”21(p4)
Thornhill and Wafer22 describe a sim-
ilar endeavor in which 6 steps of the
Brookfield Model23 were used:
1. Identify a critical incident
2. Note personal extent and con-
sequences
3. Identify and challenge as-
sumptions
4. Change importance of context
5. Imagine and explore alterna-
tives
6. Think about long-term out-
comes.
Using the Brookfield Model, Thornhill
and Wafer instructed their nursing stu-
dents to critically analyze a clinical sit-
uation.
A “meta-cognition journal exer-
cise” developed by Susan Fogarty,24 of
Ferris State University, contained a se-
ries of questions that provided struc-
ture to help nursing students think
through a clinical situation. This exer-
cise then became the basis for the
Critical Thinking Self-Reflection Tool
(Figure 1). The tool was piloted in a
300 Level medical/surgical course in
Spring 1999.
Tool Construction
The assessment team enthusiastically
supported the pilot project but ques-
tioned whether Fogarty’s statements
compared favorably with Facione’s
definition of critical thinking that had
been adopted by the BSN faculty.
The assessment team speculated that
if an exercise could be developed to
reflect the adopted critical thinking
definition, then there would be
greater assurance that the students’
could be guided to think more criti-
cally and that this could be reliably
measured.
The next statements in the Criti-
cal Thinking Self-Reflection Tool were
compared with the Facione’s defini-
tion of critical thinking (Table 1). Fa-
cione identified 6 competencies of
critical thinking: interpretation, analy-
sis, evaluation, inference, explanation,
and self-regulation.11 Each of the com-
petencies has 1 or more characteristics
(see Table 1, left column).
Finally, the assessment team com-
pared Facione’s competencies and
characteristics with the statements in
the Critical Thinking Self-Reflection
Tool. Table 1 (right column) contains
the self-reflective statements of the
new critical thinking tool. Statements
in the tool compared favorably to 1 or
more of Facione’s characteristics (in
bold type).
Validity and Reliability
The Critical Thinking Self-Reflection
Tool is congruent with the definition of
critical thinking used throughout the
BSN curriculum. Comparing the self-
reflection instrument statements with
Facione’s competency characteristics
148 NURSE EDUCATOR Volume 29, Number 4 July/August 2004

3. provided the team with the validity for
a sound qualitative instrument.
Reliability was determined by an-
swering the question: “Do clinical in-
structors evaluate students self-reflec-
tion papers similarly?” Members of the
assessment team established interrater
reliability of the instrument at 79%
with the use of independent review
and rating of the same set of student
reflections.
The self-reflection tool under-
went further revision and scrutiny by
the assessment team and the BSN fac-
ulty. The instrument as represented in
Figure 1 has been further modified for
this publication. The actual instrument
is 2 pages in length to allow for stu-
dents’ written responses. The custom-
ary school and course heading, a
place for the student’s name, and date
have been removed.
Implementation
The Critical Thinking Self-Reflection
Tool is introduced in the first 200 level
clinical course of the nursing pro-
gram. Students are initiated into criti-
cal thinking terminology and process.
Each item of the self-reflection tool is
presented, explained, and an example
is given. Students complete their first
written self-reflection in class follow-
NURSE EDUCATOR Volume 29, Number 4 July/August 2004 149
Table 1. Comparison of Self-reflection Questions With Critical Thinking Definition
Competencies
Facione Competencies Critical Thinking Self-reflection Questions*
INTERPRETATION Describe, factually, a significant event. Describe your point of
1. Identifies verbal and written data and nonverbal view. What other points of view should be considered?
cues. Recognizes client problems and strengths.
2. Explains the purpose, theme, or point of view of
written material.
3. Considers others‘ points of view.
4. Distinguishes own interpretation from actual data.
ANALYSIS Describe your thoughts and feelings about the event.
1. Identifies the reasons/opinions/arguments pro and con.
2. Examines variables and data for relationships.
3. Differentiates fact from opinion.
4. Analyzes implications of alternative decisions.
EVALUATION Identify your assumptions and describe your point of view
1. Reflects and analyzes the reasons/arguments. about this event.
2. Judges the credibility of sources of information.
3. Evaluates rationale to support conclusions.
4. Judges the data, information, and arguments using
appropriate criteria.
5. Appraises the value of data information.
INFERENCE What is the purpose of reflecting on this? What is the purpose
Identify the elements necessary to draw reasonable of reflecting on this? What question is at issue?
conclusions. Where did your assumptions come from? Were they
valid? What information was available to you to help
you understand this event? What other information might
you need?
EXPLANATION What concepts or theory were useful as you analyzed your
1. Describe the reasoning process followed in reaching thinking about this event?
the conclusions.
2. Justify one’s reasoning and conclusions in terms
of evidence.
3. Construct graphic representation of relationships
among variables.
SELF-REGULATION What are the implications/consequences of your thinking
1. Continually monitor, reflect on, and question about event? What conclusions you can draw? What implica-
one’s own thinking in relation to all tions for your professional future?
the foregoing steps in the reasoning process.
2. Reconsider interpretations or judgments based
on further analysis of facts or added information.
3. Examine own view for bias or self-interest.
*See Figure 1.

4. ing reflection on a situation that is im-
portant to them. Upon completion of
the class activity, the instructors pro-
vide feedback to the students and re-
visions are made as needed. Subse-
quent to didactic explanation, the
instrument is implemented a second
time in the clinical setting. The stu-
dent reflects on a clinical situation or
an event that he or she considers sig-
nificant.
The Critical Thinking Self-Reflec-
tion Tool was implemented across the
BSN program in the Spring of 2002.
Students perform written self-reflec-
tion twice per semester following a
clinical situation or incident identified
by the student as significant.
One student reflected upon a
new awareness that occurred during a
clinical rotation in a mental health fa-
cility. The student interviewed 2 pa-
tients with dissociative identity disor-
der (DID). She described her feelings
as, “I see people with so much poten-
tial, but they have a problem that is
keeping them from this.” The assump-
tions that the student identified in-
cluded the media portrayal of patients
in a mental health facility as “crazy”
and “lower functioning people” and
that this condition does not affect
“well-educated people.” Next, the stu-
dent explored the basis of the as-
sumptions as coming from the media
portrayal and no prior personal con-
tact with this condition. With further
exploration on the condition and the
interviews of the 2 patients, she con-
cluded that her assumptions and point
of view were faulty as she achieved a
new awareness, “I came out [of this
experience] with a new way of view-
ing these patients...and it can happen
to anyone.”
The Critical Thinking Self-Reflec-
tion Tool is used as both a teaching
and evaluation instrument with in-
creasing expectations as the student
advances through the program. To
allow for the development of critical
thinking skills, expectations for stu-
dents were leveled. The tool contains
11 items. The 200 and 300 level stu-
dents are expected to address 7 of the
11 items. The required 7 items are
specified in the assignment.
A simple grading rubric was cre-
ated by the assessment team. If the
student addresses the item appropri-
ately, a “+” was entered in the left
margin. If the student did not address
the item appropriately, a “0” was as-
signed and suggestions for improve-
ment were provided in writing. A sat-
isfactory for the 200 and 300 level
students is 7 of the 7 required items
addressed appropriately.
The 400 level students complete
all 11 of the items on the tool and 9 of
11 must be addressed appropriately
for a satisfactory grade. If a satisfac-
tory evaluation is not achieved, the
student receives feedback from the in-
structor and the assignment is revised
until a satisfactory is attained. In this
way, formative evaluation guides the
students to think more critically and
make the process of their thinking vis-
ible to themselves and the instructor.
The scores from the self-reflections
become part of each student’s written
clinical evaluation for the course. The
self-reflections then become a part of
the student’s portfolio that is reviewed
each semester.
Evaluation
Course evaluations indicated that the
critical thinking instrument was valu-
able to students and faculty. In addi-
tion, the self-refection instrument pro-
vided the instructor an opportunity to
give feedback to students as a way of
fostering critical thinking skills. A
comparison of the first semester score
in the baccalaureate 200 level and 400
level have shown an increase in the
implementation of critical thinking
skills in clinical situations at the 400
level. The results indicated that 66% of
the 200 level students (N = 37) suc-
cessfully completed the 7 of 11 items
on the first attempt and 100% on
the second attempt. In the 400 level
(N = 19), 100% completed 9 of 11
items successfully on the exercise in 2
concurrent courses. Because data
have been collected on the refined
version of the refection instrument for
only 1 semester, it must be interpreted
with caution.
150 NURSE EDUCATOR Volume 29, Number 4 July/August 2004
Critical Thinking Self-reflection
Critical thinking appraisal assists the student to enhance awareness,
practice, and documentation of critical thinking skills. Students will
complete each of the elements below. A satisfactory grade is achieved
when 9 of the 11 elements are evaluated as satisfactory. If a satisfactory
grade is not achieved, the student will remediate with his or her instructor
and the assignment will be revised until a satisfactory grade is achieved.
1. Describe, factually, a significant event that occurred in the practicum
setting (such as an ethical dilemma, value conflict, a difficult situation or
a new awareness).
2. Describe your thoughts and feelings about the event.
3. Identify your assumptions and describe your point of view about the
event.
4. Analyze your thinking about 2 and 3 above. Discuss each of the
following elements.
a. What is the purpose of your reflecting about the event?
b. What is the question at issue here?
c. What other points of view should be considered?
d. Where did your assumptions come from? Were they valid?
e. What information was available to you to help you understand the
event? What other information might you need?
5. Discuss the thinking process about the event.
a. What concepts or theory were useful as you analyzed your
thinking about the event?
b. What are the implications and/or consequences of your thinking
about the event?
6. Discuss what conclusions you can draw from this analysis and what
implications you can make for your professional future.
Figure 1. Self-reflection tool.

5. Students evaluated the self-reflec-
tion exercise at the end of the pro-
gram and most indicated that it was
beneficial, but required additional
time beyond their usual clinical pa-
perwork. Comments from students in-
cluded, “It gave me the opportunity to
look at an event from several points
of view, analyze the situation and gain
a better understanding;” “It helps me
to be able to think back on certain
events to see what I have learned;”
and “It helps me to reflect on the sit-
uation and provides a sense of closure
to a bad event or celebration for a
good event.” Other comments by the
students suggested that the exercise
revived “bad feelings” that were asso-
ciated with the event; a few students
remarked that they had difficulty iden-
tifying a significant clinical event on
which to reflect.
Similarly, BSN faculty were in-
vited to evaluate the process. At least
1 faculty member from each course
responded to the survey. All faculty
who responded stated the critical
thinking exercise was valued and
should be retained. One faculty mem-
ber noticed “a real change from 200
level to 400 level” with regard to stu-
dents “question[ing] their assump-
tions” and identifying “alternative
ways of looking at an event.” Other
comments indicated that the student
critical thinking self-reflections, “pro-
vided structure for depth of thinking,”
were “a good exercise to get them to
think” and helped them to obtain
“some insight into their thinking
processes.” The major disadvantage
noted by all the responders was the
increased faculty workload to read
and score the assignment. While the
faculty were unanimous in their rec-
ommendation to retain the practice,
they recommended a reduction in the
frequency of the assignment or the
number of items in the reflection.
Conclusions
The Critical Thinking Self-Reflection
Tool overcomes several limitations of
quantitative measurement by its con-
cept specificity, applicability to clini-
cal nursing situations, and use as a de-
velopmental tool throughout the pro-
gram. The development of a
qualitative instrument that is consis-
tent with the concepts of critical
thinking in the curriculum and imple-
mented systematically throughout the
program benefits both nursing stu-
dents and faculty to demonstrate stu-
dent development of these skills as
well as educational effectiveness.
References
1. Daly WM. The development of an al-
ternative method in the assessment of
critical thinking as an outcome of
nursing education. J Advan Nurs.
2001;36(1):120-129.
2. McCarthy P, Schuster P, Zehr P, et al.
Evaluation of critical thinking in a bac-
calaureate nursing program. J Nurs
Educ. 1999;38(3):142-144.
3. Brock A, Butts JB. On target: A model
to teach baccalaureate nursing stu-
dents to apply critical thinking. Nurs
Forum. 1998;33(3):5-10.
4. Stone CA, Davidson LJ, Evans JL, et al.
Validity evidence for using a general
critical thinking test to measure nurs-
ing students’ critical thinking. Holistic
Nurs Pract. 2001;15(4):65-74.
5. National League for Nursing Accredit-
ing Commission’s Accreditation Man-
ual for Post Secondary and Higher De-
gree Programs in Nursing. New York,
NY: NLNAC; 2000.
6. National League for Nursing Accredit-
ing Commission’s Accreditation Man-
ual and Interpretive Guidelines by
Program Type. NLNAC Publication:
New York; 2002.
7. Adams MH, Whitlow JF, Stover LM, et
al. Critical thinking as an educational
outcome: An evaluation of current in-
struments of measurement. Nurs Educ.
1996;21(3):23-32.
8. Brown HN, Sorrell JM. Connecting
across the miles: Interdisciplinary col-
laboration in the evaluation of critical
thinking. Nurs Connect. 1999;12(2):43-
47.
9. O’Sullivan PS, Blevins-Stephens WL,
Smith FM, et al. Addressing the na-
tional league for nursing critical think-
ing outcome. Nurs Educ. 1997;22(1):
23-29.
10. American Philosophical Association.
Critical Thinking: A Statement of Ex-
pert Consensus for Purposes of Educa-
tional Assessment and Instruction.
Millbrae, Calif: California Academic
Press; 1990.
11. Facione PA, Facione NC. The Califor-
nia Critical Thinking Skills Test: Test
Manual. Millbrae, Calif: California
Academic Press; 1992.
12. Facione PA, Facione NC, Sanchez CA.
The California Critical Thinking Dis-
positions Inventory: Test manual. Mill-
brae, Ca: California Academic Press;
1992.
13. Watson G, Glaser E. Watson-Glaser
Critical Thinking Appraisal Annual.
New York, NY: Macmillan; 1980.
14. Angel BF, Duffy M, Belyea, M. An evi-
dence-based project for evaluating
strategies to improve knowledge ac-
quisition and critical-thinking perfor-
mance in nursing students. J Nurs
Educ. 2000;39(5):219-228.
15. Cooper NJ. The use of narrative in the
development of critical thinking. Nurse
Education Today. 2000;20(7): 513-518.
16. Niedringhaus LK. Using student writ-
ing assessments to assess critical think-
ing skills: A holistic approach. Holistic
Nurs Pract. 2001;15(3):9-17.
17. Sorrell JM, Brown HN, Silva MC, et al.
Use of writing portfolios for interdisci-
plinary assessment of critical thinking
outcomes of nursing students. Nurs
Forum. 1997;32(4):12-24.
18. Usher K, Francis D, Owens J. Reflec-
tive writing: A strategy to foster critical
inquiry in undergraduate nursing stu-
dents. Australian Journal of Advanced
Nursing. 1999;17(1):7-12.
19. Jasper MA. Nurses’ Perceptions of the
value of written reflection. Nurse Edu-
cation Today. 1999;19:452-463.
20. Kok J, Chabeli MM. Reflective journal
writing: How it promotes reflective
thinking in clinical nursing education:
A students’ perspective. Curationis: S
African J Nurs. 2002;25(3):35-42.
21. Paul E, Elder L. How to Improve Stu-
dent Learning. Dillon Beach, Calif:
Foundation for Critical Thinking; 2002.
22. Thornhill SK, Wafer MS. Clinical pro-
cess learn to improve critical thinking.
Nurs Connect. 1997;10(3):51-59.
23. Brookfield S. On impostership, cul-
tural suicide, and other dangers: How
nurses learn critical thinking. J Contin
Educ Nurs. 1993;5(24):197-205.
24. Fogarty S. Meta-Cognition Journal Exer-
cise. Ferris State University. Personal
communication. Used with permission.
NURSE EDUCATOR Volume 29, Number 4 July/August 2004 151