1. THE USE OF COGNITIVE TASK ANALYSIS TO INVESTIGATE HOW MANY
EXPERTS MUST BE INTERVIEWED TO ACQUIRE THE CRITICAL
INFORMATION NEEDED TO PERFORM A CENTRAL VENOUS CATHETHER
PLACEMENT
by
Craig W. Bartholio
A Dissertation Presented to the
FACULTY OF THE USC ROSSIER SCHOOL OF EDUCATION
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF EDUCATION
December 2010
Copyright 2010 Craig W. Bartholio

2. ii
Dedication
There are many people that help support me during the entire three years of
completing this doctoral program and dissertation. I want to first thank my parents
for believing in me and supporting the whole process through multiple means. They
have always been a huge believer in my abilities even when I had my own doubts.
Secondly, I would like to thank my extended family for all their verbal
encouragement, baby sitting, inspiring phone conversations, editing, and all around
understanding of my multiple priorities and commitments.
To my three children, Chloe, Noah, and Hannah. Each of you has spent
several nights sleeping next to Daddy while he typed late into the night, and
sometimes into the next morning. I want to thank you for understanding all the times
I could not “play” with you because Daddy had to “study” or “write”. I have a lot of
playing activities to catch up on with each of you.
Lastly, I want to thank my wife for all her continued support. Without her
encouragement, frank realism, and editing abilities (far superior to mine), and the
ability to always understand the real perspective of importance in life, I share this
accomplishment with you.

3. iii
ACKNOWLEDGEMENTS
I would like to take this time to express my deepest gratitude, appreciation,
admiration, and thankfulness to the faculty on my dissertation committee: Dr.
Richard Clark, Dr. Kenneth Yates, and Dr. Maura Sullivan without whom I would
not have been able to accomplish this huge undertaking.
To Dr. Richard Clark, I am in continual awe of your knowledge and ability to
express the complex in simple understandable terms. I want to offer sincere
gratitude for all of the time, patience, and sharing of his knowledge that allowed me
to accomplish this feat. I am thirsty for knowledge in the area of cognitive task
analysis and look forward to continuing to add to the current body of knowledge in
this field.
To Dr. Kenneth Yates, first of all thank you for inspiring me to switch from
the K-12 Leadership concentration to the Educational Psychology concentration. It
has been a wonderful and thrilling ride. I appreciate all the significant amount of
time effort you have provided me in achieving this accomplishment. You have
provided many wonderful insights into what it takes to write a dissertation. I am
deeply indebted to you.
To Dr. Maura Sullivan, without you initial support and guidance, I’m not sure
I would be writing this acknowledgement. Thank you for your patience and
understanding of this dissertation process. The insights on organizing my writing,

4. iv
continual support and the sharing of your significant other to the good of the cause,
thank you.
Finally, to all my fellow cohort members Eko, Joon, Julia, Mary Ann, and
Lesile, and especially Patrick, thanks for your wonderful continual support.

5. v
TABLE OF CONTENTS
Dedication ii
Acknowledgements iii
List of Tables vii
List of Figures viii
Abstract x
Chapter 1 1
Statement of the Problem 1
Review of the Literature 5
Current Trends in Surgical Training 5
Development of Expertise 7
Expertise in Medicine 8
Expertise in Surgery 9
Knowledge Types 10
Declarative Knowledge 11
Procedural Knowledge 13
Automaticity 14
Automaticity and Expert Recall 16
Automaticity and Expert Recall in Medicine and
Surgery 18
Cognitive Task Analysis 20
Defining Cognitive Task Analysis 20
CTA Methodology 21
Effectiveness Studies using CTA 23
Effectiveness of CTA in Surgical Training 29
Limitations of CTA 35
Number of Experts Required for CTA 37
Summary 39
Purpose of the Study 40

6. vi
Chapter 2: Method 42
Design 42
Subjects 43
Data Collection 44
Semi-structured CTA Interviews 44
CTA Coding Scheme and Procedure 45
CTA Protocol and creating a six-Subject Matter “Gold Standard”
CVC Protocol 46
Data Analysis 47
Chapter 3: Results 52
Coding and Inter-rater reliability 52
Summary 74
Chapter 4: Conclusions 76
Research Questions 76
Summary 84
Limitations and Implications 85
Conclusion 87
References 89
Appendices
Appendix A: Non-Repeating Combinations of SME Protocols 98
Appendix B: CTA Coding Scheme 99
Appendix C: CTA Gold Standard for Central Venous Catheter 100

7. vii
LIST OF TABLES
Table 1: Excerpt from Central Venous Catheter Protocol Spreadsheet,
Single Expert 48
Table 2: Excerpt from Central Venous Catheter Placement Protocol
Spreadsheet: Multiple Experts 50
Table 3: CVC Gold Standard Sections and Corresponding Number
Of Items per Section 53
Table 4: Percentage of Knowledge Acquired from an Expert When
Compared to the Six-Expert CVC Gold Standard Protocol
(Experts A-F) 54
Table 5: The average percent of Total Knowledge, Action Steps,
Decision Steps per Number of Experts when compared to a
Six-SME Gold Standard Protocol 56
Table 6: Average Percentage Increase of Total Knowledge, Action Steps,
And Decision Steps Acquired from Multiple Groups of Expert
When compared to a Six-expert Gold Standard CVC Protocol 57
Table 7: Average Percentage of Objectives, Reasons, and Risks Acquired
From Multiple Groups of Experts When Compared to a Six
Subject Matter Expert Gold Standard Protocol 63
Table 8: Average Percentage of Indications, Contraindications, and
Standards Acquired from Multiple Groups of Experts When
Compared to a Six Subject Matter Expert Gold Standard Protocol 67
Table 9: Average Percentage of Equipment and Tasks acquired from
Multiple Groups of Experts when compared to a Six Subject
Matter Expert Gold Standard Protocol 72
Table 10: Quantity of Experts Recommended if a 10% Marginal
Utility in Knowledge Acquisition is Expected 75

8. viii
LIST OF FIGURES
Figure 1: Percentage of Acquired Knowledge Based on Six-Expert
Gold Standard CTA CVC Procedure Protocol as a Function
Of the Number of Experts 58
Figure 2: Average Percentage Increase of Knowledge as a Function
Of an Additional Expert for Total Possible Items in the
Gold Standard CVC Procedure Protocol 59
Figure 3: Average Percentage Increase in Action Step Acquired
Knowledge as a Function of the Number of Experts 60
Figure 4: Average Percentage Increase of Action Steps as a
Function of Additional Experts 60
Figure 5: Average Percentage Increase of Gold Standard
Decision Steps as a Function of the Number of Experts 61
Figure 6: Average Percentage Increase of Decision Steps as a
Function of Additional Experts 62
Figure 7: Percentage of Knowledge Acquired of Risks as a
Function of the Number of Experts 63
Figure 8: Average Increase of Acquired Knowledge of Risks as a
Function of Added Experts 65
Figure 9: Percentage of Knowledge Acquired for Reasons to Perform
The CVC Procedure Correctly 65
Figure 10: Average Percentage Increase in Acquired Knowledge in the
Area of Reasons Based on a Six-Expert Gold Standard
CVC protocol 66
Figure 11: Percentage of Indications to Perform the CVC Procedure
Based on a Six-Expert Gold Standard as a Function of the
Number of Experts 67

9. ix
Figure 12: The Average Percentage of Acquired Knowledge Gained as a
Function of Additional Experts in the Area of Indications to
Perform a CVC Procedure 68
Figure 13: Percentage of Knowledge Acquired in the Area of
Contraindications as a Function of the Number of Experts 69
Figure 14: The Average Percentage Increase in the Amount of
Acquired Knowledge in the Area of Contraindications
As a Function of the Number of Experts 70
Figure 15: Percentage of knowledge acquired for standards of CVC
Procedure as a function of the number of experts 71
Figure 16: Average Percentage Increase of Acquired Knowledge for
Standards Based on a Six-Expert Gold Standard CVC Protocol 71
Figure 17: The Average Percentage Gain of Acquired Knowledge of
Needed Equipment as a Function of Additional Experts 73
Figure 18: The Average Percentage Gain of Acquired Knowledge of
Needed Equipment as a Function of Additional Experts 74

10. x
ABSTRACT
The purpose of this study was to examine the amount of relevant information
experts provide and fail to provide when asked to describe how to perform a
complex task in enough detail for students to perform the task. In this study,
medical experts where interviewed because their past successes and failures at
the task are known and so it could be determined that all experts had succeeded
consistently at the task being described. Past research has suggested that because
experts have both conscious and unconscious automated knowledge they may not
“know what they know” and so not be able to completely describe how to make
critical decisions during task performance. A version of Cognitive Task Analysis
designed to support training was used to interview medical school faculty and
analyze their description of a controversial trauma procedure in order to
determine the average percentage of knowledge that was acquired from a single
expert and how much additional knowledge is acquired from each succeeding
expert interviewed. After analysis, it was determined that the amount of
knowledge acquired from one expert was about 63 percent and the average
increase of acquired knowledge with the second expert was about 16% and the
third expert added another 8 percent. Past studies have reported considerably
lower percentages of relevant information captured with Cognitive Task Analysis
– about 30 percent of decisions are typically reported for example. Thus it was

11. xi
hypothesized that information about the controversial procedure examined in
this study may have become more conscious to the experts interviewed because
they had been discussing it among themselves and reading accounts in journals.

12. 1
CHAPTER 1
STATEMENT OF THE PROBLEM
Surgical skills training has evolved in recent years from the traditional “see one-
do one-teach one” model (Halsted, 1904) to methods that focus more on instruction
by expert surgeons and various forms of technology, such as computer based
learning; virtual reality (VR); and high-fidelity bench models (Vozenilek, Huff,
Reznek, & Gordon, 2004). These technologies increase surgical residents exposure to
a wide array of procedures while practicing their proficiency of technical skills prior
to real surgical exposure (Aggarwal, Grantcharov, & Darzi, 2007; Reznick &
MacRae, 2006;Vozenilek et al., 2004).
The common thread for both the traditional apprenticeship and current surgical
skills training programs is that expert surgeons are often relied upon as the “master
teacher” to inform and the curriculum content, training procedures, and simulation
software. However, research indicates that experts can omit up to 70% of
information when explaining to others how to perform a task (Clark, Pugh, Yates,
Early, & Sullivan, 2008). Additionally, the transfer of knowledge from experts to
novices is not easily accomplished (Hinds, Patterson, & Pfeffer, 2001). This may
have serious consequences for surgical training in which surgical experts may not
provide a full account of the knowledge and skills required to perform a procedure
(Abernethy, Poolton, Masters, and Patil, 2008). Thus, surgical residents may be
provided incomplete knowledge about the behavioral tasks of a procedure, but, more

13. 2
important, the decision-making and cognitive processes that are inherently involved
with performing these procedures (Jacklin, Sevdalis, Darzi, & Vincent, 2008).
Cognitive task analysis is a generic reference to a variety of methods that have
been demonstrated to effectively capture the knowledge and skills experts use to
perform complex tasks (Crandall, Klein, & Hoffman, 2006; Hoffman & Militello,
2009; Clark, Feldon, van Merrienboer, Yates, & Early, 2008). There are many
definitions of cognitive task analysis (CTA). For example, Crandall, Klein, and
Hoffman (2006) define CTA as a “family of methods used for studying and
describing reasoning and knowledge [including] the activities of perceiving and
attending that underlie performance of tasks [and] the cognitive skills and strategies
needed to respond adeptly to complex situations” (p. 3). Whereas, Clark et al. (2008)
define CTA as the usage of “interviews and observation strategies to capture a
description of knowledge that experts use to perform complex tasks.” The common
thread among these definitions of CTA is that the goal of CTA is to capture the
underlying knowledge and skills experts use to solve complex tasks.
Information captured from experts through CTA has been shown to be effective
for training novice physicians (Velmahos, Toutouzas, Silin, Chan, Clark, Theodorou,
& Maupin, 2004; Sullivan, Brown, Peyre, Salim, Martin, Toowfugh, & Grunwald,
2007; and Luker, Sullivan, Peyre, Sherman, & Grunwald, 2008). For example,
Velmahos et al. (2004) found that surgical residents who received CTA based
instructions on how to perform a CVC procedure performed better on a 14 point
procedure checklist when compared to a control group taught through traditional

14. 3
methods. Additionally, Sullivan et al. (2007) found that general surgery residents
performed significantly better on post -instruction technical assessments at both a
one-month and six-month benchmarks when instructed on how to perform a
percutaneous tracheostomy (PT) utilizing a CTA based curriculum. In another study,
Luker et al. (2008) found that although technical skills are developed through task
exposure, the insertion of a CTA based multimedia instructional program
significantly increased the knowledge and skills of the surgical residents completing
flexor tendon repair procedure. And, in a recent study, Tirapelle (2010) found that
randomly assigned surgical residents who were provided CTA-based instruction
improved in their knowledge and skills on how to conduct an Open Cricothyrotomy
when compared to a control group taught with traditional methods. These studies
emphasize the positive impact a CTA based curriculum has on surgical residents’
knowledge and skills in performing complex procedures.
It is often recommended in the CTA literature that multiple experts be consulted
to increase the validity and reliability of CTA results (Yates, 2007; Yates & Feldon,
in press). However, many researchers note that the recommendation to use multiple
experts creates a “bottleneck” due to the additional time and resources required
during the CTA process (Hoffman, Crandall, and Shadbolt, 1998; Hoffman, Shadbolt,
Burton, & Klein, 1995). As such, a question often asked is how many experts are
required to capture the optimal knowledge needed to perform a complex task? There
is a paucity of research that has studied this problem. An exception is Chao and
Salvendy (1994), who examined different techniques of acquiring knowledge to

15. 4
diagnose computer software programs. While Chao and Salvendy (1994) found that
a single expert provided an average of 27% to 40%, depending on the software-
debugging task, the percentage of knowledge and skills increased as they interviewed
additional subject matter experts up to a total of six individuals. Chao & Salvendy
(1994) concluded three experts were needed to acquire the optimum critical
knowledge and skills needed to solve a complex task.
To date, there have been no published studies within the field of surgical training
that examine the number of surgical experts required to capture the optimum
knowledge and skills required to perform complex procedures. A study conducted
concurrently with this study sought to examine this question for the cricothyrotomy
procedure and found that 3 experts were required to capture the actions and decision
steps necessary to perform the procedure (Crispen, 2010). The current investigation
seeks to replicate the methodology used by Crispen (2010) to determine the optimal
number of experts required for the procedure to place a central venous catheter.
As such, this study poses two research questions: 1) How much information about
a central venous catheter placement procedure does a single expert provide when
compared to the combined contributions of a comprehensive “gold standard” protocol
based on interviews with six physician subject matter experts; 2) How much critical
information is gained from each additional interview about a central venous catheter
placement?

16. 5
REVIEW OF THE LITERATURE
To understand the contributions of cognitive task analysis and its potential benefit
to medical education, we must first review the current trends in surgical training and
the inherent complexities of relying on expert physicians as instructors/mentors for
training surgical residents. A review of both declarative and procedural knowledge
types is critical to understanding how experts construct knowledge and sometimes
have difficulty communicating acquired knowledge. Often, subject matter experts
(SMEs) omit critical information when describing how to perform a task thus.
Research shows that surgeons can omit up to 70% of the critical information needed
to perform a procedure (Clark, Pugh, Yates, Sullivan, 2008). Thus, capturing this
information is a critical component to helping train surgical novices. Cognitive task
analysis provides a method to capture this omitted critical information. Therefore, a
review of the use of CTA and it’s reported effectiveness both within and outside the
medical field is presented. The review of the literature section will conclude with a
summary of recent studies that examined how many experts are required to capture
the critical knowledge needed to perform a complex task.
Current Trends in Surgical Training
Surgical training has experienced a paradigm shift over the last several decades
from the Halstedain-based apprenticeship model to the development of surgical skills
training facilities (Hamdorf and Hall, 2000). This shift is due to advancements in
technology and the need to expose surgical residents to multiple procedures to
increase the knowledge and skills of surgical residents outside the operating room

17. 6
(Scott, Cendan, Pugh, Minter, Dunnington, & Kozar, 2008). As a result, university
medical education facilities are modifying their programs to utilize discrete,
procedure specific trainings, in an effort to provide the needed skills training for
proficient physicians (Sachdeva, Bell, Britt, Tarpley, Blair, & Tarpley, 2007). A
surgeon’s developing knowledge and skills in completing a variety of medical
procedures are determined by the frequency of practice. Surgical skills training
centers provide an opportunity for surgical residents to develop their skills to a level
of automaticity outside the operative room. The use of virtual reality and other human
performance simulators can help develop the knowledge and skills of surgical
residents (Reznick and MacRae, 2006).
Evidence of increased surgical knowledge and skills has been validated through
studies in short-term, specific procedure concentrated classes. Surgical interns who
were trained using a human patient stimulator (HPS) saw an increase the participant’s
trauma management skills and overall self-confidence in their abilities (Marshall,
Smith, Gorman, Krummel, Haluck, & Cooney, 2001). Additionally, third-year
residents indicated through self-reports that their knowledge and skills generally
increased as a result of participating in a small group training sessions covering four
surgical procedures [foley catheter placement; nasogastric tube insertion/removal;
I.V. placement; and arterial stick] (Meyers, Meyer, Stewart, Dreesen, Barrick, Lange,
& Farrell, In press). Grantcharov and Reznick (2008) indicate two common elements
within teaching procedural technical skills for surgery: pre-patient training and
training in a clinical situation. While the pre-patient focuses theory and simulation

18. 7
curriculum, clinical situation training incorporates an expert instructor for
demonstration, instruction, and evaluation purposes.
The common element of all medical education is the reliance on surgical experts.
Experts are often characterized by the coordination of their highly organized
knowledge structure and their advanced technical skills within a specific domain
(Ericsson, Charness, Feltovich, & Hoffman, 2006). However, Sleeman and Brown
(1982) indicate that much of an expert’s knowledge is tacit and not readily available
to the expert during retrieval tasks. Therefore, relying on experts for educating
surgical residents is a major limitation to the current training methods. A brief review
of the characteristics of experts, experts in medicine, and the affect of experts on
medical training will be reviewed to establish an understanding of the complexity
involved with experts in the field of medicine and the underlying problems with the
expert’s ability to transfer their knowledge to novices.
Development of Expertise
Although expertise can be defined in many ways (Feldon, 2007), it is commonly
agreed upon that expertise is defined as the possession of a large body of knowledge
and procedural skills within a specific domain (Chi, Glaser, & Rees, 1982; Ericsson et
al., 2006; Feldon, 2007). Expertise includes the development of automaticity through
deliberate practice (Ericsson et al., 2006) that reduces the cognitive demand in
performing procedural tasks. Cognitive demand is lowered through automaticity by
reducing the number of decisions that require conscious thought. Therefore,
automaticity provides the benefit of processing speed in problem solving, but inhibits

19. 8
conscious monitoring and modification of skills during task performance. The
inability to adapt routine behavior could lead to lower performance levels when faced
with atypical tasks. Feldon (2007) discussed the need for experts to have a developed
level of adaptivity when faced with atypical tasks (adaptive expertise). Adaptivity
provides the ability to adapt to novel task constraints while consistently demonstrate
optimal performance. In contrast, routine experts fail to maintain high levels of
performance when faced with changing task conditions. As a result of their high
performance level and cognitive efficiency in problem solving abilities, experts are
often called upon to provide information for training and expert systems, and are
revered socially as an authority figure within a specific field.
Expertise in Medicine
According to Ericsson et al. (2006) a significant portion of the research literature
on medical expertise focuses on diagnosis on medical problems. Boshuizen and
Schmidt (1992) investigated the different use of biomedical and clinical knowledge
by novice, intermediate, and expert-level participants in a study in clinical reasoning.
They defined biomedical knowledge as understanding the principle processes
underlying the manifestation of a disease (e,g, virus or bacteria, organ or organ
systems) and clinical knowledge as knowledge of the attributes in which diseases can
manifest themselves, the related overt and covert signs of the disease and overall case
management heuristics. Boshuizen and Schmidt (1992) found that with increased
levels of expertise, individuals utilized less biomedical knowledge and more clinical
knowledge while sequencing a clinical diagnosis. The increase in clinical knowledge

20. 9
is due to an experts developed “illness scripts” which provide the ability of experts
to mediate the current presented task based on prior actual worked cases.
In a study to analyze how subject matter experts use personal causal models for
diagnostic purposes, Patel and Groen (1986) provided a written medical case to seven
cardiologist specialists. After a brief review of the case, the participants were asked
to provide a written account of the case and also provide an underlying
pathopysiology of the case including a diagnosis. Patel and Groen (1986) found that
the experts who provided an accurate diagnosis utilized forward reasoning in their
causal models, while participants who provided inaccurate diagnosis utilized
hypothetical testing, including backwards reasoning. Similarly, in their research on
expertise and problem solving, Chi et al. (1982) noted that one of the qualitative
differences between experts and novices is their approach to a presented task. While
novices solve problems while working backwards with hypothesis testing heuristics,
experts tend to perform an initial qualitative analysis of the problem and utilize
forward reasoning strategies mediating the current task with similarly stored
representations based on prior experience.
Expertise in Surgery
Acquiring surgical skills requires the development of cognitive and psychomotor
abilities (Hamdorf & Hall, 2000; Peters, Fired, Swanstrom, Soper, Sillin, Schirmer et
al., 2003). According to Hamdorf and Hall (2000), motor dexterity is developed
through three phases: cognition, integration, and automation. The cognition phase
consists of understanding the task. The integration stage incorporates the motor

21. 10
movements specific to the procedure. Lastly, automation infers that the movements
needed for a procedure have become non-conscious and do not rely on continuous
external stimuli. As individuals develop surgical motor and cognitive skills it is
imperative to minimize error development (Porte, Xeroulis, Reznick, & Dubrowski,
2007; Abernethy et al., 2008).
In summary, the attainment of expertise requires the acquisition of extensive
conceptual knowledge and automated procedural skills that allows experts to solve
problems within minimal cognitive demand (Feldon, 2007). To obtain the level of
expertise, in any domain, takes about ten years of deliberate practice with continuous
feedback (Ericsson, 1996; Pavlik and Anderson, 2008). Therefore, to acquire the level
of expertise in medicine and surgery, in particular, requires extensive practice. Two
underlying themes that distinguish experts from non-experts are their knowledge base
and their rapid cognitive processing abilities (Chi et al., 1982).
Due to the automaticity of an expert’s knowledge and skills, conscious monitoring
of thought processes is difficult, therefore an expert’s description of problem solving
is typically limited to observable actions. An understanding of knowledge types
(declarative and procedural) and their function in developing expertise provides a
further understanding of the requirements for capturing needed information from
experts for training complex tasks, such as surgical procedures.
Knowledge Types
Knowledge is the representation of learned information that can be expressed in a
variety of settings for different purposes, actions, or means (Markus, 2001; Spender

22. 11
1996; de Jong & Ferguson-Hessler, 1996; & Anderson, Krathwol, Airasian,
Cruikshank, Mayer, Pintirich, Raths, & Wittrock, 2001). Research indicates that
individuals have a limited capacity in the ability to consciously process information
and there is evidence that people have the ability to process four plus or minus one
bits of information within their working memory (Cowan, 2001). Working memory
is the theorized space that individuals “pay attention to” and encode new information
into long-term memory. Long-term memory is considered unlimited in its capacity
and is associated to encoded chunks of information and are accessible based on their
retrieval structures (Ericcson and Kintsch, 1995). In this section, two types of
knowledge – declarative and procedural – will be discussed and the interactions
between the two that comprise expertise
Declarative Knowledge
Cognitive psychologists have defined declarative knowledge in many ways.
Ormrod (2008) describes declarative knowledge as information retrievable from an
individual’s long-term memory into their short-term memory as concepts, principles,
general knowledge, and recollections of life events/ experiences. Schunk (2000)
suggests that declarative knowledge is comprised of facts, actions within a story, and
the organization of a passage, with reference to information obtained from a novel or
piece of literature. Another definition by Schraw (2006), postulates that declarative
knowledge encompasses facts, concepts, and the mediated relationships among
learned concepts that create an integrated holistic conceptualization within a specific
domain of knowledge. Declarative knowledge is often subdivided into two

23. 12
categories, semantic knowledge comprised of organized schemata regarding
individual concepts (e.g., vascular system) and episodic knowledge (time specific
events or episodes in life, e.g. prom, wedding, & birth of a child) (Schraw, 2006).
Declarative knowledge is considered explicit knowledge (which is easily
retrievable) that is developed through encoding information from the environment.
Declarative knowledge is retrievable information that can articulate the answers to
question of what and why with regards to concepts and facts (Paris, Lipson &
Wixson, 1983; Anderson & Schunn, 2000; Hoffman & Militello, 2009). These
encoded chunks are retrievable from an individual’s long-term memory into their
short-term (i.e. working memory) for immediate needed purposes. Retrieved
declarative knowledge guides individuals in completing performance-based problem
solving tasks (Anderson, Bothell, Byrne, Douglas, Lebiere, & Qin, 2004). For
example, a surgeon knows what equipment to use for a procedure and why the
surgery is needed and has the ability to describe what and why
According to Anderson’s (1996) Adaptive Character of Thought (ACT-R) theory
declarative knowledge is developed as one encodes chunks of information from their
environment. Anderson (1996) describes the declarative chunks of knowledge as
“schema-like structures” that are categorically specific with tangible connections to
other related encoded content. Knowledge structures develop through the
accumulation and toning of individual units of information (chunks) that in whole
produce complex cognition. Although, these chunks of knowledge represent factual
information that an individual can express (Anderson & Schunn, 2000), the

24. 13
acquisition of declarative knowledge does not occur in a vacuum, but rather in
conjunction with the acquisition of procedural knowledge.
Procedural Knowledge
Anderson (1996) contrasts declarative and procedural knowledge by describing
declarative knowledge as “chunks” of information, and procedural knowledge as
“production rules” that are based on the encoded chunks of information. Specifically,
Anderson (1996) suggests that production rules represent procedural knowledge of
the conditions (when) and actions (how) to perform a goal-oriented task that are
based on the individual’s declarative knowledge structures. Further, a production rule
can only be activated once a precise environmental state meets the set conditions for
the production rule to be carried out. For example, a surgical procedure consists of
multiple decisions that need to be made based on the presenting conditions within that
surgery. Thus, production rules symbolize the condition and action needed in
response to the attainment of specific goals.
Production rules consist of IF, THEN statements and are context embedded
(Anderson, 1996) In medical procedures, for example, IF there is bright red pulsating
blood in a syringe, THEN the needle must have punctured an artery. Thus, procedural
knowledge is “when and how to” perform an action or “make a decision” while
completing a task or solving a problem. These production rules (procedural
knowledge) are tacit and not easily retrievable to the individual. In surgery,
procedural knowledge consists of the decisions a physician makes on “when and
how” to perform certain actions during the surgery. In contrast, declarative

25. 14
knowledge is the information the individual knows about the surgery and its
components (knowledge of the anatomy and the instruments used for a procedure.)
When they practice, individuals fine-tune their skills and declarative knowledge
gradually evolves into procedural knowledge (Pirolli & Anderson, 1985; Stadler,
1989; Anderson & Fincham, 1994; Anderson, Fincham, & Douglass, 1997;
Anderson, Fincham & Douglass, 1999; Pavlik & Anderson, 2008; and Knowlton &
Moody, 2008.). Ericsson et al. (2006) indicate that procedural knowledge is
considered to be the final state of the learning process: the combination of acquired
declarative knowledge used to solve problems that become automated through
deliberate practice.
In medicine, through deliberate practice, a highly experienced surgeon can
process the cues of a medical procedure and perform the tasks within the procedure to
a level of automaticity. This reduces the attention demands on the surgeon (What is
the next step in this procedure?) and frees up cognitive resources and enabling the
physician to shift attention to other cognitive actions such as directing the supporting
nurses, self-monitoring effectiveness of the actions taken, and the status of the patient
(changing conditions).
Automaticity
Automaticity is a means of restructuring same procedures so that working
memory is largely circumvented, freeing cognitive resources for other cognitive
demanding tasks (Ericsson, Charness, Feltovich, & Hoffman, 2006). Ericsson et al.,
(2006) discuss how automaticity is central to the development of expertise and

26. 15
practice is the means to achieving automaticity. Expertise involves automated basic
strokes. Experts perform an extreme amount of practice. Such experience,
appropriately conducted, can yield effective, major behavior and brain changes.
Through the act of practice (with reflective feedback, self-regulation, & monitoring of
one’s progress) the character of cognitive operations changes in a manner that:
a) Improves the speed of the operations
b) Improves the smoothness of the operations
c) Reduces the cognitive demands of the operations.
Thus releasing cognitive resources (attention) for other functions such as
planning, self-monitoring, situational awareness. Appropriate planning includes the
use of a developed algorithm to problem solve. Self-monitoring focuses on the error
correction procedures conducted within a task. Situational awareness refers to being
observant of the current conditions during a task.
According to Ericsson et al. (2006), two major difficulties exist with the
development of automaticity. First, individual’s performing at this level of cognitive
automaticity are not as attentive to new cues that fall outside their learned procedure
and may overlook the otherwise salient cue, nor are individuals cognizant of all the
discrete steps involved within their actions. For example, while performing a medical
procedure, a surgeon may continue to follow previously learned behaviors when the
present conditions indicate alternative techniques may be more beneficial.
Additionally, when describing their actions multiple steps may be omitted due to their
automated knowledge. Second, once automaticity has been achieved, the ability for

27. 16
individuals to increase their skills diminishes. Once individuals learn a task to the
level of automaticity, they lose the ability to augment their performance without
purposeful actions, both physically and mentally, which Ericsson calls deliberate
practice. Deliberate practice is 1) actively seeking out novel demanding tasks, and 2)
monitoring, through guided introspection, actions and decisions while performing a
task.
In summary, expertise is the interaction of declarative knowledge (the What, the
concepts, process, and principles involved in labeling items in the world) and
procedural knowledge (the how and when in terms of actions and decisions necessary
in completing a task). In terms of surgical residents, their declarative knowledge
consists of the conscious academic preparation and skills they have learned up to an
intermediate level of proficiency. Their procedural knowledge consists of utilizing
their declarative knowledge in completing medical procedures. As individuals
develop expertise, their knowledge becomes highly organized and integrated
structures within their domain. Their skills become highly developed through
deliberate practice, reflection, and corrective feedback.
Automaticity and Expert Recall
Cognitive psychology describes how individuals transform encoded information
through practice to develop heuristics in problem solving. These heuristics are often
automated procedures that are tacit and unconscious to the individual. Polanyi (1962)
initially referred to tacit knowledge as certain cognitive processes in context with
problem-solving behaviors that are inaccessible to the conscious mind. Tacit

28. 17
knowledge refers to the individual’s inability to recall the explicit knowledge used
during a series of events while completing a task (Gourlay, 2004).
Expert instructors are often called up to provide needed information and resources
to novices through curriculum development and teaching. Hinds, Patterson, and
Pferrer (2001) found that due to the cognitive characteristics of experts, that is, a
highly organized, hierarchal system of interconnected abstract concepts, accompanied
by advanced procedural automated knowledge, the transfer of knowledge from
experts to novices has severe limitations. In their first study, Hinds et al. (2001)
investigated the different instructional styles used by beginner and expert instructors
when training novice individuals about how to understand simple electronic concepts
by completing an electronic circuit. Hinds et al. (2001) found that experts utilized
more abstract concepts and statements when compared to beginner instructors who
trained novices with more concrete statements. For instance, only 9% of experts
provided direct information about the requirement of properly connecting the wires to
the spring coil, which is essential for the electrical connection to be completed, while
90% of beginners provided this information. Hinds et al. suggest that experts
educating individuals in this process may omit critical information that novices need
to develop their conceptual knowledge within this domain. According to Hinds
(1999), as experts develop their automaticity, their simplification of facts and
concepts precludes them from recalling specific details and task imbedded
complexities that novices require for problem solving in this domain.

29. 18
Automaticity and expert recall in medicine and surgery
An individual’s automated procedural (tacit) knowledge has systemic
consequences in the medical field. Surgical experts usually train novice physicians
(Vadcard & Luengo, 2004). Research indicates experts have greater perceptual,
cognitive, motor, attention and personal reflective feedback capabilities when
compared with non-experts (Abernethy, Poolton, Masters, & Patil, 2008). Physicians
develop their declarative and procedural knowledge through years of experience and
practice (Cauraugh, Martin, & Martin, 1999). Physicians develop and hone their skills
by conducting multiple procedures and develop ‘rules of thumb’ (Andre, Borgquist,
Foldevi, & Molstad, 2002) which are tacit and unconscious to the individual. As a
result of their continued practice, their skills becomes automated and transition from
being knowledge that is on a conscious level to knowledge that is tacit and illicit
without deliberate probing (Abernethy, Poolton, Masters, & Patil, 2008). Novice
medical students relying on the attending physician for critical information regarding
a procedure may not receive the knowledge required to perform adequately due to the
attending physician’s inability to recall the needed information (Hamdorf & Hall,
2000).
The consequences of expertise and automaticity on expert recall was clearly
establish through the research of Clark, Pugh, Yates, Early, & Sullivan (2008) in their
investigation of capturing the declarative and procedural knowledge from trauma
surgeons on how to perform an emergency femoral artery shunt procedure. Nine
trauma surgeons provided an unaided (no- CTA condition) account of the shunt

30. 19
procedure. When their description of the surgery was compared to a gold standard
femoral shunt surgical protocol, it was determined that they omitted an average of
68.75% of the procedural steps performed in the surgery (Clark et al., 2008). As a
result of their automated knowledge and skills, the trauma surgeons within the study
were not able to recall the full accounts of the procedure. Theoretically, experts may
omit up to 70% of the procedural knowledge involved within a surgical procedure
during the instruction of novices. However, differing results were obtained when one
trauma surgeon was interviewed using CTA-based methodology. When the acquired
knowledge from the one trauma surgeon (CTA condition) was compared to the gold
standard protocol, Clark et al., (2008) calculated only a 30% omission of the
procedural steps. The significant variable in capturing the procedural steps from the
trauma surgeons was the use of an interview based on cognitive task analysis
methodologies, suggesting that cognitive task analysis shows promise as a
methodology to capture the automated, procedural (tacit) knowledge from experts.
The following sections includes a review of how cognitive task analysis is
defined; the appropriate CTA methodologies to capture the intended critical
knowledge and skills sought; the effectiveness of CTA outside medicine and within
surgical training; and the barriers and limitations to using CTA. Finally, the
importance in determining the number of experts and the methods utilized in
determining the optimal number of experts that are the most reliable and valid will be
reviewed.

31. 20
Cognitive Task Analysis
Until the second half of the 20th
century, behavioral task analysis was the primary
method to understand task performance by detailing the observable behaviors as an
individual performs a task. Although specific physical attributes of problem solving
skills could be quantified, there was little consideration for the unobservable thought
processes taking place inside the individual’s mind. Prior to the use of cognitive task
analysis, individuals developed and used a behavior task analysis to capture the
needed information from experts on how to complete tasks (Clark et al., 2008).
Although a behavior task analysis can describe the observable steps involved in
completing a task, it fails to capture the critical information about the cognitive
processes required for how to complete the task. Cognitive task analysis developed
out of the need to capture these previously unobservable decisions, analyses,
judgments, and other cognitive processes.
Defining cognitive task analysis
There are a considerable number of definitions for cognitive task analysis.
Cognitive task analysis (CTA) refers to a variety of tools and techniques for
describing the knowledge and skills required to perform a task that yields information
about the unconscious thought processes and goal structures that underlie the
observable task performance (Schraagen, Chipman, & Shalin, 2000). CTA
techniques capture a representation of the knowledge and strategies that have been
retained by the individual until some time after the event in question (Feldon, 2007).
Furthermore, cognitive task analysis (CTA) uses a variety of methods to capture

32. 21
experts conscious and non-conscious automated knowledge needed to perform a
complex or critical task (Crandall, Klein, & Hoffman, 2006). The common theme
found in these definitions is the use of multiple techniques that elicit knowledge,
facilitate data analysis, and represent the content and structure of the participant’s
knowledge within a specific task domain.
CTA has been used to capture expertise in a variety of working environments.
Researchers have used a cognitive task analysis framework to examine expert
performance to capture the cognitive task demands on an Intensive Care Unit team
(Fackler, Watts, Grome, Miller, Crandall, & Pronovost, 2009); to capture the
cognitive demands and indicators of business pilot expertise when faced with various
weather conditions (Latorella, Pliske, Hutton, & Chrenka, 2001); to investigate the
characteristics and decision making of expert warning forecasters of severe weather
(Hahn, Rall, & Klinger, 2003); and to develop a preliminary visual design aide
(heuristic) and prescriptive evaluation criteria to assist intelligence analyst efficiently
sift through data overload and develop an appropriate response to situational
questions outside their area of expertise (Patterson, Woods, Tinapple, & Roth, 2001).
CTA methodology
Cooke (1994) identified over 100 different types of cognitive task analysis
methods. This enormous variety of CTA methodologies can make it difficult for
practitioners to choose the correct method for the end result purposes of the research.
Cooke (1994) grouped the CTA methods reviewed into three broad families of CTA
techniques: 1) Observations and interviews; 2) process tracing; and 3) conceptual

33. 22
techniques. As noted in Clark et al. (2008), observations and interviews includes
talking with experts and watching them perform tasks. Process tracing involves the
use of think-aloud protocol or subsequent recall of an expert’s performance in a
specific task, while conceptual techniques are used to capture structured conceptual
knowledge within a domain (Clark et al., 2008).
Clark et al., (2008) reviewed the CTA literature and found that individual’s who
conduct CTA research typically follow a five-step procedure: 1) Collect preliminary
knowledge; 2) Identify knowledge representation; 3) Apply focused knowledge
elicitation methods; 4) Analyze and verify data acquired; and 5) Format results for the
intended application (Schraagen et al., 2000 and Clark et al., 2008,).
Collecting preliminary knowledge incorporates gaining initial knowledge of the
domain (bootstrapping) and developing a sequence of main steps that will guide the
CTA investigation. This initial step includes the analysis of documents and
identification of potential experts to be utilized within the study. The information
gained in the preliminary stage allows the analyst to examine the overall task in
question and identify possible sub-tasks and types of knowledge required for experts
to perform the complex task. Once the analyst has identified the knowledge
representations involved in the complex task, it is important to choose the appropriate
CTA method of knowledge elicitation (Clark et al., 2008; Yates, 2007) depending on
the knowledge type the analyst is trying to capture. Differing CTA methods elicit
either conceptual or procedural knowledge (or both) (Yates, 2007) with varying
efficiency (Hoffman, Crandall, and Shadbolt, 1998). One of the more effective ways

34. 23
to conduct a CTA and capture both declarative and procedural knowledge is
through a semi-structured interview (Yates and Clark, in review).
After applying the appropriate focused knowledge elicitation methods, acquired
data analysis and verification involves multiple steps. This process includes a)
coding the transcribed interviews to identify the conceptual knowledge (concepts,
processes, and principles), procedural knowledge (action and decision steps), goals,
standards, equipment and materials (Clark, 2004); b) formatting the output for review
and verification by the interviewed subject matter experts (SME); and c) finally the
analyst aggregates all documents into one standard “gold standard” format (Clark et
al., 2008). The final CTA document details the declarative and procedural knowledge
required to complete the actions and decision steps involved with performing the task.
Often, these products are utilized in the development of instructional materials (see
Velmahos et al., 2004) and expert systems (Schraagen et al., 2000).
Research has demonstrated that CTA is effective to acquire the critical conceptual
knowledge and automated skills utilized by experts in task performance (Clark et al.,
2008; Yates, 2007). The following sections will discuss the effectiveness of using
cognitive task analysis methods within a variety of settings, concluding with a review
of CTA within surgical training.
Effectiveness Studies Using Cognitive Task Analysis
Webster’s dictionary (1999) defines effectiveness as “producing a definite or
desired result: efficient” (p.454). Research in the effectiveness of CTA has a variety
of components, such as the chosen knowledge elicitation technique (domain specific),

35. 24
the type of knowledge captured or the usefulness of the knowledge and its
application capabilities (Hoffman, Crandall, & Shadbolt, 1998). In addition, the
effectiveness of a CTA study depends on the results meeting or surpassing some set
criteria in order to be considered effective. In experimental CTA research, the
effectiveness of a study is dependent on the acquired knowledge types utilizing a
specific methodology (Clark et al., 2008; Yates, 2007; Yates and Feldon, in press).
The use of cognitive task analysis has shown to be effective within a variety of areas,
including addressing data overload and human-computer interaction (Patterson,
Woods, Tinapple, & Roth, 2001), piloting under various weather conditions
(Latorella, Pliske, Hutton, & Chrenka, 2001); emergency response teams and white-
water rafting (O’Hare, Wiggins, Williams, & Wong, 1998); air traffic control
knowledge (Redding, Cannon, Lierman, Ryder, Purcell, & Seamster, 1991);
information retrieval through library resources (Pejtersen, 1989); and medical
education (Crandall & Getchell-Reiter, 1993; Johnson, Healey, Evans, Murphy,
Crawshaw, & Golud, 2005; Sullivan, Ortega, Wasserberg, Kaufman, Nyquist, &
Clark, 2008).
Redding et al. (1991) investigated the use of an integrated task analysis
methodology to extrapolate the expert knowledge structures of air traffic controllers
in an effort to produce instructional content for new curriculum. Five full
performance level air traffic controllers were videotaped solving four different
problem scenarios. Each participant provided a verbal protocol while reviewing their
performance giving a description of their immediate goals and decision-making

36. 25
processes while solving each of the four problems. As a result of the collective
verbal protocols, mental models, task decompositions, and individual task models
were constructed. The results of the CTA indicated a significant difference between
expert and non-expert air traffic controllers. In comparison to novices, experts
utilized fewer management strategies at a greater effective rate than novices.
Although it was noted that experts had a vastly greater repertoire of strategies to draw
upon, their ability to synthesize the situation better than novices allowed them to
choose specific appropriate strategies to solve the problem at a faster effective rate,
unlike novices who sometimes exhausted their limited strategies to find a workable
solution. The results from the cognitive task analysis were integrated into a
standardized curriculum that provided novice air traffic controllers a consistent and
deeper understanding to solving problems within their field (Redding et al., 1991).
While the CTA conducted within the Redding et al., (1991) study resulted in a
standardized curriculum for future trainings, Johnson, Cumming, & Omodei (2008)
concluded that the knowledge and skills captured from bushfire fighters would be
inappropriately used in a universal training guide, but would be more appropriate for
training at different levels of leadership. Johnson et al., (2008) interviewed 90
experienced bushfire fighters from 2005-2006 in order to capture the conceptual
knowledge and the decision making process that occurred at various levels of
firefighting leadership positions with regards to a worst case scenario. Johnson et
al.’s, (2008) investigation was in reaction to the post hoc reality of poor decision
making in recent bushfires in the Australian countryside that lead to two fatalities.

37. 26
Semi-structured interviews (adapted from the Critical Decision Method: CDM;
Klein, Calderwood, & MacGregor, 1989) were conducted in multiple probes. Initially
to capture the series of events with additional probes aimed at capturing the actions
and decisions occurring within specific incidents.
An analysis of the transcribed interviews indicated that the responses from the
bushfire fighters could be classified into three major themes: A fire focus; a fire
fighter focus; and a community focus. Additionally, Johnson et al. (2008) indicated
the results followed a trend that matched the level of leadership to the respondent’s
primary focus (i.e. higher levels of leadership displayed more of a community focus).
The CTA utilized captured the decision making process at all levels of leadership, but
due to the complexity at each level, a universal training protocol would not meet the
varied needs of the leadership positions. An interesting finding during the
investigation was the occurrence of experienced bushfire fighters omitting of any
references to a “worst case scenario” during their interviews which could lead to poor
decision making under certain contexts (Johnson et al., 2008). The omitted reference
to a ‘worst case scenario’ from experienced bushfire fighters is consistent with the
research literature on expertise and automated knowledge (Ericsson et al., 2006).
Cognitive task analysis provided a means for Johnson et al., (2008) to capture the
automated decision-making bushfire fighters would execute within a specific context
and for different leadership positions.
Cognitive task analysis has been shown to be effective in capturing decision-
making made by medical professionals (Crandall et al., 1993; & Jacklin, Sevdalis,

38. 27
Darzi, & Vincent, 2008). For example, in an effort to elicit the critical cues
neonatal intensive care unit nurses used to evaluate the health and immediate future
needs of newborns within their care, Crandall and Getchell-Reiter (1993) conducted
semi-structured interviews with nineteen registered nurses. The semi-structured
interviews produced 33 incidents of immediate, life threatening situations, most often
cases of sepsis or systemic bacterial infection, requiring urgent care within minutes or
over the next few hours. The results indicated NICU nurses were alert to a number of
critical indicators of sepsis not found in the medical literature used to train neonatal
care nurses.
In a second study, Crandall & Getchell-Reiter (1993) conducted semi-structured
interviews with five experienced female NICU nurses utilizing a critical decision
method (CDM). Critical decision method is a retrospective knowledge elicitation
technique developed by Klein, Calderwood, & Macgregor (1989). Each nurse was
interviewed about three incidents that lead her to the correct diagnosis of infant
septsis, an incorrect diagnosis of the same nature, and a time when the baby
developed sepsis when not originally detected by the current methods. The
information obtained from the initial study along with the secondary study of sepsis-
related incidents was used to create a sepsis assessment guide that became a learning
aid within training NICU nurses.
As individuals gain expertise, their ability to communicate their knowledge to
novices diminishes (Hinds, Patterson, and Pferrer, 2001). Sullivan et al. (2008)
investigated whether expert surgeons omitted actions and decisions when teaching a

39. 28
colonoscopy to second -year postgraduate residents. Three expert surgeons were
videotaped during an instructional session during which they provided a detailed
account of both the action and decision steps involved in a colonoscopy procedure.
Shortly after the expert’s instructional session, the experts participated in a free recall
of the colonoscopy procedure, focusing on the detail and completeness of the
information provided during the prior teaching session. Following their free recall, an
analyst conducted a CTA with each expert individually in a series of structured
interviews to capture the conceptual and procedural knowledge required to perform
the procedure. As a result of the knowledge captured through the CTA, a 26-step
procedural checklist and a 14-point ‘cognitive demands checklist’ (p.21) were
created. In comparing the experts free recall to the CTA checklist, Sullivan et al.,
(2008) found that the experts omitted 50% to 74% of the ‘essential how to steps’ and
57-75% of the critical decisions involved in the colonoscopy procedure. Although
the expert surgeons in the study were recognized as outstanding surgical educators,
they were not able to articulate all of the action and decision steps involved in the
task. This is consistent with the expertise literature that suggests the knowledge and
skills of experts are highly automated and not easily retrievable when prompted.
(Ericsson et al., 2006), and, moreover, because experts often make errors when
describing how to perform a task (Clark and Estes, 1996, Clark et al., 2008), they
often omit critical information they believe that they have communicated to their
students (Clark, 2006; Feldon, 2004).

40. 29
In summary, the effectiveness of cognitive task analysis is often measured by
the type of knowledge captured and sufficiency for the intended application. The
studies reviewed provide insight about how the use of CTA successfully captures
both the conceptual knowledge and the procedural skills experts use to perform
complex tasks. While there are numerous studies that examine CTA for the purposes
of capturing expertise (See Cooke, 1994 and Yates, 2007) there is a paucity of studies
that investigate integrating the knowledge and skills of experts into training protocols
in the surgical domain (exceptions- see Luker et al., 2008; Sullivan et al., 2007;
Velmahos et al., 2004; and Tirapelle, 2010).
Effectiveness of CTA in Surgical Training
Cognitive task analysis (CTA) has been shown to make a significant difference in
increasing the knowledge and skills of surgical residents (Velmahos et al., 2004;
Bathalon, Martin, and Dorin, 2004; Johnson et al., 2005; Luker et al., 2008; Sullivan
et al., 2007; and Tirapelle, 2010). The following review of studies illustrates the
significant benefit CTA brings to surgical training. The proceeding sections will
discuss the benefits CTA provides to surgical training including capturing both action
and decision steps within a procedure; utilizing CTA to create training materials that
produce immediate acquisition of knowledge and skills when comparing CTA- based
curriculum versus traditional methods; and CTA and long term maintenance of
acquired knowledge and skills.
Cognitive task analysis is often utilized to capture the expertise in order to create
training protocols for novices. Johnson et al. (2005) followed CTA principles to

41. 30
create training guides for five commonly performed interventional radiology
procedures. Each task analysis captured the action and decision steps involved in the
procedure. For example, the CTA conducted for an arterial needle puncture captured
101 action steps and 24 decision steps from the experts interviewed. Johnson et al.,
(2005) indicated the collective knowledge gained through conducting the five CTA’s
would provide a basis for establishing a standard of practice in conducting these
interventional radiology procedures. Johnson et al. (2005) proposed that the
knowledge and skills captured through CTA could be utilized to develop novice
training and assessment simulations focusing on both the action and decision steps
conducted during a procedure. As a result, CTA protocols would rapidly increase a
novice’s knowledge and skill development. Additionally, the CTA could be used as a
cue for experienced operators before they conduct a rarely performed procedure. An
expert’s automated knowledge and skills impacts their ability to modify ingrained
task performance (Ericsson et al., 2006). Therefore, reviewing the action and
decision steps within a procedure would help create a revised mental model to follow
during task performance. Overall, Johnson et al. (2005) hoped that their investigation
would lead to additional studies that focused on less common interventional radiology
procedures, thus highlighting the benefits CTA in its ability to capture automated
decision steps for training purposes.
In a study to examine the efficacy of CTA-based instructional protocols,
Velmahos et al. (2004) sought to determine if training new interns through instruction
developed with the results of cognitive task analysis made a difference in the

42. 31
knowledge and technical skills of interns when compared to those trained with
traditional methods. Twenty-six new surgical interns were randomly assigned to
receive instruction on how to perform a central venous catheterization (CVC) through
traditional methods or through training based on a cognitive task analysis. Pre-test
scores indicated no differences between control (n=12) and experimental (n=14)
groups. Velmahos et al. (2004) found a significant difference between groups in the
posttest mean score when comparing the CTA group with the traditional group (11.0
+- 1.86 versus 8.64 +- 1.82, P=0.03). Velmahos et al. (2004) concluded that the
course taught with the results of cognitive task analysis was more effective in
increasing the cognitive knowledge and technical skills of interns on how to conduct
a CVC procedure.
In another study, Luker et al. (2008) investigated the use of a CTA based multi-
media instructional program and its effect on increasing the surgical skills knowledge
of residents in repairing a flexor tendon. Luker et al. (2008) utilized three flexor
tendon repair subject matter experts to construct a CTA detailing the knowledge and
skills needed to perform the surgery. The principal investigator designed a CTA-
based multimedia curriculum highlighting the critical decision points and skills
needed to complete a flexor tendon repair surgery. Ten surgical residents were asked
to perform flexor tendon repairs on three different occasions with a post-procedure
assessment to assess the residents understanding of the tasks and the critical decision
points within the procedure. Any increase in the participant’s knowledge and skills
from the first and second performance was deemed a result of practice and was

43. 32
considered the “control group” within the study. A learning session using a CTA-
based multi-media instructional presentation was conducted in between the second
and third flexor tendon procedure. Luker et al., (2008) concluded that changes in the
post procedure scores resulted from the CTA-based presentation. The results
indicated that the mean difference between the first and second procedure (control
group) was 13.2 for conceptual knowledge and 8.5 for understanding the advantages
and disadvantages of each decision within the procedure. However, the mean
difference between the second and third procedure was 34.0 for conceptual
knowledge and 19.4 for decision points. Luker et al., (2008) discovered that although
the group improved due to shear practice, the use of CTA in surgical training
significantly increased the knowledge and skills needed to perform a flexor tendon
surgery.
In a recent study, Tirapelle (2010) investigated the effects of a CTA based
curriculum versus traditional surgical training methods on how to perform an Open
Cricothyrotomy. The composition of the twenty-six medical student participants
included 3rd
year medical students, 2nd
year post-graduate students, and 3rd
year post-
graduate students. Pre-test measures showed no differences between the randomly
assigned experimental (n=12) and the control (n=14) groups in terms of experience
F(6,19) = .414; p = .860) or gender F(6,19) = .396; p = .873, or pre-test assessments
=5.4 (experimental) and =5.5 (control) out of 17 possible points. All assessment
and training materials for this investigation were developed from a gold standard
CTA report derived from interviewing six subject matter experts on how to perform

44. 33
an Open Cricothyrotomy. Tirapelle (2010) utilized Clark’s (2004) Guided
Experiential Learning (GEL) protocol in the development of the training materials.
Both groups (experimental and control) received a 30-minute instructional session on
how to conduct the procedure. The control group received instructions through
existing course materials and course structure, while the experimental group was
provided a CTA-based curriculum. Both groups were provided opportunities to
practice an Open Cricothyrotomy on inanimate models following the instruction in a
guided practice format. Post-instruction analysis was conducted on the participant’s
acquired knowledge and skills through individual assessments and ratings from expert
instructors and surgeons participating in the presentation of the training materials.
While the post instruction assessment showed no significant differences in the
acquired conceptual knowledge between the groups ( = 14.5, experimental and 13.9,
control; (t(22) = 0.55, p = .59), a significant difference was found when Tirapelle
(2010) measured the acquired procedural knowledge ( =17.75, experimental and
=16, control; t(21) = 2.08, p = .050) between the groups. Tirapelle (2010)
concluded that CTA-based instruction had a significant positive effect on the
acquisition of procedural knowledge and performance when compared to traditional
expert-led surgical skills instruction.
While the research of Velmahos et al. (2004), Luker et al. (2008) and Tirapelle
(2010) all found CTA- based curriculum produced a greater immediate increase in the
knowledge and skills of surgical interns, Bathalon et al. (2004) provided evidence that
the use of CTA for surgical instruction has long-term benefits as well. Bathalon et al.

45. 34
(2004) randomly assigned 44 first year medical students into three different
instructional groups on how to perform a cricthyroidotomy procedure. The first group
(n=16) received instruction from the traditional ATLS protocol. The second group
(n=13) learned the same procedure using both cognitive task analysis and kinesiology
principles. The third group (n=15) was instructed with a combination of CTA,
kinesiology principles, mental imagery practiced daily, and debriefing. The results
indicated that the groups taught with a CTA-based curriculum performed better
initially and also maintained their skill level over 12 months when compared to the
first group. The group who received training using a combination of CTA,
kinesiology principles, mental imagery practice, and debriefing showed the highest
skill acquisition and long-term maintenance.
In another study, Sullivan et al. (2007) investigated the effectiveness of a CTA-
based curriculum to instruct novices on the knowledge and skills required to
successfully complete a percutaneous tracheostomy (PT) placement as compared to
students trained by traditional methods. Sullivan et al. (2007) randomly assigned 20
postgraduate surgery residents into either the control group (N=11) and provided
traditional PT training, or the CTA group (N=9) and provided CTA-based training.
The results from Sullivan et al. (2007) study indicated that there were no significant
differences between the groups prior to instruction. However, post-instruction
assessment results indicated that a significant difference existed between the control
and experimental (CTA) group after instruction. The CTA group scored significantly
higher mean averages than the control group in the technical aptitude assessment at 1

46. 35
month (CTA: 43.5+- 3.7, control 35.2 +- 3.9, P=0.001). Although an attrition of
skills was indicated at the 6-month reevaluation assessment for both groups (CTA: 39
+- 4.2, Control: 31.8 +- 5.8, P=0.004), the group that received a CTA-based
curriculum still performed significantly better than the group who received traditional
instructional methods. Therefore, the expertise CTA captures, when utilized in
instructional materials, plays a significant role in encoding knowledge and skill
development into long-term memory.
In summary, research has shown using CTA is effective for surgical training.
Cognitive task analysis is effective in capturing expertise to create training protocols
for standards of practice (Johnson et al., 2005). Additionally, others have developed
CTA-based instructional materials where the results showed significant gains in
knowledge and skills of surgical interns when compared to traditional methods
(Velmahos et al., 2004; Luker et al., 2008; and Tirapelle, 2010). While others
provided evidence CTA has beneficial acquisition of knowledge and skills with long-
term maintenance (Bathalon et al., 2004 and Sullivan et al., 2007). CTA can help
obtain insights into the automated expert conceptual and procedural knowledge of
domain specific tasks and separate the steps into individual discrete teachable units
for others to learn. While there are several positive outcomes when utilizing cognitive
task analysis methods, there are some limitations with CTA.
Limitations of Cognitive Task Analysis
Although the research on cognitive task analysis consistently shows that CTA
training is more effective than traditional training methods, there are several

47. 36
limitations to using CTA techniques. An important limitation and the one most
commonly identified in the literature is the amount of time and effort involved in
conducting a CTA. Additional limitations include the automaticity of expert’s
knowledge and the ability to acquire their expertise. Next, these limitations are
discussed in detail.
The time necessary to conduct a cognitive task analysis can be time intensive
(Chao & Salvendy, 1994). According to Clark & Estes (2008), one hour of capturing
expertise requires approximately 30-35 hours of effort (Grunwald, Clark, Fisher,
McLaughlin, Piepol, 2004). Sullivan et al., (2008) estimated it took approximately 30
hours to complete the cognitive task analysis in their study. Additionally, Hoffman et
al., (2004) solidified the human cost of knowledge acquisition by documenting the
human hours required to complete components of their investigation utilizing CTA
methods. For instance, Hoffman et al., (2004) utilized CTA methods in transferring
the information in the Terrain Analysis Data Base (TADB) into 150 concept maps.
The total person hours to complete the 150 concept maps was calculated at 187.5-225
person hours. Each individual concept map took approximately 75-90 person hours
to complete.
One of the major limitations of cognitive task analysis focuses on the process of
knowledge acquisition (Hoffman et al., 1998). There are two main components to
knowledge acquisition including acquiring experts to interview and capturing their
expertise. An overall factor involved in these components is the time and effort
required in conducting a CTA. While there are enough experts to interview and

48. 37
capture the desired procedural knowledge for training purposes, their time to
participate in a CTA may be limited. Patience, adaptivity to the expert’s schedule,
and the ability to mediate logistic situations are essential to obtaining an environment
to conduct the CTA. Once these issues have been resolved, a bottleneck still exists in
capturing the expertise due to their highly organized cognitive structures.
The automated knowledge and skills of experts is a natural barrier to acquiring
their expertise. Sleeman and Brown (1982) indicate that much of an expert’s
knowledge is tacit and not readily available to the expert during retrieval tasks. Glaser
(1985) derived that experts have the ability to discuss the “what” and “why” in
performing a task, but a conscious analysis of their verbal recall is required to capture
the “how and when”, that is the decisions and reasoning necessary to perform a task.
As a result experts automated knowledge may lead to retrieval inaccuracies during the
knowledge elicitation process (Feldon, 2007). As a consequence, dependence on a
single expert for a CTA investigation can provide incomplete results for intended
purposes. Therefore, the necessity of acquiring multiple experts may be an additional
limitation to capture the required knowledge.
Number of Experts Required for CTA
There is a scarcity of research indicating how many experts are required to
optimally capture the critical information necessary to problem-solve tasks. A notable
exception is Chao and Salvendy’s (1994) study recommending the use of three
experts. They randomly assigned twenty-four expert computer science students to
one of four knowledge elicitation techniques. The dependent variable was the

49. 38
percentage of procedural knowledge captured through a knowledge elicitation
technique. The independent variables included the three computer programming
tasks (diagnosis, debugging, and interpretation) and the four knowledge elicitation
methods utilized (protocol, interview, induction, and repertory grid). The diagnosis
task combined with the repertory grid knowledge elicitation method indicated the
greatest gains in procedural knowledge from 40% from a single expert to 87% from a
total of six experts. For the other two tasks, the use of protocol knowledge elicitation
method showed the greatest gains in procedural knowledge obtained from one to six
subject matter experts: Debugging: 37% to 88%; Interpretation: 27% to 62% (Chao &
Salvendy, 1994). Chao and Salvendy (1994) found that the percentage of procedural
knowledge increased as they interviewed additional subject matter experts up to a
total of six individuals. As a result of their findings, Chao and Salvendy (1994)
recommended interviewing three subject matter experts based on a 10% marginal
utility cost-benefit analysis.
A review of the surgical training literature indicates a variability of the number of
experts utilized for CTA investigations. For example, Velmahos et al. (2004) made
use of two subject matter experts; Sullivan et al. (2007) gained knowledge from three
subject matter experts; Luker et al. (2008) utilized three subject matter experts in their
investigation; Johnson et al. (2006) employed two or three subject matter experts to
create their CTA protocols; and Sullivan et al. (2008) developed their CTA from three
subject matter experts. In an effect to create a decision map based on a diagnosis of
symptomatic gallstones, Jacklin, Sevdalis, Darzi, and Vincent (2008) utilized a

50. 39
structured interview technique with 10 experienced physicians in the area of
gallstones. It appears common in the CTA research to utilize two to three experts for
the purposes of capturing expertise.
Currently, only two known studies exist recommending the number of subject
matter experts needed to conduct a cognitive task analysis: for solving computer
related problems (Chao and Salvendy, 1994): and for surgical procedures (Crispen,
2010). In a concurrent investigation, Crispen (2010) investigated the optimal number
of subject matter experts to be interviewed in acquiring the critical expertise required
to complete a surgical procedure. Crispen (2010) found that four subject matter
expert’s were optimal in capturing the knowledge and skills needed to perform an
Open Cricothyrotimy procedure. This was the first empirical study that has identified
a recommended number of subject matter experts to be interviewed to capture the
expertise needed to complete a surgical procedure.
Summary
Medical training centers are charged with producing competent physicians who
are technically proficient in a variety of procedures (Aggarwal, Grantcharo, & Darzi,
2007). Within every educational program, expert instructors are charged with
providing needed information and resources to help novices develop their expertise.
Due to an experts highly developed declarative and procedural knowledge, their
ability to share their expertise with novices is limited. Cognitive task analysis (CTA)
has been shown to be effective in capturing expert’s automated knowledge in a
variety of fields (O’Hare, Wiggins, Williams, & Wong, 1998; Redding, Cannon,

51. 40
Lierman, Ryder, Purcell, & Seamster, 1991; Pejtersen, 1989; and Chao &
Salvendy, 1994). CTA investigations within the field of medicine are aimed at
capturing critical decisions an expert physician makes during a procedure. This
acquired knowledge has been proven beneficial in the development of expert-based
instructional protocols (Johnson et al., 2005). These CTA based training aids help
novice physicians develop declarative and procedural knowledge for a variety of
medical procedures.
Although there are significant benefits of CTA techniques, there are certain
limitations of this methodology. The two major limitations of cognitive task analysis
are 1) the process of knowledge acquisition and 2) the time required to conduct CTA
to provide a useable product for training purposes. The research literature has
multiple examples of studies indicating the benefits of using experts in capturing
expertise needed to perform a task/ procedure (Lyons, 2009). Currently, there is no
established recommended number of subject matter experts required when conducting
a CTA capturing the necessary conceptual and procedural knowledge for intended
purposes. Only two studies have provided recommended number of SMEs in CTA
research: a study conducted by Chao and Salvendy (1994) resulted in their
recommendation of three experts in solving computer based problems; and Crispen
(2010) recommended interviewing four experts for a surgical procedure.
Purpose of the study
Informed by the study conducted by Chao & Salvendy (1994), the purpose of this
investigation is to explore critical information that is gained from a single expert

52. 41
position and the information gained from each additional expert CTA interview.
The information acquired from this investigation will allow us to propose an answer
to the following question: How many experts are recommended to interview in order
to collect the critical knowledge (procedural steps and decisions) needed to perform a
(surgical) procedure? The results of this investigation aims to establish a
recommended number of experts one would need to conduct CTA interviews with in
order to capture their expertise to develop a gold standard protocol to conduct a
medical procedure. Establishing a recommended number of experts to interview has
significant implications in long-term cost-benefit savings of time and effort inherent
in conducting CTA investigations.
To review, the research questions are:
1) How much information about a central venous catheter placement procedure
does a single expert provide when compared to a six-subject matter expert gold
standard protocol?
2) How much Critical Information is gained from each additional CTA interview
about the Central Venous Catheter Procedure?

53. 42
CHAPTER 2: METHOD
The central venous catheter placement is a procedure that can be performed in
emergency and non-emergency situations. In medical procedures, a central venous
catheter ("central line", "CVC", "central venous line" or "central venous access
catheter") is a catheter placed into a large vein in the neck (internal jugular), chest
(sublcavian) or groin (femoral) . A CVC is used to administer medication or fluids,
and directly obtain cardiovascular measurements such as the central venous pressure.
As noted, the procedure is designed to gain access to a central vein for multiple
reasons, including rapid fluid infusion. The central venous catheter procedure was
used as the sample procedure to answer the proposed research questions. The current
study utilized a CTA based methodology. The proceeding sections review the overall
study design, how the subject matter experts were acquired, and how the data was
collected and analyzed.
Design
The current descriptive investigation, in general, replicates the research questions
proposed by Chao and Salvendy (1994), While Chao and Salvendy (1994) had 100%
knowledge of all possible errors within their study, the current investigation started
without a gold standard to measure expert’s elicited knowledge. Prior research
(Velmahos et al., 2004) on the central venous catheter placement produced a 14-point
checklist of tasks to complete during the procedure. This was created through the
collaborative work between two CTA experts and two experts on CVC. Although a
checklist has previously been created, it was never qualified as a “gold standard” for

54. 43
how to perform the CVC procedure. The current investigation began by collecting
conceptual and procedural knowledge from experts discussing how to complete a
CVC procedure through semi-structured interviews. The combined experts’ elicited
knowledge was aggregated into a gold standard CVC protocol. The gold standard
protocol represented 100% of total knowledge about the CVC procedure. The gold
standard was utilized to quantify the total conceptual and procedural knowledge
acquired from each expert to answer the research questions. As indicated above this
data analysis was in part a replication of the work completed by Chao & Salvendy
(1994).
Subjects
A convenience sampling procedure was used to acquire the six expert physicians
who participated in this investigation. The participating physicians included four
trauma physicians; one critical care internist; and one anesthesiologist at whom work
in large medical centers in the Los Angeles area. The participants were considered
experts due to their extensive years of successfully performing the CVC procedure.
Prior CTA based investigations used a range between two to ten experts. Six experts
were utilized in the current investigation to determine the optimal number of experts
needed to capture the needed critical information and to have a large enough sample
to validate the average amount of information gained from additional experts without
expanding undue human time, effort, and cost. The research literature indicates
between two and five subject matter experts are recommended to complete a valid
CTA investigation.

55. 44
Data Collection
The procedures on how to conduct the CTA for this investigation followed the
description provided by the work of Clark, Feldon, van Merrienboer, Yates, and Early
(2008) who provided details on the five most common elements of a CTA
investigation. The data collection for this investigation was conducted in four parts::
1) semi-structured CTA interviews of physicians with expertise on conducting CVC
procedure; 2) coding of interview manuscript; 3) creation of a six-expert based gold
standard CVC procedure protocol; and 4) analysis of the elicited knowledge from
both (a) single-expert and (b) non-repeated grouping of experts (i.e 2 SME’s, 3
SME’s up to 6 SME’s) against the gold standard.
Semi-structured CTA interviews
During the semi-structured cognitive task interview, the experts were asked a
series of questions that focused on the major tasks and potential problems a surgeon
could encounter when conducting a CVC procedure. Attention was focused on
obtaining the indications and contraindications on when to and when not to perform
the procedure. In addition to obtaining the overall procedural objective, the analyst
inquired about the expert’s knowledge of the benefits and any potential risks
performing the procedure. During their description of the CVC procedure, each
expert was asked to identify the equipment needed at the various stages within the
procedure. Upon confirmation of the major tasks, the experts were asked to describe
the specific actions they perform at each major task and sub task(s) within the CVC
procedure. During the interview process, the analyst asked probing questions about

56. 45
the action and decision steps involved in the procedure to uncover any alternatives
to the decision steps being made and the criteria for choosing such alternative actions.
Lastly, the experts were asked to provide sensory information (touch, hear, or smell)
that a surgeon utilizes for an action or decision step. All interviews were recorded in
audio and transcribed into a manuscript.
CTA Coding Scheme and Procedure
Groups of two to three trained coders utilized a coding scheme developed by
Expert Knowledge Solutions (2009), to code the interview manuscripts. The analysts’
reviewed the transcriptions and coded items (words or phrases) into one of the
following categories: action or decision steps, equipment, indications to perform the
procedure, contra-indications on when not to perform the procedure, benefits, time
and accuracy details, along with any sensory information. Additional attention was
paid to uncover if any statements concealed covert action or decision steps or other
items that were considered critical information that required coding. Overall, The
coders compared their individual coding results and resolved coding disagreements
through discussion. Inter-rater coding reliability was calculated using Cohen’s Kappa
to assess the consistency of the coding process. Once the coding of the document was
completed and an inter-rater reliability calculation indicated a 99% inter-rater
reliability, the coded information was transferred into a CTA protocol.

57. 46
CTA Protocol and creating a six-subject matter “gold
standard” CVC protocol
The coded items within each of the transcripts were formatted into individual
CTA protocols that listed all the relevant information involved in conducting the
central venous catheter placement (CVC) procedure from each expert interviewed.
Each expert reviewed their own CTA protocol for data verification and agreement.
Any necessary modification, deletions, or additions to the document were made at
that time. Upon inputting the changes provided by each expert, the experts reviewed
their own CTA protocols for verification of the edits and any other possible
corrections made by the expert. Upon final edits from the experts, the six individual
protocols were aggregated to create a gold standard CTA protocol.
The aggregation process started with combining similar statements that were
made by the experts regarding either action or decision steps. When different
statements were made regarding a similar task, they were molded into one action or
decision step (for example, “advance needle forward’ and ‘walk your needle’ are two
similar statements indicating the same action- pushing the needle forward into the
patient). Partial action steps were combined into larger steps to create complete and
efficient action steps. For example, the individual action steps of “gather materials’,
‘prepare equipment’, and ‘validate usability’ were combined to create one complete
action step. This aggregation process was completed for every section of the CTA
protocol.

58. 47
After the six-subject matter expert CVC CTA protocol was created, it went
through two rounds of editing. In the first round, each expert reviewed the six-expert
CTA protocol independently for verification of data and editing purposes. All edits
were incorporated into the final CTA protocol by the researchers. A final editing
review was conducted by one of the original experts in the presence of an analyst for
immediate verification of data and sequencing of events within the CVC procedure.
These final edits were incorporated into the six-expert CTA job aid protocol. This
final version became the ‘gold standard’, which encompassed the complete details of
the central venous catheter placement procedure.
Data Analysis
The gold standard CTA protocol was used to calculate the percentage of
agreement from each of the individual subject matter expert CTA protocols. The
individual items within the CVC gold standard CTA protocol were individually
entered into a Microsoft word spreadsheet document. Each item was manually
graded for completeness in quantifying the information obtained from each expert.
Every discrete item that appears on the individual expert’s CTA protocol and matches
the ‘gold standard’ received one point. The discrete items included conditions,
equipment, action steps, and decision steps for example. Any item that was included
on the gold standard and not the subject matter expert’s CTA protocol, received zero
points. Zero points are awarded for any item that appears on the gold standard but not
on the subject matter expert’s initial CTA report (See table 1 for an example).

59. 48
Table 1
Equipment CTA Gold
Standard
Expert A Expert B Expert C Expert D Expert E Expert F
X-ray
(A72;
B1066;
C48; F175;
D381; E92)
1 1 1 1 1 1 1
Personal
protectiv
e gear
(A91;
D60)
1 1 0 0 1 0 0
(Excerpt from CVC CTA protocol spreadsheet document.)
In an effort to provide evidence to answer the first research question, “How much
information does one expert provide when compared to a six subject matter gold
standard protocol? – an ‘acquired knowledge’ score for each subject matter expert’s
CTA protocol was calculated. An acquired knowledge score is the calculated
percentage of total knowledge obtained from each expert, per section of CTA
protocol, that was calculated based on the number of items captured from each expert
in comparison to the six subject matter expert gold standard protocol. For example,
per CTA interview, there are 30 pieces of equipment needed to complete the CVC
procedure, as identified by the six-SME gold standard CTA protocol. Each expert
provided varied amounts of equipment items in their initial interviews (i.e., Expert
“A”-23 equipment items; Expert “B”-7 equipment items; and Expert “C”-12
equipment items.) An acquired knowledge score for total equipment items was
calculated by dividing the number of items obtained by the total possible items, i.e. (#
of items per expert)/ 30). Therefore, the acquired knowledge from Expert A was

60. 49
23/30 or 77%. In other words, Expert “A” provided 77% of the total number of
equipment items needed when compared to the CTA gold standard protocol.
Acquired knowledge scores were calculated for each of the sub sections within the
gold standard protocol, including action steps, decision steps, equipment, reasons for
performing the procedure, risks for not performing the procedure correctly,
indications to perform the procedure, and contraindications on when not to perform
the procedure or in determining site selection for the CVC procedure.
To answer the second research question- how much critical information is gained
from the addition of subsequent experts- an ‘acquired knowledge’ score was
calculated for each non-repeating groups of two, three, four, five, and six subject
matter expert groups for all sub-sections of the CTA CVC protocol. This calculation
will be represented by the cumulative total number of the procedure’s items captured
from each non-repeating group of subject matter experts divided by the total number
of items in the gold standard. The groupings were created by pairing the protocols
from the experts in non-repeating combinations, A SME combination refers to the
combined captured knowledge of the stated number of experts per grouping (i.e.,
Two-pair combinations: (Experts AB; AC; AD); Three SME combinations (Experts
ABC; ABD; ABE). There were 15 non-repeating two SME pair combinations, 20
non-repeating three SME combinations, 15 non-repeating four SME combinations, 6
non-repeating five SME combinations, and one six-SME combination (see appendix
‘A’ for complete list of two through six SME non-repeating group combinations).

61. 50
Acquired knowledge scores were calculated for all subject matter combinations
by using the same procedure for the first research question: if a discrete item was
present in either of the experts CTA, that item was awarded one point and if not
present a zero will be inputted into the corresponding spreadsheet cell. Using an
excel spread sheet the total acquired knowledge for each paired SME combinations
were calculated for all 10 sections (i.e. objective, conditions, equipment, etc.) within
the CTA CVC protocol and the site location subsections (Internal jugular, subclavian,
and femoral sites). For each SME combination group an average was calculated from
adding up all the raw scores from the paired SMEs and dividing by the total number
possible for each sub-section (example- Standards). The same calculations were
performed for each subject matter combinations of three, four, five, and six expert
groups (see Table 2 for a two SME calculation for the sub-section: Standards). A sum
of the points from each of the subject matter expert pairings were divided by the total
number of gold standard items to provide the total percentage agreement with the
gold standard.
Table 2
Standards CTA Gold Standard SME A SME B Combined
The time frame for the CVC
procedure ranges from 2 minutes
(F50) -10 minutes (E86) with an
average of 5 minutes (A445;
B1101; E86)
1 1 1 1
Observable indications of success:
Chest x-ray (E92) should show:
1 0 0 0
1. The catheter is in the superior
vena cava. (B1081)
1 0 1 1
2. Clear lung fields (B1076) 1 0 1 1

62. 51
3. Easily draw blood back from the
catheter (E89)
1 0 0 0
4. Easily flush fluid into the
catheter (E90)
1 0 0 0
Total # of Standards 5 1 3 3
Percent of Standard per SME 100 20 60 60
(Excerpt from CVC CTA protocol spreadsheet document.)
(Table 2, Continued)
Since the ‘gold standard’ CTA protocol is a combination of the acquired
knowledge from all six physicians with expertise in conducting a CVC procedure, the
amount of total information acquired with each additional expert will increase up
until the sixth expert. In conducting cognitive task analysis interviews, there exists a
point of diminishing marginal return on the investment of time and human effort.
Chao & Salvendy (1994) utilized a marginal utility of 10% representing this point of
diminishing marginal return. The marginal utility of acquired knowledge was
distinguished as the change in increased acquired knowledge due to the additional
expert. In formula form, marginal utility = Change in total utility/ change in quantity.
In alignment with Chao & Salvendy (1994), the marginal utility for the current study
was set at the position where the additional acquired knowledge from an additional
expert was calculated to be less than 10%.

63. 52
CHAPTER 3: RESULTS
Conducting CTA investigations constitutes significant investments in time and
effort to complete. An unanswered barrier to conducting a CTA research project is
deciding on the number of experts one would need to interview. Each expert
interviewed consists of a dedicated amount of time and effort to procure an expert,
capture their knowledge, analyze the findings, and represent their knowledge in a
working document. Establishing a standard number of experts to interview will
reduce the cost-benefit of conducting CTA investigations by providing future analyst
a set standard equated with reliable data. Knowing a set number of SMEs to interview
will minimize extraneous cost and effort from interviewing too many experts.
Coding and Inter-Rater Reliability
Each coder was instructed to review the transcript to capture the overall objective
of the procedure, any conditions for performing or contraindications, as well as noting
required action steps, decision steps, equipment, and the major tasks necessary to
perform the procedure. The analysts compared all documentation coding and the
inter-rater reliability was calculated at 99.46%. Each protocol document was
formatted into separate CTA protocols. Each CTA protocol was reviewed and edited
by the expert who was interviewed. The six individual CTA job aids were aggregated
to create a six-expert gold standard CTA CVC placement procedure protocol. Table 3
indicates the CVC gold standard CTA protocol subsections and the number of items
within each sub-section. The total items were calculated through adding the number
of individual concepts, actions, or decisions within each sub-section.

64. 53
Table 3: CVC Gold Standard Sections and Corresponding Number of Items per
Section.
Gold Standard Item Total Number Gold Standard
Item
Total Number
Objective 1 Standards 5
Risks 17 Equipment 30
Reasons 3 Tasks 8
Indications 4 Action Steps 44
Contraindications 4 Decision Steps 14
Research question #1: How much information about a central venous
catheter placement procedure does a single expert provide when compared to a six-
subject matter expert gold standard protocol?
The six-expert gold standard protocol was transferred to an Excel spreadsheet for
analysis. For each individual expert CTA protocol, a value of 1 was given to each
discrete item that represented a single action, decision step, condition, equipment, etc.
A column in the excel spreadsheet was established for each of the six expert CTA
protocols. The total score from each expert was calculated by adding all the values
together. The percentage of acquired knowledge was calculated by dividing the
obtained total for each of the six individual scores by the total possible number of
items from the six-expert CVC gold standard protocol. Sub-totals were also
calculated for the procedures action steps, decision steps, conditions, equipment,
reasons for performing the procedure, and risks for not performing the procedure
correctly.
To answer the first research question, the results of the study are displayed in
Table 4, which indicates the percentage of acquired knowledge obtained from

65. 54
individual experts in comparison to the six-expert CTA CVC “gold standard”
protocol based on a single CTA interview. The range of total knowledge acquired
from a single expert when compared to a six-subject matter expert gold standard was
43% to 73% with an average of 57%. The range of acquired action steps from a
single expert when compared to a six-subject matter expert gold standard from a
single from 50% to 89% with an average of 70%. The average acquired knowledge
for decision steps from a single expert when compared to a six-subject matter expert
gold standard was 65% with a range of 57% to 71%.
Table 4
Percentage of Knowledge Acquired from One Expert When Compared to a Six
Subject Matter Expert Gold Standard Protocol
SME 1 SME 2 SME 3 SME 4 SME 5 SME 6
Total
Knowledge
Acquired
65 48 43 73 61 54 57
Action Steps 75 66 50 89 70 70 70
Decision Steps 57 71 64 64 71 64 65
Objectives 100 100 100 100 100 100 100
Reasons 0 0 67 0 67 33 28
Risks 35 0 0 82 0 0 20
Indications 75 75 75 75 75 75 75
Contra-
indications
50 50 25 25 50 75 46
Standards 20 60 0 0 60 0 23
Recommended
Equipment
77 23 40 67 63 50 53