Cognitive Task Analysis of Teams Yogesh Tadwalkar March 22, 2004

Yogesh Tadwalkar

March 22, 2004

Traditional Task Analysis Task Analysis aims to optimize work performance by matching tasks with human capabilities and limitations Human work is comprised of both physical and cognitive activities Physical tasks are observable (Overt) However, cognitive processes that lead to physical actions are not (Covert)

Task Analysis aims to optimize work performance by matching tasks with human capabilities and limitations

Human work is comprised of both physical and cognitive activities

Physical tasks are observable (Overt)

However, cognitive processes that lead to physical actions are not (Covert)

Cognitive Task Analysis (CTA) Extension of traditional task analysis techniques to uncover information about: - Work domain, mental demands on operator - Goal Structures - Knowledge and Strategies - Thought Processes and Decision Making Thus, providing a holistic, complete view of human work

Extension of traditional task analysis techniques to uncover information about:

- Work domain, mental demands on operator

- Goal Structures

- Knowledge and Strategies

- Thought Processes and Decision Making

Thus, providing a holistic, complete view of human work

Goals of CTA Performance improvement through: Better Task and Interface Design Ideal Personnel Selection Comprehensive Training and Instructions Systematic Performance Assessment Improved Planning

Performance improvement through:

Better Task and Interface Design

Ideal Personnel Selection

Comprehensive Training and Instructions

Systematic Performance Assessment

Improved Planning

Cognitive Task Analysis of Teams

Why Conduct Team CTA? Because 2 + 2 ≠ 4 Individuals are often required to perform in teams Optimum individual performances may not lead to final mission accomplishment if team co-ordination is unsatisfactory Thus, individual CTA would not be sufficient for designing optimum task performance

Because 2 + 2 ≠ 4

Individuals are often required to perform in teams

Optimum individual performances may not lead to final mission accomplishment if team co-ordination is unsatisfactory

Thus, individual CTA would not be sufficient for designing optimum task performance

Team CTA As defined by Klein: A reflection and representation of the team mind / Knowledge Description of the cognitive skills required for effective team performance Elaboration of the team decision requirements Determination of types of expertise found in effective teams

As defined by Klein:

A reflection and representation of the team mind / Knowledge

Description of the cognitive skills required for effective team performance

Elaboration of the team decision requirements

Determination of types of expertise found in effective teams

Objectives of Team CTA Team performance enhancement through: Restructuring and Resizing the team Designing better: Information technology Information management strategies Human-computer interfaces Decision support systems and Communication Developing methods for team training and selection

Team performance enhancement through:

Restructuring and Resizing the team

Designing better:

Information technology

Information management strategies

Human-computer interfaces

Decision support systems and Communication

Developing methods for team training and selection

Team Environment

Team Environment Tasks are decomposed and assigned to different team members Team members may or may not have direct contact with each other (co-located vs. dispersed teams) Tasks can be performed serially or in parallel or overlapping fashion Team members possess, acquire and process information, make decisions, solve problems and make plans

Tasks are decomposed and assigned to different team members

Team members may or may not have direct contact with each other (co-located vs. dispersed teams)

Tasks can be performed serially or in parallel or overlapping fashion

Team members possess, acquire and process information, make decisions, solve problems and make plans

Team Environment Teams are intelligent entities (Thordsen and Klein, 1989) Team members possess, acquire and process information Communicate with each other Make decisions Solve problems and Make plans

Teams are intelligent entities

(Thordsen and Klein, 1989)

Team members possess, acquire and process

information

Communicate with each other

Make decisions

Solve problems and

Make plans

Types of Teams Planning Teams Military Command Emergency Response at nuclear plants Action Teams Air Traffic Control SWAT

Planning Teams

Military Command

Emergency Response at nuclear plants

Action Teams

Air Traffic Control

SWAT

Cognitive Processes of Teams What to measure through Team CTA methods

Cognitive Processes of Teams Methods for team CTA should focus at capturing these 5 processes (Klein): 1. Shared mental models - Team Goals, team member roles 2. Control of attention - Information seeking, filtering, communication 3. Shared situation awareness - Assessment of dynamism, prediction of future 4. Application of strategies - To make decisions, solve problems and plan 5. Metacognition - Self-monitoring, sensing trouble, making alternative plans

Methods for team CTA should focus at capturing these 5 processes (Klein):

1. Shared mental models

- Team Goals, team member roles

2. Control of attention

- Information seeking, filtering, communication

3. Shared situation awareness

- Assessment of dynamism, prediction of future

4. Application of strategies

- To make decisions, solve problems and plan

5. Metacognition

- Self-monitoring, sensing trouble, making alternative plans

A Model for Team Cognition Knowledge Mental Models Attitudes Expectations Skills Member A Member C Member B Team Performance Pre-task Knowledge Shared Mental Models Strategies Dynamic Knowledge Control of Attention Shared Situation Awareness Metacognition (Stout, Cannon-Bowers, and Salas, 1996)

Pre-task Knowledge

Shared Mental Models

Strategies

Dynamic Knowledge

Control of Attention

Shared Situation Awareness

Metacognition

1. Shared Mental Models The degree to which team members have the same understanding of key processes (Cannon-Bowers, Sales, & Converse, 1992) - Knowledge about overall team goals and mission objectives e.g. Football game - Roles and responsibility of each team member - Team-mate characteristics and preferences - Task Procedures, sequences and timing - Use and relationship of equipment

The degree to which team members have the same understanding of key processes (Cannon-Bowers, Sales, & Converse, 1992)

- Knowledge about overall team goals and mission objectives e.g. Football game

- Roles and responsibility of each team member

- Team-mate characteristics and preferences

- Task Procedures, sequences and timing

- Use and relationship of equipment

Shared Mental Models and Team Performance Shared mental models affect team performance significantly (Oransu, 1990) : Anticipating team member behaviours and information requirements Performing tasks from a common frame of reference Attending to, interpreting, communicating about, and responding to the world more similarly than individuals with discrepant knowledge (Rentsch & Hall, 1994)

Shared mental models affect team performance significantly (Oransu, 1990) :

Anticipating team member behaviours and information requirements

Performing tasks from a common frame of reference

Attending to, interpreting, communicating about, and responding to the world more similarly than individuals with discrepant knowledge (Rentsch & Hall, 1994)

Methods to Elicit and Represent Shared Mental Models

Variation of Goal-Directed Analysis (Woods & Hollnagel, 1987) To solicit knowledge of Team Goals and Task Objectives Document Analysis and Observation Separate interviews with each team member, followed by group interview Representation: Agreement Metric Task Goal Sub-Goal 1 Sub-Goal 2 Sub-Goal 3 Task 1 Agreement Metric Team Goals For Each Member For Whole Team Task 2

Document Analysis and Observation

Separate interviews with each team member, followed by group interview

Representation: Agreement Metric

Variation of Task-Action Mapping (Coury et al., 1991) To solicit knowledge of Task Procedures, Sequence and Timing Document Analysis and Observation Ask each team member to describe key team concepts, associated tasks and specific actions Representation: Concept Maps Description Task 1 Task 2 Task 3 Action 1 What (Steps) Who When Concept 1 e.g. Evacuation Action 1 What (Steps) Who When

Document Analysis and Observation

Ask each team member to describe key team concepts, associated tasks and specific actions

Representation: Concept Maps

Action 1

What (Steps)

Who

When

Action 1

What (Steps)

Who

When

Bootstrapping (Hoffman, Shadbolt, Burton & Klein 1995) To solicit knowledge of Roles and Responsibilities Bootstrapping Documentation Individual Interviews Observation Group Interviews May not adequately capture dynamic, contextual data like misinterpretations, omissions due to environmental factors

Bootstrapping

Adapted Critical Incident Method (Flanagan, 1954; Critical Decision Method by Klein, 1989) To solicit knowledge of Roles and Responsibilities, Strategies, Attitudes and Personal Characteristics Document Analysis and Recall of a critical past event e.g. fire, accident, high-stake mission. Ask each team member to describe various decisions taken, action triggers, who did what vs. who was supposed to do what Describe how each teammate reacted to the situation (perception, prediction, response time, stress, flexibility in co-ordination, etc.)

Document Analysis and Recall of a critical past event e.g. fire, accident, high-stake mission.

Ask each team member to describe various decisions taken, action triggers, who did what vs. who was supposed to do what

Describe how each teammate reacted to the situation (perception, prediction, response time, stress, flexibility in co-ordination, etc.)

2. Control of Attention The way a team engages in information management - Information seeking - Filtering irrelevant information - Allocating attention to important functions - Properly distributing messages Refers to the team’s working memory The way it uses limited resources for processing simultaneous messages (Klein)

The way a team engages in information management

- Information seeking

- Filtering irrelevant information

- Allocating attention to important functions

- Properly distributing messages

Refers to the team’s working memory

The way it uses limited resources for processing simultaneous messages (Klein)

Methods to Elicit and Represent Control of Attention

Scenario based Questionnaires (Blickensderfer, 1997) To solicit knowledge of Information Management Strategies, and Communication Skills Document Analysis, observations and interviews to create a questionnaire covering a team work scenario, given to each team member Answers to outline relevant information protocol for the scenario, missing information, cues for identifying missing data, strategies to acquire and distribute information, and possible communication bottlenecks

Document Analysis, observations and interviews to create a questionnaire covering a team work scenario, given to each team member

Answers to outline relevant information protocol for the scenario, missing information, cues for identifying missing data, strategies to acquire and distribute information, and possible communication bottlenecks

3. Shared Situation Awareness Extent to which team members have the same interpretation of ongoing events (Cannon-Bowers, Sales, & Converse, 1992) Is giving full data to each member a remedy for discrepant situation Awareness? How different team members perceive dynamic situations and events How they form divergent impressions e.g. police in a riot situation How they update each other

Extent to which team members have the same interpretation of ongoing events (Cannon-Bowers, Sales, & Converse, 1992)

Is giving full data to each member a remedy for discrepant situation Awareness?

How different team members perceive dynamic situations and events

How they form divergent impressions e.g. police in a riot situation

How they update each other

Methods to Elicit and Represent Shared Situation Awareness

Adapted Situation Awareness Global Assessment Technique (SAGAT) (Endsley, 1995) To solicit knowledge of compatible situation awareness Bootstrapping to create a mission simulation Asking team members to perform mission Freezing mission after specific intervals Asking each team member to analyze the situation and predict outcome and strategize Compare results for all members and assess compatibility and reasons for discrepancy

Bootstrapping to create a mission simulation

Asking team members to perform mission

Freezing mission after specific intervals

Asking each team member to analyze the situation and predict outcome and strategize

Compare results for all members and assess compatibility and reasons for discrepancy

Situation Awareness Linked Indicators Adapted to Novel Tasks (SALIANT) (Muniz et al. , 1998) To solicit knowledge of compatible situation awareness 1. Delineate behaviours theoretically linked to team situations (24 behaviours, 5 clusters). Being able to predict next event Managing resources (technical, system , internal, team) Managing time Feeling of being in control; able to implement elegant solutions Taking the right decision at the best moment Detecting mismatches, etc. 2. Development of task scenarios 3. Identification of specific, observable responses for each scenario 4. Development of a script (to be administered to a team member) 5. Observation form: Presence or Absence of response behavior

To solicit knowledge of compatible situation awareness

1. Delineate behaviours theoretically linked to team situations (24 behaviours, 5 clusters).

Being able to predict next event

Managing resources (technical, system , internal, team)

Managing time

Feeling of being in control; able to implement elegant solutions

Taking the right decision at the best moment

Detecting mismatches, etc.

2. Development of task scenarios

3. Identification of specific, observable responses for each scenario

4. Development of a script (to be administered to a team member)

5. Observation form: Presence or Absence of response behavior

4. Application of Strategies Every team has a prepared list of steps, and routines for decision making, problem solving and planning However, a team also develops shortcuts and workarounds not codified in procedures, as it gains experience A skillful team knows its: Key decision makers (to be supported) Key sources of expertise (to be consulted) Time is critical in strategy execution

Every team has a prepared list of steps, and routines for decision making, problem solving and planning

However, a team also develops shortcuts and workarounds not codified in procedures, as it gains experience

A skillful team knows its:

Key decision makers (to be supported)

Key sources of expertise (to be consulted)

Time is critical in strategy execution

Methods to Elicit and Represent Team Strategies

Retrospective Protocol Analysis (Means, 1993) To solicit knowledge of agreement on strategies Based on non-interruption Use video-tapes of similar mission or actual team performance recorded previously Asking each team member to analyze the situation and justify strategies taken Analyze team agreement

Based on non-interruption

Use video-tapes of similar mission or actual team performance recorded previously

Asking each team member to analyze the situation and justify strategies taken

Analyze team agreement

5. Metacognition A team needs to monitor itself during ongoing task performance To determine whether and when it is running into difficulty Predict consequences of the difficulty Determine when and how it needs to shift its strategies e.g. Police at riot situation

A team needs to monitor itself during ongoing task performance

To determine whether and when it is running into difficulty

Predict consequences of the difficulty

Determine when and how it needs to shift its strategies e.g. Police at riot situation

Methods to Elicit and Represent Metacognition

Mission Simulation (Klein) To solicit knowledge of self-monitoring or metacognition Bootstrapping to create a mission simulation Asking team members to perform mission Observing how team members recognize and collect more information about critical incidents Conducting individual interviews with each team member to analyze the video-recording Analyzing gaps in team agreement

Bootstrapping to create a mission simulation

Asking team members to perform mission

Observing how team members recognize and collect more information about critical incidents

Conducting individual interviews with each team member to analyze the video-recording

Analyzing gaps in team agreement

Representing Team CTA Results

Methods for Representing Team CTA Results Dealing with the ‘Envisioned World Problem’ (Woods, in press) Creating prototypes and simulations to gauge how team CTA would impact the future Iterative testing World Team CTA Prototypes of Changed World

Dealing with the ‘Envisioned World Problem’ (Woods, in press)

Creating prototypes and simulations to gauge how team CTA would impact the future

Iterative testing

Limitations / Challenges in Applying Team CTA Methods

Challenges in Applying Team CTA Methods Knowledge elicitation Knowledge is internalized and even experts find it hard to verbalize it Retrospective account of work may differ from reality Capturing dynamism In many cases, team strategies differ vastly from one situation to another Generalizing based on a few situations may be inappropriate

Knowledge elicitation

Knowledge is internalized and even experts find it hard to verbalize it

Retrospective account of work may differ from reality

Capturing dynamism

In many cases, team strategies differ vastly from one situation to another

Generalizing based on a few situations may be inappropriate

Summary Team CTA is critical to obtain a holistic (both overt and covert tasks) view of human work Teams are intelligent entities with 5 key cognitive processes (Klein) Shared mental models Control of attention Shared situation awareness Application of strategies to make decisions, solve problems and plan Metacognition

Team CTA is critical to obtain a holistic (both overt and covert tasks) view of human work

Teams are intelligent entities with 5 key cognitive processes (Klein)

Shared mental models

Control of attention

Shared situation awareness

Application of strategies to make decisions, solve problems and plan

Metacognition

Summary The 5 key processes can be spread over 2 types of team knowledge Pre-task knowledge Dynamic Knowledge Team CTA Methods should aim at eliciting team knowledge of these 5 processes

The 5 key processes can be spread over 2 types of team knowledge

Pre-task knowledge

Dynamic Knowledge

Team CTA Methods should aim at eliciting team knowledge of these 5 processes

Summary Team CTA can yield significant performance improvement of teams by: Restructuring and Resizing the team Designing better: Information technology Information management strategies Human-computer interfaces Decision support systems and Communication Developing methods for team training and selection

Team CTA can yield significant performance improvement of teams by:

Restructuring and Resizing the team

Designing better:

Information technology

Information management strategies

Human-computer interfaces

Decision support systems and Communication

Developing methods for team training and selection

References Gary Klein Cognitive Task Analysis of Teams (1995) Cannon-Bowers, Sales, Baker Analyzing knowledge requirements in team tasks Shared Mental Models in Expert Team Decision Making (1993) Hoffman, R.R. Use of Critical Incidents to Elicit Knowledge (1995) Oransu, J. Shared Mental Models and Crew Performance (1990) Coury, B.G., Motte,S., & Seiford, L.M. Capturing and representing decision processes in the design of an information system Muniz, E.J., Stout, R.J., Bowers & Salas A methodology for measuring team situation awareness: Situation Awareness Linked Indicators Adapted to Novel Tasks (SALIANT) (1998) Woods, D.D. & Hollnagel, E. Mapping cognitive demands in complex problem solving worlds (1987)

Gary Klein

Cognitive Task Analysis of Teams (1995)

Cannon-Bowers, Sales, Baker

Analyzing knowledge requirements in team tasks

Shared Mental Models in Expert Team Decision Making (1993)

Hoffman, R.R.

Use of Critical Incidents to Elicit Knowledge (1995)

Oransu, J.

Shared Mental Models and Crew Performance (1990)

Coury, B.G., Motte,S., & Seiford, L.M.

Capturing and representing decision processes in the design of an information system

Muniz, E.J., Stout, R.J., Bowers & Salas

A methodology for measuring team situation awareness: Situation Awareness Linked Indicators Adapted to Novel Tasks (SALIANT) (1998)

Woods, D.D. & Hollnagel, E.

Mapping cognitive demands in complex problem solving worlds (1987)

Questions ?

Cognitive Task Analysis of Teams Thank You !