2. Polly Tremoulet, PhD.
Lockheed Martin
Advanced Technology Laboratories
Performance Augmentation through
Cognitive Enhancement (PACE)
HCI International / AugCog International
July 25, 2005
3. Overview
• Background
– Augmented Cognition
– Sensors and Cognitive States
– Mitigation Strategies
• System
– Design Goals
– Component descriptions
– Task selection
– Modality selection
• Ongoing and future work
4. Sensors
Background: Augmented Cognition
• Goal: Maximize operator cognitive performance in
dynamic, complex operational environments
• Approach: Biophysical sensor technology assesses
operator cognitive state
– Detects, predicts, avoids overload to reduce
operator error and maximize effectiveness
• Benefit: Improve operator
performance
– Increase situation
understanding
– Reduce errors
– Improve accuracy
Domain
Simulation
Cognitive
State
Assessor
User
5. Current Suite of Sensors
• EEG
– Placement:
• Monopolar placement of sensors along middle of head
• Bipolar placement of sensors on both sides of the head
– Sensors: Electrodes
– Preprocessing: None
• EKG
– Placement: Traditional placement on left and right shoulders and abdomen
– Sensors: Electrodes
– Preprocessing: None
• GSR
– Placement: Non-traditional placement on toes rather than fingers
– Sensors: Electrodes
– Preprocessing: None
• Down-selection criteria:
– Correlation with performance
– Interoperability with other sensors
– Physical discomfort for users/subjects
– Portability and robustness in operational environments
7. Mitigation Strategies
• Pacing
– Delegation
– Defer
– Decomposition
• Intelligent Sequencing
– Ordering based on modality and priority
• Modality switching
– Changing presentation modality based upon capacity
• Multi-modality reinforcement
8. Initial Gauge and Mitigation Options
Gauge Trigger Mitigation logic
Workload Above
threshold
Pacing = change timing of Secondary tasks
Decomposition = break down Primary and/or
Secondary tasks
Arousal Below Range
Above
threshold
Request attention / alert
Offload/delegate work
Offload/delegate work
Decomposition
Spatial
WM
High compared
to Verbal WM
Sequencing
Verbal Modality Shift
Chunking
Verbal WM High compared
to Spatial WM
Sequencing
Spatial Modality Shift
Chunking
9. PACE High-Level Software
Architecture
External
Application
User Environment
Director (ED)
Active Task
Manager
(ATM)
Task
Information
Manager (TIM)
Cognitive
State
Assessor
(CSA)
System
Interface
Director
(SID)
Adaptive
Workload
Director
(AWD)
Delegation
Manager (DM)
Delegated
Tasks
Proposed
Tasks
Cognitive
State
Sensor
Data
Task
Interactions
Presented
Tasks
Proposed
Tasks
New
Tasks
User
Actions
User
Performance
Configuration
Files
Task
Attributes
10. Overarching Architectural
Concepts• Domain Neutrality
– In order to provide the most generally useful and reusable system, as many
components as possible are written without reference to domain.
– Configuration files allow tasks, priorities, and application information to be
specified per-domain
– Certain components include domain-specific extensions to manage
domain-specific logic
• Component Separability
– CommsProvider interface allows easy exchange of underlying
communications layer
– All components operate independently, subscribing for and publishing
particular types of messages through CommsProvider
– Allows reconfiguration of system to separate machines and eases
integration with other applications
11. Configuration Files
• Purpose: Allow per-domain and
run-time configuration of tasks
• Used primarily by TIM but also used
by Environment Director and potentially others
• XML-based formats for each configuration file:
– Augmentation – configuration and selection of mitigation strategies
– Modalities – specification of modalities in which tasks may be
presented
– Presentation – specification of modalities supported by external
applications
– Priorities – assignment of priorities and urgency of different types of
tasks
App
User ED ATM TIM
CSA SID AWD DM
Conf
12. Task Information Manager
• Purpose: Manage the creation,
evaluation, and decomposition of
individual tasks
• Creates new tasks in response to external stimulus
• Implementation for TTWCS experiments creates tasks based on a
scenario script
• Monitor performance of the user to provide feedback and potentially
influence mitigations
• Perform task decomposition and combination (not currently being used
as a mitigation)
App
User ED ATM TIM
CSA SID AWD DM
Conf
13. Adaptive Workload Director
• Purpose: Manage the set of tasks
awaiting user attention
• Maintains priority-based queue of
pending tasks
• Maintains dependency graph indicating tasks which are dependent upon
the completion of other tasks before they may be presented to the user
• Proposes tasks to present to the System Interface Director
• Tasks are proposed upon completion of a task, rejection of a proposed task,
and on a periodic update (10 sec.)
• Tasks to propose are selected based on their priority and how long they’ve
been waiting in the queue
• Tasks which are rejected can be replaced on the queue, sent to the TIM for
decomposition into smaller tasks, or sent to the Delegation Manager for the
task to be handled elsewhere
App
User ED ATM TIM
CSA SID AWD DM
Conf
14. Delegation Manager
• Purpose: Reassign tasks to a peer,
either another human user or an
intelligent agent
• The functionality of the DM is not
being used for TTWCS, as only a single
operator is responsible for handling all tasks
App
User ED ATM TIM
CSA SID AWD DM
Conf
15. System Interface Director
• Receives periodic updates of cognitive state from Cognitive State
Assessor
• Receives task proposals from the Adaptive Workload Director
• Using cognitive state and currently active mitigation strategies
decides whether to accept the proposed task or to reject the task,
sending it back to the Adaptive Workload Director
• Accepted tasks are passed on to the Environment Director
App
User ED ATM TIM
CSA SID AWD DM
Conf
•Purpose: Perform mitigations based
on the current cognitive state of the user
16. • Purpose: Evaluate the current
cognitive state of the user
• Currently implemented as Proxy to
Labview implementation
• Labview performs data exchange with sensor systems via established
protocol and executes neural network function
• Gauge values are sent out of CSA to the System Interface Director
• Also includes capability to provide current performance as inputs to
neural network, but this is not currently used in TTWCS domain
App
User ED ATM TIM
CSA SID AWD DM
Conf
Cognitive State Assessor
17. Environment Director
• Purpose: Manage the presentation of
tasks through the external application
• Monitors the modalities currently
being used on all external applications by tasks which currently have user
attention
• Receives proposed tasks from the System Interface Director
• Examines tasks and attempts to select a presentation modality based on
the task’s preferred modality as well as the application’s modality
capabilities
• If no available modality can be found to successfully present the task, it
will be rejected and sent back to the Adaptive Workload Director
App
User ED ATM TIM
CSA SID AWD DM
Conf
18. Active Task Manager
• Purpose: Manage the progression of
actions associated with individual tasks
• Receives newly presented tasks and
user actions associated with tasks from Environment Director
• Determines the appropriate next step in the task whenever a user takes
an action, sending out system actions to the Environment Director
• For TTWCS, interacts with Expert Model to generate a score of the user’s
performance on completed tasks
App
User ED ATM TIM
CSA SID AWD DM
Conf
19. jTTWCS Application
• Purpose: Provide to the operator an
interface through which experimental
tasks may be performed
• Recognize and forward user-initiated actions
– Alert Responses
– Retargetting Solutions
– Coverage Indications
• React to system-initiated actions
– Begin New Scenario
– Add Emergent Target
– Display Alert Question
• Provide Expert Model to score user responses for each type of task
App
User ED ATM TIM
CSA SID AWD DM
Conf
20. Launch Area
Preplanned
Health and Status
points
Primary (Default)
d- Target
Guidance
Waypoint
Loiter Pattern
Alternate (Flex)
f-Target
Branch
Point
Time-critical
(emergent) e-
Target
The Tactical Tomahawk cruise missile represents the next generation of
cruise missiles with:
–On-board mission planning
–Inflight retargeting
–Battle damage assessment
This weapon will now be able to service high-priority, time-critical
targets, more quickly and effectively.
3. Emergent (e-target) Missions
1. Default (d-target) Missions
2. Flex (f-target) Missions
Tactical Tomahawk Application Domain
21. TTWCS Problem Space: increasing
cognitive demands
• Launch Area Coordinator (LAC) acting as strike controller
will need to:
– Review Exception Reports
– Re-allocate missions to shooters on ships
– Review Waiver Reports
– De-conflict and re-allocate missiles & air tracks
– Review shooter casualty reports
– Re-allocate and order backup
– Monitor missiles
– Re-target and Re-strike
• Apply learned heuristics:
• Who’s in range? Who’s been on station longer? Who will be off-
station earliest? What is my resource availability?
22. Task Selection in PACE
• Tasks are inserted by application
or TIM’s task generator
• Tasks are decomposed into forest of subtasks, as
needed
– E.g. two button clicks two trees
• (Sub)tasks assigned priorities and inserted into a queue
– Prioritization function of insertion time, urgency, etc
• Proposed tasks are examined by SID and compared to
CSA’s most recent assessment of cognitive workload
• Appropriate modality for next task in queue is selected
App
User ED TIM
CSA SID AWD DM
Conf
ATM
23. Modality Selection in PACE
• Each task is defined with a preferred modality
– E.g. alerts prefer text-window panels, but may be
delivered via speech
• Application interface specifies all possible modalities
for each task and quality rankings for each modality
• SID examines available modalities and proposed task.
– Task rejected if no slots available in any possible modality,
o/w
• SID accepts task and designates it for modality of
greatest utility
– Utility = combination of task preference and application’s
modality quality and user’s cognitive capacity for task
24. Additional LM ATL Components
Developed
• Log Analyzer
– Data combined from multiple ACES XML log files into one, easy to read
spreadsheet
– ACES logs quickly distributed to Sub-Contractors
• Scenario Generator
– Enables realistic, rapid creation of scenarios by all groups
– Playback enables review of scenarios at different speeds
• ACES (AugCog Experimental System) Launcher and Distribution Tool
– Every component of the ACES system can be started up or shutdown by
pressing a button
– Simple install script
• All required libraries are included
• Runs “out of the box” with no compilation or compatibility issues
25. Future Directions
• Mitigation Strategy research
– Appropriate application of delegation
– Multi-modal reinforcement strategies
• Using task context to control application of
mitigations
• Transitioning PACE to the field:
– HCI evaluation: work in progress
– Training operators to use complex applications
– Improving command and control operator
performance in operational environments
26.
27. Why this is NOT just Advanced
HCI
Cognitive Model
Measured
Verbal Task
Performance is
Optimal
Task System
inhibits
Mitigation
CSA
Hysteresis
and
Smoothing
Neural
Network
Verbal Gauge
Spatial Gauge
Sensors
Verbal
Only Task
1.Anticipates when gauge will
reach threshold
2.Threshold is set to avoid
becoming seriously
overloaded.
Task System
turns on
Mitigation
28. PACE Architecture
Cognitive
Workload
Assessor
External
Actuators/
Sensors
Tasks delegated
to other operators
or software agents
New
Tasks
Maintains a virtual work
environment that
optimizes
communication
between operator and
machine
Maintains
definition and state
of all operator
tasks both current
and historic
Measures the
operator’s ability to
handle the current
and projected
workload
Optimizes
presentation of
current tasks within
the operator’s virtual
work environment
Maintains a plan that
optimizes the
operator’s ability to
handle the current
workload
Human Work Space Task Space External
Task
Information
Manager
System
Interface
Director
Environment
Director Operator
Actions
Domain & Application Independent
Domain & Application Dependent
Adaptive
Workload
Director
• Manages Tasks, Alerts
and Contexts
• Monitors User
Performance
• Listens to Cognitive
Workload Level
• Directs Cognitive
Augmentations
– Sequencing
– Pacing
– Modality Shifts
– Chunking
– Delegation
29. LM ATL AugCog Environment:
Augmented Cognition Experimental System – ACES
• Experiment environment
– Controlled
– Repeatable
– Scorable
– Portable
• Provides realistic and discrete events
• Isolates memory-intensive tasks
• Separable spatial and verbal activities
• Modular: able to gradually increase
complexity
31. Sensor Data Processing Systems
• Sensor Data Processing Systems connect directly to a set of sensors
• Minimal processing is performed on that data, producing a periodic report
on all pertinent sensor values
• Sensor data is passed through the network to the CSA System
Sensors
Sensor Data
Processing
System
Semi-
Processed
Sensor
Data
CSA
System
32. CSA System
• The CSA System receives sensor data from the various Sensor Systems.
• Using an ANN, these sensor values are processed into a set of Gauge
values.
• Current gauge values are periodically sent to the Task System to affect its
mitigation strategy.
CSA
System
Semi-
Processed
Sensor
Data
Task
System
Gauge
DataSensor Data
Processing
System
33. Task System
• Task System receives Gauge Data from CSA System.
• Based on current Cognitive State, additional tasks are proposed to the user
or rescheduled if Cognitive State indicates a potential overload
• Tasks which are deferred due to Cognitive State are retained and re-
proposed at a later time when the user’s state is more conducive to
completing that task.
Task
System
Tasks
CSA
System
Gauge
Data
User
34. Neural Network Vital Statistics
• Inputs: 234 excluding fNIR, 252 including fNIR
– For each feed, 3 inputs: now, 0.5 sec ago, 1.0 sec ago
– GSR: 1 x 3
– IBI: 1 x 3
– fNIR: 6 x 3
– EEG: 74 x 3
• Combination of direct measurements and calculated values such
as vigilance
– Pupillometry: 2 x 3
• Outputs: 2
– Spatial Working Memory
– Verbal Working Memory
• Hidden/Internal Nodes: 200 (single hidden layer)
• Type: Feed-forward
• Training Method: Standard Back-propagation
35. Building the Neural Network
• Data Collection
– Collected data during several scenario runs for 3 subjects
– Subjects performed same types of tasks to be used during CVE
• Training
– Untrained network created in Stuttgart Neural Network Simulator
(SNNS)
– SNNS provided with data files from scenarios
– 1000 training epochs executed
• Standard back-propagation, no momentum factor, learning rate =
0.2
– Resulting network converted to C-function to be embedded within
Labview sensor pre-processing system
• Other experimentation
– Other networks and training methods were attempted and this was the
best combination found
36. Using the Neural Network
• CSA System
– Reads sensor values
– Passes them to them to the Neural Network every 0.5 seconds
• Neural Network
– Processes sensor data and returns gauge value estimations
• PACE System Interface Director
– Examines current cognitive state
– Perform hysteresis and smoothing on cognitive state values
• If user has been in high verbal memory workload for at least 5
seconds, postpone low-priority verbal tasks
• If user has been in high spatial memory workload for at least 5
seconds, postpone low-priority spatial tasks
37. Task Description and Stimuli
Retarget task
– Reassign missiles to service higher priority emergent targets instead of
their default target destinations.
• Goal is to service as many emergent targets as possible, while
maintaining coverage on as many high and medium default targets
as possible.
– Tactical Targeting
Alert task
– Respond to questions from a commanding officer about an ongoing
strike
– Commander and Team Online Interruptions
Location task
– Upon Inquiry, determine what targets can/cannot be covered based
on missile coverage zones.
– Situation Awareness
38. Benefits of AugCog in TTWCS domain
• Augmented Cognition system in TTWCS environment will increase
operator performance
– Number of missiles simultaneously monitored
– Number of alerts handled successfully
– Overall number of emergent targets handled correctly
– Enable operators to employ new capabilities effectively:
• Redirection and flex missions
• Multiple engagements
• Overlapping strike packages
• Augmented Cognition system in TTWCS environment will reduce
manning
Editor's Notes
The simultaneous occurrence of tasks between these major groups of tasks will really produce cognitive bottlenecks. The TSC must have the “battlegroup” and joint force persepectives.
