1. Project Echo and
Speech Recognition Software
Matthew N. Gray
NIFS Student Intern – Summer 2016
Airspace Operations and Safety Program
Crew Systems and Aviation Operations Branch
Mentors: Dr. Angela Harrivel, Chad Stephens
1

2. Crew State Monitoring (CSM) – Motivation
• National airspace is becoming increasingly
busy and more complex.
• If we could stop just one fatal commercial
aviation accident, we could save hundreds
of lives and a billion dollars.
• Loss of Control – Inflight (LOC-I)
o Largest Category of aircraft-related Fatal
Events
• CSM aims to improve pilot operational
efficiency during safety-critical operations
by:
o Improving the human-machine interface in
aircraft
o Aiding in pilot attention training 3 Boeing Statistical Summary of Commercial Worldwide Jet
Transport Accidents, 2011. Includes only accidents involving
turbofan or turbojet airplanes with max takeoff weight > 60,000
lbs., referenced in the CAST Airplane State Awareness Joint
Safety Analysis Team Final Report, June 17, 2014
2

3. Background – Crew State Monitoring (CSM)
• Data Collection
o fNIRS, EEG, EKG, Resp., GSR, Eye-
tracking, etc.
• Data Synchronization
o MAPPS Software
• Signal Pre-Processing
o Filtering, Feature Extraction
• Machine Learning
o Classification Algorithm
o Model Evaluation
• Real-time State Indicator
Display
o High/Low Workload, Distraction,
Inattentional Blindness, Confirmation
Bias, etc.
Figure courtesy of Charles Liles
and the LaRC Big Data and Machine Information Team3

4. Background – AFDC 2.0 and SHARP 1.0
• Augmented Flight Deck
Countermeasures (AFDC
2.0)
o Seeks to improve human-
machine interaction among
safety critical operations of
airplanes
• Scenarios for Human
Attention Recovery using
Psychophysiology (SHARP
1.0)
o Supports CAST goals by
measuring crew cognitive
states in simulators during
safety critical operations SHARP Display
4

5. Muse
SmartEye
Spire
Empatica E4
BIOPAC fNIR100B
EEG
fNIRS
Heart Rate (PPG)
GSR
Temp.
Accel.
Respiratory Rate
Eye-tracking
Project Echo
Sensors
Data
Acquisition
Data
Synchronization
MAPPS or MAPPS Equivalent:
• Lab Streaming Layer (LSL)
• iMotions
• XTObserver
Signal
Processing
Classification
Algorithm
Hidden Markov
Models (HMMs)
Neural Netowrks (NNs)
Machine
Learning
Display
Indicate Cognitive State:
• High/Nominal Workload
• Channelized Attention
• Diverted Attention
• Confirmation Bias
• Inattentional Blindness

6. Project Echo – SmartEye® Eye-Tracking System
• Hardware Setup
• Camera Calibration
• Building World Model
• Head Profile Creation
• Gaze Calibration
• Real-time data transfer
(future)
SmartEye® Setup with corresponding
World Model
6

7. Project Echo – Empatica E4 Wristband
• Measures
o Heart Rate – HR (PPG)
o Galvanic Skin Response – GSR
o Skin Temperature
o Acceleration/Movement
• Lightweight, non-invasive,
portable
• Real-time data streaming
through Matlab
o TCP/IP Connection
o Stored in text file
7

8. Speech Recognition Software – Motivation
• Talking could affect
classification accuracy during
runs
oMore/varying cognitive
activation
oIrregular breathing
oMovement
• Automatic labeling of speech
vs. other noises for future
analysis
Talking
8

9. Speech Recognition – MFCCs
• Mel Frequency Cepstral
Coefficients (MFCCs)
oFeature derived from audio
signal
oRepresents shape of vocal tract
through short-time frequency
analysis
oApplies filter to frequency
response of signal to emulate
human hearing
 Humans can distinguish low
frequencies easier than high
frequencies
Cross-sectional shapes of vocal tracts
Cochlear frequency map showing logarithmic
frequency resolution of human hearing 9

10. Speech Recognition – MFCCs (cont.)
Steps:
• Calculate moving window PSD
o Assume ‘stationarity’ at 25ms window
sizes
o Apply Hamming filter to windowed signal
• Create Mel filter bank (shown to the
right)
• Multiply each windowed PSD by each
filter in filter bank
• Sum powers in each binned and filtered
frequency for each frame
• Take log and discrete cosine transform
of summed powers
• Produces array of 12 coefficients for
each windowed time series
×
=
10

11. Future Work
• Acquire data real-time from SmartEye®
system
• Parse out data stream from Empatica in
Matlab into separate text files
• Synchronize data streams from all
devices with MAPPS or MAPPS
equivalent (LSL, etc.)
• Incorporate in-house preprocessing and
machine learning scripts
• Input MFCCs from audio signals into
Hidden Markov Model machine learning
classifier
11

12. Acknowledgements
• Angela Harrivel, PhD
• Chad Stephens, PhD Candidate
• Kyle Ellis, PhD
• Ray Comstock, PhD
• Kellie Kennedy, PhD Candidate
• Nick Napoli
• Katrina Colucci-Chang
• Will Hollingsworth
• Alex Liang
12