Pilot workload varies greatly during a typical flight. Technological advances are designed to reduce pilot workload and improve safety by eliminating pilot tasks during the busiest phases of flight - approach and landing. But do these new systems actually increase pilot workload? How do we measure the workload on the pilot? How do we measure safety? Justin Stephen Brown

1. IMPROVEMENTS TO SITUATIONAL AWARENESS DURING APPROACH AND
LANDING THROUGH ENHANCED AND SYNTHETIC VISION SYSTEMS
by
Justin Stephen Brown
A Graduate Capstone Project
Submitted to the Worldwide Campus
in Partial Fulfillment of the Requirements of the Degree of
Master of Aeronautical Science
Embry-Riddle Aeronautical University
Worldwide Campus
Pensacola, FL
May 2011

2. ii
IMPROVEMENTS TO SITUATIONAL AWARENESS DURING APPROACH AND
LANDING THROUGH ENHANCED AND SYNTHETIC VISION SYSTEMS
by
Justin Stephen Brown
This Graduate Capstone Project
was prepared under the direction of the candidate’s Research Committee Member,
Mr. William L. Little, Adjunct Associate Professor, Worldwide Campus
and the candidate’s Research Committee Chair,
Dr. Peter B. Walker, Adjunct Assistant Professor, Worldwide Campus and has been
approved by the Project Review Committee. It was submitted
to the Worldwide Campus in partial fulfillment of
the requirements for the degree of
Master of Aeronautical Science
PROJECT REVIEW COMMITTEE:
___________________________________
William L. Little, MAS
Committee Member
___________________________________
Peter B. Walker, Ph.D.
Committee Chair

3. iii
ACKNOWLEDGEMENTS
I would like to thank my wife, Tamara Brown, MPT, for providing the support
and stamina required to care for our daughters while I worked on the Capstone project.
When we started down this journey we were still somewhat newlyweds. We managed
through two PCS moves, the birth of two daughters, two months of daily visits to the
NICU and countless evenings spent in front of the computer. I cannot thank Tamara
enough for her love and patience throughout the five years I've spent working on my
graduate education. Without her, this never would have happened.
I would also like to recognize my parents for teaching me the meaning of
dedication and hard work. Thank you for creating a strong work ethic in me.
I am also very thankful to my committee members who helped me finish this
project remotely with consistent and reliable communication. You all deserve a raise.
Finally, I am sincerely thankful to God for blessing me with family, friends, co-
workers and the means necessary that allowed me to achieve my goals.

4. iv
ABSTRACT
Researcher: Justin Stephen Brown
Title: Enhanced and Synthetic Vision System’s Improvement to Situational
Awareness During Approach and Landing
Institution: Embry-Riddle Aeronautical University
Degree: Master of Aeronautical Science
Year: 2011
In June 1999, American Airlines flight 1420 overran runway 04R in Little Rock,
Arkansas and fell into a ravine killing 10 people including the captain. This accident
occurred during a thunderstorm and contains many of the characteristics involved in
approach and landing accidents. The Flight Safety Foundation’s (FSF) Approach and
Landing Accident Reduction (ALAR) study noted that 76 percent of all jet and turboprop
aircraft accidents involved landing overruns, loss of control, runway incursion, non-
stabilized approaches, or controlled flight into terrain (Killers in aviation, 1998).
Enhanced and synthetic vision systems are technological advances that intend to reduce
pilot workload, especially during the final phase of flight. The primary aim of this
research was to use simulation studies, proto-type research and statistical data to compare
mishap rates and pilot workload during visual flight rules (VFR) and instrument flight
rules (IFR) approaches to landing using traditional navigational aids and enhanced and
synthetic vision technology.

5. v
TABLE OF CONTENTS
Page
PROJECT REVIEW COMMITTEE ii
ACKNOWLEDGEMENTS iii
ABSTRACT iv
TABLE OF CONTENTS v
LIST OF TABLES viii
LIST OF FIGURES ix
Chapter
I INTRODUCTION 1
Background of the Problem 1
Researcher's Work Role and Setting 5
Statement of the Problem 6
Limitations 7
Acronyms 8
II REVIEW OF RELEVANT LITERATURE AND RESEARCH 10
History of Navigational and Landing Aids 10
Developments in SVS 15
Developments in EVS 18
Situational Awareness Measurement 19

6. vi
Prior Research of SVS and EVS 21
SVS for CFIT Prevention 22
SVS Symbology 23
Blending EVS and SVS 25
Statement of the Hypothesis 26
III RESEARCH METHODS 27
Research Model 27
Survey Population 27
Sources of Data 28
The Data Collection Device 28
Instrument Pretest 29
Instrument Reliability 30
Procedures 30
Treatment of Data 30
IV RESULTS 31
V DISCUSSION 40
VI CONCLUSION 44
VII RECOMMENDATIONS 46
REFERENCES 48
APPENDICES 52

7. vii
A BIBLIOGRAPHY 52
B BASELINE HUD FOR IMC-DAY CONDITION 53
C SVS HUD FOR IMC-DAY CONDITION 54
D EVS HUD FOR IMC-DAY CONDITION 55
E COMBO HUD FOR IMC-DAY CONDITION 56
F BASELINE HUD FOR IMC-NIGHT CONDITION 57
G SVS HUD FOR IMC-NIGHT CONDITION 58
H EVS HUD FOR IMC-NIGHT CONDITION 59
I COMBO HUD FOR IMC-NIGHT CONDITION 60

8. viii
LIST OF TABLES
Table Page
1 Pilot Flight Time in Hours 29
2 ANOVA Results for RMSE Data 31
3 Summary of F-Test Results on SAGAT Scores 36
4 Results of Post-Hoc Analysis for Display Effect on Pilot SA Types 38

9. ix
LIST OF FIGURES
Figure Page
1 Four Course Radio Range 11
2 Very High Frequency Omnidirectional Radio Range 11
3 Early Forms of Runway Landing Aid Lighting 13
4 Elements of the Instrument Landing System 14
5 Universal Avionics SVS Display 17
6 EVS Using FLIR on Approach 18
7 RMSEs for Each Display Configuration 32
8 RMSEs for IMC Conditions 33
9 RMSEs for Display Configurations by IMC Condition 34
10 RMSEs for Leg by Display Configuration 35
11 Overall SA Scores for Each Display Configuration 37
12 SA Scores for Three Different Types of SA Being Measured 38

10. 1
CHAPTER I
INTRODUCTION
Background of the Problem
The Flight Safety Foundation (FSF) conducted a study in 1998 with intentions of
reducing approach and landing accidents ("Killers in Aviation", 1998). The FSF study
found that 76 percent of all jet and turboprop aircraft accidents involved landing
overruns, loss of control, runway excursions, non-stabilized approaches, or controlled
flight into terrain (CFIT) ("Killers in Aviation", 1998). More specifically the FSF study
found that nearly 60% of ALAs involved CFIT ("Killers in Aviation", 1998). Similarly,
a study conducted by the National Aeronautics and Space Administration (NASA) in
2005 found that 66 percent of all jet accidents occur during the landing or final phase of
flight (NASA, 2006). The study also noted that 81 percent of these ALAs occurred
during instrument meteorological conditions (IMC) (NASA, 2006).
Take for example, the mishap of American Airlines flight 1420. In June 1999,
American Airlines flight 1420 overran runway 04R in Little Rock, Arkansas and fell into
a ravine killing 10 people including the captain. This accident occurred during a
thunderstorm and contains many of the characteristics involved in approach and landing
accidents (ALAs.) The ability of the pilot to determine critical information through
perceived visual cues of the outside environment may be limited by time of day and
various weather phenomena such as fog, rain, snow, and clouds. The Flight Safety
Foundation (FSF) conducted a study in 1998 with intentions of reducing approach and
landing accidents ("Killers in Aviation", 1998).
Various systems have been developed to reduce this accident rate and enhance

11. 2
safety and overcome the issues with limited outside visibility for the pilot, such as radio
navigation, instrument landing systems (ILS), ground proximity warning systems
(GPWS). More recent developments include moving map displays, Global Positioning
System (GPS) capability for improved navigational accuracy, terrain awareness warning
systems (TAWS) and enhanced GPWS.
Regardless of the technological advancement, the aircraft information display
concepts require pilots to perform various mental transformations of display data to a
mental picture of what the outside environment may look like (Prinzel et al., 2002). For
example, the TAWS technology may help to mitigate some factors causing CFIT, its use
generally follows the information processing model of warn then act and, therefore
requires the pilot to be reactive rather than proactive in dealing with terrain hazards
(Prinzel, Comstock, Glaab, Kramer, and Arthur, 2002).
Snow and Reising (1999) stated that what is needed in terms of aircraft
information systems in intuitive technologies that improve pilot SA with respect to spatial
orientation (relative to terrain and flight path) without requiring the pilot to divert visual
attention and cognitive resources away from possible external events and primary flight
references. A proactive system that can help prevent (versus just warn a pilot of) a
potential collision with terrain is needed (Prinzel et al., 2004a).
NASA and its industry partners have designed and prototyped crew-vehicle
interface technologies that strive to proactively overcome aircraft safety issues due to
low-visibility conditions by providing the operational benefits of clear day flight through
cockpit displays, regardless of the actual outside visibility conditions (Bailey, Parrish,
Kramer, Harrah, and Arthur, 2002).

12. 3
Enhanced and synthetic vision systems are technological advances that intend to
reduce pilot workload, especially during the final phase of flight. The titles are
occasionally used interchangeably; individually they are quite different and sometimes
competing technologies.
Synthetic Vision/System (SV/SVS) is a computer generated display image of the
out-of-the-cockpit scene topography based on aircraft attitude, high-precision navigation
instrumentation, and data on the surrounding terrain, obstacles, and cultural features.
SVS databases have been built to support this display technology with real-time integrity
in order to ensure pilot detection of real obstacles and to plan and verify accurate flight
navigation.
Prinzel et al. (2002) suggested that this display concept presents information to
the pilot with a level of accuracy and realism to flying under visual meteorological
conditions (VMC), regardless of the actual weather conditions. Laboratory and field
research experiments have successfully demonstrated both the safety and capability
benefits of SVS technologies in flight (Snow, Reising, Kiggett, and Barry, 1999), landing
(Prinzel et al., 2004a; Schnell, Kwon, Merchant, and Etherington, 2004) and taxi
operations (Wilson, Hoovey, Foyle, Williams, 2002). Prinzel et al. (2004a) suggests that
SVS display concepts are expected to reduce the occurrence of accident precursors
including:
• Pilot loss of attitude awareness
• Pilot loss of altitude awareness
• Pilot loss of vertical and lateral spatial awareness
• Pilot loss of terrain and traffic awareness on approach

13. 4
• Confusion regarding escape or go-around path after recognition of
flight problem
• Pilot loss of SA in relating to the runway environment and
incursion
• Unclear flight path guidance on initial approach
An Enhanced Vision/System (EV/EVS) uses electronic sensors such as forward
looking infrared radar (FLIR) or millimeter-wave radar (MMWR) to augment or enhance
the natural vision while flying an aircraft. Such instruments are used to penetrate weather
phenomena such as rain, snow, fog, and haze.
In 2007 business jets began incorporating EVS displays as a night-vision
technology to complement other traditional navigational aids. Based on the development
of EVS technology the U.S. Federal Aviation Regulations (FAR), Section 91.175 was
amended such that pilots conducting straight-in approaches may now operate aircraft
below publicized Decision Heights (DH) or Minimum Descent Altitudes (MDA), when
using an approved EVS presented on a HUD (Bailey, Kramer and Prinzell, 2006).
EVS instruments are designed to help the pilot see permanently located
obstructions such as buildings, trees, towers, power lines, and terrain in general and are
normally displayed on a heads-up display (HUD). The intended use of EVS parallels
SVS in that both strive to improve safety and SA in low visibility conditions that may
otherwise cause major flight accidents and to provide the operational benefits of VMC
(Bailey et al., 2006).
EVS and SVS technologies are designed to improve situational awareness (SA)

14. 5
for pilots flying the aircraft (PF) and non-flying pilots (NPF) alike. Investigative
interviews and the National Transportation and Safety Board (NTSB) research has shown
that accidents similar to the American Airlines crash, involve pilots that 1) are
overwhelmed with tasks during an IMC and/or night-time approach or; 2) have lost SA
during an IMC and/or night-time approach (Prinzel, Hughes, Arthur and Kramer, 2003).
It is hypothesized that EVS and SVS technology improvements will reduce the
rate of ALA accidents when compared to other navigational aids and traditional
instrument approach procedures by increasing SA and reducing pilot workload during
high-risk phases of flight such as the final approach to landing in low visibility situations.
Despite the technology’s infancy, NASA, the Federal Aviation Administration,
and many other private entities have tested this technology rigorously. The quantifiable
methods typically center on pilot workload and traditional physiological measurement
methods of stress and anxiety (Hughes, 2005c). Practical applications of this research
have promoted further mainstream research to determine safety improvements, allowing
for more efficient air carrier movement especially in congested areas susceptible to poor
weather. Financial effects can only be estimated until implementation is more complete.
Researcher’s Work Role and Setting
The researcher holds numerous Federal Aviation Administration (FAA) licenses
including Airline Transport Pilot (ATP) and a Boeing 737-300 type rating. Other
licenses and certificates include Airplane Single Engine Land (ASEL), Airplane Mutli-
Engine Land (AMEL), Military Instructor Pilot, Military Test Pilot, and Military Crew
Resource Management (CRM) Facilitator. He has served for 10 years as a Naval Aviator
with numerous sorties in wartime theaters.

15. 6
The researcher became interested in improvements of SA during low visibility
approaches to landing while deployed to Misawa, Japan. The deployment spanned the
winter months, and Misawa, Japan is located in the northern part of the main Japanese
island of Honshu. During this deployment, the researcher encountered numerous whiteout
conditions while attempting to land in Misawa. The researcher was flying as copilot
during a hazardous emergency landing during a snowstorm onto a snow-covered runway.
During this landing, the emergency arresting cable was covered by snow and
unknowingly left rigged and unreported, resulting in significant damage to the aircraft
upon landing. From this incident spawned seven years of research and education on how
to improve safety of flight during the segment of the flight that requires the most
situational awareness and removal of cluttering information.
Statement of the Problem
The FAA has authorized synthetic vision systems and enhanced vision systems
for use in flight with certain limitations. The popularity and integration has risen sharply
in its relatively short lifespan. Many pilots have not yet been exposed to the training
required to become proficient in use of EVS or SVS yet exposure is destined to come.
Without question this technology will offer more information to the pilot when visual
cues are no longer available.
The purpose of this project was to determine if EVS and SVS systems improve
SA during the approach and landing phase of flight, or if the addition of the new
technology increased avionic clutter causing a decrease in SA. How is SA or pilot
workload measured? Do experimental results show a decrease in pilot workload? Do
the results show improved performance? The importance of this study showed whether

16. 7
the implementation of additional technology in cockpit could reduce clutter and improve
the pilot’s ability to safely land an aircraft in less than VFR conditions.
Limitations
In general, conducting an experiment with real aircraft in real scenarios to
examine the effects of certain cockpit displays on human performance is difficult and
expensive. Even simulator experiments can prove to be costly and time consuming to
gather a large enough sample size of pilots, high tech instruments, etc.
Additionally, EVS and SVS are young technological advancements. Therefore, a
relatively small number of experiments or studies have been conducted on the effects on
human performance. The researcher detailed a meta-analytic approach to the relevant
experiments and studies in Chapter II regarding the early researchers involved in the
advent of synthetic and enhanced vision systems and their impact on pilot performance
and accident reduction.
Additionally, the researcher noted that the primary scientific study in his research
was Kim's simulator study (2009). Kim set the p-value at .05 to determine outliers that
occur by chance. However, Kim's sample size (see Table 3), involved 49 samples, which
would result in approximately 2 outliers with a p-value of .05. Perhaps Kim's results
would be more accurate with a p-value of .01.

17. 8
Acronyms
ALAR – Approach and Landing Accident Reduction
AGL – Above Ground Level
ALA – Approach and Landing Accidents
ALA – Approach and Landing Accidents
ANOVA – Analysis of Variance
CAA – Civil Aeronautics Administration
CFIT – Controlled Flight Into Terrain
CRM – Crew Resource Management
DH – Decision Height
DIME – Database Integrity Monitoring Equipment
DV – Dependent Variables
EV – Enhanced Vision
EVS – Enhanced Vision System
FAA – Federal Aviation Administration
FAR – Federal Aviation Regulation
FLIR – Forward Looking Infrared Radar
FMS – Flight Management System
FOV – Field of View
FSF – Flight Safety Foundation
FTE – Flight Technical Errors
GPWS – Ground Proximity Warning System
HDD – Heads-Down Display
HITS – Highway-In-The-Sky
HUD – Heads-Up Display
ICAO – International Civil Aviation Organization
IFD – Integration Flight Deck
IFR – Instrument Flight Rules
ILS – Instrument Landing System
IMC – Instrument Meteorological Conditions
MDA – Minimum Descent Altitude
MMWR – Millimeter Wave Radar
NASA – National Aeronautics and Space Administration
ND – Navigation Displays
NPF – Pilot Not Flying
NTSB – National Transportation and Safety Board
PF – Pilot Flying
PFD – Primary Flight Display
RMSE – Root Mean Square Error
SA – Situational Awareness
SAGAT – Situation Awareness Global Assessment Technique
SART – Situation Awareness Rating Technique
SAS – Statistical Analysis Software
STARs – Standard Terminal Arrival Routes

18. 9
SV – Synthetic Vision
SVS – Synthetic Vision System
SWORD – Subjective Workload Dominance
TLX – Task Load Index
USAF – United States Air Force
VHF – Very High Frequency
VFR – Visual Flight Rules
VOR – Very High Frequency Omnidirectional Radio Range
WAAS – Wide Area Augmentation System

19. 10
CHAPTER II
REVIEW OF RELEVANT LITERATURE AND RESEARCH
History of Navigational and Landing Aids
In the early days of flight, there were no navigation aids to help pilots find their
way. Pilots flew by looking out of their cockpit window for landmarks or by using road
maps. These visual landmarks and maps were fine for daylight operations, but airmail
operated around the clock and the need for mail to be delivered by air expanded.
In 1919, U.S. Army Air Service Lieutenant Donald L. Bruner began using bonfires to
help with night navigation (Preston, 1998). In February 1921, an airmail pilot named
Jack Knight flew the first all-night flight from North Platte, Nebraska to Chicago using
bonfires lit by Post Office staff, farmers and the public (Preston, 1998).
In 1926, the Aeronautics Branch of the Department of Commerce took over
responsibility of building lighted airways and by 1933, 18,000 miles of airway had lights
and 1,500 beacons were in place (Preston, 1988). Regardless of the complexity of
lighting systems, pilots were still required to maintain visual contact with the ground.
Radio communications developed rapidly during this same period allowing for
weather reports to be passed via two-way radio signals and teletypewriter. This let pilots
plan their flight path and diverts around foul weather to maintain VMC. In 1929, Army
Lt. James H. Doolittle became the first pilot to use aircraft instrument guidance solely for
take off, fly a set course, and land. Lt. Doolittle used a four-course radio (Figure 1)
range and radio marker beacons to indicate his distance from the runway (Preston, 1988).
An altimeter displayed his altitude, and a directional gyroscope with artificial horizon
helped Lt. Doolittle control his aircraft’s attitude, without seeing the ground. These

20. 11
technologies became the basis for many future developments in aviation navigation.
Figure 1. Four Course Radio Range
In May 1941, the Civil Aeronautics Administration (CAA) opened its first ultra-
high frequency radio range system for scheduled airline navigation, later expanding to
use such equipment to 35,000 miles of federal airways. In 1944, the CAA began testing a
static-free, very high frequency (VHF) omnidirectional radio range (VOR) (Figure 2)
that allowed pilots to navigate by watching a dial and needle on the instrument panel
rather than by listening to the radio signal audibly (Preston, 1988).
Figure 2. Very High Frequency Omnidirectional Radio Range (VOR)
By the middle of 1952, 45,000 miles of VHF and VOR airways, referred to as
Victor airways, supplemented the 70,000 miles of federally maintained low frequency

21. 12
airways. The CAA began to shut down the low and medium frequency four course radio
ranges (Preston, 1998). In 1961, the FAA began using distance-measuring equipment
(DME) on its entire system. DME allowed aircraft to determine their distance from
known checkpoints in order to confirm their position. DME with VOR greatly improved
accuracy in instrument approaches during less than visual conditions and during
nighttime operations.
By 1973 the last airway light beacon from the system in the 1920s was shut down
and by 1982, the first of 950 new navigation aids equipped with solid-state construction
and advanced features was installed. With the advancement of radar implementation into
the air traffic control system, air traffic made large improvements in managing the
increasing flow of traffic while also increasing safety.
Developments in landing aids progressed parallel to developments in navigational
aids. Airports began using rotating lights at the landing field so they could be found after
dark. In the early 1930s, airports installed the earliest forms of approach lighting. These
indicated the correct angle of descent and the correct alignment with the runway. The
approach path was called glidepath or glideslope. The colors of the lights, their rates of
flash became standard worldwide based on International Civil Aviation Organization
(ICAO) standards.
Radio navigation aids assisted in landing, such as the four-course radio range, in
which the pilot was guided by the strength of Morse code signals. The introduction of
the slope-line approach system was a first in landing aids. Developed in the 1940s, the
aid consisted of lights in a row (Figure 3) that allows the pilot a simple funnel of two
rows that led to the end of the runway (Komons, 1989). The system was inexpensive to

22. 13
build and operate and variations of this same lighting system are still used in airports
worldwide.
Figure 3. Early Forms of Runway Landing Aid Lighting; the top image shows a daytime
view; the lower image illustrates a nighttime view
The instrument landing system (ILS) incorporated the best features of both
approach lighting and radio beacons with higher frequency transmissions. The ILS
painted an electronic picture of the glideslope onto a pilot’s cockpit instrument panel.
The first landing of a scheduled U.S. passenger airline using ILS was on January 26,
1938 as a Pennsylvania-Central Airlines Boeing 247 arrived in Pittsburgh from
Washington, D.C. and landed in a snowstorm using only the ILS system (Komons, 1989).
More than one type of ILS system (Figure 4) was tried and eventually the

23. 14
adaption consisted of a course indicator (localizer) that showed drift either left or right
from runway centerline, glidepath to signify drift above or below glideslope, and two
marker beacons to show progress of the approach to the landing field or runway
(Komons, 1989). Approach lighting and other visibility equipment are part of the ILS
and also aid the pilot in landing. The ILS remains very similar to the initial system and
continues to be relied upon as an accurate instrument approach landing aid as of the
writing of this paper.
Figure 4. Elements of the Instrument Landing System
The advent of the GPS provided an alternative source of precision information for
instrument approaches. In the U.S., the Wide Area Augmentation System (WAAS) has
been available to provide precision guidance to Category I standards since 2007. Satellite
based avionics acted as the foundation for the majority of improvements to precision
instrument navigational supplements and landing aids.
Similar to the rapid changes in technological advances for navigational aids in the
1920s and 1930s, the 2000s witnessed rapid upgrades in possibilities to improve aircrew
and passenger safety. However, the addition of more and more information proved to be
more of a distraction in certain situations, such as cell phone use while driving, than of an

24. 15
improvement. Therefore, technological progress was tempered with caution to eliminate
unnecessary information in circumstances that demand high levels of situational
awareness. This created a need for repeated and even continual experiments to measure
pilot performance while exposed to additional information with the objective to remove
unnecessary and distracting data.
In this section, various recent and historical research and studies pertaining to
EVS and/or SVS are reviewed from a human factors perspective and the influence on
situational awareness and pilot workload. It was the author’s intent to discuss the studies
and their assessment of specific features or configurations of EVS and/or SVS
technologies on pilot performance, in order to understand their effect on reducing pilot
workload and improving situational awareness during the approach and landing phases of
flight. Due to the difficulty in measuring situational awareness, the methods used during
these studies will also be discussed.
Developments in SVS
Throughout aviation history, developers have made attempts to improve safety and
increase the ability to successfully aviate and navigate through poor weather and poor
visibility conditions. For example, Jeppesen researched a detailed terrain database and
precision navigation sensors that would be a part of future-generation, three-dimensional
(3-D) flight management system (FMS.) The FMS would supply an on-board SVS with
position, altitude and heading data, along with terrain features, elevations and contours.
The SVS then projects a synthetic image of the outside world onto a HUD or HDD,
similar to the computer generated visuals in a flight simulator (see Figure 5) (Ramsey,
2004).

25. 16
Early developments of SVS had to overcome an Achilles’ heel: access to an
accurate terrain database that could be approved for IFR flight. The U.S. military
controlled access to one of the most precise terrain databases ever developed: the Defense
Mapping Agency’s 90 meter terrain survey that was developed for cruise missile
guidance and strike aircraft attack missions. A major step forward in terrain mapping
came with the February 2000 launch of the space shuttle Endeavor and its 11-day shuttle
radar topography mission (Ramsey, 2004). This accelerated the development of a system
flying in general aviation (GA) aircraft. Coupled with Jeppesen navigation charts, the
system displays a highway-in-the-sky that guides pilots through a series of waypoints,
standard terminal arrival routes (STARS), and ILS approaches to landing. Terrain,
obstructions, and traffic can also be depicted on the display (Ramsey, 2004).

26. 17
Figure 5. Universal Avionics SVS display
Developments in EVS
Enhanced vision systems are designed to use electronic sensors to augment or
enhance the natural vision while flying an aircraft (see Figure 6) (Hughes, 2005b). These
instruments are designed to help the pilot see permanently located obstructions, such as
buildings, trees, towers, power lines, and are normally displayed on a HUD.

27. 18
Figure 6. EVS using FLIR on approach
FLIR provides excellent EVS image resolution and is relatively affordable
technology. Also, FLIR systems are compact and installation requires only minor
modifications to the airframe. FLIR could provide pilots with a high-resolution display
of the terrain and runway environment when attempting to make an approach to an
isolated airport on a clear night. However, fog and clouds severely degrade FLIR
performance, rendering it almost useless especially if the clouds are dense.
Clouds and fog, on the other hand, don’t affect MMWR, although heavy
precipitation may reduce its effective range. Yet MMWR doesn’t have as high of a

28. 19
resolution as FLIR. Compared to FLIR systems, MMWR systems are relatively
expensive and hard to adapt to civil aircraft because of their bulky antennas. A
MMWR’s ultra-short wavelength may require changes to an aircraft’s radome to ensure
that it is transparent to three and nine millimeter radio waves (George, 1995).
Situational Awareness Measurement
Due to the difficulty and ambiguity surrounding situational awareness, there are
also difficulties in measuring situational awareness during studies such as those discussed
in this paper. The research, experiments and studies reviewed in the following chapters
use different SA measurement methods.
Situational awareness subjective workload dominance (SA-SWORD) is used to
assess and compare the pilot SA when using two or more different cockpit displays or
interfaces. The SWORD technique is a subjective workload assessment tool that has
been used both retrospectively and predictively. SWORD uses subjective paired
comparisons of tasks in order to provide a rating of workload for each individual task.
When using SWORD, participants rate one task’s dominance over another in terms of
workload imposed.
Situational awareness global assessment technique (SAGAT) is an objective,
diagnostic and sensitive metric that is highly validated for use in a variety of applications.
SAGAT has been successfully used to directly and objectively measure operator SA in
evaluating avionics concepts, display designs and interface technologies. With SAGAT,
mission simulations are frozen randomly, the system displays are turned off and the
simulation is suspended while operators quickly answer questions about their current
perceptions of the situation. Operator perceptions are then compared to the real situation

29. 20
to provide an objective measurement of SA.
The situational awareness rating technique (SART) is a direct self-rating measure
of SA that is more complex than a simple Likert scale. In 1989, R.M. Taylor developed
the SART by eliciting knowledge from pilots and aircrew. Through statistical techniques
he created the SART which consists of ten dimensions of questions to measure SA.
Cooper-Harper rating scale is a set of criteria used by pilots and flight test engineers
to evaluate the handling qualities of aircraft during flight tests. The scale ranges from 1
to 10, with 10 indicating the worst handling characteristics and 1 indicating the best. The
criteria are evaluative and therefore the scale is considered subjective.
The U.S. Air Force’s Revised Workload Estimation Scale (a 7-point Likert scale) is
a subjective technique used to measure mental workload in comparison with two or more
dependent variables.
NASA Task Load Index (NASA-TLX) is a subjective workload assessment tool.
NASA-TLX allows users to perform subjective workload assessments on operators
working with various human-machine systems. NASA-TLX is a dimensional rating
procedure that derives an overall workload score based on a weighted average of ratings
on six subscales. These subscales include mental demands, physical demands, temporal
demands, own performance, effort and frustration.
Root mean square error (RMSE) is a frequently used measure of the differences
between values predicted by a model or an estimator and the values actually observed
from the subject being modeled or estimated. In Kim’s study, RMSE was used to
measure the errors in flight path control by calculating the distance away from centerline
an aircraft or simulator drifted during the study.

30. 21
Prior Research of EVS and SVS
The use of EVS and SVS technology is expected to reduce aircraft accidents, in
particular CFIT, due to the display information enhancing pilot SA under low visibility
conditions, similar to a night-time or inclement approach to landing. Several studies have
been conducted to assess this expectation and to investigate the effect of specific
EVS/SVS features on pilot performance. In general, studies have focused on the effects
of display sizes and corresponding field of view (FOV), guidance images, and tunnel
images on flight path tracking performance, SA, workload, or subjective display ratings
(Kim, 2009).
Researchers such as Prinzel, Schnell, Arthur, Bailey, Hughes, McKinley, Kramer
and others used these measurement techniques to assess the pilot’s ability to safely
maintain an aircraft in stable flight through experiments with symbology, ground
proximity, instrument approach precision, and other variables discussed below. The
previous research has mainly focused on portions or certain aspects of the
implementation of new displays with more information into the cockpit. Sang-Hwan
Kim’s (2009) experiment combined the research into a measurement of all aspects into a
quantifiable study on how pilots respond to an efficient display of additional information.
The results of their work created the basis for Kim’s simulator experiment in 2009, which
combined numerous variables into one study.
Early SVS and EVS studies and articles were centered on the rapidly changing
technology and its integration into mainstream aviation. Prinzel et al. (2002) conducted
flight tests to evaluate the effects of three display concepts. These design concepts were
tested with the primary objective to determine which, if any, of the concepts improved

31. 22
SA and reduced pilot workload in IMC conditions on final approach. In general, Prinzel
et al. confirmed the hypotheses that SVS would provide safety and performance benefits
over traditional navigational instruments (2002).
With a similar objective, Prinzel et al. in another study in 2004 conducted two
experiments to examine the efficacy of SVS displays and to develop field of view (FOV)
and terrain texture recommendations for cockpit display design. In one of their
experiments, they investigated the effects of different types of displays for presenting
SVS information, two types of textures (photorealistic and generic) and two runway
conditions on performance, subjective preference ratings, workload and SA (using SA-
SWORD). Results demonstrated that the different display sizes did not affect flight
performance and that the use of the HUD for presenting SVS information reduced lateral
path error, as compared to the HDD. A reduction in lateral path error led to confirming
their hypothesis that a HUD would reduce pilot workload and improve situational
awareness over a HDD.
SVS for CFIT Prevention
The studies reviewed above have focused on nominal flight operations; however,
other research has been conducted to examine the efficacy of SVS technology for CFIT
prevention in off-nominal situations (Prinzel et al, 2003). In an experiment by Prinzel et
al. (2003), 10 display concepts, including two baseline conditions (a round-dials display
and a primary flight display), and various SVS textures were used to assess operator
CFIT detection. Results revealed that the use of SVS, in general, improved CFIT
detection.
In a second experiment, Prinzel et al (2003) evaluated four display concepts by

32. 23
measuring flight performance, SA and workload during a go-around situation.
Situational awareness was measured using the SART and SA-SWORD methods.
Workload was measured using modified Cooper - Harper ratings. Results confirmed that
the use of the SVS allowed pilots to detect CFIT more efficiently than baseline concepts.
SVS Symbology
These experiments demonstrated the general efficacy of the SVS concept.
Consequently, the effects of guidance and tunnel images, combined with SVS
technology, were investigated. Prinzel et al. (2004c) conducted two experiments to
compare different tunnel and guidance symbology concepts for synthetic vision display
systems presented on HDDs and HUDs. They evaluated the efficacy of these concepts
during complex, curved approaches under instrument meteorological conditions (IMC).
In the first experiment, they focused on a SVS primary flight display (PFD) and
examined four tunnel concepts compared to a baseline (no tunnel) configuration. They
also assessed three guidance symbologies by measuring mental workload using the
United States Air Force’s (USAF) Revised Workload Estimation Scale, SA using SART
and SA-SWORD, a subjective questionnaire, and RMSE flight path control measurement
(Kim, 2009). The results of the first experiment revealed the baseline condition to be
worse than other conditions including tunnel concepts, in terms of path control, workload
and SA. The second experiment evaluated two pathway tunnel concepts and two forms
of guidance for a HUD. Overall, the results demonstrated that presenting any kind of
tunnel feature could produce better performance in terms of RMSE, workload and SA.
Schnell et al. (2004) also evaluated a SVS HDD against traditional navigational
aids through conventional glass cockpit displays to assess whether SVS technology could

33. 24
improve pilot performance, SA and workload. Schnell et al. included navigation displays
(ND) in their simulation setup for providing pilots with more realistic flight situation
information. SA was measured using SAGAT, mental workload using the NASA-TLX
workload assessment tool, flight technical errors (FTE) and eye movements of pilots
when using three different configurations of flight decks.
The configurations included a conventional PFD with displays, a SVS PFD with
navigational displays, and a conventional PFD with an external display. This is an
exocentric display that depicts the planned flight path in the context of the surrounding
terrain. The depiction is centered on the aircraft (Kim, 2009). Results demonstrated, in
general, the use of the SVS display format to improve pilot performance by generating
reduced flight technical errors (FTEs), lower workload scores and short overall visual
scan length. Interestingly, there was no significant difference in SA across display
conditions. That is, the SVS PFD with terrain representation did not seem to improve the
terrain awareness of the pilot.
Schnell et al. (2004) deduced that pilots relied on and trusted the pathway tunnel
to the extent that they did not feel they needed to devote much attention to the aircraft-
terrain situation. Schnell et al. (2004) generalized that pilot workload measures were
lower in the SVS condition than with the conventional PFD.
Problems with using SVS technology included standardizing the quality of
database-oriented system information and controlling the quality of matching real and
synthetic information closely enough to certify the database for practical applications.
Bailey et al. (2002) introduced two possibilities of reducing SVS information errors: 1)
development of a complimentary system called Database Integrity Monitoring Equipment

34. 25
(DIME); and 2) incorporating use of EVS, real-time and non-database elements, blended
into the SVS display.
Blending EVS and SVS
EVS and SVS have been perceived as separate technologies, both designed to
improve pilot situational awareness but neither technology completely provided the entire
picture outside the cockpit. NASA developed a Sensor Enhanced – SVS (SE – SVS)
concept, which utilizes the beneficial aspects of EVS and SVS while mitigating the
negative aspects of each concept (e.g., calibration in SVS images with terrain and poor
EVS display quality due to meteorological conditions) (Bailey et al, 2002).
Bailey, Kramer and Prinzel (2006) compared the general effects of blending
EVS/SVS concepts with and without pathway tunnel images. The comparisons focused
on the fusion of the two displays during low-visibility and landing operations. In the
experiments, four HUD display concepts were tested. This included simulated approach
to landing with the use of SVS displays on approach then transitioned to an EVS image at
500’ above ground level (AGL). This transition was designed primarily because an EVS
camera can only give a usable image when the aircraft is low and close to the ground
(Bailey et al., 2006). Bailey et al. (2006) measured flight path errors and pilot control
inputs during each experimental trial. They also collected subjective questionnaires,
workload ratings, and SA ratings using SA – SWORD and SART. The results of the
study showed that significant improvements in pilot SA without increases in workload
could be provided by the fusion display and the pathway tunnel image (Bailey et al.,
2006).
These results confirmed previous studies (Alexander, Wickens and Hardy, 2003;

35. 26
McKinley, Heidhausen, Cramer and Krone, 2005; Wickens, Horrey, Nune and Hardy,
2004), which showed a synthetic tunnel image to improve flight performance. Through
these studies, it can be stated that the critical objective of improving flight performance
and reducing pilot workload might be largely facilitated by the presentation of tunnel
images in a HUD (Kim, 2009).
Statement of Hypothesis
To summarize, the studies mentioned in this literature review support evidence
that advanced synthetic images in cockpit displays, including blended or fused images
with EVS, synthetic tunnels and terrain features for approach to landing, improve flight
performance and/or pilot SA, and reduce workload (Alexander et al., 2003; McKinley et
al., 2005; Prinzell et al., 2002, 2003; Wickens et al., 2004). Highway-in-the-sky (HITS)
or tunnel images have demonstrated to be a significant factor in improving pilot
performance, particularly flight path accuracy (Alexander et al., 2003).
The findings of combined terrain feature utilities SVS – HUD and EVS – HUD in
Kim's study were hypothesized to improve pilot SA and generated lower mental
workload for pilots than traditional navigational aids (Arthur et al., 2005). Additionally,
the null hypothesis was determined that Kim's experiment would show no change in the
results following the baseline simulator flights as compared to the other test flights.

36. 27
CHAPTER III
RESEARCH METHODOLOGY
Research Model
The Graduate Capstone Project included a meta-analytic approach of multiple studies
and experiments in collaboration to answer the hypothesis. The project employed a
descriptive quantitative research model involving a laboratory experiment conduct by
S.H. Kim in 2009 as the primary focus. Kim's results were collected via SAGAT
measurement devices and three dependent variables (DV) were analyzed with a one-way
analysis of variance (ANOVA) and f-tests.
A meta-analytic approach is a statistical examination of multiple scientific studies and
not an actual scientific study itself. This approach can be used in situations where large
sample sizes are not available, such as in this case with few scientific studies based on the
dependent variable of situational awareness. The strength of a meta-analytic approach
can more powerfully estimate the true effect size as opposed to a smaller effect size
derived in a single study under a given set of assumptions and conditions. The weakness
of this approach was that the sources of bias were not controlled by the method.
The experiment was first developed in consideration of the data collection method
that would be most valid. As previously mentioned, measuring SA is difficult and the
methods are limited. The SAGAT method was employed to quantify the results of Kim’s
experiment (2009).
Survey Population
The sample size in Kim's study was eight pilots recruited to participate in the lab
experiment. The requirements for selection included previous flying experience in

37. 28
commercial aircraft with glass cockpit displays. Due to the relatively recent
developments in EVS and SVS, experience with the use of this technology was not
expected. Data from the demographic survey for the eight pilots include: all pilots were
male; the average age was 58.6 with a standard deviation of 14.4 years; all pilots had
glass cockpit experience; the average flight hour experience was 11,043.8 hours with a
standard deviation of 7,893.1; three pilots had experience with a HUD display in either
actual flight or in a simulator; two pilots had experience with SVS systems (mean of 4
hours) and one pilot had EVS experience (6 hours) in a simulator (Kim, 2009).
Sources of Data
Primary sources of data include task load measurement study conducted by Kim in
2009 at North Carolina State University. Other studies, as cited previously, have been
conducted on specific methods or specific modes of EVS/SVS. However, Kim’s study
(2009) was most appropriate for the specific task of testing the hypothesis that pilot
workload during approach and landing is decreased with the use of EVS and SVS.
The Data Collection Device
Each pilot in Kim’s study completed nine trials. Eight of these trials followed a
within-subjects variable design and one additional trial for collected subject verbal
protocols to measure cognitive task analysis. The data from the first eight trials were
collected using SAGAT measurements of situational awareness. Responses to SAGAT
queries represent a binomial variable – either correct or incorrect. The nature of this
measurement violates quantified statistical test assumptions. With the use of a valid
arcsine function, the data was transformed and traditional parametric data analysis can be
satisfied (Kim, 2009). One-way ANOVAs were then conducted on the transformed

38. 29
SAGAT scores for overall SA, SA by levels and SA by types.
The ninth and last trial data was collected through verbal questioning and
measurement based on the NASA Task Load Index (NASA-TLX) model. Another one-
way ANOVA was used to normalize the rating scores for individual workload scaling.
In addition, all nine trials included a physiological workload measurement – incremental
heart rates (∆HR) – compared to the HR during rest (baseline HR) (Kim, 2009). A single
variable ANOVA was conducted to analyze the effect of display conditions, visibility,
nighttime effects on HR.
Instrument Pretest
All data collection devices were pre-tested to ensure their accuracy and
comprehensiveness. Kim’s experiment (2009) was conducted on a low-fidelity simulator
developed by Integration Flight Deck (IFD) inputs from NASA’s Langley Research
Center, which provided pilots with a full-mission simulator. An ex-Air Force C-130
check pilot with experience with advanced HUDs and experience with NASAs IFD
simulator tested the different HUD configurations in each trial and validated the
simulator.
Table 1.
Pilot Flight Time Data in Hours
Pilot Hours Mean Standard Deviation
Total Instrument Time 3781.3 4064.7
Total Night Time 3660.0 3471.8
Total Flight Time 11043.8 7893.1
Total Time Last 12 Months 230.6 204.8

39. 30
Instrument Reliability
SAGAT is the most widely validated of all SA measurement techniques (Endsley,
2000). According to Endsley (2000) numerous studies have been performed to assess the
validity of the SAGAT method and the evidence suggests that it is a valid metric of SA.
The study by Endsley (2000) included two sets of simulation trials on four fighter pilots
with mean scores of .98, .99, .99 and .92.
The SAGAT method, when used to measure SA through the low-fidelity simulator,
maintains its reliability through simplified binomial variable testing. Either correct or
incorrect responses were measured through pilot inputs on the simulator and then
collected and stored on the simulator’s PC-based computer.
Procedures
Kim’s simulator experiment was conducted on four different segments of a typical
commercial flight. The pilots flew these segments first without EVS/SVS to establish a
baseline. Then the pilots flew using SVS and EVS separately followed by a combination
of EVS and SVS. The dependent variables (DV) measured during these trials were
spatial awareness, system awareness and task awareness, which are all functions of SA.
Treatment of Data
All statistical analyses were performed using Statistical Analysis Software (SAS) (Kim,
2009). Residual plots were used to examine and verify linearity, constant variance of
error terms, and independence of error terms and normality of error term distribution
(Kim, 2009). An alpha level of 0.05 was used to identify any significant effects and
interactions.

40. 31
CHAPTER IV
RESULTS
Flight Path Control Performance
The results of the experiment are listed in the tables below including the ANOVA
results on RMSE data for each of the display configurations (baseline, SVS, EVS,
combination), ANOVA results on pilot SA (overall and by SA type – spatial awareness,
system awareness, and task awareness), and ANOVA results for pilot workload (Kim,
2009).
Flight path control performance is illustrated in Table 2 and in Figure 7.
Table 2
ANOVA Results for RMSE Data
Source df Type III SS Mean Square F P
Display 3 0.324819 0.108273 14.37 <.0001
IMC 1 0.458698 0.458698 60.89 <.0001
Leg 3 0.021173 0.007058 0.94 0.4236
IMC Leg 3 0.0582812 0.017604 2.34 0.0746
Display IMC 3 0.211148 0.070383 9.34 <.0001
Display Leg 9 0.176666 0.01963 2.61 .0007
Display IMC Leg 9 0.07069 0.007854 1.04 .407

41. 32
Figure 7. RMSEs for each display configuration
ANOVA results (see Figure 8) also revealed that there was a significant effect of
the IMC condition on tracking performance (F(1,223)=60.9, p<.0001). The IMC-day
condition (M=14.3) was associated with greater errors in the tracking task than the IMC-
night condition (M=12.0) (Kim, 2009).
`
12.9
15.7
11.3
14.3
0
2
4
6
8
10
12
14
16
18
RMSE
Display
Con0iguration
RMSEs
for
Display
Con0iguration
Baseline
SVS
EVS
Combo

42. 33
Figure 8. RMSEs for IMC Conditions.
There was no significant effect of leg and no interaction of IMC condition and leg
on RMSE. However, ANOVA results revealed an interaction effect among the display
and IMC conditions. Figure 9 shows the interaction plot. This graph indicates the SVS-
HUD under IMC-day condition produced higher RMSE than the same HUD during IMC-
night conditions. In general, IMC-day conditions were associated with higher RMSE
then IMC-night conditions (Kim, 2009).
0.0
2.0
4.0
6.0
8.0
10.0
12.0
14.0
16.0
Visibility
Con=iguration
RMSE
RMSEs
for
IMC
Condition
IMC-­‐Day
IMC-­‐Night

43. 34
Figure 9. RMSEs for display configuration by IMC condition.
ANOVA results also revealed a significant interaction effect between display and
flight leg on RMSE (F(9,223)=2.61, p=.007). Figure 10 shows the RMSEs for each
display configuration for the four simulator legs. The SVS-HUD yielded higher RMSEs
and the EVS-HUD produced lower RMSEs across legs. However, the Combo-HUD in
Leg 4 generated higher tracking error than the other displays in other legs. There was no
three-way interaction effect among the experimental manipulations (F(9,223)=1.04,
p=.4070).
12.8
19.0
12.5
14.2
12.1
11.7
10.4
12.0
0.0
2.0
4.0
6.0
8.0
10.0
12.0
14.0
16.0
18.0
20.0
Basleline
SVS
EVS
Combo
RMSE
Display
Con0iguration
by
IMC
Condition
IMC
Day
IMC
Night

44. 35
Figure 10. RMSEs for leg by display configuration.
Pilot Situational Awareness
Results of an ANOVA applied to pilot SA revealed significant main and
interaction effects of display configuration, IMC condition, and leg of flight on SAGAT
scores including overall SA, and for various levels of SA. Table 3 presents a summary of
F-test results on the situational awareness SAGAT scores.
12.3
13.2
11.9
12.5
14.6
14.9
15.5
13.5
10.3
12.0
12.7
10.7
12.7
12.2
12.4
15.3
0.0
2.0
4.0
6.0
8.0
10.0
12.0
14.0
16.0
18.0
Leg
1
Leg
2
Leg
3
Leg
4
RMSE
Leg
(Phase
of
Flight)
Leg
by
Display
Con0iguration
Baseline
SVS
EVS
Combo

45. 36
Table 3
Summary of F-test Results on SAGAT Scores
Ind.
Vrbls
Overall
SA
Levels of SA Types of SA
Level 1 Level 2 Level 3
Spatial
Awareness
System
Awareness
Task
Awareness
Display
F(3,224)=3.04 F(3,224)=2.37 F(3,224)=1.42 F(3,224)=1.73 F(3,224)=1.1 F(3,224)=4.55 F(3.224)=.82
p=.0300 p=.0712 p=.2374 p=.1614 p=.3517 p=.0041 p=.4823
IMC
F(1,224)=1.22 F(1,224)=.44 F(1,224)=5.99 F(1,224)=56 F(1,224)=2.3 F(1,224)=3.99 F(1,224)=3.39
p=.2701 p=.5096 p=.0151 p=4555 p=.1306 p=.0469 p=.0668
Leg
F(3,224)=.76 F(3,224)=2.28 F(3,224)=1.15 F(3.224)=10.62 F(3,224)=9.37 F(3,224)=10.36 F(3,224)=11.12
p<.0001 p=.0799 p=.3292 p<.0001 p<.0001 p<.0001 p<.0001
Display
IMC
F(3,224)=.76 F(3,224)=.88 F(3.224)=1.45 F(3.224)=.14 F(3.224)=.57 F(3,224)=.99 F(3, 224)=1.68
p=.5188 p=.4532 p=.2289 p=.9334 p=6334 p=.3978 p=.1711
Display
Leg
F(9,224)=1.91 F(9,224)=1.32 F(9,224)=1.45 F(9,224)=1.66 F(9,224)=.55 F(9,224)=2.31 F(9,224)=.84
p=.0509 p=.2282 p=.1686 p=.1009 p=.8386 p=.0167 p=.5827
IMC
Leg
F(3,224)=1.77 F(3.224)=.66 F(3,224)=.56 F(3,224)=4.28 F(3,224)=.97 F(3,224)=2,82 F(3,224)=.97
p=.1544 p=.5748 p=.6412 p=.0059 p=.4064 p=.0386 p=.4091
Display
IMC
Leg
F(9,224)=.62 F(9,224)=.63 F(9,224)=1.08 F(9,224)=.86 F(9,224)=1.05 F(9,224)=.42 F(9,224)=.22
p=.7786 p=.7722 p=.3754 p=.5604 p=.4030 p=.92237 p=.9911
ANOVA results revealed significant effects of HUD configuration
(F(3,224)=3.04, p=.03) and leg of flight (F(3,224)=9.75, p<.0001) on overall SA score.
A post-hoc analysis categorized the display into two groups: one group consisted of SVS,
Baseline, and the Combination of EVS and SVS, condition, which produced higher SA
scores; and another group consisted of Baseline, Combo, and EVS conditions, which
were associated with lower SA scores (see Figure 11) (Kim, 2009).

46. 37
Figure 11. Overall SA Scores for Each Display Configuration
Regarding the effect of display configuration on the three types of pilot SA,
ANOVA results revealed that a significant effect on pilot system awareness
(F(3,224)=4.55, p=.0041). Table 4 shows the post-hoc analyses for the effect of display
configuration on the three types of pilot SA and Figure 12 illustrates the same
information. It can be observed that system awareness was degraded by the EVS-HUD
while the other display effects were comparable.
ANOVA results also revealed the effect of IMC condition to be significant on
system awareness was associated with higher SA scores for system awareness than the
IMC-night condition (Kim, 2009).
44
46
48
50
52
54
56
58
60
Display
Con=iguration
Percent
Correct
for
SA
Queries
Overall
Scores
for
Display
Con0iguration
Baseline
SVS
EVS
Combo

47. 38
Table 4
Results of post-hoc analysis for display effect on pilot SA types
Spatial Awareness System Awareness Task Awareness
Baseline (M=47.9%)
SVS (M=47.2%)
EVS (M=40.5%)
Combo (M=51.8%)
Baseline (M=61.5%)
SVS (M=61.2%)
EVS (M=41.9%)
Combo (M=51.4%)
Baseline (M=78.2%)
SVS (M=85.3%)
EVS (M=83.3%)
Combo (M=78.2%)
Figure 12. SA Scores for three different types of SA being measured according to display
configuration
Cognitive Task Analysis
The results of the cognitive task analysis (CTA) included three types of outcomes:
lists of sequential tasks, lists of non-sequential tasks and critical outcomes from pilots.
Patterns of pilot shifting attention about the HUD were examined.
47.9
61.5
78.2
47.2
61.2
85.3
40.5
41.9
83.3
51.8
51.4
79.4
0
10
20
30
40
50
60
70
80
90
Spatial
Awareness
System
Awareness
Task
Awareness
Percent
Correct
Answers
Types
of
SA
SA
Scores
for
Types
of
SA
by
Display
Con0iguration
Baseline
SVS
EVS
Combo

48. 39
There were no specific differences in the sequential tasks among the experimental
manipulations, which suggested the display configurations and IMC conditions do not
affect sequential flight tasks (Kim, 2009). Trigger events were based on flight scenario
and confirmed by observations of common pilot behaviors in the simulator. Examples of
trigger events included ATC broadcasts, altimeter announcements, and critical changes in
flight status (e.g. runway appearing through clouds, intercepting final approach). Each
event or task consisted of an associated flow of elementary actions.
The analysis of non-sequential tasks revealed required pilot behaviors in
controlling the aircraft independent of the task timing. Non-sequential tasks included
tracking the flight path marker (FPM), airspeed control, altitude control, heading control,
vertical speed control, and radar altitude. Throughout the verbal protocol and analysis,
Kim revealed that some tasks have several alternative methods of performance based on
pilot preference, from training or experience as well as the status of the flight (2009).
Based on analysis of non-sequential tasks, inferences can be made on how
specific behaviors required information affect pilot workload and SA in task
performance. For example, Kim noticed two methods for acquiring vertical speed. The
first method observed included the use of the vertical speed indicator (VSI), while the
second method was pilot inference of the current speed based on the vertical motion of
the flight path marker between the horizon and pitch control (2009).
Similar to the sequential tasks analysis, the non-sequential behaviors among
display configurations and IMC conditions did not reveal differences in pilot
performance due to the display features, such as terrain imagery.

49. 40
CHAPTER V
DISCUSSION
Flight Path Control
Errors in tracking were evaluated as a measure of flight path control performance
using RMSE calculations. It was assumed that higher RMSEs would be associated with
less attention to the FPM indicative of degraded path control performance. Flight path
control performance was affected by pilot display configuration and visibility condition
(Kim, 2009). These results were in-line with the expectation for the HUD content to
drive variation in flight performance under the various environment conditions.
EVS generated lower RMSE scores, the Baseline and Combo conditions were
comparable to each other and SVS induced the greatest flight path control. This was not
in-line with the expectation that the Baseline display would generate the lowest errors in
flight path control followed increasingly be SVS, EVS and Combo displays.
Primarily, the grid lines depicting terrain features in Kim's experiment, generated
by the SVS were often confused by pilots with the FPM and the tunnel features which
also consisted of lines. This confusion may have diverted pilot attention from the FPM in
order to discriminate other features from the SVS imagery and caused higher tracking
errors. Secondly, the thermal returns, or moisture images, from the EVS did not appear
to produce pilot confusion. EVS imagery appeared to compel pilots to focus more on the
FPM in the display and this produced lower tracking errors. Finally, due to the negating
effects of SVS increasing RMSE and EVS decreasing RMSE, the Combo display was
comparable to the Baseline display configurations.

50. 41
Pilot Situational Awareness
Pilot SA was measured using SAGAT techniques and was affected by the
experimental manipulations in Kim's experiment. There were several results that
contradicted the hypothesis.
Overall SA scores were higher with SVS displays and lower for EVS displays,
while the effect of each of these configurations was not different from the Baseline and
Combo displays. The use of EVS degraded system awareness, which involved pilot
understanding of iconic information in the display (see Figure 12.) Degradation in SA
while using EVS may be attributed to the thermal features frequently washing-out iconic
features that presented system information and pilot focus on the FPM to perform the
tracking task which caused negligence in attending to system information. Therefore,
pilots who used EVS features produced higher tracking performance (lower RMSEs) but
had lower system awareness. This is suggestive of a cognitive tunneling effect. This is
caused by the thermal features from the EVS in the HUD.
The visibility conditions had no effect on overall situational awareness. However,
there were significant effects on specific levels and types of SA. While IMC during
night-time operations were associated with higher levels of SA scores, it induced lower
system awareness scores. This is suggestive that night flying increased pilot
comprehension of overall flight information with decreased understanding of system
information in the HUD (Kim, 2009). Additionally, mean scores for other levels and
types of SA across display configuration and legs were higher in the IMC-night
condition.
No significant interaction effects were found on overall SA scores. However,

51. 42
there were significant effects on several levels and types of pilot SA. Display
configuration caused differences in system awareness among the different legs of flight.
The use of SVS produced higher system awareness in Legs 1 and 2, but the Baseline
configuration produced higher system awareness scores for Legs 3 and 4 (Kim, 2009).
This may be explained by the actual terrain and runway being visible through the out-of-
cockpit view.
Among the three types of pilot SA, spatial awareness and task awareness were not
affected by display configuration and visibility condition. The effects of most
experimental manipulations were significant on system awareness.
Pilot Workload
Both subjective and physiological pilot workload measures (NASA TLX and
heart rate measurements) were not affected by the experimental manipulations, including
display configurations or IMC conditions. Although a previous study (Kaber et al., 2008)
found subjective workload rating measured by NASA-TLX to be affected by levels of
display clutter, Kim's study could not reveal the impact of display configuration on the
experiment and nature of the task (2009). This result was unexpected and may be
attributed to the constraints on the experiment and the nature of the task. The primary
assignment for the experimental pilots was not actual flight control but tracking the flight
path.
Cognitive Task Analysis Results
Three types of results were derived from the pilot think aloud portion of the
experiment. The number of tasks and amount of information for each leg of flight
affected system and task awareness. The non-sequential task analysis demonstrated how

52. 43
pilots control the aircraft and which alternative behaviors can be employed to perform the
specific task along with the relevant information. Pilot comments during the think-aloud
session provided useful information for interpreting the effect of the display
configurations on performance. The use of SVS terrain features produced confusion with
the tunnel and FPM features. EVS caused a cognitive tunneling effect that compelled
pilots to concentrate on the FPM instead of iconic information. The use of terrain
features should not be considered after the runway is visible during VMC conditions
because of the potential to create display clutter effects.

53. 44
CHAPTER VI
CONCLUSIONS
The objective of this study was to determine if advancements in avionics
technology, such as EVS and SVS in HUD configurations, could improve flight path
control, pilot situational awareness and pilot workload during landing approach under
instrument meteorological conditions. In this study, a quasi meta-analytic approach was
used to compare results from numerous experiments. A heavy focus was placed on S. H.
Kim's experiment in 2009 that tested pilot performance under various situations involving
EVS and SVS variables.
Flight Path Control
In Kim's study, videos of HUD content for approach to runway 16R at KRNO
were prepared and then presented to pilots on a lab simulator (Kim, 2009). The pilots
were tasked with tracking a flight path in the stimuli while following a flight scenario and
understanding the flight situation. Several pilot studies mentioned earlier in this project
by Prinzell, Schnell, Kramer, Hughes, etc. revealed that the use of tunnel features
improved pilot performance. Kim's experiment revealed that HUD content with EVS
images of terrain features were most effective for facilitating SA on system status
information as compared to providing pilots with spatial information (2009).
Pilot Workload
Both subjective and physiological pilot workload measurements (NASA-TLX and
heart rate changes, respectively) were not affected by increased demands in the
experimental manipulations. Initially, this was thought to not fall in line with the
hypothesis that pilot workload would not be the same under the experimental conditions

54. 45
(Kim, 2009). However, it was expected that the subjective and physiological
measurements would show a parallel movement with the experimental variables. Since
pilot workload measurements remained unchanged under increased demands, it was
determined that, in effect, pilot workload would decrease under normal demands.
Therefore, it can be said that EVS and SVS improve pilot workload conditions during
landing approach under IMC conditions.
Cognitive Task Analysis
Analyses on the subjective-questioning, think-aloud sessions and verbal probes
revealed three types of results. The number of tasks and amount of information for each
leg of flight affected system awareness and task awareness. This may be explained by
the practice effect whereby the pilots became more familiar with the simulator. The non-
sequential task analysis demonstrated how pilots control the aircraft and which
alternative habits can be employed to perform the specific task along with relevant
information. The use of SVS terrain features produced confusion with the tunnel and
flight path marker features (Kim, 2009). EVS caused a cognitive tunneling effect that
compelled pilots to concentrate on the FPM instead of other important information. The
use of terrain features should not be considered after the runway is visible on the
instrument approach because of the potential to create display clutter effects.

55. 46
CHAPTER VII
RECOMMENDATIONS
Based on the results of Kim's experiment and the previous studies and research,
directions for future research include investigating advanced HUD design and the use of
HDD for SVS flight path control during landing approach prior to the runway
environment coming into view. Primarily, the effects of SVS during more realistic flight
conditions or perhaps actual flight conditions should be further evaluated on a larger,
more heavily funded scale. Various avionics manufacturers have employed SVS in more
of a mainstream modality and its benefits have proven effective in managing pilot
workload and improving situational awareness in low visibility conditions (Hughes,
2006). EVS features with HUD usage have proven more costly and less likely than SVS
to become mainstream. Again, further experiments with wider breadth to provide a more
realistic flight scenario can more accurately measure RMSE on EVS effects on spatial
awareness past the traditional decision height position.
Also, more studies should be conducted regarding the design of advanced HUDs
to determine the optimal features to include. Iconic features can prove confusing to pilots
when too many features are presented. Likewise, too few features foster a visual search
for more information, which then increases the pilot workload. Therefore, cognitive task
analyses involving the best use of symbols, numbers and out-of-cockpit images should be
studied to determine the optimum display configuration.
Although it may be possible for pilots to manipulate display configurations in
order to achieve most feature combination or display properties under particular flight
conditions, this may cause additional cognitive workload for pilots. It would be desirable

56. 47
to present pilots with optimal display feature combinations or display sets according to
dynamic changes in flight situations. The HUD interface in Kim's experiment presented
pilots with a runway outline and glideslope reference line instead of tunnel features when
the aircraft descended below 500' AGL, which was preferred by most pilots.
In short, numerous future studies and experiments are required before bold
statements regarding EVS in the landing transition can be made. Many of the Level C
and D full flight simulators in use by airlines and military training facilities would serve
as testing and training devices for future SVS and EVS research.

57. 48
REFERENCES
Alexander, M. J., Wickens, C. D., & Hardy, T. J. (2003). Examining the effects of
guidance symbology, display size, and field of view on flight performance and
situation awareness. Proceedings of the Human Factors and Ergonomics Society
47th
Annual Meeting. Santa Monica, CA: Human Factors and Ergonomics
Society.
Arthur, J. J., Kramer, L. J., & Bailey, R . E. (2005). Flight test comparison between
Enhanced Vision (FLIR) and Synthetic Vision Systems. Proceedings of
Enhanced and Synthetic Vision (SPIE) 2005, Vol. 5802, 25-36.
Bailey, R. E., Parrish, R. V., Kramer, L. J., Harrah, S., & Arthur, J. J. (2002). Technical
challenges in the development of a NASA Vision System Concept. Proceedings
of the North Atlantic Treaty Organization (NATO) Enhanced and Synthetic
Vision Systems.
Bailey, R. E., Kramer, L. J., & Prinzell, L. J. (2006). Crew and display concepts
evaluation for synthetic/enhanced vision systems. Proceedings of the SPIE,
Enhanced and Synthetic Vision Conference 2006.
Endsley, M. R. & Garland, D. J. (2000). Situation awareness analysis and measurement.
Mahwah, NJ: Lawrence Erlbaum Associates.
Flight Safety Foundation. (2007). Approach and landing safety demands improvement. In
Approach and landing accident reduction (Data Systems and Analysis).
George, F. (1995). EVS: Is it feasible? Business & Commercial Aviation, 6, 74-77.
Hughes, D. (2004). Clearing the fog. Aviation Week & Space Technology, 161, 48.
Hughes, D. (2005). MMW ready to roll. Aviation Week & Space Technology, 163, 62.

58. 49
Hughes, D. (2005). Picture this. Aviation Week & Space Technology, 163, 65.
Hughes, D. (2005). Seeing and believing. Aviation Week & Space Technology, 163, 59.
Hughes, D. (2006). Virtual vfr for bizjets. Aviation Week & Space Technology, 165, 66.
Kaber, D.B., Kim, S-H, Kaufmann, K. Veil, T., Alexander, A.L., Stelzer, E.M., Prinzel,
L. & Allamandola, A. (2008). Modeling the effect of HUD visual properties,
pilot experience and flight scenario. Proceedings of the NASA Aviation Safety
Technical Conference. Denver, CO. October 21-23.
Killers in aviation: FSF task force presents facts about approach-and-landing and
controlled-flight-into-terrain accidents. (1998). Flight Safety Digest, November -
December 1998.
Kim, S.H. (2009). Examining and explaining the effects of non-iconic conformal features
in advanced heads-up displays on pilot performance. Retrieved from ProQuest.
(UMI: 3357736)
Komons, N. (1989). Bonfires to beacons. Smithsonian Institution Press. Washington,
D.C.
McKinley, J. B., Heidhausen, E., Cramer, J. A., & Krone, N. J. (2005). Flight testing of
an airborne SVS with Highway-In-The-Sky on a head-up display. Proceedings of
the Digital Avionics System Conference, DASC 2005, 2, 13.C.1-1-13.C.1.1-9.
NASA. (2006). Synthetic vision would give pilots clear skies all the time. In (NASA,
Ed.) Find It @ NASA. Retrieved 25 July 2006, from
http://www.nasa.gov/centers/langley/news/factsheets/SynthVision.html.

59. 50
NTSB American Airlines 1420 report (McDonnell Douglas MD-82, Registration:
N215AA). (Docket Management System (DMS) No. DCA99MA060). (2002).
Washington, D.C. (10).
Preston, Edmund. (1998) FAA historical chronology, civil aviation and the federal
government 1926-1996. Department of Transportation, Federal Aviation
Administration Office of Public Affairs, Washington.
Prinzell, L. J., Kramer, L. J., Comstock, J. R., Bailey, R. E., Hughes, M. F., & Parish, R.
V. (2002). NASA synthetic vision EGE flight test. Proceedings of the Human
Factors and Ergonomics Society 46th
Annual Meeting. Santa Monica, CA:
Human Factors and Ergonomics Society.
Prinzell, L. J., Hughes, M. F., Arthur, J. J., & Kramer, L. J. (2003) Synthetic vision CFIT
experiments for GA and commercial aircraft: “A picture is worth a thousand
lives”. Proceeding of the Human Factors and Ergonomics Society 47th
Annual
Meeting. Santa Monica, CA: Human Factors and Ergonomics Society.
Prinzell, L. J., Comstock, J. R., Glaab, L. J., Kramer, L. J., & Arthur, J. J. (2004). The
efficiency of head-down and head-up synthetic vision display concepts for retro
and forward-fit of commercial aircraft. International Journal of Aviation
Psychology, 14(1), 53-57.
Prinzell, L. J., Kramer, L. J., Arthur, J. J., Bailey, R. E., & Comstock, R. J. (2004).
Comparison of head-up and head-down “Highway-In-The-Sky” tunnel and
guidance concepts for synthetic vision displays. Proceeding of the Human
Factors and Ergnonomics Soceity 48th
Annual Meeting. Santa Monica, CA.
Human Factors and Ergonomics Society.

60. 51
Prinzell, L. J., Arthur, J. J., Kramer, L. J., Bailey, R. E. (2004). Pathway concepts for
head-down synthetic vision displays. Proceedings of the SPIE, Enhanced and
Synthetic Vision 2004. 5423-02.
Ramsey, J. (2004). Synthetic vision: no longer futuristic. Aviation Today. 14, 48.
Schnell, T. Kwon, Y., Merchant, S. & Etherington, T. (2004). Improved flight technical
performance in flight decks equipped with synthetic vision information system
display. International Journal of Aviation Psychology, 14, 79-102.
Snow, M.P., Reising, J.M., Kiggett, K.K. U & Barry, T.P. (1999). Flying complex
approaches using a head-up display: effects on visibility and display type. In
Proceedings of the 10th International Aviation Psychology Symposium
Tailwind and snow forced Southwest 737-700 off runway. (2005, January 20). Flight
International, 168(5016), pp. 7.
Wickens, C. D., Alexander, A. L., Horrey, W. J., Nune, A. & Hardy, T. J. (2004). Traffic
and flight guidance depiction on a synthetic vision system display: The effects of
clutter on performance and visual attention allocation. Proceedings of the Human
Factors and Ergonomics Society 48th
Annual Meeting. Santa Monica, CA:
Human Factors and Ergonomics Society.
Wilson, J.R., Hoovey, B.L., Foyle, D.C., & Williams, J.L. (2002) Comparing pilot’s taxi
performance, situation awareness and workload using command-guidance,
situation-guidance and hybrid head-up display symbologies. In Proceedings of
the Human Factors and Ergonomics Society 46th
Annual Meeting. Santa
Monica, CA: Human Factors and Ergonomics Society.

61. 52
APPENDIX A
BIBLIOGRAPHY
American Psychological Association (2010). Publication manual of the American
Psychological Association (6th ed.) Washington, D.C.
Bender, A. Clark, R., Hanrahan, P., Harsha, W., McMasters, B., Murphy, E. et al.
(Eds.) (2005). Graduate/technical management capstone project
guidelines (6th ed.). Daytona Beach, FL: Embry-Riddle Aeronautical
University, Extended Campus
StatPlus (Version 5.7.8) (2009) [Computer Software]. Vancouver, B.C.
AnalystSoft Inc.

62. 53
APPENDIX B
BASELINE HUD FOR IMC-DAY CONDITION
!"#
!
!! !
!
!! ! ! !
$%&'()#"*+,*#-./)0%1)#234#56(#789:;.<#=61;%>%61*#
!"#$%$ !"#$&$
!"#$'$ !"#$($

63. 54
APPENDIX C
SVS HUD FOR IMC-DAY CONDITION
!"#
!
!! !
!
!!
$%&'()#*+,-+#./.#012#34(#56789:;#<4=9%>%4=+
!"#$%$ !"#$&$
!"#$'$ !"#$($

64. 55
APPENDIX D
EVS HUD FOR IMC-DAY CONDITION
!"#
## #
!
!!
$%&'()#*+,-+#./0#123#45(#6789:;<#=5>:%?%5>+
!"#$%$ !"#$&$
!"#$'$ !"#$($

65. 56
APPENDIX E
COMBO HUD FOR IMC-DAY CONDITION
!"#
## #
!
!!
$%&'()#*+,!+#-./0.#123#4.(#56-789:#;.<8%=%.<+
!"#$%$ !"#$&$
!"#$'$ !"#$($

66. 57
APPENDIX F
BASELINE HUD FOR IMC-NIGHT CONDITION
!"#
## #
!
!!
$%&'()#*+,-+#./0)1%2)#345#67(#89:;2%&<=#>72?%=%72+
!"#$%$ !"#$&$
!"#$'$ !"#$($

67. 58
APPENDIX G
SVS HUD FOR IMC-NIGHT CONDITION
!!"
"" " "
!
!!
#$%&'(")*+,*"-.-"/01"23'"45678$%9:";38<$:$38*
!"#$%$ !"#$&$
!"#$' !"#$($

68. 59
APPENDIX H
EVS HUD FOR IMC-NIGHT CONDITION
!"#
## #
!
!!
$%&'()#*+,-+#./0#123#45(#6789:%&;<#=5:>%<%5:+
!"#$%$ !"#$&$
!"#$'$ !"#$($

69. 60
APPENDIX I
COMBO HUD FOR IMC-NIGHT CONDITION
!"#
## #
!
!!
$%&'()#*+,,+#-./0.#123#4.(#56-78%&9:#;.8<%:%.8+!
!"#$%$ !"#$&$
!"#$'$ !"#$($