The Interaction of Navigation Instructions and
Visual Attention in Dynamic Automotive Environments*
Hendrik Koesling & Ronan G. Reilly
Department of Computer Science
National University of Ireland Maynooth
Maynooth, Co. Kildare
Ireland
e-mail: hendrik@cs.nuim.ie
* Part-funded by the German Science Foundation (DFG grant KO2350/1-1)

Overview
Navigation in Dynamic Automotive Environments
Driver-assistant devices such as navigation systems are a common feature
in many modern vehicles (e.g. Barfield & Dingus, 1998). Processing
navigation instructions, however, is a challenging task that has to be
accomplished in parallel to the actual driving task and often distracts drivers
(e.g Recarte & Nunes, 2000).
A flexible guidance of attention could optimise the cognitive workload spent
on each task and improve driving safety: In dynamic driving environments,
navigation messages should be adapted according to the current traffic
situation. To verify this hypothesis, we investigated how attending to
INTRODUCTION
dynamic road scenes while following navigation messages (“visuo-linguistic
interaction”) affects message comprehension and driver performance.
METHOD To quantify this relation, we suggest to modify a model that successfully
describes the incremental generation of representations from spoken
language in static scenes by serially mapping navigation instructions onto
gaze trajectories (e.g. Allopenna et al., 1998; Eberhard et al., 1995; Spivey
RESULTS
et al., 2002). This may allow us to develop optimal properties of navigation
messages for specific traffic situations.
CONCLUSIONS Koesling & Reilly: The Interaction of Navigation Instructions and Visual Attention in Dynamic Automotive Environments

Paradigm
Driving while following navigation instructions
• Task
INTRODUCTION
METHOD
- A dual task? Not really! It is a multiple task:
a) Actual driving (e.g. steering, changing gears, etc.)
RESULTS
b) Observation of traffic, environment and mirrors
c) Listening to navigation messages
CONCLUSIONS Koesling & Reilly: The Interaction of Navigation Instructions and Visual Attention in Dynamic Automotive Environments

Paradigm
Driving while following navigation instructions
• Challenges
INTRODUCTION
METHOD
- Complex tasks
- Complex environments
RESULTS
- Parallel processing of all tasks required
- Dynamically changing environment
CONCLUSIONS Koesling & Reilly: The Interaction of Navigation Instructions and Visual Attention in Dynamic Automotive Environments

Paradigm
Driving while following navigation instructions
Empty road, clear environment Busy traffic, cluttered environment
• Goal
INTRODUCTION - Optimise cognitive workload spent on each task in order to improve
driving safety
• Goal approach
METHOD - Dynamic adaptation of navigation messages according to current
traffic situation
• Methodological approach
RESULTS - Use distribution of (visual) attention as indicator for cognitive workload
- Eye tracking: Recording of eye movements to monitor visual attention
(e.g. Just & Carpenter, 1987)
CONCLUSIONS Koesling & Reilly: The Interaction of Navigation Instructions and Visual Attention in Dynamic Automotive Environments

Paradigm Focus: Visuo-Linguistic Interaction
Attending to road scenes while listening to navigation instructions
• Tasks & Effects
- Observation of traffic and environment
- Listening to navigation messages
- Effects on message comprehension and driver performance
• Key factors
- Scene complexity
- Linguistic structure of instruction messages
INTRODUCTION
- Message timing
• Accounting for driving situation dynamics
METHOD - Assess visual attention distribution before, during and after instruction
• Relating instruction processing, visual attention and driver performance
- Incremental generation of representations from spoken language (e.g.
RESULTS
Allopenna et al., 1998; Eberhard et al., 1995; Spivey et al., 2002)
- Map gaze trajectories onto navigation instructions
CONCLUSIONS Koesling & Reilly: The Interaction of Navigation Instructions and Visual Attention in Dynamic Automotive Environments

Research Questions & Hypotheses
1. Which properties of navigation instructions, traffic situations and their
interaction particularly affect message understanding?
2. How does the distribution of attention change between the different
phases of driving while following navigation instructions?
3. Can we – analogous to the incremental generation of representations
INTRODUCTION
from spoken language in static scenes – serially map gaze
trajectories onto navigation instructions in dynamic situations?
METHOD
4. Does a correlation exist between possible disturbances of this
mapping and a deterioration of driver responses?
RESULTS
CONCLUSIONS Koesling & Reilly: The Interaction of Navigation Instructions and Visual Attention in Dynamic Automotive Environments

Experiment Details
Participants
- n = 80 experienced drivers
- 50% male, 50% female
- driving experience: M = 9.4 years (SD = 2.3)
Stimuli
- 24 video sequences of road scenes
- overlaid with auditory navigation instructions (turning directions)
Experimental task
1. ”Left/Right”-response: Turn left or right at the specified junction
INTRODUCTION 2. “Safe/Unsafe”-response: Is it safe to turn at the junction or not?
Communication of the decision
1. Left or right response button, respectively (during video presentation)
METHOD
2. Verbal (e.g., “safe”; after video presention)
Apparatus
RESULTS - SR Research Eyelink II eye tracker
- “VDesigner” visual programming environment for eye-tracking experiments
(e.g. Koesling, Clermont & Ritter, 2001)
CONCLUSIONS Koesling & Reilly: The Interaction of Navigation Instructions and Visual Attention in Dynamic Automotive Environments

Experiment Details
Independent variables
• Visual - Scene complexity (number of vehicles & turning alternatives)
• Linguistic - Message complexity (message length & number of propositions)
- Verb position
• Temporal - Message timing (in relation to turning point)
Dependent variables
INTRODUCTION • Response correctness
• Number of fixations NF
METHOD • Gaze duration GD
• NF and GD separated for
- time intervals (before, during and after message presentation) and
RESULTS
- spatial stimulus regions (verbal reference, relevant road furniture, vehicles
and irrelevant areas)
CONCLUSIONS Koesling & Reilly: The Interaction of Navigation Instructions and Visual Attention in Dynamic Automotive Environments

Experiment Set-Up
Eye-tracking
laboratory
INTRODUCTION
METHOD
EyeLink II eye tracker
RESULTS
CONCLUSIONS Koesling & Reilly: The Interaction of Navigation Instructions and Visual Attention in Dynamic Automotive Environments

1. Comprehension of Instructions
Response correctness
100
90
correctness %
80
70
60
INTRODUCTION
"left/right" "safe/unsafe"
decision type
METHOD
Significant variation with
• Scene complexity
RESULTS Independent of all factors
• Message complexity
• Message timing
CONCLUSIONS Koesling & Reilly: The Interaction of Navigation Instructions and Visual Attention in Dynamic Automotive Environments

1. Comprehension of Instructions
Safe/Unsafe response
Scene complexity Message complexity
100 100
F(1; 76) = 57.33; p < 0.001 F(2; 152) = 61.96; p < 0.001
s/u correctness %
s/u correctness %
90 90
80 80
70 70
60 60
simple complex simple intermediate complex
scene complexity message complexity
INTRODUCTION
Message issue time Verb position
100 100
F(1; 76) = 77.02; p < 0.001 No significant effect
s/u correctness %
s/u correctness %
METHOD 90 90
80 80
70 70
RESULTS 60 60
early late start end
message issue time verb position
CONCLUSIONS Koesling & Reilly: The Interaction of Navigation Instructions and Visual Attention in Dynamic Automotive Environments

1. Comprehension of Instructions
Summary
• Highly correct “right/left” responses, independent of all factors
• Considerable errors in “safe/unsafe” responses, varying with
• message complexity
• scene complexity
• message issuing time
• but not with verb position
• Significantly higher safe/unsafe error rates
INTRODUCTION
• for complex messages
• for complex scenes
METHOD
• when messages are issued late
• Interaction between message complexity, message timing and verb position
[F(3; 76) = 60.42; p < 0.001]
RESULTS
• Safe/unsafe decision correctness drops significantly for complex, late
messages with verb at end of instruction
CONCLUSIONS Koesling & Reilly: The Interaction of Navigation Instructions and Visual Attention in Dynamic Automotive Environments

2. Distribution of Attention
Number of fixations rNF
Vehicle areas Relevant route areas
80 80
F(2; 152) = 150.23; p < 0.001 F(2; 152) =112.72; p < 0.001
rNF vehicles %
60 60
rNF route %
40 40
20 20
0 0
before during after before during after
time interval (relative to msg.) time interval (relative to msg.)
INTRODUCTION
Instruction reference areas Irrelevant areas
80 80
F(2; 152) = 162.90; p < 0.001 No significant effect
rNF irrelevant %
rNF instruc %
METHOD 60 60
40 40
20 20
RESULTS 0 0
before during after before during after
time interval (relative to msg.) time interval (relative to msg.)
CONCLUSIONS Koesling & Reilly: The Interaction of Navigation Instructions and Visual Attention in Dynamic Automotive Environments

2. Distribution of Attention
Gaze duration rGD
Vehicle areas Relevant route areas
80 80
F(2; 152) = 145.11; p < 0.001 F(2; 152) =128.82; p < 0.001
rGD vehicles %
rGD route %
60 60
40 40
20 20
0 0
before during after before during after
time interval (relative to msg.) time interval (relative to msg.)
INTRODUCTION
Instruction reference areas Irrelevant areas
80 80
F(2; 152) = 165.44; p < 0.001 No significant effect
rGD irrelevant %
rGD instruc %
METHOD 60 60
40 40
20 20
RESULTS 0 0
before during after before during after
time interval (relative to msg.) time interval (relative to msg.)
CONCLUSIONS Koesling & Reilly: The Interaction of Navigation Instructions and Visual Attention in Dynamic Automotive Environments

2. Distribution of Attention
Gaze trajectories
INTRODUCTION
METHOD
RESULTS
CONCLUSIONS Koesling & Reilly: The Interaction of Navigation Instructions and Visual Attention in Dynamic Automotive Environments

2. Distribution of Attention
Gaze trajectories
INTRODUCTION
METHOD
RESULTS
CONCLUSIONS Koesling & Reilly: The Interaction of Navigation Instructions and Visual Attention in Dynamic Automotive Environments

2. Distribution of Attention
Gaze trajectories
before instruction
during instruction
after instruction
INTRODUCTION
METHOD
RESULTS
CONCLUSIONS Koesling & Reilly: The Interaction of Navigation Instructions and Visual Attention in Dynamic Automotive Environments

2. Distribution of Attention
Gaze trajectories
before instruction
during instruction
after instruction
INTRODUCTION
METHOD
RESULTS
CONCLUSIONS Koesling & Reilly: The Interaction of Navigation Instructions and Visual Attention in Dynamic Automotive Environments

2. Distribution of Attention
General visual processing pattern
• Before instruction
• Coarse scene overview
• Several brief fixations, mainly foveating vehicles
• Fixations connected by long-range saccades
• During instruction
• Focus of attention centers on areas verbally referred to, i.e. guided by
INTRODUCTION
navigation instruction
• Some attention spent on relevant road furniture and route information
• Little attention is being paid to other vehicles
METHOD
• After instruction
RESULTS
• Focus of attention shifts back to vehicles’ paths and relevant route
areas for action planning
CONCLUSIONS Koesling & Reilly: The Interaction of Navigation Instructions and Visual Attention in Dynamic Automotive Environments

2. Distribution of Attention
Special visual processing patterns
• Inefficient visual processing
• Complex scene, message issued early, verb at start, e.g. “Turn left at
second junction.”
• Saccades to first junction immediately after “left” was issued
• Distraction caused by redundancies
• “Turn left into minor road after bridge” when only one turn left exists
INTRODUCTION
• Sequential mapping of message constituents onto visual reference
points leads to processing of redundant information instead of other
task-relevant areas
METHOD • Most critical in complex scenes
• However, redundancies may be used as “back-up” in simple scenes
RESULTS
CONCLUSIONS Koesling & Reilly: The Interaction of Navigation Instructions and Visual Attention in Dynamic Automotive Environments

3. & 4. Incremental Generation of Representations
Model: Mapping gaze trajectories onto navigation instructions
• Procedure
• Coding instructions and gaze trajectories in symbol sequences
• Symbols represent words and fixations
• Fixation locations determine fixation symbols (according to stimulus area)
• Vehicle areas change dynamically
• Comparison of symbol sequences indicates how (scene) representations
are generated
INTRODUCTION
METHOD
“Turn left at second junction”
RESULTS A B C D
CONCLUSIONS Koesling & Reilly: The Interaction of Navigation Instructions and Visual Attention in Dynamic Automotive Environments

3. & 4. Incremental Generation of Representations
Model: Mapping gaze trajectories onto navigation instructions
• Findings
• In general, the fixation sequence closely follows the instruction sequence
• The visual focus is thus guided by the navigation instruction, very similar
to the incremental generation of representations from spoken language in
static scenes
• However, depending on scene and message parameters, the incremental
generation can be interrupted. Guidance of visual attention by instructions
pauses …
INTRODUCTION
… when relevant route
… when other vehicles approach. information is attended to.
METHOD
Instruction processing is resumed, skipping previous message part
RESULTS
Most likely for complex Most likely for complex
scenes and messages scenes and late messages
CONCLUSIONS Koesling & Reilly: The Interaction of Navigation Instructions and Visual Attention in Dynamic Automotive Environments

Summary and Conclusions
• Analysis of eye movements revealed how visuo-linguistic interaction
affected visual attention
• Distribution of attention changes considerably between the different phases
of driving while following navigation instructions
• When a navigation instruction is issued, the visual focus is largely guided by
the message
• However, relevant scene information and vehicle movements are also taken
into account
INTRODUCTION
• Scene and message complexity and message timing affect the processing
of this additional information
METHOD
• If the sequential processing of information directly referred to in the
instruction is interrupted for too long, the resumption of message
processing is critical and leads to increased error rates for message
RESULTS
comprehension
CONCLUSIONS Koesling & Reilly: The Interaction of Navigation Instructions and Visual Attention in Dynamic Automotive Environments

Summary and Conclusions
• Perception and processing of navigation messages are greatly compromised
when visual attention moves away from items currently referred to by the
instruction, suggesting a strong visuo-linguistic interaction between message
perception (auditory), message processing (linguistic) and referencing
(visual) as well as message comprehension (cognitive)
• It appears that verbal and visual information cannot be processed in parallel
when an asynchrony exists between the current verbal input and the focus of
visual attention
INTRODUCTION • The attempt to synchronise the auditory and visual input streams often fails
as message comprehension deteriorates in the dynamic, time critical
environments tested in the study
METHOD
• It could thus be beneficial for navigation systems to dynamically generate (or
adapt) its verbal messages depending on the current traffic situation
RESULTS • Navigation instruction generation guided by scene content could optimise the
processing of concurrent tasks and thus improve driving safety
CONCLUSIONS Koesling & Reilly: The Interaction of Navigation Instructions and Visual Attention in Dynamic Automotive Environments

Thank you for your attention!
Hendrik Koesling & Ronan G. Reilly
The Interaction of Navigation Instructions and Visual Attention in Dynamic Automotive Environments
NUI Maynooth, Ireland
e-mail: hendrik@cs.nuim.ie