This document discusses the UiT Autonomous Ship Program and its research on technologies to support autonomous maritime transportation systems. It proposes a ship intelligence framework (SIF) that uses deep neural networks (DNNs) trained on large datasets to mimic human ship navigator behavior. The goals are to overcome issues with ship controllability and replace human navigators. A decision support system would provide an adequate safety buffer to help DNNs handle unexpected situations. The framework is conceptualized based on factors behind successful self-driving cars, and aims to train DNNs using real-world ship navigation data to achieve accurate autonomous control.

1. Introduction Ship Intelligence Framework Important Concepts Navigation and Control Platform Conclusions
UiT Autonomous Ship Program
Current Status
Lokukaluge P. Perera1, Peter Wide1, Bjørn-M. Batalden1, Ricardo Pascoal 1 &
Brian Murray1
1Department of Technology and Safety
UiT The Arctic University of Norway
Tromso
April 2019
UiT The Arctic University of Norway April 2019 1 / 23

2. Introduction Ship Intelligence Framework Important Concepts Navigation and Control Platform Conclusions
Outline
1 Introduction
Transportation Systems
Ship Maneuvering
System Intelligence
2 Ship Intelligence Framework
Introduction
3 Important Concepts
4 Navigation and Control Platform
Ship Model
Vessel Systems
Onshore Command Center
5 Conclusions
UiT The Arctic University of Norway April 2019 2 / 23

3. Introduction Ship Intelligence Framework Important Concepts Navigation and Control Platform Conclusions
Transportation Systems
Autonomous Navigation
Autonomous navigation will play an important role in future transportation
systems.
The technologies required for autonomous navigation in land transportation
systems, i.e. self-driving cars such as Tesla, Uber, and Waymo, are in a mature
phase when the environment is structured, i.e. well-defined roads and
communication networks.
The required technological advancements for autonomous transportation
systems in an unstructured environment are subject to more challenging
navigation constraints.
Not only the required technologies for maritime transportation systems can
be more complex and still in a development phase, the infrastructure is in
general inadequate.
A considerable amount of infrastructure and technology challenges has been
encountered by maritime transportation systems in relation to autonomous
navigation.
This project proposes to research on the required fundamental technologies to
support future maritime transportation systems operation under autonomous
conditions.
This requires an understating of the challenges associated with ship
navigation and finding appropriate solutions.
UiT The Arctic University of Norway April 2019 3 / 23

4. Introduction Ship Intelligence Framework Important Concepts Navigation and Control Platform Conclusions
Ship Maneuvering
Vessel Controllability Problem
Vessel position (i.e. the centre of vessel
rotation) : P(x(t), y(t))
Course-speed vector : V(t)
Heading (surge) vector : u(t)
Sway velocity : v(t)
Drift angle : β(t)
Ship manoeuvring consists of complex rigid body motions.
This is due to large bandwidth of nonlinear hydrodynamic and wind force and
moment interactions between environment and the vessel hull and
superstructure, which often generate unexpected and undesirable ship
motions.
Vessels often have a heading vector that deviates from the course-speed
vector, resulting in a drift angle.
Since vessels are not navigating in fully-defined ship routes, one vessel can
encounter other vessels in its vicinity with various course-speed and heading
vectors.
Ship navigators should be aware of such encounters associated with higher
collision risk.
UiT The Arctic University of Norway April 2019 4 / 23

5. Introduction Ship Intelligence Framework Important Concepts Navigation and Control Platform Conclusions
Ship Maneuvering
Vessel Controllability Problem (cont.)
Vessel position (i.e. the centre of vessel
rotation) : P(x(t), y(t))
Course-speed vector : V(t)
Heading (surge) vector : u(t)
Sway velocity : v(t)
Drift angle : β(t)
Full controllability of vessels with rudder and propeller actuators is not
possible and is especially demanding under rough weather conditions.
Vessels present sway-yaw manoeuvring interactions and are thus
under-actuated systems with heavy inertia.
The course-speed vector cannot be measured or estimated with enough
accuracy, i.e. not enough sensor measurements.
The centre of vessel rotation can also change due to the environmental
loads.
Not only the respective navigation vectors but also their positions can change
without any requests.
Vessels are considered as slow response systems with considerable
time-delays in response to discrete control requests.
UiT The Arctic University of Norway April 2019 5 / 23

6. Introduction Ship Intelligence Framework Important Concepts Navigation and Control Platform Conclusions
System Intelligence
Classical Mechanics 2 AI
Various advanced controllers based on classical mechanics have been
introduced by the research community to address this ship controllability
problem , however the outcomes are still not satisfactory.
The main reason is that these mathematical models may not adequately
capture the complexities in ship motions; therefore controller robustness
and/or stability cannot be preserved during ship manoeuvres.
The controller inputs, i.e. rudder and propeller control inputs, are not
continuous and the controller outputs, i.e. heading and course-speed vectors,
may not have adequate accuracy and/or associated time-delays, hence
controller performance can be further degraded.
Though conventional ship auto-pilot systems are based on similar approaches,
such systems may not able to handle complex navigational constraints.
Ocean going vessels are still navigated by humans, especially in cluttered
navigation zones and rough weather, using their knowledge and experiences to
overcome complex vessel motions.
Our aim is to overcome these issues in ship navigation by introducing Artificial
Intelligence (AI) into the ship controllability problem.
That has categorized as Cloning Ship Navigator Behavior.
UiT The Arctic University of Norway April 2019 6 / 23

7. Introduction Ship Intelligence Framework Important Concepts Navigation and Control Platform Conclusions
System Intelligence
Self-driving Cars 2 Autonomous Ships
This project proposes to capture and mimic navigator knowledge and
experiences by using deep learning, i.e. deep neural networks (DNNs), as a
ground-breaking technology that facilitates a better solution to control vessels.
A considerable number of sensors should also be on-board and the respective
data should be fused in a perception framework to achieve this objective.
In self-driving cars, the respective driver is successfully replaced by DNNs
trained to mimic human behaviour.
The success in self-driving cars is due to three main factors 1 :
1 collecting and analysing large-scale real-world driving data
sets, including sensor and high definition video/image data, to
support deep learning based digital drivers.
2 holistic understanding of how human drivers interact with
vehicle automation technologies by observing video/image and
vehicle motion data, driving characteristics, human knowledge
and experiences with the new technologies during the training
phase.
3 adequate safety buffer to save lives by identifying how
technology and other related factors can be used during the
self-driving phase, i.e. the execution phase.
1. A. Fridman, et al., MIT Autonomous Vehicle Technology Study : Large-Scale Deep Learning Based Analysis of
Driver Behavior and Interaction with Automation lessArXiv2017
UiT The Arctic University of Norway April 2019 7 / 23

8. Introduction Ship Intelligence Framework Important Concepts Navigation and Control Platform Conclusions
System Intelligence
Human-AI-Technology-Regulations Interactions
UiT The Arctic University of Norway April 2019 8 / 23

9. Introduction Ship Intelligence Framework Important Concepts Navigation and Control Platform Conclusions
System Intelligence
Deep Neural Networks (DNNs)
This is a considerable deviation from conventional control approaches in ship
navigation developed in the last decade.
Instead of control flow logics and if-then-else statements, DNNs consist of
state-of-the-art Neural Networks with many layers and millions to one billion
parameters, i.e. the weights of the respective neurons, of nonlinear activation
functions.
Convolutional neural networks are the most popular DNNs for self-driving cars
and those network parameters are adjusted via back-propagation type
approaches.
DNNs require a large amount of real-world vessel navigation data and
hundreds of thousands to millions of forward and backward training
iterations to achieve higher accuracy in navigator behaviour.
DNNs consist of large numbers of identical neurons with a highly parallel
structure that can be mapped to GPUs (Graphics processing unit) naturally to
obtain a higher computational speed when compared to CPUs based training.
UiT The Arctic University of Norway April 2019 9 / 23

10. Introduction Ship Intelligence Framework Important Concepts Navigation and Control Platform Conclusions
System Intelligence
Adequate Safety Buffer
It would be difficult build DNNs as a robust and safety critical system purely
by human training.
Unexpected and undesirable motion and navigation conditions can be
encountered by ship navigation situations.
If the DNNs have not seen such situations during its training phase and
generalization is poor in the execution phase, that could create undesirable
behavior.
A decision support layer with adequate information sources should support
the DNNs to overcome such situations.
Situation awareness and collision avoidance (SACA) is identified as the
minimal decision support facility required to support the training and
execution phases.
This can create an adequate safety buffer to avoid possible collision or
near-miss situations, by identifying moving and stationary objects around the
vessel domain.
DNNs are integrated into the ship intelligence framework (SIF) created in a
conceptual level by the UiT autonomous ship program, while considering the
same main factors of self-driving cars.
UiT The Arctic University of Norway April 2019 10 / 23

11. Introduction Ship Intelligence Framework Important Concepts Navigation and Control Platform Conclusions
Introduction
SIF (cont.)
UiT The Arctic University of Norway April 2019 11 / 23

12. Introduction Ship Intelligence Framework Important Concepts Navigation and Control Platform Conclusions
Introduction
SIF in The Training Phase
UiT The Arctic University of Norway April 2019 12 / 23

13. Introduction Ship Intelligence Framework Important Concepts Navigation and Control Platform Conclusions
Introduction
SIF in The Execution Phase
UiT The Arctic University of Norway April 2019 13 / 23

14. Introduction Ship Intelligence Framework Important Concepts Navigation and Control Platform Conclusions
Introduction
A View from the Bridge
Information Visualization Platform (IVP) on the bridge.
Decision support system for ship navigators during the training phase and DNNs during
the execution phase.
Required ship route as the Digital Ship Route (DSR)
Actual & Predicted ship route as the Advanced Ship Predictor (ASP).
Local scale with ship performance and navigation data.
Global scale with AIS data.
The Situation Awareness and Collision Avoidance (SACA) Module.
Target Detection and Tacking Unit (TDTU)
Collision Risk Assessment Unit (CRAU)
UiT The Arctic University of Norway April 2019 14 / 23

15. Introduction Ship Intelligence Framework Important Concepts Navigation and Control Platform Conclusions
Advanced Ship Predictor
FIGURE – Local scale FIGURE – Global scale
UiT The Arctic University of Norway April 2019 15 / 23

16. Introduction Ship Intelligence Framework Important Concepts Navigation and Control Platform Conclusions
Situation Awareness and Collision Avoidance (SACA)
FIGURE – Local scale FIGURE – Global scale
UiT The Arctic University of Norway April 2019 16 / 23

17. Introduction Ship Intelligence Framework Important Concepts Navigation and Control Platform Conclusions
Ship Model
Autonomous Vessel
Small scale vessel will be purchased.
UiT The Arctic University of Norway April 2019 17 / 23

18. Introduction Ship Intelligence Framework Important Concepts Navigation and Control Platform Conclusions
Vessel Systems
GNSS & INS System
UiT The Arctic University of Norway April 2019 18 / 23

19. Introduction Ship Intelligence Framework Important Concepts Navigation and Control Platform Conclusions
Vessel Systems
Engine, Rudder & Propulsion System
UiT The Arctic University of Norway April 2019 19 / 23

20. Introduction Ship Intelligence Framework Important Concepts Navigation and Control Platform Conclusions
Vessel Systems
Other Sensors
UiT The Arctic University of Norway April 2019 20 / 23

21. Introduction Ship Intelligence Framework Important Concepts Navigation and Control Platform Conclusions
Vessel Systems
Communication System
UiT The Arctic University of Norway April 2019 21 / 23

22. Introduction Ship Intelligence Framework Important Concepts Navigation and Control Platform Conclusions
Onshore Command Center
Initial Experiments
Ferry crossing type maneuvers will be conducted under this vessel, initially.
Autonomous Test-Site in Tromso will be developed with the legal requirements.
Ashore Remote-controlled Center will be developed near the test-site.
Ashore Operational Center will be developed in UiT to visualize sensor data for
teaching and research purpose.
UiT The Arctic University of Norway April 2019 22 / 23

23. Introduction Ship Intelligence Framework Important Concepts Navigation and Control Platform Conclusions
Any Question?
UiT The Arctic University of Norway April 2019 23 / 23