Webinar presentation on the UiT Autonomous Ship Program, including recent research activities.

1. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
UiT Autonomous Ship Program
An Overview
Lokukaluge Prasad Perera1
1Associate Professor
Department of Technology and Safety
UiT The Arctic University of Norway
Tromso
June, 2020
UiT The Arctic University of Norway 1 / 28

2. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Outline
1 Introduction
Transportation Systems
Classical Mechanics
Machine Learning
2 Ship Intelligence Framework
Building Blocks
Ship Intelligence Framework
3 Vessel Sensors & Systems
Sub-Systems
4 Remote Operation Center
Possible Facilities
5 Situation Awareness
Recent Research Topics
Bridge Simulator Experiments
Sea Trials
6 Conclusions
Communication
UiT The Arctic University of Norway 2 / 28

3. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Transportation Systems
Autonomous Navigation
Autonomous navigation will play an important role in future transportation
systems.
The technologies required for autonomous navigation in land transportation
systems, i.e. self-driving cars such as Tesla, Uber, and Waymo, are in a mature
phase and the environment is structured, i.e. well-defined roads and
communication networks.
The required technological advancements for autonomous transportation
systems in an unstructured environment are subject to more challenging
navigation constraints.
Not only the required technologies for maritime transportation systems can
be more complex and still in a development phase, but also the present
infrastructure is inadequate, in general.
A considerable amount of infrastructure and technology challenges has been
encountered by maritime transportation systems in relation to autonomous
navigation.
The required fundamental technologies to support future maritime
transportation systems under autonomous navigation should be developed.
This requires an understating of the challenges associated with ship navigation
and finding appropriate solutions.
UiT The Arctic University of Norway 3 / 28

4. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Classical Mechanics
Vessel Controllability Problem
Vessel position (i.e. the centre of vessel
rotation) : P(x(t), y(t))
Course-speed vector : V(t)
Heading (surge) vector : u(t)
Sway velocity : v(t)
Drift angle : β(t)
Ship manoeuvring consists of complex rigid body motions.
This is due to large bandwidth of nonlinear hydrodynamic and wind force and
moment interactions between environment and the vessel hull and
superstructure, which often generate unexpected and undesirable ship
motions.
A vessel can have a heading vector that deviates from the course-speed
vector, resulting in a drift angle.
Since vessels are not navigating in fully-defined ship routes, one vessel can
encounter other vessels in its vicinity with various course-speed and heading
vectors.
Ship navigators should be aware of such encounters associated with higher
collision risk.
UiT The Arctic University of Norway 4 / 28

5. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Classical Mechanics
Vessel Controllability Problem (cont.)
Vessel position (i.e. the centre of vessel
rotation) : P(x(t), y(t))
Course-speed vector : V(t)
Heading (surge) vector : u(t)
Sway velocity : v(t)
Drift angle : β(t)
Full controllability of vessels with rudder and propeller actuators is not
possible and is especially demanding under rough weather conditions.
Vessels present sway-yaw manoeuvring interactions and are thus
under-actuated systems with heavy inertia.
The course-speed vector cannot be measured or estimated with enough
accuracy, i.e. not enough sensor measurements.
The centre of vessel rotation can also change due to the environmental
loads.
Not only the respective navigation vectors but also their positions can change
without any requests.
Vessels are considered as slow response systems with considerable
time-delays in response to discrete control requests.
UiT The Arctic University of Norway 5 / 28

6. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Machine Learning
Classical Mechanics 2 AI
Various advanced controllers based on classical mechanics have been
introduced by the research community to address this ship controllability
problem , however the outcomes are still not satisfactory.
The main reason is that these mathematical models may not adequately
capture the complexities in ship motions; therefore controller robustness
and/or stability cannot be preserved during ship manoeuvres.
The controller inputs, i.e. rudder and propeller control inputs, are not
continuous and the controller outputs, i.e. heading and course-speed vectors,
may not have adequate accuracy and/or associated time-delays, hence
controller performance can be further degraded.
Though conventional ship auto-pilot systems are based on similar approaches,
such systems may not able to handle complex navigational constraints.
Ocean going vessels are still navigated by humans, especially in cluttered
navigation zones and rough weather, using their knowledge and experiences to
overcome complex vessel motions.
Our aim is to overcome these issues in ship navigation by introducing Artificial
Intelligence (AI) into the ship controllability problem.
That has categorized as Cloning Ship Navigator Behavior.
UiT The Arctic University of Norway 6 / 28

7. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Machine Learning
Self-driving Cars 2 Autonomous Ships
This study proposes to capture and mimic navigator knowledge and
experiences by using deep learning, i.e. deep neural networks (DNNs), as a
ground-breaking technology that facilitates a better solution to control vessels.
A considerable number of sensors should also be on-board and the respective
data should be fused in a perception framework to achieve this objective.
In self-driving cars, the driver is successfully replaced by DNNs trained to
mimic human behavior, due to three main factors 1 :
collecting and analyzing large-scale real-world
driving data sets, including sensor and high definition
video/image data, to support deep learning based
digital drivers.
holistic understanding of how human drivers
interact with vehicle automation technologies by
observing video/image and vehicle motion data, driving
characteristics, human knowledge and experiences with
the new technologies during the training phase.
adequate safety buffer to save lives by identifying how
technology and other related factors can be used during
the self-driving phase, i.e. the execution phase.
1. A. Fridman, et al., MIT Autonomous Vehicle Technology Study : Large-Scale Deep Learning Based Analysis of
Driver Behavior and Interaction with Automation lessArXiv2017
UiT The Arctic University of Norway 7 / 28

8. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Building Blocks
The Key Pillars
UiT The Arctic University of Norway 8 / 28

9. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Building Blocks
Deep Neural Networks (DNNs)
This is a considerable deviation from conventional control approaches in ship
navigation developed in the last decade.
Instead of control flow logics and if-then-else statements, DNNs consist of
state-of-the-art Neural Networks with many layers and millions to one billion
parameters, i.e. the weights of the respective neurons, of nonlinear activation
functions.
Convolutional neural networks are the most popular DNNs for self-driving cars
and those network parameters are adjusted via back-propagation type
approaches.
DNNs require a large amount of real-world vessel navigation data and
hundreds of thousands to millions of forward and backward training
iterations to achieve higher accuracy in navigator behaviour.
DNNs consist of large numbers of identical neurons with a highly parallel
structure that can be mapped to GPUs (Graphics processing unit) naturally to
obtain a higher computational speed when compared to CPUs based training.
UiT The Arctic University of Norway 9 / 28

10. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Building Blocks
Adequate Navigation Safety
It would be difficult build DNNs as a robust and safety critical system purely
by human training.
Unexpected and undesirable motion and navigation conditions can be
encountered by ship navigation situations.
If the DNNs have not seen such situations during its training phase and
generalization is poor in the execution phase, that could create undesirable
behavior.
A decision support layer with adequate information sources should support
the DNNs to overcome such situations.
Situation awareness and collision avoidance (SACA) is identified as the
minimal decision support facility required to support the training and
execution phases.
This can create an adequate safety buffer to avoid possible collision or
near-miss situations, by identifying moving and stationary objects around the
vessel domain.
DNNs are integrated into the ship intelligence framework (SIF) created in a
conceptual level by the UiT autonomous ship program, while considering the
same main factors of self-driving cars.
UiT The Arctic University of Norway 10 / 28

11. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Ship Intelligence Framework
A General Framework
UiT The Arctic University of Norway 11 / 28

12. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Ship Intelligence Framework
The Training Phase
UiT The Arctic University of Norway 12 / 28

13. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Ship Intelligence Framework
The Execution Phase
UiT The Arctic University of Norway 13 / 28

14. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Ship Intelligence Framework
Bridge View
Information Visualization Platform (IVP).
Decision support system for ship navigators during the training phase and DNNs during
the execution phase.
Required ship route as the Digital Ship Route (DSR)
Actual & Predicted ship route as the Advanced Ship Predictor (ASP).
Local scale with ship performance and navigation data.
Global scale with AIS data.
Situation Awareness and Collision Avoidance (SACA) Module.
Target Detection and Tacking Unit (TDTU)
Collision Risk Assessment Unit (CRAU)
UiT The Arctic University of Norway 14 / 28

15. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Sub-Systems
Sub-system 1
UiT The Arctic University of Norway 15 / 28

16. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Sub-Systems
Sub-system 2
UiT The Arctic University of Norway 16 / 28

17. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Sub-Systems
Sub-system 3
UiT The Arctic University of Norway 17 / 28

18. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Possible Facilities
Remote Operation Center
This center will be developed inside UiT.
The center will have remote controlled facilities of the vessel.
The sea trial data will be uploaded into the servers in the center.
The vessel will be remote controlled by the center.
UiT The Arctic University of Norway 18 / 28

19. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Recent Research Topics
Advanced Ship Predictor
Work done by a PhD student : TBA.
FIGURE – Advanced Ship Predictor in Local scale
Work done by a PhD student : Brian Murray.
FIGURE – Advanced Ship Predictor in Global scale
UiT The Arctic University of Norway 19 / 28

20. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Bridge Simulator Experiments
Simulated Experiments
FIGURE – UiT Bridge Simulator
UiT The Arctic University of Norway 20 / 28

21. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Bridge Simulator Experiments
Collision Avoidance Decisions
The respective Collision Risk in each ship encounter situation should be
estimated.
The same collision risk should be used to formulate appropriate Collision
Avoidance Decisions.
The respective collision avoidance decisions by humans and systems should have
a Good Consistency, since the digital helmsman is trained by human navigators.
The combination of those two components, i.e. the collision risk estimation and
collision avoidance decisions, can be categorized a decision support feature
and can be used by both humans and systems to make appropriate collision
avoidance actions.
Collision avoidance actions : Speed Change(to increase or decrease) and
Course Change (to the starboard and port).
UiT The Arctic University of Norway 21 / 28

22. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Bridge Simulator Experiments
Collision Avoidance Actions
Collision avoidance actions should be executed through ship propulsion and
rudder control systems.
The alteration of course is the most preferred collision avoidance action, since
low speed conditions can also reduce ship maneuverability.
Since both humans and systems convert the collision avoidance decisions into
actions, their Regulatory Compliance should also be evaluated.
Any inconsistency between human and system collision avoidance actions
can be eliminated to preserve the integrity of system intelligence.
The respective regulatory compliance in relation to possible COLREGs failure
situations under several simulated system decision making situations in ship
navigation should be investigated.
That outcome can also help to improve digital helmsman behavior and identify
possible regulatory modifications to support future vessels.
UiT The Arctic University of Norway 22 / 28

23. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Bridge Simulator Experiments
Possible Regulatory Failures
UiT The Arctic University of Norway 23 / 28

24. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Sea Trials
Realistic Ship Navigation Data Collection
UiT The Arctic University of Norway 24 / 28

25. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Sea Trials
Cognitive ability of Human & Digital Helmsman
UiT The Arctic University of Norway 25 / 28

26. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Communication
Our Team
Peter Wide, Lokukaluge Prasad Perera, Gudmund Johansen, Bjørn-Morten
Batalden, Ricardo Pascoal.
UiT The Arctic University of Norway 26 / 28

27. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Communication
Social Media
WebPage
https ://en.uit.no/prosjekter/prosjekt?p_document_id=668855
Linkedin
https ://www.linkedin.com/company/autonomousuit
YouTube
https ://www.youtube.com/channel/UCdRmcUaSn-i4oAktp11VWdA
Researchgate
https ://www.researchgate.net/project/UiT-Autonomous-Ship-Program
UiT The Arctic University of Norway 27 / 28

28. Introduction Ship Intelligence Framework Vessel Sensors & Systems Remote Operation Center Situation Awareness Conclusions
Communication
Any Question?
UiT The Arctic University of Norway 28 / 28