1. Rocking the boat with big
data
www.bigdataocean.eu
Dr. Spiros Mouzakitis (NTUA)
Deputy Project Manager
GiannisTsapelas(NTUA)
Senior Researcher

2. Around 80% of global trade by volume is carried
by sEA
….and around 70% of global
trade by value

3. CHALLEN
GESo Undeveloped sharing of Blue Data between
enterprises and entities of the maritime domain
and other domains
o Lack of agreed standards and formats
o Huge potential from cross-sectorial blue data
applications – still unexploited
o Out-of-the-box Big Data solution that enables
advanced queries and analytics on cross-sector
data - missing

4. Develop a Maritime Big Data platform
that delivers out-of-the-box, value-added data and
analytic services for maritime applications
by exploiting cross-sector data streams
4

6. BigDataOcean platform (services)
Find maritime data
Query / interlink datasets Compose analytic services
Visualize datasets
Create (real-time) dashboards
Request on-demand services

7. Port
Authorities
Maritime
Research
Centers
Ship-owning
companies
Maritime
software
suppliers
Service
providers at
sea level
Maritime
advisory firms

8. BigDataOcean end-user applications
Vessel Fault
Detection,
Predictive
Maintenance
and Fuel
Consumption
Reduction
Maritime
security and
anomaly
detection
Oil spill
modeling
Wave power
generation
8BDV PPP Technical Committee Meeting

9. Case 1 - Fault Prediction & Proactive Maintenance
and Fuel Consumption
9

10. Case 1 - Fault Prediction & Proactive Maintenance
and Fuel Consumption
10
• Damage and mechanical failures detection and predictive
maintenance of vessel equipment
• Investigation of the impact of the environmental conditions
and the operational decisions taken on the vessel's fuel
consumption
Goal
• Ship owners, Maritime EquipmentConstructors
Stakeholders

11. BigDataOcean Solution
Analytics and knowledge
base about fuel
consumption and repairs
Prediction models for
maintenance and fuel
consumption
Maritime Regulations, Flag
& Ship Classification
Provisions
Plant Maintenance
System for Main Engine
& spare parts
Vessel
routes
In-situ Observations
Cross-domain
Forecast Data
Port information
BigDataOcean
Solution

15. Business Benefits
● Shipping companies and Maritime Equipment Constructors
 Minimum repairs, maintenance cost and fuel consumption
 Maximum vessels’ use and financial benefit
 Minimize environment impact
 Reliability and innovation
 Advanced Data analytics related to maintenance and fuel
consumption

16. Case 2 – Mare protection – Oil Spill dispersion
forecast
16

17. Case 2 – Mare protection – Oil Spill dispersion
forecast
17
• Provide to the end users oil spill drift forecasting and
simulation services for the marine environment
• Enhance the efficiency in managing oil spill pollution risks.
Goal
• Emergency Response Companies, National Entities
(PublicAuthorities), NGOs, Marine Research Institutes,
Shipping companies and Oil drilling companies
Stakeholders

18. Natura /
Protecte
d areas
In-situ Observations
BigDataOcean Solution
Graphical output
Cross-domain
Forecast Data
POSEIDON
OSM
Oil spill scenario
submission
Location, Rate,
Nature & characteristics
Ocean Circulation
Forecast
Weather
Forecast
BigDataOcean
Solution
AIS
Data
BigDataOcean
Wave
Forecast
Oil Spill Dispersion Forecast
Acquisition
High Risk PollutionAreas
UnderwaterAccident

22. Business Benefits
● Extended knowledge, models and enriched datasets
● New products addressed to environment protection
organizations and maritime authorities for rapid intervention
against oil spills in the sea
● Control and limit impact and damage on the coast and on
essential resources and structures.
● Efficiency in the protection of the marine environment and of
the marine life.

23. Case 3 – Wave Power as Clean Energy Source
23

24. Case 3 – Wave Power as Clean Energy Source
24
• Evaluation of wave energy potential and contribution to
development of wave energy solutions.
Goal
• Offshore Renewables Service Providers, Offshore Pilot
Zone Concessionaires,WEC developers, Energy
Producers, Hydrographic Centres
Stakeholders

25. BigDataOcean Solution
Waves In-situ
Observations
Cross-domain
Vessel
routes
Port
s
Protected
areas
Wave models
output
Instruments
Data
BigDataOcean
Solution
Wave ResourceAssessment
Wave Energy study and
Forecast
DataVisualisations

28. Business Benefits
● Wave resource characterization in your selected location or
area, based in historical data.
● Wave forecast.
● Assessment of Wave Energy Converters energy generation,
allowing to compare your device with others.
● Forecast of energy generation for your WEC device.

29. Case 4 – Security and anomaly path detection

30. Case 4 – Security and anomaly path detection
• Identify vessel routes based on their motion patterns to act
proactively and minimise threats at sea.
Goal
• Port authorities, Ocean Observatories, Port/Cargo
Community systems,Transport and Logistics companies,
Harbour Pilots and Maritime Consultants
Stakeholders

31. BigDataOcean solution
AIS
Anomaly Detection
BigDataOcean
Solution
Visualization
Anomaly
detection
services
Analytics
AIS
data
Weather Data
Location of Sea Ports
List of “High Risk” Vessels
List of security
incidents & vessels
implicated
Nautical Information Maps
Feedback from vessel’s
crew & domain experts

34. Business Benefits
● Effectively handle the information volume from tracking
technologies and AIS data
● identify patterns of behavior and vessel risk profile
● proactively minimize the impact of possible threats

35. BigDataOcean Consortium
35
35
Development
Semantics
Pilot
Pilot
PilotPilot
Requirements
Exploitation
Coordination &
Development
Pilot

36. Timeplan
Analysis and Development
2017 6/2019
Start
MS4:
Prototype
MS2:
Requirements
and needs
Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2
MS1:
Maritime data
value chain
definition
MS3:
Methodology
& Architecture
MS5: Final
Platform
MS6:
Business
Plan,
Lessons
Learnt
Evaluation
Market preparation
We are here

37. Lessons learned: exploitation
37
BDO Commercial Partnership for the platform
1. BDO Platform Services
2. BDO End-UsersApplication Services
3. BDO On-Demand Data Services and custom solutions
4. BDO Non-Technical Package
• Training
• Consulting
• Data science coaching
• Contract deals with maritime companies
• Focus on B2B applications

38. Semantic Challenges
● Plethora of file formats and metadata standards coming from diverse data sources
● Ability to perform advanced queries that combine those big datasets
● Enable ad hoc queries
Our Approach
● Automatic ingestion system to a harmonized, semantically aware schema (BDO
canonical Model)
● Based on DCAT, NETCDF convention standards (cfconventions.org)

39. Performance Challenges
● Performance issues
 POSTGRESQL and relational databases typically used in legacy maritime applications –
Serious scalability issues for the size of the datasets
 NoSQL databases -> Good performance for simple queries, but bad performance for
queries that combine data
 Distributed, wide column stores (e.g. Cassandra) -> better optimised for known queries
Our Approach
● Presto with Apache Hive

40. Lessons learned: current solution
Solution for Query Designer
queries
Solution for further query
optimization
● Caching JOIN queries performed by
users
● Pre-join datasets for pilot
applications
● Adoption of Apache Parquet storage
and ORF file formats
HDFS
Query performance over million of rows: from 5 minutes to 5 seconds!

41. BDO Architecture

42. Demo
www.bigdataocean.eu

43. Questions
www.bigdataocean.eu