Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

BDVe Webinar Series - Big Data Ocean - Rocking the boat with Big Data


Published on

BigDataOcean brings a digital revolution to the maritime industry by creating a large maritime big data infrastructure that enables collaborative, data-driven intelligence. BigDataOcean will allow analytics based on diverse data resources, coming from public and private providers. In this webinar, Spyros Mouzakitis and Giannis Tsapelas will present a demo of the BigDataOcean platform and discuss the challenges and lessons learned so far.

Published in: Data & Analytics
  • You might get some help from ⇒ ⇐ Success and best regards!
    Are you sure you want to  Yes  No
    Your message goes here
  • I can advise you this service - ⇒ ⇐ Bought essay here. No problem.
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

BDVe Webinar Series - Big Data Ocean - Rocking the boat with Big Data

  1. 1. Rocking the boat with big data Dr. Spiros Mouzakitis (NTUA) Deputy Project Manager GiannisTsapelas(NTUA) Senior Researcher
  2. 2. Around 80% of global trade by volume is carried by sEA ….and around 70% of global trade by value
  3. 3. CHALLEN GESo Undeveloped sharing of Blue Data between enterprises and entities of the maritime domain and other domains o Lack of agreed standards and formats o Huge potential from cross-sectorial blue data applications – still unexploited o Out-of-the-box Big Data solution that enables advanced queries and analytics on cross-sector data - missing
  4. 4. Develop a Maritime Big Data platform that delivers out-of-the-box, value-added data and analytic services for maritime applications by exploiting cross-sector data streams 4
  5. 5. BigDataOcean platform (services) Find maritime data Query / interlink datasets Compose analytic services Visualize datasets Create (real-time) dashboards Request on-demand services
  6. 6. Port Authorities Maritime Research Centers Ship-owning companies Maritime software suppliers Service providers at sea level Maritime advisory firms
  7. 7. BigDataOcean end-user applications Vessel Fault Detection, Predictive Maintenance and Fuel Consumption Reduction Maritime security and anomaly detection Oil spill modeling Wave power generation 8BDV PPP Technical Committee Meeting
  8. 8. Case 1 - Fault Prediction & Proactive Maintenance and Fuel Consumption 9
  9. 9. Case 1 - Fault Prediction & Proactive Maintenance and Fuel Consumption 10 • Damage and mechanical failures detection and predictive maintenance of vessel equipment • Investigation of the impact of the environmental conditions and the operational decisions taken on the vessel's fuel consumption Goal • Ship owners, Maritime EquipmentConstructors Stakeholders
  10. 10. BigDataOcean Solution Analytics and knowledge base about fuel consumption and repairs Prediction models for maintenance and fuel consumption Maritime Regulations, Flag & Ship Classification Provisions Plant Maintenance System for Main Engine & spare parts Vessel routes In-situ Observations Cross-domain Forecast Data Port information BigDataOcean Solution
  11. 11. Business Benefits ● Shipping companies and Maritime Equipment Constructors  Minimum repairs, maintenance cost and fuel consumption  Maximum vessels’ use and financial benefit  Minimize environment impact  Reliability and innovation  Advanced Data analytics related to maintenance and fuel consumption
  12. 12. Case 2 – Mare protection – Oil Spill dispersion forecast 16
  13. 13. Case 2 – Mare protection – Oil Spill dispersion forecast 17 • Provide to the end users oil spill drift forecasting and simulation services for the marine environment • Enhance the efficiency in managing oil spill pollution risks. Goal • Emergency Response Companies, National Entities (PublicAuthorities), NGOs, Marine Research Institutes, Shipping companies and Oil drilling companies Stakeholders
  14. 14. Natura / Protecte d areas In-situ Observations BigDataOcean Solution Graphical output Cross-domain Forecast Data POSEIDON OSM Oil spill scenario submission Location, Rate, Nature & characteristics Ocean Circulation Forecast Weather Forecast BigDataOcean Solution AIS Data BigDataOcean Wave Forecast Oil Spill Dispersion Forecast Acquisition High Risk PollutionAreas UnderwaterAccident
  15. 15. Business Benefits ● Extended knowledge, models and enriched datasets ● New products addressed to environment protection organizations and maritime authorities for rapid intervention against oil spills in the sea ● Control and limit impact and damage on the coast and on essential resources and structures. ● Efficiency in the protection of the marine environment and of the marine life.
  16. 16. Case 3 – Wave Power as Clean Energy Source 23
  17. 17. Case 3 – Wave Power as Clean Energy Source 24 • Evaluation of wave energy potential and contribution to development of wave energy solutions. Goal • Offshore Renewables Service Providers, Offshore Pilot Zone Concessionaires,WEC developers, Energy Producers, Hydrographic Centres Stakeholders
  18. 18. BigDataOcean Solution Waves In-situ Observations Cross-domain Vessel routes Port s Protected areas Wave models output Instruments Data BigDataOcean Solution Wave ResourceAssessment Wave Energy study and Forecast DataVisualisations
  19. 19. Business Benefits ● Wave resource characterization in your selected location or area, based in historical data. ● Wave forecast. ● Assessment of Wave Energy Converters energy generation, allowing to compare your device with others. ● Forecast of energy generation for your WEC device.
  20. 20. Case 4 – Security and anomaly path detection
  21. 21. Case 4 – Security and anomaly path detection • Identify vessel routes based on their motion patterns to act proactively and minimise threats at sea. Goal • Port authorities, Ocean Observatories, Port/Cargo Community systems,Transport and Logistics companies, Harbour Pilots and Maritime Consultants Stakeholders
  22. 22. BigDataOcean solution AIS Anomaly Detection BigDataOcean Solution Visualization Anomaly detection services Analytics AIS data Weather Data Location of Sea Ports List of “High Risk” Vessels List of security incidents & vessels implicated Nautical Information Maps Feedback from vessel’s crew & domain experts
  23. 23. Business Benefits ● Effectively handle the information volume from tracking technologies and AIS data ● identify patterns of behavior and vessel risk profile ● proactively minimize the impact of possible threats
  24. 24. BigDataOcean Consortium 35 35 Development Semantics Pilot Pilot PilotPilot Requirements Exploitation Coordination & Development Pilot
  25. 25. Timeplan Analysis and Development 2017 6/2019 Start MS4: Prototype MS2: Requirements and needs Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 MS1: Maritime data value chain definition MS3: Methodology & Architecture MS5: Final Platform MS6: Business Plan, Lessons Learnt Evaluation Market preparation We are here
  26. 26. Lessons learned: exploitation 37 BDO Commercial Partnership for the platform 1. BDO Platform Services 2. BDO End-UsersApplication Services 3. BDO On-Demand Data Services and custom solutions 4. BDO Non-Technical Package • Training • Consulting • Data science coaching • Contract deals with maritime companies • Focus on B2B applications
  27. 27. Semantic Challenges ● Plethora of file formats and metadata standards coming from diverse data sources ● Ability to perform advanced queries that combine those big datasets ● Enable ad hoc queries Our Approach ● Automatic ingestion system to a harmonized, semantically aware schema (BDO canonical Model) ● Based on DCAT, NETCDF convention standards (
  28. 28. Performance Challenges ● Performance issues  POSTGRESQL and relational databases typically used in legacy maritime applications – Serious scalability issues for the size of the datasets  NoSQL databases -> Good performance for simple queries, but bad performance for queries that combine data  Distributed, wide column stores (e.g. Cassandra) -> better optimised for known queries Our Approach ● Presto with Apache Hive
  29. 29. Lessons learned: current solution Solution for Query Designer queries Solution for further query optimization ● Caching JOIN queries performed by users ● Pre-join datasets for pilot applications ● Adoption of Apache Parquet storage and ORF file formats HDFS Query performance over million of rows: from 5 minutes to 5 seconds!
  30. 30. BDO Architecture
  31. 31. Demo
  32. 32. Questions