Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

PHIDIAS HPC – Building a prototype for Earth Science Data and HPC Services

75 views

Published on

High-Performance Computing (HPC) technology is becoming increasingly important as a key driver to push European economic growth and Scientific Research. A comprehensive tool that can support the development of a wide array of scientific domains (like Big Data, earth observation and ocean study) and impact societal challenges as well.

The Webinar aims at introducing the Phidias HPC initiative to the European HPC and Research community, including main features, expected impact and advantages for Research & HPC ecosphere. The project is paving the way to increase the HPC and Data capacities of the European Data Infrastructure by pursuing the following objectives:

- Building a prototype for earth scientific data
- Enabling Open Access to HPC Services
- Strengthening FAIRisation
- Creating a framework combining computing, dissemination and archiving resources.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

PHIDIAS HPC – Building a prototype for Earth Science Data and HPC Services

  1. 1. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854. PHIDIAS HPC - Building a prototype for Earth Science Data and HPC services Webinar | February 13, 2020, 10:00 AM CEST
  2. 2. PHIDIAS HPC - Building a prototype for Earth Science Data and HPC services 10:00 – 10:10 – PHIDIAS offering to HPC community – Boris Dintrans, PHIDIAS Project Coordinator, CINES 10:10 – 10:20 – Improving the efficiency of the intelligent screening of environmental satellite data – Pascal Prunet, WP4 Leader, SPASCIA 10:20 – 10:30 – On demand image processing for environmental monitoring : challenges and uses cases coming from earth observation data – Jean- Christophe Desconnets, WP5 Leader, IRD 10:30 – 10:35 – Q&A 10:35 – 10:45 – Boosting the use of cloud services for marine data studies – Cecile Nys, WP6 member, IFREMER 10:45 – 10:55 – AI Services targeting user communities – Aleksi Kallio, Development Manager of data analytics, CSC – IT Centre of Science Ltd 10:55 – 11:00 – Q&A 11:00 – 11:05 – Final remarks 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc 2
  3. 3. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854. PHIDIAS offering to HPC Community Boris DINTRANS, PHIDIAS coordinator and CINES director Centre Informatique National de l’Enseignement Supérieur (CINES), Montpellier, France
  4. 4. CINES: one of the three national supercomputing centers in France 4 TGCC: Joliot-Curie IDRIS: Jean Zay HPC (Occigen) IT hosting Archiving 60 people Budget ~10M€/y National & EU scope 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  5. 5. Projet overview Scope and objectives of the action: the Action’s overall objective is to build a prototype for Data/High Performance Computing (HPC) services based on Earth sciences cases. In this respect, the consortium will develop and provide new services to discover, manage and process spatial and environmental data produced by research communities tackling scientific challenges such as atmospheric, marine and earth observation issues. Duration: 01/09/2019 to 01/09/2022 (3 years) Budget: 3,519,476€ CEF Grant: 2,639,607€ (75%) Project Officer: Mark Vella Muskat from INEA (Innovation & Network Executive Agency) Project number: Action n° 2018-EU-IA-0089 513.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  6. 6. Consortium with 13 partners (8 FR, 2 FI, 1 NL, 1 IT, 1 BE) CINES: Centre Informatique National de l’Enseignement Supérieur (France, Coordinator) CERFACS: Centre Européen de Recherche et de Formation Avancée en Calcul Scientifique (France) CNRS: Centre National de la Recherche Scientifique (France) CSC: IT Centre for Science Ltd. (Finland) Geomatys (France) IRD: Institut de Recherche pour le Développement (France) IFREMER: Institut Français de Recherche pour l‘Exploitation de la Mer (France) MARIS: Mariene Informatie Service MARIS BV (Netherlands) Néovia: Néovia Innovation (France) Spascia (France) SYKE: Finnish Environment Institute (Finland) Trust-IT Services (Italy) ULIEGE: Université de Liège (Belgium) 613.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  7. 7. Four main challenges that must be addressed [Challenge #1] Handling the diversity of data coming from the Earth System Research Infrastructure. [Challenge #2] Developing and testing transversal methods and tools that can be applied to data coming from other scientific domains such as health and environment data. [Challenge #3] Scalability of Data processing tools. [Challenge #4] Industrialization and strength development of HPC/HPDA/AI workflows. 713.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  8. 8. The Data Terra research infrastructure brings the data to PHIDIAS 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  9. 9. Data Terra = 3 scientific use cases for PHIDIAS [Use case #1] Intelligent screening of large amount of satellite data for detection and identification of anomalous atmospheric composition events (WP4, leader: SPASCIA)  AERIS [Use case #2] Big data Earth Observations: processing on-demand for environmental monitoring (WP5, leader: IRD)  THEIA [Use case #3] Ocean (WP6, leader: IFREMER)  ODATIS 913.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  10. 10. The generic and fully-integrated PHIDIAS workflow 1013.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  11. 11. Main project organization 11 Management Board: acting a as the decision-making body of the consortium Technical Board: acting as the supervisory body for the execution of the Action Scientific and users Committee: gather various stakeholders of the HPC community as well as users and citizen communities, and act as an advisory entity for PHIDIAS Security Committee: address transversal security issues and identify Information Technology (IT) risks in the workflows developed during the Action 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  12. 12. 24/04/2020 footer 12 WP1 - Management WP2 – Compute & storage workflow WP3 – Technical coordination WP4 – Satellite data WP5 – Environmental monitoring WP6 – Ocean use case WP7 – Dissemination T1.1: Project contractual management T1.2: Project operational day-to-day management T1.3 – Scientific and Users Committee T2.1: HPC, HPDA and storage services T2.2: Storage backend and archiving services T2.3: On-the-fly computing T3.1: Technical coordination T3.2: Common Metadata Repository T3.3: End-users web Common interactive Processing Services T3.4: Common Portal for data discovery, access and processing T4.1: Implementation of PCA filtering techniques for detection of exceptional atmospheric event T4.2: Adaptation and preparation of tools, data and environment for the data screening processing of Sentinel 5 precursor data T4.3: Processing of 1 year of global S5P data for detection of extreme atmospheric events T4.4.1: UC1 - Exploitation for monitoring and alert service development T4.4.2: UC2 - Scientific exploitation of the product T5.1: EO data processing chains for massive and on-demand execution T5.2: UI web environment dedicated for on-demand execution T5.3: Data workflows for discovery, access of EO raw data and products T6.1: Improvement of long-term stewardship of data T6.2: Improvement of data storage for services to users T6.3: Marine data processing workflows for on-demand processing T6.4: Data inter-comparison, collection and visualization T7.1: Communication and Outreach T7.2: Engagement with user and stakeholder communities T7.3: EU-wide cross-dissemination & concertation T7.4: Sustainability Path & Funding models CINES CINES IRST SPACIA IRD IFREMER TRUST-IT NEOVI A NEOVIA CINES Géomaty s Ulieg e 9 x Delivrable CINES PMs T4.5: Update the service, and prepare for the exploitation Of Copernicus Atmosphere satellite data MARI S 9 3 6 1 8 1 8 Task Leader
  13. 13. Thank-you Boris Dintrans, CINES & PHIDIAS Coordination, phidias-hpc.eu boris.dintrans@cines.fr 13.02.2020 PHIDIAS Webinar | 13.02.2020 | www.phidias-hpc.eu | @PhidiasHpc 13
  14. 14. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854. Intelligent screening of satellite data for air quality and climate Pascal Prunet, SPASCIA CEO & WP4 Coordinator Webinar | February 13, 2020
  15. 15. 15 Independent SME : R&D in space science and environment Processing, analysis and exploitation of remote sensing observations of the Earth atmosphere from space :  For supporting the definition, development, calibration & validation of satellite missions/systems/instruments.  For environmental applications (air quality, atmospheric composition, climate, meteorology). Created in March 2016, located near Toulouse, France 11 staff members : PhDs with complementary skills in Space and Earth Sciences Working for space agencies, research institutes, industries In strong collaboration with European research and scientific institutes SPASCIA is acting for a better use of Earth observations from space THE COMPANY SPASCIA Space Science Algorithmics Satellite mission/systems IASI, IASI-NG, MTG-IRS, MicroCarb, S5P, CO2M Atmosphere and environment Atmsopheric chemistry, GHG, air quality, Cllimate Level 1 data simulation, processing, cal/val Level 2 geophysical products retrieval, caracterisation, analysis Level 3 & 4 atmospheric fields innovation, exploitation 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  16. 16. 16 Earth and environment challenges at local, regional and global scales : continuous increase of human activities modify the atmospheric composition Impact on environment (climate; soil, water and air quality; biodiversity) Impact on health, on economy European opportunities for better adressing these challenges : Europe is entering in the era of operational measurement of the air composition from space, for the analysis and forecasting of chemical weather and climate monitoring PHIDIAS FOR ATMOSPHERE : CONTEXT First measurements already available : The European operational atmospheric composition observation from space currently benefits from data provided by IASI and GOME onboard the Metop satellites, as well as Sentinel 5 precursor (S5P/TROPOMI) 4 Operational satellites from 2021 : Metop-NG, MeteoSat Third Generation, Sentinel 4 and 5 Development of the European strategy for the measurement of GHG for monitoring human emissions Copernicus Operational forecasting services for atmospheric chemistry (CAMS) and Climate (C3S) 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  17. 17. 17 From 2021, European atmospheric sounding missions will deliver each day several TB (terabytes) of raw datacubes at high spatial/temporal/spectral resolutions. This represents an unprecedented amount of atmospheric data, with improved quality and coverage.  Increasing difficulties for properly dealing with all available information  key challenge : provide the capacity of identifying and focusing on useful data, e.g., by targeting scenes of interest in view of their dedicated processing or exploitation. PHIDIAS addresses those needs by using HPC and HPDA capacities : intelligent screening approaches for the exploitation of large amounts of satellite atmospheric data in an operational context, for detection and identification of atmospheric composition events. . CHALLENGES AND OBJECTIVES 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  18. 18. 18 PHIDIAS ADDED VALUE Earth observation from space is underexploited Huge quantity of data, increasing information content and accuracy, complex processes from data to information Operational forecasting systems Efficient, real-time data assimilation in models, but could not use « extreme » data related with « extreme events » Reduced added value of the data Research groups, scientific institutes Innovative and well adapted to event studies but case by case exploitation of the data Not exhaustive exploitation, no real-time analysis Screening (detection and filtering) on the fly of relevant data Prototype service Realtime Detection & analyse of extreme events, pollutant plumes, other targeted events Dedicated services Early warning, monitoring, decision support Added-value : Robust detection, objective filtering Added-value : Verification, improvement Innovation 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  19. 19. 19 DATA FOR USE CASE DEMONSTRATION This use case propose to Develop, test and prototype the approach with Sentinel 5 Precursor (S5P) data/products S5P : first atmospheric Sentinel for air quality and climate Launched Oct. 13 2017, 7 years lifetime UV-Vis-NIR-SWIR naidr view spectrometer, with enhanced radiometric performances, spatial resolution (7x3.5 km2) and temporal revisit (Global daily coverage) 500 GB 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  20. 20. 20 Objective : Test, implement and demonstrate intelligent screening of large amount of satellite data for detection and identification of anomalous atmospheric composition events Two processing prototypes would be proposed for development and test : 1. PCA-based screening of L1 data (SWIR) for detection of extreme events. Based on experience and methods developed for IASI, implementation and consolidation of algorithms and tools for generic processing of atmospheric spectra recorded by S5P. 2. New AI methods for objective/automatic detection of plumes from L2 products (CO, others ? (CH4 NO2, SO2) : SPASCIA experience with physical methods analyzing significant signal enhancements WP4 OBJECTIVES 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  21. 21. 21 Objective : Test, implement and demonstrate intelligent screening of large amount of satellite data for detection and identification of anomalous atmospheric composition events Two processing prototypes would be proposed for development and test : 1. PCA-based screening of L1 data (SWIR) for detection of extreme events. Based on experience and methods developed for IASI, implementation and consolidation of algorithms and tools for generic processing of atmospheric spectra recorded by S5P. Product : Real Time, operational detection, qualification and monitoring of rare or extremely strong events Potential users : Atmospheric research, Operational air chemistry services (CAMS) for forecast validation, quality control, improvements, future services of alerts and decision support 1. New AI methods for objective/automatic detection of plumes from L2 products (CO, others ? (CH4 NO2, SO2) : SPASCIA experience with physical methods analyzing significant signal enhancements WP4 OBJECTIVES 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  22. 22. 22 PCA-Based filtering on L1 data – illustration with IASI data Results from work by SPASCIA/HYGEOS/LATMOS, funded by CNES Nominal case: denoising filter Extreme case: identification of the residual signal Reconstruction of "scores" maps: Rapid detection of anomalous situations Analysis of « residuals" : Interpretation of the anomalous event 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  23. 23. 23 PCA-Based filtering on L1 data – illustration with IASI data Results from work by SPASCIA/HYGEOS/LATMOS, funded by CNES Big fires in Indonésie, 2015 PCA-based detection of the fire event Cloud fraction Operational CO product 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  24. 24. 24 Objective : Test, implement and demonstrate intelligent screening of large amount of satellite data for detection and identification of anomalous atmospheric composition events Two processing prototypes would be proposed for development and test : 1. PCA-based screening of L1 data (SWIR) for detection of extreme events. Based on experience and methods developed for IASI, implementation and consolidation of algorithms and tools for generic processing of atmospheric spectra recorded by S5P. 2. New AI methods for objective/automatic detection of plumes from L2 products (CO, others ? (CH4 NO2, SO2) : SPASCIA experience with physical methods analyzing significant signal enhancements Product : automatic Identification of pollutant plumes and sources in satellite maps Potential users : Necessary input for processing and services of pollutant emission quantification, pollution monitoring : Atmospheric research, Operational system of GHG and pollutant emission monitoring (e.g., CO2 emission survey for supporting Paris Climate Agreement), end users services WP4 OBJECTIVES 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  25. 25. 25 Plume detection for pollutant emission quantification TNO anthr. CO2 emission at 6x6 km2 (WP2) From atmospheric gradients … … to anthropogenic emissions XCO2 COSMO-GHG simulations Provided by EMPA (from H2020 ECMWF CHE project) 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  26. 26. 26 Methods for objective/automatic detection of plumes from L2 products from S5P (CO, CH4 NO2, SO2) : Experience with physical methods analyzing significant signal enhancements Innovative AI methods : efficient detection of plumes and associated source(s) S5P products Plume detection for pollutant emission quantification 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  27. 27. 27 Space-based Earth observation will provide amazing amount of complex, informative data : How to extract and use comprehensively this increasing information ? Operational Models and systems fail to exploit the non-understood data Research face to strong limits for analyzing all the underlined information and science PHIDIAS will provide new tools and approaches to deal with the measurements and the information, allowing better and more efficient use of the data, and pave the ways for finding new paradigms. GOAL XCO2 IFS model simulation provided by ECMWF (from H2020 ECMWF CHE project) : 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  28. 28. Thank-you Pascal Prunet, SPASCIA Pascal.prunet@spascia.fr 13.02.2020 PHIDIAS Webinar | 13.02.2020 | www.phidias-hpc.eu | @PhidiasHpc 28
  29. 29. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854. On demand image processing for environmental monitoring: challenges and uses cases coming from earth observation data WP 5 leader : Jean-Christophe Desconnets (IRD, ESPACE-DEV)
  30. 30. 30 Working primarily in partnership with Mediterranean and inter- tropical countries on the science of global development issues. Multidisciplinary research: health and society, climate change, humanitarian and political crises, agriculture and biodiversity The French National Research Institute for Sustainable Development Publications : 1300/Year 65 Research Unit 52 % co-publications with South Countries 2048 Agents Budget : 230 M€/Year 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  31. 31. Data and HPC Challenges for land surface community • Late 2000s, shared diagnosis among scientific community and public authorities : Under-utilization of satellite imagery to monitor environment Challenges Facilitate selection and image analysis activities for end-users Issues Technological (Big data and scalability) Openess and reusability (FAIR data) 31 Accessing relevant images Having the skills Finding support Getting appropriate equipments 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  32. 32. State of the art : data land surface interoperable catalogs Catalog GEOSUD of VHR images SPOT6, 14 000 raw images 32 Catalog PEPS of Radar & optical Sentinel products >> 10 Millions of raw images 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  33. 33. State of the art : on demand-processing services Data pre-processing services (GEOSUD IDS V2) : extraction, subsetting, segmentation, classification … 3313.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  34. 34. State of the art : land surface processing chains Coming from scientific remote sensing community THEIA Scientific Expertise Center (SEC) working around ten themes : Agriculture, Forest, Urban, Coastline, Health, Water… Outputs : scientifically validated processing chains for environmental monitoring 34 Land cover IOTA Chain Soil moisture with VR spatial resolution 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  35. 35. Objectives of WP5 Technical Sub-objectives Dedicated environment adapted to the target users Ability to produce maps over large areas in a systematic manner Open the dissemination of processsing chains outputs in FAIR way Improvement of data reusability in perspective of EOSC Leveraging AI techniques to provide alternative image classification methodologies Merging scientific experimental algorithms and catalogs with community user-needs 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  36. 36. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854. Two examples of uses cases coming from environmental monitoring community
  37. 37. Sentinel-1/Sentinel-2-derived Soil Moisture product at Plot scale (S2MP) over agricultural areas Importance of monitoring of the soil moisture in agricultural areas Target Users PHIDIAS issues Rationale Free access to Sentinel-1 (SAR satellite) and Sentinel-2 (optical High resolution 10x10 m) with high revisit HPC : Interactive and on-demand processing on specific zone and a large temporal depth Data : Big data access and data outputs FAIRness Scientific : water cycle modelisation purposes Farming sector : mapping of irrigation activities 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  38. 38. Remote sensing images processing with artificial intelligence: application to land cover mapping and super-resolution Taking advantage of IA libraries maturity and GPU architecture to apply deep network for classification images purposes Target Users PHIDIAS issues Rationale Super-resolution of Sentinel-2 images 10m  1.5m HPC : GPU architecture to test scalability of IA approach over national territory temporal depth Data : Big data access and data outputs FAIRness Scientific : experimental mode (notebook) Public authority : very high resolution LU/LC monitoring Processing of multi modal imagery for land cover mapping using semantic segmentation. (SPOT & Sentinel-2) 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  39. 39. Architecture targeted (logical view) user services HPDA /HPC facilities Discovery Data services On demand processing IA RF Data Terra Web Portal (land surface focus) Data and computing layer Data, services, process catalog Notebooks Interactive Web IHM Data sources Processing libraries & framework 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  40. 40. Take away message WP5 Challenges Taking advantage of HPC architecture and big data infrastructure to facilitate selection and image analysis activities for environmental monitoring Use cases • Targeted to land surface community: scientific and public authorities • Capibility to produce on-demand new mapping products at very temporal and spatial resolution (soil moisture, LU/LC, …) • Provide dedicated and interactive user environment to select, configure and execute processing chains • Open and FAIR diffusion of outputs data in reusable manner 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  41. 41. Thank-you Jean-Christophe Desconnets, IRD, ESPACE-DEV PHIDIAS WP 5 leader 13.02.2020 PHIDIAS Webinar | 13.02.2020 | www.phidias-hpc.eu | @PhidiasHpc 41
  42. 42. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854. PHIDIAS HPC - Building a prototype for Earth Science Data and HPC services Webinar | February 13, 2020, 10:00 AM CEST
  43. 43. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854. Boosting the use of Cloud Services for marine data studies Cécile NYS, IFREMER Assistant Manager Ocean Data Cluster – ODATIS Phidias WP6 member Webinar | February 13, 2020
  44. 44. WP6 “Use-case 3 – Ocean” overview Boosting the use of Cloud Services for marine data studies Combine and collocate data from several data sources (in situ & satellite) Enhancing data archiving (most observation cannot be reproduced)  facilitate data reuse Facilitate and speed up co-localisation of data from different sources Adopting new data structures (based on big-data technologies) DataCubes NoSQL databases (numerical data) : Cassandra, MongoDB, etc. Semantic Web (text data) Providing on demand data browsing and processing facilities 4413.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  45. 45. WP6 “Use-case 3 – Ocean” overview Focus on two geographical areas North Atlantic Ocean Baltic Sea 4513.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  46. 46. Ocean observation Ocean  difficult ecosystem to observe Constantly evolving Seasonally Specific phenomenon evolving quickly : waves, currents, plankton blooms, etc. Not easily accessible, especially the deep seas (costs of equipment deployment)  Requires several complementary systems (from satellite to underwater vehicles, but also buoys, etc.)  Importance of inter-comparisons and co-processing of all produced data 4613.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  47. 47. Input Data Initial tests on specific variables Temperature and Salinity  drive the ocean circulation (geostrophic currents, turbulence, water masses) Chlorophyll  indicator of primary production and eutrophication Aggregate and process data (in situ & satellite) from several infrastructures Copernicus Marine Environmental Monitoring Services (marine.copernicus.eu) & associated WEkEO DIAS (www.wekeo.eu) SeaDataNet European Research Infrastructure (www.seadatanet.org) EMODnet (DG-Mare) : European Marine Observation and Data Network (www.emodnet.eu) 4713.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  48. 48. Input Data 4813.02.2020 PHIDIAS Webinar | 13.02.2020 | CMEMS – North Atlantic CMEMS – Baltic Sea SeaDataNet & EMODnet In situ data Atlantic Iberian Biscay Irish Ocean- In- Situ Near Real Time Observations Baltic Sea- In Situ Near Real Time Observations Satellite data North Atlantic Surface Chlorophyll Concentration from Satellite observations (monthly) Baltic Sea, Ocean Colour Chlorophyll (daily observation)
  49. 49. Improvement of Data Storage Present data structures (e.g. collections of NetCDF files) of in-situ marine data are not very efficient, due to the large number of files and/or the heterogeneity of data observations Challenges Access quickly to a few number of observations within a large number of observations  data visualization and data selections on web portals Access to large number of observations within a large number of observations  data processing or parallel processing (e.g. machine learning algorithms, interpolation software such as DIVA, etc.) Solutions Data visualization  test of No-SQL databases (e.g. Cassandra - cassandra.apache.org) Data processing  test of “data cubes” structures such as Parquet (columnar storage format from the Haddoop ecosystem - parquet.apache.org) 4913.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  50. 50. On demand Data Processing “Virtual Research Environment” for users, allowing Jupyter Notebook as a basis to develop workflows Annotate, compare, conserve and share expertise Access to different data structures in one environment (access to specified data on premise or remotely) Scripting in various languages (Python, R, Julia, etc.) Access to Pangeo components Use of Diva software Software for gridding data, using a finite-element method Developed in Julia programming language Extension for satellite images GIS features Visualisation of inputs (study area), outputs (compare results of different processes) and images Existing tools : Diva online, Sextant, Geomatys, etc. 5013.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  51. 51. Data Visualisation and inter-comparison 2 case-studies Surface Salinity in North Atlantic CTD (SeaDataNet), Argo Floats (CMEMS), SMOS satellite, Chlorophyll in North-East Atlantic and Balitic Sea CTD and bottles (SeaDataNet) BGC Argo floats (ARGO GDAC) Ferrybox Sentinel 2 images (DIAS WEkEO) Visual comparisons (input and output) using GIS features Software comparisons Using Pangeo components Using DIVA software “Declouding” processing for satellite images (visible and infra-red) 5113.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  52. 52. Thank-you Cécile NYS & Gilbert MAUDIRE, IFREMER PHIDIAS WP6 leader Phidias@Ifremer.fr / Cecile.Nys@Ifremer.fr / Gilbert.Maudire@Ifremer.fr 13.02.2020 PHIDIAS Webinar | 13.02.2020 | www.phidias-hpc.eu | @PhidiasHpc 52
  53. 53. AI service targeting user communities Aleksi Kallio PHIDIAS webinar 13.2.2020 53
  54. 54. 54 Non-profit state organization with special tasks Headquarters in Espoo, datacenter in Kajaani Owned by state (70%) and all Finnish higher education institutions (30%) Turn over in 2018 44,9M€ Circa employees in 2018 350
  55. 55. CSC’s solutions 5513.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc 55 Computing and software Data management and analytics for research Support and training for research Research administration Solutions for managing and organizing education Solutions for learners and teachers Solutions for educational and teaching cooperation Hosting services tailored to customers’ needs Identity and authorisation Management and use of data ICT platforms, Funet network and data center functions are the base for our solutions
  56. 56. CSC in PHIDIAS 56 WP2: 12 PM Intelligent data workflows on HPC/HPDA environment Optimisation of distributed large scale AI/ML workloads WP6: 3 PM Supporting Ocean use case “The other computing center partner together with CINES” 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  57. 57. Our AI user communities 57 Researchers in capacity driven fields Researchers in established data driven fields Researchers in emerging data driven fields Research IT management Research eInfrastructures Public institutions looking for data driven solutions57 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  58. 58. Services and solutions for artificial intelligence 58 • Mapping of needs, possibilities and competences • Defining and implementing AI projects • Data science, machine learning, data engineering, AI and BI expertise AI & BI consultation • Practical courses • Methods support for AI and data analytics Training and user support (ministry funded) • Cloud services for building intelligent systems • HPC computing for ambitious machine learning • Software services for displaying data (Notebooks) • Sensitive data environment for personal data Computing services (ministry funded)58 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  59. 59. NeIC iOBS (Improved Observation Usage in Numerical Weather Prediction) 5913.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  60. 60. Training and user support 60 • Data analytics with R • Data visualisation • Big data analytics with Apache Spark Data science courses • Practical machine learning • Practical machine learning for spatial data • Practical deep learning Machine learning courses • Supporting researchers on using our environment and choosing right methods • Data science, machine learning, data engineering and AI expertise Methods support from service desk 60 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  61. 61. Computing services 61 • Rahti container cloud (Kubernetes / OpenShift) • Apache Spark for big data, Apache Kafka for streaming data, GPU capacity coming • cPouta IaaS cloud • GPU capacity (Nvidia P100), IO acceleration (SSD) • Allas object storage system Cloud services • Puhti and Puhti-AI • Nvidia V100 GPU’s with fast interconnect • Coming next: Mahti supercomputer and Lumi EuroHPC supercomputer HPC computing services • Notebooks provides easy-to-use environments for working with data and programming Software services • ePouta sensitive data IaaS cloud • GPU capacity (Nvidia V100) Sensitive data services 61 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  62. 62. 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  63. 63. EuroHPC pre-exascale supercomputer ecosystem
  64. 64. LUMI supercomputer One of the world’s fastest computer systems Performance more than tenfold compared to Europe's fastest supercomputer today 450 m2 PINTA-ALA Computing power equivalent to Peak Performance Data reading speed from disk to memory corresponds to the simultaneous operation of MacBook Pro computers 600 000 Size of a basketball court 200+ Pflop/s 1 system Blu-ray Players 18 500 450m2
  65. 65. facebook.com/CSCfi twitter.com/CSCfi youtube.com/CSCfi linkedin.com/company/csc---it-center-for-science Kuvat CSC:n arkisto ja Thinkstock github.com/CSCfi Aleksi Kallio aleksi.kallio@csc.fi www.csc.fi
  66. 66. Thank-you for joining us! 13.02.2020 PHIDIAS Webinar | 13.02.2020 | www.phidias-hpc.eu | @PhidiasHpc 66
  67. 67. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854. PHIDIAS HPC - Building a prototype for Earth Science Data and HPC services Webinar | February 13, 2020, 10:00 AM CEST

×