Skip to main content
The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
PHIDIAS HPC - Building a prototype
for Earth Science Data and HPC
services
Webinar | February 13, 2020, 10:00 AM CEST
PHIDIAS HPC - Building a prototype for
Earth Science Data and HPC services
10:00 – 10:10 – PHIDIAS offering to HPC community – Boris Dintrans,
PHIDIAS Project Coordinator, CINES
10:10 – 10:20 – Improving the efficiency of the intelligent screening of
environmental satellite data – Pascal Prunet, WP4 Leader, SPASCIA
10:20 – 10:30 – On demand image processing for environmental monitoring :
challenges and uses cases coming from earth observation data – Jean-
Christophe Desconnets, WP5 Leader, IRD
10:30 – 10:35 – Q&A
10:35 – 10:45 – Boosting the use of cloud services for marine data studies –
Cecile Nys, WP6 member, IFREMER
10:45 – 10:55 – AI Services targeting user communities – Aleksi Kallio,
Development Manager of data analytics, CSC – IT Centre of Science Ltd
10:55 – 11:00 – Q&A
11:00 – 11:05 – Final remarks
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc 2
The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
PHIDIAS offering to HPC
Community
Boris DINTRANS, PHIDIAS coordinator and CINES director
Centre Informatique National de l’Enseignement Supérieur
(CINES), Montpellier, France
CINES: one of the three national supercomputing
centers in France
4
TGCC: Joliot-Curie
IDRIS: Jean Zay
HPC (Occigen) IT hosting Archiving
60 people
Budget ~10M€/y
National & EU scope
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Projet overview
Scope and objectives of the action: the Action’s overall objective is to build a prototype for
Data/High Performance Computing (HPC) services based on Earth sciences cases. In
this respect, the consortium will develop and provide new services to discover, manage and
process spatial and environmental data produced by research communities tackling scientific
challenges such as atmospheric, marine and earth observation issues.
Duration: 01/09/2019 to 01/09/2022 (3 years)
Budget: 3,519,476€
CEF Grant: 2,639,607€ (75%)
Project Officer: Mark Vella Muskat from INEA (Innovation & Network Executive Agency)
Project number: Action n° 2018-EU-IA-0089
513.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Consortium with 13 partners (8 FR, 2 FI, 1 NL, 1 IT, 1 BE)
CINES: Centre Informatique National de l’Enseignement Supérieur (France, Coordinator)
CERFACS: Centre Européen de Recherche et de Formation Avancée en Calcul Scientifique (France)
CNRS: Centre National de la Recherche Scientifique (France)
CSC: IT Centre for Science Ltd. (Finland)
Geomatys (France)
IRD: Institut de Recherche pour le Développement (France)
IFREMER: Institut Français de Recherche pour l‘Exploitation de la Mer (France)
MARIS: Mariene Informatie Service MARIS BV (Netherlands)
Néovia: Néovia Innovation (France)
Spascia (France)
SYKE: Finnish Environment Institute (Finland)
Trust-IT Services (Italy)
ULIEGE: Université de Liège (Belgium)
613.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Four main challenges that must be addressed
[Challenge #1] Handling the diversity of data coming from the
Earth System Research Infrastructure.
[Challenge #2] Developing and testing transversal methods and
tools that can be applied to data coming from other scientific
domains such as health and environment data.
[Challenge #3] Scalability of Data processing tools.
[Challenge #4] Industrialization and strength development of
HPC/HPDA/AI workflows.
713.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
The Data Terra research infrastructure brings the data to
PHIDIAS
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Data Terra = 3 scientific use cases for PHIDIAS
[Use case #1] Intelligent screening of large amount of satellite data
for detection and identification of anomalous atmospheric
composition events (WP4, leader: SPASCIA)  AERIS
[Use case #2] Big data Earth Observations: processing on-demand
for environmental monitoring (WP5, leader: IRD)  THEIA
[Use case #3] Ocean (WP6, leader: IFREMER)  ODATIS
913.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
The generic and fully-integrated PHIDIAS workflow
1013.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Main project organization
11
Management Board: acting a as the decision-making body of the consortium
Technical Board: acting as the supervisory body for the execution of the Action
Scientific and users Committee: gather various stakeholders of the HPC community as well as users and
citizen communities, and act as an advisory entity for PHIDIAS
Security Committee: address transversal security issues and identify Information Technology (IT) risks in the
workflows developed during the Action
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
24/04/2020 footer 12
WP1 - Management WP2 – Compute & storage workflow WP3 – Technical coordination
WP4 – Satellite data WP5 – Environmental monitoring WP6 – Ocean use case
WP7 – Dissemination
T1.1: Project contractual management
T1.2: Project operational day-to-day management
T1.3 – Scientific and Users Committee
T2.1: HPC, HPDA and storage services
T2.2: Storage backend and archiving services
T2.3: On-the-fly computing
T3.1: Technical coordination
T3.2: Common Metadata Repository
T3.3: End-users web Common interactive Processing
Services
T3.4: Common Portal for data discovery, access and
processing
T4.1: Implementation of PCA filtering techniques for
detection of exceptional atmospheric event
T4.2: Adaptation and preparation of tools, data and environment for
the data screening processing of Sentinel 5 precursor data
T4.3: Processing of 1 year of global S5P data for detection of
extreme atmospheric events
T4.4.1: UC1 - Exploitation for monitoring and alert service
development
T4.4.2: UC2 - Scientific exploitation of the product
T5.1: EO data processing chains for massive and on-demand
execution
T5.2: UI web environment dedicated for on-demand
execution
T5.3: Data workflows for discovery, access of EO
raw data and products
T6.1: Improvement of long-term stewardship of data
T6.2: Improvement of data storage for services to users
T6.3: Marine data processing workflows for on-demand
processing
T6.4: Data inter-comparison, collection and visualization
T7.1: Communication and Outreach
T7.2: Engagement with user and stakeholder communities
T7.3: EU-wide cross-dissemination & concertation
T7.4: Sustainability Path & Funding models
CINES CINES IRST
SPACIA IRD IFREMER
TRUST-IT
NEOVI
A
NEOVIA
CINES
Géomaty
s
Ulieg
e
9
x
Delivrable
CINES PMs
T4.5: Update the service, and prepare for the exploitation
Of Copernicus Atmosphere satellite data
MARI
S
9 3
6
1
8
1
8
Task Leader
Thank-you
Boris Dintrans, CINES & PHIDIAS Coordination, phidias-hpc.eu
boris.dintrans@cines.fr
13.02.2020 PHIDIAS Webinar | 13.02.2020 | www.phidias-hpc.eu | @PhidiasHpc 13
The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
Intelligent screening of satellite
data for air quality and climate
Pascal Prunet, SPASCIA CEO & WP4 Coordinator
Webinar | February 13, 2020
15
Independent SME : R&D in space science and environment
Processing, analysis and exploitation of remote sensing observations of the Earth
atmosphere from space :
 For supporting the definition, development, calibration & validation of satellite
missions/systems/instruments.
 For environmental applications (air quality, atmospheric composition, climate, meteorology).
Created in March 2016, located near Toulouse, France
11 staff members : PhDs with complementary skills in Space and Earth Sciences
Working for space agencies, research institutes, industries
In strong collaboration with European research and scientific institutes
SPASCIA is acting for a better use of Earth observations from space
THE COMPANY
SPASCIA
Space Science Algorithmics
Satellite mission/systems
IASI, IASI-NG, MTG-IRS, MicroCarb, S5P, CO2M
Atmosphere and environment
Atmsopheric chemistry, GHG, air quality, Cllimate
Level 1 data simulation,
processing, cal/val
Level 2 geophysical
products retrieval,
caracterisation,
analysis
Level 3 & 4
atmospheric fields
innovation, exploitation
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
16
Earth and environment challenges at local, regional and global scales : continuous increase of
human activities modify the atmospheric composition
Impact on environment (climate; soil, water and air quality; biodiversity)
Impact on health, on economy
European opportunities for better adressing these challenges : Europe is entering in the era of
operational measurement of the air composition from space, for the analysis and forecasting of chemical
weather and climate monitoring
PHIDIAS FOR ATMOSPHERE : CONTEXT
First measurements already available : The European operational atmospheric composition
observation from space currently benefits from data provided by IASI and GOME onboard the
Metop satellites, as well as Sentinel 5 precursor (S5P/TROPOMI)
4 Operational satellites from 2021 : Metop-NG, MeteoSat Third Generation, Sentinel 4 and 5
Development of the European strategy for the measurement of GHG for monitoring human emissions
Copernicus Operational forecasting services for atmospheric chemistry (CAMS) and Climate (C3S)
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
17
From 2021, European atmospheric sounding missions will deliver each day several TB (terabytes) of raw
datacubes at high spatial/temporal/spectral resolutions. This represents an unprecedented amount of
atmospheric data, with improved quality and coverage.
 Increasing difficulties for properly dealing with all available information
 key challenge : provide the capacity of identifying and focusing on useful data, e.g., by targeting scenes of interest in
view of their dedicated processing or exploitation.
PHIDIAS addresses those needs by using HPC and HPDA capacities : intelligent screening approaches for
the exploitation of large amounts of satellite atmospheric data in an operational context, for detection and
identification of atmospheric composition events.
.
CHALLENGES AND OBJECTIVES
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
18
PHIDIAS ADDED VALUE
Earth observation from space is underexploited
Huge quantity of data, increasing information content and
accuracy, complex processes from data to information
Operational forecasting systems
Efficient, real-time data assimilation in models,
but could not use « extreme » data related with
« extreme events »
Reduced added value of the data
Research groups, scientific institutes
Innovative and well adapted to event studies
but case by case exploitation of the data
Not exhaustive exploitation,
no real-time analysis
Screening (detection
and filtering) on the
fly of relevant data
Prototype service
Realtime Detection &
analyse of
extreme events, pollutant
plumes, other targeted
events
Dedicated services
Early warning, monitoring, decision support
Added-value :
Robust detection,
objective filtering
Added-value :
Verification,
improvement
Innovation
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
19
DATA FOR USE CASE DEMONSTRATION
This use case propose to Develop, test and prototype the approach with Sentinel 5 Precursor (S5P) data/products
S5P : first atmospheric Sentinel for air
quality and climate
Launched Oct. 13 2017, 7 years lifetime
UV-Vis-NIR-SWIR naidr view spectrometer,
with enhanced radiometric performances,
spatial resolution (7x3.5 km2) and temporal
revisit (Global daily coverage)
500 GB
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
20
Objective : Test, implement and demonstrate intelligent screening of large amount of satellite data for detection
and identification of anomalous atmospheric composition events
Two processing prototypes would be proposed for development and test :
1. PCA-based screening of L1 data (SWIR) for detection of extreme events. Based on experience and methods
developed for IASI, implementation and consolidation of algorithms and tools for generic processing of atmospheric
spectra recorded by S5P.
2. New AI methods for objective/automatic detection of plumes from L2 products (CO, others ? (CH4 NO2, SO2) :
SPASCIA experience with physical methods analyzing significant signal enhancements
WP4 OBJECTIVES
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
21
Objective : Test, implement and demonstrate intelligent screening of large amount of satellite data for detection
and identification of anomalous atmospheric composition events
Two processing prototypes would be proposed for development and test :
1. PCA-based screening of L1 data (SWIR) for detection of extreme events. Based on experience and methods
developed for IASI, implementation and consolidation of algorithms and tools for generic processing of atmospheric
spectra recorded by S5P.
Product : Real Time, operational detection, qualification and monitoring of rare or extremely strong events
Potential users : Atmospheric research, Operational air chemistry services (CAMS) for forecast validation, quality control,
improvements, future services of alerts and decision support
1. New AI methods for objective/automatic detection of plumes from L2 products (CO, others ? (CH4 NO2, SO2) :
SPASCIA experience with physical methods analyzing significant signal enhancements
WP4 OBJECTIVES
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
22
PCA-Based filtering on L1 data – illustration with IASI data
Results from work by SPASCIA/HYGEOS/LATMOS, funded by CNES
Nominal
case:
denoising
filter
Extreme
case:
identification
of the
residual
signal
Reconstruction of
"scores" maps:
Rapid detection of
anomalous situations
Analysis of « residuals" :
Interpretation of the
anomalous event
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
23
PCA-Based filtering on L1 data – illustration with IASI data
Results from work by SPASCIA/HYGEOS/LATMOS, funded by CNES
Big fires in Indonésie,
2015
PCA-based detection of the fire event
Cloud fraction
Operational CO product
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
24
Objective : Test, implement and demonstrate intelligent screening of large amount of satellite data for detection
and identification of anomalous atmospheric composition events
Two processing prototypes would be proposed for development and test :
1. PCA-based screening of L1 data (SWIR) for detection of extreme events. Based on experience and methods
developed for IASI, implementation and consolidation of algorithms and tools for generic processing of atmospheric
spectra recorded by S5P.
2. New AI methods for objective/automatic detection of plumes from L2 products (CO, others ? (CH4 NO2, SO2) :
SPASCIA experience with physical methods analyzing significant signal enhancements
Product : automatic Identification of pollutant plumes and sources in satellite maps
Potential users : Necessary input for processing and services of pollutant emission quantification, pollution monitoring : Atmospheric
research, Operational system of GHG and pollutant emission monitoring (e.g., CO2 emission survey for supporting Paris Climate
Agreement), end users services
WP4 OBJECTIVES
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
25
Plume detection for pollutant emission
quantification
TNO anthr. CO2 emission at 6x6 km2
(WP2)
From atmospheric gradients
…
… to anthropogenic
emissions
XCO2 COSMO-GHG simulations
Provided by EMPA (from H2020 ECMWF CHE project)
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
26
Methods for objective/automatic detection of plumes from L2 products from S5P (CO, CH4 NO2, SO2) :
Experience with physical methods analyzing significant signal enhancements
Innovative AI methods : efficient detection of plumes and associated source(s)
S5P products
Plume detection for pollutant emission
quantification
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
27
Space-based Earth observation will provide amazing amount of complex, informative data :
How to extract and use comprehensively this increasing information ?
Operational Models and systems fail to exploit the non-understood data
Research face to strong limits for analyzing all the underlined information and science
PHIDIAS will provide new tools and approaches to deal with the measurements and the information, allowing better and
more efficient use of the data, and pave the ways for finding new paradigms.
GOAL
XCO2 IFS model simulation
provided by ECMWF
(from H2020 ECMWF CHE project)
:
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Thank-you
Pascal Prunet, SPASCIA
Pascal.prunet@spascia.fr
13.02.2020 PHIDIAS Webinar | 13.02.2020 | www.phidias-hpc.eu | @PhidiasHpc 28
The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
On demand image processing for
environmental monitoring: challenges
and uses cases coming from earth
observation data
WP 5 leader : Jean-Christophe Desconnets
(IRD, ESPACE-DEV)
30
Working primarily in partnership with Mediterranean and inter-
tropical countries on the science of global development issues.
Multidisciplinary research: health and society, climate change,
humanitarian and political crises, agriculture and biodiversity
The French National Research Institute for
Sustainable Development
Publications : 1300/Year
65 Research Unit
52 % co-publications with
South Countries
2048 Agents
Budget : 230 M€/Year
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Data and HPC Challenges for land surface community
• Late 2000s, shared diagnosis among scientific community and public
authorities : Under-utilization of satellite imagery to monitor environment
Challenges
Facilitate selection and image analysis activities for end-users
Issues
Technological (Big data and scalability)
Openess and reusability (FAIR data)
31
Accessing relevant images
Having the skills
Finding support
Getting appropriate equipments
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
State of the art : data land surface interoperable
catalogs
Catalog GEOSUD of VHR images
SPOT6, 14 000 raw images
32
Catalog PEPS of Radar & optical
Sentinel products
>> 10 Millions of raw images
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
State of the art : on demand-processing services
Data pre-processing services (GEOSUD IDS V2) : extraction,
subsetting, segmentation, classification …
3313.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
State of the art : land surface processing chains
Coming from scientific remote sensing community
THEIA Scientific Expertise Center (SEC) working around ten themes :
Agriculture, Forest, Urban, Coastline, Health, Water…
Outputs : scientifically validated processing chains for environmental monitoring
34
Land cover IOTA Chain Soil moisture with VR spatial resolution
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Objectives of WP5
Technical Sub-objectives
Dedicated environment adapted to the target users
Ability to produce maps over large areas in a systematic manner
Open the dissemination of processsing chains outputs in FAIR way
Improvement of data reusability in perspective of EOSC
Leveraging AI techniques to provide alternative image classification
methodologies
Merging scientific experimental algorithms and catalogs
with community user-needs
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
Two examples of uses cases coming from
environmental monitoring community
Sentinel-1/Sentinel-2-derived Soil Moisture
product at Plot scale (S2MP) over agricultural
areas
Importance of monitoring of the soil moisture in
agricultural areas
Target Users
PHIDIAS issues
Rationale
Free access to Sentinel-1 (SAR satellite) and
Sentinel-2 (optical High resolution 10x10 m) with
high revisit
HPC : Interactive and on-demand
processing on specific zone and a large
temporal depth
Data : Big data access and data outputs
FAIRness
Scientific : water cycle modelisation
purposes
Farming sector : mapping of irrigation
activities
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Remote sensing images processing with artificial
intelligence: application to land cover mapping
and super-resolution
Taking advantage of IA libraries maturity and
GPU architecture to apply deep network for
classification images purposes
Target Users
PHIDIAS issues
Rationale
Super-resolution of Sentinel-2 images 10m  1.5m
HPC : GPU architecture to test scalability of IA
approach over national territory
temporal depth
Data : Big data access and data outputs
FAIRness
Scientific : experimental mode (notebook)
Public authority : very high resolution
LU/LC monitoring
Processing of multi modal imagery for land cover
mapping using semantic segmentation. (SPOT &
Sentinel-2)
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Architecture targeted (logical view)
user
services
HPDA /HPC facilities
Discovery Data services
On demand
processing
IA RF
Data Terra Web Portal (land surface focus)
Data and
computing layer
Data,
services,
process
catalog
Notebooks
Interactive Web IHM
Data sources Processing libraries & framework
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Take away message
WP5 Challenges
Taking advantage of HPC architecture and big data infrastructure to
facilitate selection and image analysis activities for environmental monitoring
Use cases
• Targeted to land surface community: scientific and public authorities
• Capibility to produce on-demand new mapping products at very temporal and spatial
resolution (soil moisture, LU/LC, …)
• Provide dedicated and interactive user environment to select, configure and execute
processing chains
• Open and FAIR diffusion of outputs data in reusable manner
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Thank-you
Jean-Christophe Desconnets, IRD, ESPACE-DEV
PHIDIAS WP 5 leader
13.02.2020 PHIDIAS Webinar | 13.02.2020 | www.phidias-hpc.eu | @PhidiasHpc 41
The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
PHIDIAS HPC - Building a prototype
for Earth Science Data and HPC
services
Webinar | February 13, 2020, 10:00 AM CEST
The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
Boosting the use of Cloud
Services for marine data studies
Cécile NYS, IFREMER
Assistant Manager Ocean Data Cluster – ODATIS
Phidias WP6 member
Webinar | February 13, 2020
WP6 “Use-case 3 – Ocean” overview
Boosting the use of Cloud Services for marine data studies
Combine and collocate data from several data sources (in situ &
satellite)
Enhancing data archiving (most observation cannot be reproduced) 
facilitate data reuse
Facilitate and speed up co-localisation of data from different sources
Adopting new data structures (based on big-data technologies)
DataCubes
NoSQL databases (numerical data) : Cassandra, MongoDB, etc.
Semantic Web (text data)
Providing on demand data browsing and processing facilities
4413.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
WP6 “Use-case 3 – Ocean” overview
Focus on two geographical areas
North Atlantic Ocean Baltic Sea
4513.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Ocean observation
Ocean  difficult ecosystem to observe
Constantly evolving
Seasonally
Specific phenomenon evolving quickly : waves, currents, plankton blooms, etc.
Not easily accessible, especially the deep seas (costs of equipment
deployment)
 Requires several complementary systems (from satellite to
underwater vehicles, but also buoys, etc.)
 Importance of inter-comparisons and co-processing of all
produced data
4613.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Input Data
Initial tests on specific variables
Temperature and Salinity  drive the ocean circulation (geostrophic
currents, turbulence, water masses)
Chlorophyll  indicator of primary production and eutrophication
Aggregate and process data (in situ & satellite) from several
infrastructures
Copernicus Marine Environmental Monitoring Services
(marine.copernicus.eu) & associated WEkEO DIAS (www.wekeo.eu)
SeaDataNet European Research Infrastructure (www.seadatanet.org)
EMODnet (DG-Mare) : European Marine Observation and Data Network
(www.emodnet.eu)
4713.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Input Data
4813.02.2020 PHIDIAS Webinar | 13.02.2020 |
CMEMS –
North Atlantic
CMEMS –
Baltic Sea
SeaDataNet & EMODnet
In situ data Atlantic Iberian Biscay Irish Ocean- In-
Situ Near Real Time Observations
Baltic Sea- In Situ Near Real Time
Observations
Satellite data North Atlantic Surface Chlorophyll
Concentration from Satellite observations
(monthly)
Baltic Sea, Ocean Colour Chlorophyll (daily
observation)
Improvement of Data Storage
Present data structures (e.g. collections of NetCDF files) of in-situ marine
data are not very efficient, due to
the large number of files
and/or
the heterogeneity of data observations
Challenges
Access quickly to a few number of observations within a large number of observations 
data visualization and data selections on web portals
Access to large number of observations within a large number of observations  data
processing or parallel processing (e.g. machine learning algorithms, interpolation
software such as DIVA, etc.)
Solutions
Data visualization  test of No-SQL databases (e.g. Cassandra - cassandra.apache.org)
Data processing  test of “data cubes” structures such as Parquet (columnar storage
format from the Haddoop ecosystem - parquet.apache.org)
4913.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
On demand Data Processing
“Virtual Research Environment” for users, allowing
Jupyter Notebook as a basis to develop workflows
Annotate, compare, conserve and share expertise
Access to different data structures in one environment (access to specified data on premise or remotely)
Scripting in various languages (Python, R, Julia, etc.)
Access to Pangeo components
Use of Diva software
Software for gridding data, using a finite-element method
Developed in Julia programming language
Extension for satellite images
GIS features
Visualisation of inputs (study area), outputs (compare results of different processes) and images
Existing tools : Diva online, Sextant, Geomatys, etc.
5013.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Data Visualisation and inter-comparison
2 case-studies
Surface Salinity in North Atlantic
CTD (SeaDataNet),
Argo Floats (CMEMS),
SMOS satellite,
Chlorophyll in North-East Atlantic and Balitic Sea
CTD and bottles (SeaDataNet)
BGC Argo floats (ARGO GDAC)
Ferrybox
Sentinel 2 images (DIAS WEkEO)
Visual comparisons (input and output) using GIS features
Software comparisons
Using Pangeo components
Using DIVA software
“Declouding” processing for satellite images (visible and infra-red)
5113.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Thank-you
Cécile NYS & Gilbert MAUDIRE, IFREMER
PHIDIAS WP6 leader
Phidias@Ifremer.fr / Cecile.Nys@Ifremer.fr / Gilbert.Maudire@Ifremer.fr
13.02.2020 PHIDIAS Webinar | 13.02.2020 | www.phidias-hpc.eu | @PhidiasHpc 52
AI service targeting user communities
Aleksi Kallio
PHIDIAS webinar 13.2.2020
53
54
Non-profit state
organization with
special tasks
Headquarters in
Espoo,
datacenter in
Kajaani
Owned by state (70%)
and all Finnish higher education institutions
(30%)
Turn over
in 2018
44,9M€
Circa
employees
in 2018
350
CSC’s solutions
5513.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
55
Computing and software
Data management and analytics for
research
Support and training for research
Research administration
Solutions for managing and
organizing education
Solutions for learners and teachers
Solutions for educational and
teaching cooperation
Hosting services tailored to
customers’ needs
Identity and authorisation
Management and use of data
ICT platforms, Funet network and
data center functions are the base
for our solutions
CSC in PHIDIAS
56
WP2: 12 PM
Intelligent data workflows on HPC/HPDA environment
Optimisation of distributed large scale AI/ML workloads
WP6: 3 PM
Supporting Ocean use case
“The other computing center partner together with CINES”
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Our AI user communities
57
Researchers in capacity driven fields
Researchers in established data driven fields
Researchers in emerging data driven fields
Research IT management
Research eInfrastructures
Public institutions looking for
data driven solutions57
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Services and solutions for artificial intelligence
58
• Mapping of needs, possibilities and competences
• Defining and implementing AI projects
• Data science, machine learning, data engineering, AI and BI expertise
AI & BI consultation
• Practical courses
• Methods support for AI and data analytics
Training and user support (ministry funded)
• Cloud services for building intelligent systems
• HPC computing for ambitious machine learning
• Software services for displaying data (Notebooks)
• Sensitive data environment for personal data
Computing services (ministry funded)58
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
NeIC iOBS (Improved Observation Usage in
Numerical Weather Prediction)
5913.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Training and user support
60
• Data analytics with R
• Data visualisation
• Big data analytics with Apache Spark
Data science courses
• Practical machine learning
• Practical machine learning for spatial data
• Practical deep learning
Machine learning courses
• Supporting researchers on using our environment and choosing right methods
• Data science, machine learning, data engineering and AI expertise
Methods support from service desk
60
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Computing services
61
• Rahti container cloud (Kubernetes / OpenShift)
• Apache Spark for big data, Apache Kafka for streaming data, GPU capacity coming
• cPouta IaaS cloud
• GPU capacity (Nvidia P100), IO acceleration (SSD)
• Allas object storage system
Cloud services
• Puhti and Puhti-AI
• Nvidia V100 GPU’s with fast interconnect
• Coming next: Mahti supercomputer and Lumi EuroHPC supercomputer
HPC computing services
• Notebooks provides easy-to-use environments for working with data and programming
Software services
• ePouta sensitive data IaaS cloud
• GPU capacity (Nvidia V100)
Sensitive data services
61
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
EuroHPC pre-exascale supercomputer ecosystem
LUMI supercomputer
One of the world’s fastest computer systems
Performance more than tenfold compared to Europe's fastest
supercomputer today
450 m2
PINTA-ALA
Computing power
equivalent to
Peak Performance
Data reading speed from
disk to memory
corresponds to the
simultaneous operation of
MacBook Pro computers
600 000
Size of a basketball court
200+
Pflop/s
1 system
Blu-ray Players
18 500
450m2
facebook.com/CSCfi
twitter.com/CSCfi
youtube.com/CSCfi
linkedin.com/company/csc---it-center-for-science
Kuvat CSC:n arkisto ja Thinkstock
github.com/CSCfi
Aleksi Kallio
aleksi.kallio@csc.fi
www.csc.fi
Thank-you for joining us!
13.02.2020 PHIDIAS Webinar | 13.02.2020 | www.phidias-hpc.eu | @PhidiasHpc 66
The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
PHIDIAS HPC - Building a prototype
for Earth Science Data and HPC
services
Webinar | February 13, 2020, 10:00 AM CEST