SlideShare a Scribd company logo
The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
PHIDIAS HPC - Building a prototype
for Earth Science Data and HPC
services
Webinar | February 13, 2020, 10:00 AM CEST
PHIDIAS HPC - Building a prototype for
Earth Science Data and HPC services
10:00 – 10:10 – PHIDIAS offering to HPC community – Boris Dintrans,
PHIDIAS Project Coordinator, CINES
10:10 – 10:20 – Improving the efficiency of the intelligent screening of
environmental satellite data – Pascal Prunet, WP4 Leader, SPASCIA
10:20 – 10:30 – On demand image processing for environmental monitoring :
challenges and uses cases coming from earth observation data – Jean-
Christophe Desconnets, WP5 Leader, IRD
10:30 – 10:35 – Q&A
10:35 – 10:45 – Boosting the use of cloud services for marine data studies –
Cecile Nys, WP6 member, IFREMER
10:45 – 10:55 – AI Services targeting user communities – Aleksi Kallio,
Development Manager of data analytics, CSC – IT Centre of Science Ltd
10:55 – 11:00 – Q&A
11:00 – 11:05 – Final remarks
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc 2
The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
PHIDIAS offering to HPC
Community
Boris DINTRANS, PHIDIAS coordinator and CINES director
Centre Informatique National de l’Enseignement Supérieur
(CINES), Montpellier, France
CINES: one of the three national supercomputing
centers in France
4
TGCC: Joliot-Curie
IDRIS: Jean Zay
HPC (Occigen) IT hosting Archiving
60 people
Budget ~10M€/y
National & EU scope
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Projet overview
Scope and objectives of the action: the Action’s overall objective is to build a prototype for
Data/High Performance Computing (HPC) services based on Earth sciences cases. In
this respect, the consortium will develop and provide new services to discover, manage and
process spatial and environmental data produced by research communities tackling scientific
challenges such as atmospheric, marine and earth observation issues.
Duration: 01/09/2019 to 01/09/2022 (3 years)
Budget: 3,519,476€
CEF Grant: 2,639,607€ (75%)
Project Officer: Mark Vella Muskat from INEA (Innovation & Network Executive Agency)
Project number: Action n° 2018-EU-IA-0089
513.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Consortium with 13 partners (8 FR, 2 FI, 1 NL, 1 IT, 1 BE)
CINES: Centre Informatique National de l’Enseignement Supérieur (France, Coordinator)
CERFACS: Centre Européen de Recherche et de Formation Avancée en Calcul Scientifique (France)
CNRS: Centre National de la Recherche Scientifique (France)
CSC: IT Centre for Science Ltd. (Finland)
Geomatys (France)
IRD: Institut de Recherche pour le Développement (France)
IFREMER: Institut Français de Recherche pour l‘Exploitation de la Mer (France)
MARIS: Mariene Informatie Service MARIS BV (Netherlands)
Néovia: Néovia Innovation (France)
Spascia (France)
SYKE: Finnish Environment Institute (Finland)
Trust-IT Services (Italy)
ULIEGE: Université de Liège (Belgium)
613.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Four main challenges that must be addressed
[Challenge #1] Handling the diversity of data coming from the
Earth System Research Infrastructure.
[Challenge #2] Developing and testing transversal methods and
tools that can be applied to data coming from other scientific
domains such as health and environment data.
[Challenge #3] Scalability of Data processing tools.
[Challenge #4] Industrialization and strength development of
HPC/HPDA/AI workflows.
713.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
The Data Terra research infrastructure brings the data to
PHIDIAS
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Data Terra = 3 scientific use cases for PHIDIAS
[Use case #1] Intelligent screening of large amount of satellite data
for detection and identification of anomalous atmospheric
composition events (WP4, leader: SPASCIA)  AERIS
[Use case #2] Big data Earth Observations: processing on-demand
for environmental monitoring (WP5, leader: IRD)  THEIA
[Use case #3] Ocean (WP6, leader: IFREMER)  ODATIS
913.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
The generic and fully-integrated PHIDIAS workflow
1013.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Main project organization
11
Management Board: acting a as the decision-making body of the consortium
Technical Board: acting as the supervisory body for the execution of the Action
Scientific and users Committee: gather various stakeholders of the HPC community as well as users and
citizen communities, and act as an advisory entity for PHIDIAS
Security Committee: address transversal security issues and identify Information Technology (IT) risks in the
workflows developed during the Action
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
24/04/2020 footer 12
WP1 - Management WP2 – Compute & storage workflow WP3 – Technical coordination
WP4 – Satellite data WP5 – Environmental monitoring WP6 – Ocean use case
WP7 – Dissemination
T1.1: Project contractual management
T1.2: Project operational day-to-day management
T1.3 – Scientific and Users Committee
T2.1: HPC, HPDA and storage services
T2.2: Storage backend and archiving services
T2.3: On-the-fly computing
T3.1: Technical coordination
T3.2: Common Metadata Repository
T3.3: End-users web Common interactive Processing
Services
T3.4: Common Portal for data discovery, access and
processing
T4.1: Implementation of PCA filtering techniques for
detection of exceptional atmospheric event
T4.2: Adaptation and preparation of tools, data and environment for
the data screening processing of Sentinel 5 precursor data
T4.3: Processing of 1 year of global S5P data for detection of
extreme atmospheric events
T4.4.1: UC1 - Exploitation for monitoring and alert service
development
T4.4.2: UC2 - Scientific exploitation of the product
T5.1: EO data processing chains for massive and on-demand
execution
T5.2: UI web environment dedicated for on-demand
execution
T5.3: Data workflows for discovery, access of EO
raw data and products
T6.1: Improvement of long-term stewardship of data
T6.2: Improvement of data storage for services to users
T6.3: Marine data processing workflows for on-demand
processing
T6.4: Data inter-comparison, collection and visualization
T7.1: Communication and Outreach
T7.2: Engagement with user and stakeholder communities
T7.3: EU-wide cross-dissemination & concertation
T7.4: Sustainability Path & Funding models
CINES CINES IRST
SPACIA IRD IFREMER
TRUST-IT
NEOVI
A
NEOVIA
CINES
Géomaty
s
Ulieg
e
9
x
Delivrable
CINES PMs
T4.5: Update the service, and prepare for the exploitation
Of Copernicus Atmosphere satellite data
MARI
S
9 3
6
1
8
1
8
Task Leader
Thank-you
Boris Dintrans, CINES & PHIDIAS Coordination, phidias-hpc.eu
boris.dintrans@cines.fr
13.02.2020 PHIDIAS Webinar | 13.02.2020 | www.phidias-hpc.eu | @PhidiasHpc 13
The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
Intelligent screening of satellite
data for air quality and climate
Pascal Prunet, SPASCIA CEO & WP4 Coordinator
Webinar | February 13, 2020
15
Independent SME : R&D in space science and environment
Processing, analysis and exploitation of remote sensing observations of the Earth
atmosphere from space :
 For supporting the definition, development, calibration & validation of satellite
missions/systems/instruments.
 For environmental applications (air quality, atmospheric composition, climate, meteorology).
Created in March 2016, located near Toulouse, France
11 staff members : PhDs with complementary skills in Space and Earth Sciences
Working for space agencies, research institutes, industries
In strong collaboration with European research and scientific institutes
SPASCIA is acting for a better use of Earth observations from space
THE COMPANY
SPASCIA
Space Science Algorithmics
Satellite mission/systems
IASI, IASI-NG, MTG-IRS, MicroCarb, S5P, CO2M
Atmosphere and environment
Atmsopheric chemistry, GHG, air quality, Cllimate
Level 1 data simulation,
processing, cal/val
Level 2 geophysical
products retrieval,
caracterisation,
analysis
Level 3 & 4
atmospheric fields
innovation, exploitation
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
16
Earth and environment challenges at local, regional and global scales : continuous increase of
human activities modify the atmospheric composition
Impact on environment (climate; soil, water and air quality; biodiversity)
Impact on health, on economy
European opportunities for better adressing these challenges : Europe is entering in the era of
operational measurement of the air composition from space, for the analysis and forecasting of chemical
weather and climate monitoring
PHIDIAS FOR ATMOSPHERE : CONTEXT
First measurements already available : The European operational atmospheric composition
observation from space currently benefits from data provided by IASI and GOME onboard the
Metop satellites, as well as Sentinel 5 precursor (S5P/TROPOMI)
4 Operational satellites from 2021 : Metop-NG, MeteoSat Third Generation, Sentinel 4 and 5
Development of the European strategy for the measurement of GHG for monitoring human emissions
Copernicus Operational forecasting services for atmospheric chemistry (CAMS) and Climate (C3S)
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
17
From 2021, European atmospheric sounding missions will deliver each day several TB (terabytes) of raw
datacubes at high spatial/temporal/spectral resolutions. This represents an unprecedented amount of
atmospheric data, with improved quality and coverage.
 Increasing difficulties for properly dealing with all available information
 key challenge : provide the capacity of identifying and focusing on useful data, e.g., by targeting scenes of interest in
view of their dedicated processing or exploitation.
PHIDIAS addresses those needs by using HPC and HPDA capacities : intelligent screening approaches for
the exploitation of large amounts of satellite atmospheric data in an operational context, for detection and
identification of atmospheric composition events.
.
CHALLENGES AND OBJECTIVES
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
18
PHIDIAS ADDED VALUE
Earth observation from space is underexploited
Huge quantity of data, increasing information content and
accuracy, complex processes from data to information
Operational forecasting systems
Efficient, real-time data assimilation in models,
but could not use « extreme » data related with
« extreme events »
Reduced added value of the data
Research groups, scientific institutes
Innovative and well adapted to event studies
but case by case exploitation of the data
Not exhaustive exploitation,
no real-time analysis
Screening (detection
and filtering) on the
fly of relevant data
Prototype service
Realtime Detection &
analyse of
extreme events, pollutant
plumes, other targeted
events
Dedicated services
Early warning, monitoring, decision support
Added-value :
Robust detection,
objective filtering
Added-value :
Verification,
improvement
Innovation
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
19
DATA FOR USE CASE DEMONSTRATION
This use case propose to Develop, test and prototype the approach with Sentinel 5 Precursor (S5P) data/products
S5P : first atmospheric Sentinel for air
quality and climate
Launched Oct. 13 2017, 7 years lifetime
UV-Vis-NIR-SWIR naidr view spectrometer,
with enhanced radiometric performances,
spatial resolution (7x3.5 km2) and temporal
revisit (Global daily coverage)
500 GB
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
20
Objective : Test, implement and demonstrate intelligent screening of large amount of satellite data for detection
and identification of anomalous atmospheric composition events
Two processing prototypes would be proposed for development and test :
1. PCA-based screening of L1 data (SWIR) for detection of extreme events. Based on experience and methods
developed for IASI, implementation and consolidation of algorithms and tools for generic processing of atmospheric
spectra recorded by S5P.
2. New AI methods for objective/automatic detection of plumes from L2 products (CO, others ? (CH4 NO2, SO2) :
SPASCIA experience with physical methods analyzing significant signal enhancements
WP4 OBJECTIVES
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
21
Objective : Test, implement and demonstrate intelligent screening of large amount of satellite data for detection
and identification of anomalous atmospheric composition events
Two processing prototypes would be proposed for development and test :
1. PCA-based screening of L1 data (SWIR) for detection of extreme events. Based on experience and methods
developed for IASI, implementation and consolidation of algorithms and tools for generic processing of atmospheric
spectra recorded by S5P.
Product : Real Time, operational detection, qualification and monitoring of rare or extremely strong events
Potential users : Atmospheric research, Operational air chemistry services (CAMS) for forecast validation, quality control,
improvements, future services of alerts and decision support
1. New AI methods for objective/automatic detection of plumes from L2 products (CO, others ? (CH4 NO2, SO2) :
SPASCIA experience with physical methods analyzing significant signal enhancements
WP4 OBJECTIVES
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
22
PCA-Based filtering on L1 data – illustration with IASI data
Results from work by SPASCIA/HYGEOS/LATMOS, funded by CNES
Nominal
case:
denoising
filter
Extreme
case:
identification
of the
residual
signal
Reconstruction of
"scores" maps:
Rapid detection of
anomalous situations
Analysis of « residuals" :
Interpretation of the
anomalous event
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
23
PCA-Based filtering on L1 data – illustration with IASI data
Results from work by SPASCIA/HYGEOS/LATMOS, funded by CNES
Big fires in Indonésie,
2015
PCA-based detection of the fire event
Cloud fraction
Operational CO product
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
24
Objective : Test, implement and demonstrate intelligent screening of large amount of satellite data for detection
and identification of anomalous atmospheric composition events
Two processing prototypes would be proposed for development and test :
1. PCA-based screening of L1 data (SWIR) for detection of extreme events. Based on experience and methods
developed for IASI, implementation and consolidation of algorithms and tools for generic processing of atmospheric
spectra recorded by S5P.
2. New AI methods for objective/automatic detection of plumes from L2 products (CO, others ? (CH4 NO2, SO2) :
SPASCIA experience with physical methods analyzing significant signal enhancements
Product : automatic Identification of pollutant plumes and sources in satellite maps
Potential users : Necessary input for processing and services of pollutant emission quantification, pollution monitoring : Atmospheric
research, Operational system of GHG and pollutant emission monitoring (e.g., CO2 emission survey for supporting Paris Climate
Agreement), end users services
WP4 OBJECTIVES
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
25
Plume detection for pollutant emission
quantification
TNO anthr. CO2 emission at 6x6 km2
(WP2)
From atmospheric gradients
…
… to anthropogenic
emissions
XCO2 COSMO-GHG simulations
Provided by EMPA (from H2020 ECMWF CHE project)
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
26
Methods for objective/automatic detection of plumes from L2 products from S5P (CO, CH4 NO2, SO2) :
Experience with physical methods analyzing significant signal enhancements
Innovative AI methods : efficient detection of plumes and associated source(s)
S5P products
Plume detection for pollutant emission
quantification
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
27
Space-based Earth observation will provide amazing amount of complex, informative data :
How to extract and use comprehensively this increasing information ?
Operational Models and systems fail to exploit the non-understood data
Research face to strong limits for analyzing all the underlined information and science
PHIDIAS will provide new tools and approaches to deal with the measurements and the information, allowing better and
more efficient use of the data, and pave the ways for finding new paradigms.
GOAL
XCO2 IFS model simulation
provided by ECMWF
(from H2020 ECMWF CHE project)
:
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Thank-you
Pascal Prunet, SPASCIA
Pascal.prunet@spascia.fr
13.02.2020 PHIDIAS Webinar | 13.02.2020 | www.phidias-hpc.eu | @PhidiasHpc 28
The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
On demand image processing for
environmental monitoring: challenges
and uses cases coming from earth
observation data
WP 5 leader : Jean-Christophe Desconnets
(IRD, ESPACE-DEV)
30
Working primarily in partnership with Mediterranean and inter-
tropical countries on the science of global development issues.
Multidisciplinary research: health and society, climate change,
humanitarian and political crises, agriculture and biodiversity
The French National Research Institute for
Sustainable Development
Publications : 1300/Year
65 Research Unit
52 % co-publications with
South Countries
2048 Agents
Budget : 230 M€/Year
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Data and HPC Challenges for land surface community
• Late 2000s, shared diagnosis among scientific community and public
authorities : Under-utilization of satellite imagery to monitor environment
Challenges
Facilitate selection and image analysis activities for end-users
Issues
Technological (Big data and scalability)
Openess and reusability (FAIR data)
31
Accessing relevant images
Having the skills
Finding support
Getting appropriate equipments
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
State of the art : data land surface interoperable
catalogs
Catalog GEOSUD of VHR images
SPOT6, 14 000 raw images
32
Catalog PEPS of Radar & optical
Sentinel products
>> 10 Millions of raw images
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
State of the art : on demand-processing services
Data pre-processing services (GEOSUD IDS V2) : extraction,
subsetting, segmentation, classification …
3313.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
State of the art : land surface processing chains
Coming from scientific remote sensing community
THEIA Scientific Expertise Center (SEC) working around ten themes :
Agriculture, Forest, Urban, Coastline, Health, Water…
Outputs : scientifically validated processing chains for environmental monitoring
34
Land cover IOTA Chain Soil moisture with VR spatial resolution
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Objectives of WP5
Technical Sub-objectives
Dedicated environment adapted to the target users
Ability to produce maps over large areas in a systematic manner
Open the dissemination of processsing chains outputs in FAIR way
Improvement of data reusability in perspective of EOSC
Leveraging AI techniques to provide alternative image classification
methodologies
Merging scientific experimental algorithms and catalogs
with community user-needs
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
Two examples of uses cases coming from
environmental monitoring community
Sentinel-1/Sentinel-2-derived Soil Moisture
product at Plot scale (S2MP) over agricultural
areas
Importance of monitoring of the soil moisture in
agricultural areas
Target Users
PHIDIAS issues
Rationale
Free access to Sentinel-1 (SAR satellite) and
Sentinel-2 (optical High resolution 10x10 m) with
high revisit
HPC : Interactive and on-demand
processing on specific zone and a large
temporal depth
Data : Big data access and data outputs
FAIRness
Scientific : water cycle modelisation
purposes
Farming sector : mapping of irrigation
activities
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Remote sensing images processing with artificial
intelligence: application to land cover mapping
and super-resolution
Taking advantage of IA libraries maturity and
GPU architecture to apply deep network for
classification images purposes
Target Users
PHIDIAS issues
Rationale
Super-resolution of Sentinel-2 images 10m  1.5m
HPC : GPU architecture to test scalability of IA
approach over national territory
temporal depth
Data : Big data access and data outputs
FAIRness
Scientific : experimental mode (notebook)
Public authority : very high resolution
LU/LC monitoring
Processing of multi modal imagery for land cover
mapping using semantic segmentation. (SPOT &
Sentinel-2)
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Architecture targeted (logical view)
user
services
HPDA /HPC facilities
Discovery Data services
On demand
processing
IA RF
Data Terra Web Portal (land surface focus)
Data and
computing layer
Data,
services,
process
catalog
Notebooks
Interactive Web IHM
Data sources Processing libraries & framework
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Take away message
WP5 Challenges
Taking advantage of HPC architecture and big data infrastructure to
facilitate selection and image analysis activities for environmental monitoring
Use cases
• Targeted to land surface community: scientific and public authorities
• Capibility to produce on-demand new mapping products at very temporal and spatial
resolution (soil moisture, LU/LC, …)
• Provide dedicated and interactive user environment to select, configure and execute
processing chains
• Open and FAIR diffusion of outputs data in reusable manner
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Thank-you
Jean-Christophe Desconnets, IRD, ESPACE-DEV
PHIDIAS WP 5 leader
13.02.2020 PHIDIAS Webinar | 13.02.2020 | www.phidias-hpc.eu | @PhidiasHpc 41
The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
PHIDIAS HPC - Building a prototype
for Earth Science Data and HPC
services
Webinar | February 13, 2020, 10:00 AM CEST
The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
Boosting the use of Cloud
Services for marine data studies
Cécile NYS, IFREMER
Assistant Manager Ocean Data Cluster – ODATIS
Phidias WP6 member
Webinar | February 13, 2020
WP6 “Use-case 3 – Ocean” overview
Boosting the use of Cloud Services for marine data studies
Combine and collocate data from several data sources (in situ &
satellite)
Enhancing data archiving (most observation cannot be reproduced) 
facilitate data reuse
Facilitate and speed up co-localisation of data from different sources
Adopting new data structures (based on big-data technologies)
DataCubes
NoSQL databases (numerical data) : Cassandra, MongoDB, etc.
Semantic Web (text data)
Providing on demand data browsing and processing facilities
4413.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
WP6 “Use-case 3 – Ocean” overview
Focus on two geographical areas
North Atlantic Ocean Baltic Sea
4513.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Ocean observation
Ocean  difficult ecosystem to observe
Constantly evolving
Seasonally
Specific phenomenon evolving quickly : waves, currents, plankton blooms, etc.
Not easily accessible, especially the deep seas (costs of equipment
deployment)
 Requires several complementary systems (from satellite to
underwater vehicles, but also buoys, etc.)
 Importance of inter-comparisons and co-processing of all
produced data
4613.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Input Data
Initial tests on specific variables
Temperature and Salinity  drive the ocean circulation (geostrophic
currents, turbulence, water masses)
Chlorophyll  indicator of primary production and eutrophication
Aggregate and process data (in situ & satellite) from several
infrastructures
Copernicus Marine Environmental Monitoring Services
(marine.copernicus.eu) & associated WEkEO DIAS (www.wekeo.eu)
SeaDataNet European Research Infrastructure (www.seadatanet.org)
EMODnet (DG-Mare) : European Marine Observation and Data Network
(www.emodnet.eu)
4713.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Input Data
4813.02.2020 PHIDIAS Webinar | 13.02.2020 |
CMEMS –
North Atlantic
CMEMS –
Baltic Sea
SeaDataNet & EMODnet
In situ data Atlantic Iberian Biscay Irish Ocean- In-
Situ Near Real Time Observations
Baltic Sea- In Situ Near Real Time
Observations
Satellite data North Atlantic Surface Chlorophyll
Concentration from Satellite observations
(monthly)
Baltic Sea, Ocean Colour Chlorophyll (daily
observation)
Improvement of Data Storage
Present data structures (e.g. collections of NetCDF files) of in-situ marine
data are not very efficient, due to
the large number of files
and/or
the heterogeneity of data observations
Challenges
Access quickly to a few number of observations within a large number of observations 
data visualization and data selections on web portals
Access to large number of observations within a large number of observations  data
processing or parallel processing (e.g. machine learning algorithms, interpolation
software such as DIVA, etc.)
Solutions
Data visualization  test of No-SQL databases (e.g. Cassandra - cassandra.apache.org)
Data processing  test of “data cubes” structures such as Parquet (columnar storage
format from the Haddoop ecosystem - parquet.apache.org)
4913.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
On demand Data Processing
“Virtual Research Environment” for users, allowing
Jupyter Notebook as a basis to develop workflows
Annotate, compare, conserve and share expertise
Access to different data structures in one environment (access to specified data on premise or remotely)
Scripting in various languages (Python, R, Julia, etc.)
Access to Pangeo components
Use of Diva software
Software for gridding data, using a finite-element method
Developed in Julia programming language
Extension for satellite images
GIS features
Visualisation of inputs (study area), outputs (compare results of different processes) and images
Existing tools : Diva online, Sextant, Geomatys, etc.
5013.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Data Visualisation and inter-comparison
2 case-studies
Surface Salinity in North Atlantic
CTD (SeaDataNet),
Argo Floats (CMEMS),
SMOS satellite,
Chlorophyll in North-East Atlantic and Balitic Sea
CTD and bottles (SeaDataNet)
BGC Argo floats (ARGO GDAC)
Ferrybox
Sentinel 2 images (DIAS WEkEO)
Visual comparisons (input and output) using GIS features
Software comparisons
Using Pangeo components
Using DIVA software
“Declouding” processing for satellite images (visible and infra-red)
5113.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Thank-you
Cécile NYS & Gilbert MAUDIRE, IFREMER
PHIDIAS WP6 leader
Phidias@Ifremer.fr / Cecile.Nys@Ifremer.fr / Gilbert.Maudire@Ifremer.fr
13.02.2020 PHIDIAS Webinar | 13.02.2020 | www.phidias-hpc.eu | @PhidiasHpc 52
AI service targeting user communities
Aleksi Kallio
PHIDIAS webinar 13.2.2020
53
54
Non-profit state
organization with
special tasks
Headquarters in
Espoo,
datacenter in
Kajaani
Owned by state (70%)
and all Finnish higher education institutions
(30%)
Turn over
in 2018
44,9M€
Circa
employees
in 2018
350
CSC’s solutions
5513.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
55
Computing and software
Data management and analytics for
research
Support and training for research
Research administration
Solutions for managing and
organizing education
Solutions for learners and teachers
Solutions for educational and
teaching cooperation
Hosting services tailored to
customers’ needs
Identity and authorisation
Management and use of data
ICT platforms, Funet network and
data center functions are the base
for our solutions
CSC in PHIDIAS
56
WP2: 12 PM
Intelligent data workflows on HPC/HPDA environment
Optimisation of distributed large scale AI/ML workloads
WP6: 3 PM
Supporting Ocean use case
“The other computing center partner together with CINES”
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Our AI user communities
57
Researchers in capacity driven fields
Researchers in established data driven fields
Researchers in emerging data driven fields
Research IT management
Research eInfrastructures
Public institutions looking for
data driven solutions57
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Services and solutions for artificial intelligence
58
• Mapping of needs, possibilities and competences
• Defining and implementing AI projects
• Data science, machine learning, data engineering, AI and BI expertise
AI & BI consultation
• Practical courses
• Methods support for AI and data analytics
Training and user support (ministry funded)
• Cloud services for building intelligent systems
• HPC computing for ambitious machine learning
• Software services for displaying data (Notebooks)
• Sensitive data environment for personal data
Computing services (ministry funded)58
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
NeIC iOBS (Improved Observation Usage in
Numerical Weather Prediction)
5913.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Training and user support
60
• Data analytics with R
• Data visualisation
• Big data analytics with Apache Spark
Data science courses
• Practical machine learning
• Practical machine learning for spatial data
• Practical deep learning
Machine learning courses
• Supporting researchers on using our environment and choosing right methods
• Data science, machine learning, data engineering and AI expertise
Methods support from service desk
60
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Computing services
61
• Rahti container cloud (Kubernetes / OpenShift)
• Apache Spark for big data, Apache Kafka for streaming data, GPU capacity coming
• cPouta IaaS cloud
• GPU capacity (Nvidia P100), IO acceleration (SSD)
• Allas object storage system
Cloud services
• Puhti and Puhti-AI
• Nvidia V100 GPU’s with fast interconnect
• Coming next: Mahti supercomputer and Lumi EuroHPC supercomputer
HPC computing services
• Notebooks provides easy-to-use environments for working with data and programming
Software services
• ePouta sensitive data IaaS cloud
• GPU capacity (Nvidia V100)
Sensitive data services
61
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
EuroHPC pre-exascale supercomputer ecosystem
LUMI supercomputer
One of the world’s fastest computer systems
Performance more than tenfold compared to Europe's fastest
supercomputer today
450 m2
PINTA-ALA
Computing power
equivalent to
Peak Performance
Data reading speed from
disk to memory
corresponds to the
simultaneous operation of
MacBook Pro computers
600 000
Size of a basketball court
200+
Pflop/s
1 system
Blu-ray Players
18 500
450m2
facebook.com/CSCfi
twitter.com/CSCfi
youtube.com/CSCfi
linkedin.com/company/csc---it-center-for-science
Kuvat CSC:n arkisto ja Thinkstock
github.com/CSCfi
Aleksi Kallio
aleksi.kallio@csc.fi
www.csc.fi
Thank-you for joining us!
13.02.2020 PHIDIAS Webinar | 13.02.2020 | www.phidias-hpc.eu | @PhidiasHpc 66
The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
PHIDIAS HPC - Building a prototype
for Earth Science Data and HPC
services
Webinar | February 13, 2020, 10:00 AM CEST

More Related Content

What's hot

D2.10 Grid-enabled Spatial Data Infrastructure serving GEOSS, INSPIRE, and UNSDI
D2.10 Grid-enabled Spatial Data Infrastructure serving GEOSS, INSPIRE, and UNSDID2.10 Grid-enabled Spatial Data Infrastructure serving GEOSS, INSPIRE, and UNSDI
D2.10 Grid-enabled Spatial Data Infrastructure serving GEOSS, INSPIRE, and UNSDIenvirogrids-blacksee
 
Cv avishek
Cv avishekCv avishek
Cv avishek
Avishek Ghosh
 
Gergely Sipos, Claudio Cacciari: Welcome and mapping the landscape: EOSC-hub ...
Gergely Sipos, Claudio Cacciari: Welcome and mapping the landscape: EOSC-hub ...Gergely Sipos, Claudio Cacciari: Welcome and mapping the landscape: EOSC-hub ...
Gergely Sipos, Claudio Cacciari: Welcome and mapping the landscape: EOSC-hub ...
EOSC-hub project
 
RINPAS Data for Decisions - Big Data and Data Communities
RINPAS Data for Decisions - Big Data and Data CommunitiesRINPAS Data for Decisions - Big Data and Data Communities
RINPAS Data for Decisions - Big Data and Data Communities
Helen Thompson
 
EMPA Materials Science and Technology - Klaus Richter
EMPA Materials Science and Technology - Klaus RichterEMPA Materials Science and Technology - Klaus Richter
EMPA Materials Science and Technology - Klaus Richter
UCL
 
Plan4all Dissemination - Georama at PCI 2010 Greece
Plan4all Dissemination - Georama at PCI 2010 GreecePlan4all Dissemination - Georama at PCI 2010 Greece
Plan4all Dissemination - Georama at PCI 2010 GreeceManolis Viennas
 
The role of INSPIRE and the significance of the Plan4all experience
The role of INSPIRE and the significance of the Plan4all experienceThe role of INSPIRE and the significance of the Plan4all experience
The role of INSPIRE and the significance of the Plan4all experience
plan4all
 
Linking EUDAT services to the EGI Fed-Cloud - EUDAT Summer School (Hans van P...
Linking EUDAT services to the EGI Fed-Cloud - EUDAT Summer School (Hans van P...Linking EUDAT services to the EGI Fed-Cloud - EUDAT Summer School (Hans van P...
Linking EUDAT services to the EGI Fed-Cloud - EUDAT Summer School (Hans van P...
EUDAT
 
Open geodata in Finland
Open geodata in FinlandOpen geodata in Finland
Open geodata in Finland
Antti Rainio
 
Inspire-hands_on-data_transformation
Inspire-hands_on-data_transformationInspire-hands_on-data_transformation
Inspire-hands_on-data_transformation
Chris Schubert
 
General overview of the INSPIRE Directive and relations to other policies
General overview of the INSPIRE Directive and relations to other policiesGeneral overview of the INSPIRE Directive and relations to other policies
General overview of the INSPIRE Directive and relations to other policiesinspireeu
 
INSPIRE principles, components and implementation
INSPIRE principles, components and implementationINSPIRE principles, components and implementation
INSPIRE principles, components and implementation
inspireeu
 
Big data in the research life cycle: technologies, infrastructures, policies
Big data in the research life cycle: technologies, infrastructures, policiesBig data in the research life cycle: technologies, infrastructures, policies
Big data in the research life cycle: technologies, infrastructures, policies
BigData_Europe
 
Building High Performance Computing Capability in the African Continent/Happy...
Building High Performance Computing Capability in the African Continent/Happy...Building High Performance Computing Capability in the African Continent/Happy...
Building High Performance Computing Capability in the African Continent/Happy...
Academy of Science of South Africa (ASSAf)
 
The building of a European Spatial Data Infrastructure , INSPIRE
The building of a European Spatial Data Infrastructure,  INSPIREThe building of a European Spatial Data Infrastructure,  INSPIRE
The building of a European Spatial Data Infrastructure , INSPIRE
inspireeu
 
Rogan esip overview
Rogan esip overviewRogan esip overview
Rogan esip overviewRebreid
 
Extreme earth overview
Extreme earth overviewExtreme earth overview
Extreme earth overview
Big Data Value Association
 
Earth Observation - DUE/VAE
Earth Observation - DUE/VAEEarth Observation - DUE/VAE
Earth Observation - DUE/VAE
Space-Applications
 
Vladimir Sahakyan - The National Research and Education Network. Problems and...
Vladimir Sahakyan - The National Research and Education Network. Problems and...Vladimir Sahakyan - The National Research and Education Network. Problems and...
Vladimir Sahakyan - The National Research and Education Network. Problems and...
Arm Igf
 
European Policies for High Performance Computing
European Policies for High Performance ComputingEuropean Policies for High Performance Computing
European Policies for High Performance Computing
Carl-Christian Buhr
 

What's hot (20)

D2.10 Grid-enabled Spatial Data Infrastructure serving GEOSS, INSPIRE, and UNSDI
D2.10 Grid-enabled Spatial Data Infrastructure serving GEOSS, INSPIRE, and UNSDID2.10 Grid-enabled Spatial Data Infrastructure serving GEOSS, INSPIRE, and UNSDI
D2.10 Grid-enabled Spatial Data Infrastructure serving GEOSS, INSPIRE, and UNSDI
 
Cv avishek
Cv avishekCv avishek
Cv avishek
 
Gergely Sipos, Claudio Cacciari: Welcome and mapping the landscape: EOSC-hub ...
Gergely Sipos, Claudio Cacciari: Welcome and mapping the landscape: EOSC-hub ...Gergely Sipos, Claudio Cacciari: Welcome and mapping the landscape: EOSC-hub ...
Gergely Sipos, Claudio Cacciari: Welcome and mapping the landscape: EOSC-hub ...
 
RINPAS Data for Decisions - Big Data and Data Communities
RINPAS Data for Decisions - Big Data and Data CommunitiesRINPAS Data for Decisions - Big Data and Data Communities
RINPAS Data for Decisions - Big Data and Data Communities
 
EMPA Materials Science and Technology - Klaus Richter
EMPA Materials Science and Technology - Klaus RichterEMPA Materials Science and Technology - Klaus Richter
EMPA Materials Science and Technology - Klaus Richter
 
Plan4all Dissemination - Georama at PCI 2010 Greece
Plan4all Dissemination - Georama at PCI 2010 GreecePlan4all Dissemination - Georama at PCI 2010 Greece
Plan4all Dissemination - Georama at PCI 2010 Greece
 
The role of INSPIRE and the significance of the Plan4all experience
The role of INSPIRE and the significance of the Plan4all experienceThe role of INSPIRE and the significance of the Plan4all experience
The role of INSPIRE and the significance of the Plan4all experience
 
Linking EUDAT services to the EGI Fed-Cloud - EUDAT Summer School (Hans van P...
Linking EUDAT services to the EGI Fed-Cloud - EUDAT Summer School (Hans van P...Linking EUDAT services to the EGI Fed-Cloud - EUDAT Summer School (Hans van P...
Linking EUDAT services to the EGI Fed-Cloud - EUDAT Summer School (Hans van P...
 
Open geodata in Finland
Open geodata in FinlandOpen geodata in Finland
Open geodata in Finland
 
Inspire-hands_on-data_transformation
Inspire-hands_on-data_transformationInspire-hands_on-data_transformation
Inspire-hands_on-data_transformation
 
General overview of the INSPIRE Directive and relations to other policies
General overview of the INSPIRE Directive and relations to other policiesGeneral overview of the INSPIRE Directive and relations to other policies
General overview of the INSPIRE Directive and relations to other policies
 
INSPIRE principles, components and implementation
INSPIRE principles, components and implementationINSPIRE principles, components and implementation
INSPIRE principles, components and implementation
 
Big data in the research life cycle: technologies, infrastructures, policies
Big data in the research life cycle: technologies, infrastructures, policiesBig data in the research life cycle: technologies, infrastructures, policies
Big data in the research life cycle: technologies, infrastructures, policies
 
Building High Performance Computing Capability in the African Continent/Happy...
Building High Performance Computing Capability in the African Continent/Happy...Building High Performance Computing Capability in the African Continent/Happy...
Building High Performance Computing Capability in the African Continent/Happy...
 
The building of a European Spatial Data Infrastructure , INSPIRE
The building of a European Spatial Data Infrastructure,  INSPIREThe building of a European Spatial Data Infrastructure,  INSPIRE
The building of a European Spatial Data Infrastructure , INSPIRE
 
Rogan esip overview
Rogan esip overviewRogan esip overview
Rogan esip overview
 
Extreme earth overview
Extreme earth overviewExtreme earth overview
Extreme earth overview
 
Earth Observation - DUE/VAE
Earth Observation - DUE/VAEEarth Observation - DUE/VAE
Earth Observation - DUE/VAE
 
Vladimir Sahakyan - The National Research and Education Network. Problems and...
Vladimir Sahakyan - The National Research and Education Network. Problems and...Vladimir Sahakyan - The National Research and Education Network. Problems and...
Vladimir Sahakyan - The National Research and Education Network. Problems and...
 
European Policies for High Performance Computing
European Policies for High Performance ComputingEuropean Policies for High Performance Computing
European Policies for High Performance Computing
 

Similar to PHIDIAS HPC – Building a prototype for Earth Science Data and HPC Services

Phidias: Steps forward in detection and identification of anomalous atmospher...
Phidias: Steps forward in detection and identification of anomalous atmospher...Phidias: Steps forward in detection and identification of anomalous atmospher...
Phidias: Steps forward in detection and identification of anomalous atmospher...
Phidias
 
BioDT for the UiO Science section meeting 2023-03-24
BioDT for the UiO Science section meeting 2023-03-24BioDT for the UiO Science section meeting 2023-03-24
BioDT for the UiO Science section meeting 2023-03-24
Dag Endresen
 
ESCAPE Kick-off meeting - Welcome (Feb 2019)
ESCAPE Kick-off meeting - Welcome (Feb 2019)ESCAPE Kick-off meeting - Welcome (Feb 2019)
ESCAPE Kick-off meeting - Welcome (Feb 2019)
ESCAPE EU
 
Sshoc kick off meeting - 1.2.3 EOSC board - Social Sciences and Humanities Op...
Sshoc kick off meeting - 1.2.3 EOSC board - Social Sciences and Humanities Op...Sshoc kick off meeting - 1.2.3 EOSC board - Social Sciences and Humanities Op...
Sshoc kick off meeting - 1.2.3 EOSC board - Social Sciences and Humanities Op...
SSHOC
 
Big Data Europe at eHealth Week 2017: Linking Big Data in Health
Big Data Europe at eHealth Week 2017: Linking Big Data in HealthBig Data Europe at eHealth Week 2017: Linking Big Data in Health
Big Data Europe at eHealth Week 2017: Linking Big Data in Health
BigData_Europe
 
Available data sources & Real-time data collection
Available data sources & Real-time data collectionAvailable data sources & Real-time data collection
Available data sources & Real-time data collection
CLEEN_Ltd
 
CeRDI Research RUN Vietnam Agriculture Group
CeRDI Research RUN Vietnam Agriculture GroupCeRDI Research RUN Vietnam Agriculture Group
CeRDI Research RUN Vietnam Agriculture Group
Helen Thompson
 
Bridging the gap to facilitate selection and image analysis activities for la...
Bridging the gap to facilitate selection and image analysis activities for la...Bridging the gap to facilitate selection and image analysis activities for la...
Bridging the gap to facilitate selection and image analysis activities for la...
Phidias
 
Social Innovation For Resource Management
Social Innovation For Resource ManagementSocial Innovation For Resource Management
Social Innovation For Resource Management
jexxon
 
PHIDIAS - Boosting the use of cloud services for marine data management, serv...
PHIDIAS - Boosting the use of cloud services for marine data management, serv...PHIDIAS - Boosting the use of cloud services for marine data management, serv...
PHIDIAS - Boosting the use of cloud services for marine data management, serv...
Phidias
 
Data management plans – EUDAT Best practices and case study | www.eudat.eu
Data management plans – EUDAT Best practices and case study | www.eudat.euData management plans – EUDAT Best practices and case study | www.eudat.eu
Data management plans – EUDAT Best practices and case study | www.eudat.eu
EUDAT
 
Cross e-Infrastructure collaborations
Cross e-Infrastructure collaborationsCross e-Infrastructure collaborations
Cross e-Infrastructure collaborations
EUDAT
 
EUDAT
EUDATEUDAT
EUDAT
EUDAT
 
BDE-SC1 Webinar: OpenPHACTS Re-engineered with Big Data Europe
BDE-SC1 Webinar: OpenPHACTS Re-engineered with Big Data EuropeBDE-SC1 Webinar: OpenPHACTS Re-engineered with Big Data Europe
BDE-SC1 Webinar: OpenPHACTS Re-engineered with Big Data Europe
BigData_Europe
 
The Developing Needs for e-infrastructures
The Developing Needs for e-infrastructuresThe Developing Needs for e-infrastructures
The Developing Needs for e-infrastructures
guest0dc425
 
Project Overview_MINKE
Project Overview_MINKEProject Overview_MINKE
Project Overview_MINKE
MinkeProject
 
Session 50 - High Performance Computing Ecosystem in Europe
Session 50 - High Performance Computing Ecosystem in EuropeSession 50 - High Performance Computing Ecosystem in Europe
Session 50 - High Performance Computing Ecosystem in EuropeISSGC Summer School
 
BDE Technical Webinar 1 : Pilot Instantiation
BDE Technical Webinar 1 :  Pilot InstantiationBDE Technical Webinar 1 :  Pilot Instantiation
BDE Technical Webinar 1 : Pilot Instantiation
BigData_Europe
 
H2020 space wp 2018 2020 call 2019
H2020 space wp 2018 2020 call 2019H2020 space wp 2018 2020 call 2019
H2020 space wp 2018 2020 call 2019
The European GNSS Agency (GSA)
 
Facing data sharing in a heterogeneous research community: lights and shadows...
Facing data sharing in a heterogeneous research community: lights and shadows...Facing data sharing in a heterogeneous research community: lights and shadows...
Facing data sharing in a heterogeneous research community: lights and shadows...
Research Data Alliance
 

Similar to PHIDIAS HPC – Building a prototype for Earth Science Data and HPC Services (20)

Phidias: Steps forward in detection and identification of anomalous atmospher...
Phidias: Steps forward in detection and identification of anomalous atmospher...Phidias: Steps forward in detection and identification of anomalous atmospher...
Phidias: Steps forward in detection and identification of anomalous atmospher...
 
BioDT for the UiO Science section meeting 2023-03-24
BioDT for the UiO Science section meeting 2023-03-24BioDT for the UiO Science section meeting 2023-03-24
BioDT for the UiO Science section meeting 2023-03-24
 
ESCAPE Kick-off meeting - Welcome (Feb 2019)
ESCAPE Kick-off meeting - Welcome (Feb 2019)ESCAPE Kick-off meeting - Welcome (Feb 2019)
ESCAPE Kick-off meeting - Welcome (Feb 2019)
 
Sshoc kick off meeting - 1.2.3 EOSC board - Social Sciences and Humanities Op...
Sshoc kick off meeting - 1.2.3 EOSC board - Social Sciences and Humanities Op...Sshoc kick off meeting - 1.2.3 EOSC board - Social Sciences and Humanities Op...
Sshoc kick off meeting - 1.2.3 EOSC board - Social Sciences and Humanities Op...
 
Big Data Europe at eHealth Week 2017: Linking Big Data in Health
Big Data Europe at eHealth Week 2017: Linking Big Data in HealthBig Data Europe at eHealth Week 2017: Linking Big Data in Health
Big Data Europe at eHealth Week 2017: Linking Big Data in Health
 
Available data sources & Real-time data collection
Available data sources & Real-time data collectionAvailable data sources & Real-time data collection
Available data sources & Real-time data collection
 
CeRDI Research RUN Vietnam Agriculture Group
CeRDI Research RUN Vietnam Agriculture GroupCeRDI Research RUN Vietnam Agriculture Group
CeRDI Research RUN Vietnam Agriculture Group
 
Bridging the gap to facilitate selection and image analysis activities for la...
Bridging the gap to facilitate selection and image analysis activities for la...Bridging the gap to facilitate selection and image analysis activities for la...
Bridging the gap to facilitate selection and image analysis activities for la...
 
Social Innovation For Resource Management
Social Innovation For Resource ManagementSocial Innovation For Resource Management
Social Innovation For Resource Management
 
PHIDIAS - Boosting the use of cloud services for marine data management, serv...
PHIDIAS - Boosting the use of cloud services for marine data management, serv...PHIDIAS - Boosting the use of cloud services for marine data management, serv...
PHIDIAS - Boosting the use of cloud services for marine data management, serv...
 
Data management plans – EUDAT Best practices and case study | www.eudat.eu
Data management plans – EUDAT Best practices and case study | www.eudat.euData management plans – EUDAT Best practices and case study | www.eudat.eu
Data management plans – EUDAT Best practices and case study | www.eudat.eu
 
Cross e-Infrastructure collaborations
Cross e-Infrastructure collaborationsCross e-Infrastructure collaborations
Cross e-Infrastructure collaborations
 
EUDAT
EUDATEUDAT
EUDAT
 
BDE-SC1 Webinar: OpenPHACTS Re-engineered with Big Data Europe
BDE-SC1 Webinar: OpenPHACTS Re-engineered with Big Data EuropeBDE-SC1 Webinar: OpenPHACTS Re-engineered with Big Data Europe
BDE-SC1 Webinar: OpenPHACTS Re-engineered with Big Data Europe
 
The Developing Needs for e-infrastructures
The Developing Needs for e-infrastructuresThe Developing Needs for e-infrastructures
The Developing Needs for e-infrastructures
 
Project Overview_MINKE
Project Overview_MINKEProject Overview_MINKE
Project Overview_MINKE
 
Session 50 - High Performance Computing Ecosystem in Europe
Session 50 - High Performance Computing Ecosystem in EuropeSession 50 - High Performance Computing Ecosystem in Europe
Session 50 - High Performance Computing Ecosystem in Europe
 
BDE Technical Webinar 1 : Pilot Instantiation
BDE Technical Webinar 1 :  Pilot InstantiationBDE Technical Webinar 1 :  Pilot Instantiation
BDE Technical Webinar 1 : Pilot Instantiation
 
H2020 space wp 2018 2020 call 2019
H2020 space wp 2018 2020 call 2019H2020 space wp 2018 2020 call 2019
H2020 space wp 2018 2020 call 2019
 
Facing data sharing in a heterogeneous research community: lights and shadows...
Facing data sharing in a heterogeneous research community: lights and shadows...Facing data sharing in a heterogeneous research community: lights and shadows...
Facing data sharing in a heterogeneous research community: lights and shadows...
 

Recently uploaded

Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Aggregage
 
The Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and SalesThe Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and Sales
Laura Byrne
 
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
Neo4j
 
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
DianaGray10
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance
 
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdfFIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance
 
GraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge GraphGraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge Graph
Guy Korland
 
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR EventsMonitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
Ana-Maria Mihalceanu
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
Alan Dix
 
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Neo4j
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance
 
Secstrike : Reverse Engineering & Pwnable tools for CTF.pptx
Secstrike : Reverse Engineering & Pwnable tools for CTF.pptxSecstrike : Reverse Engineering & Pwnable tools for CTF.pptx
Secstrike : Reverse Engineering & Pwnable tools for CTF.pptx
nkrafacyberclub
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
KatiaHIMEUR1
 
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
James Anderson
 
UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4
DianaGray10
 
Free Complete Python - A step towards Data Science
Free Complete Python - A step towards Data ScienceFree Complete Python - A step towards Data Science
Free Complete Python - A step towards Data Science
RinaMondal9
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
Kari Kakkonen
 
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdf
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfSAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdf
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdf
Peter Spielvogel
 
Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?
Nexer Digital
 
PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)
Ralf Eggert
 

Recently uploaded (20)

Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
 
The Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and SalesThe Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and Sales
 
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
 
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
 
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdfFIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdf
 
GraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge GraphGraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge Graph
 
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR EventsMonitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
 
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
 
Secstrike : Reverse Engineering & Pwnable tools for CTF.pptx
Secstrike : Reverse Engineering & Pwnable tools for CTF.pptxSecstrike : Reverse Engineering & Pwnable tools for CTF.pptx
Secstrike : Reverse Engineering & Pwnable tools for CTF.pptx
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
 
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
 
UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4
 
Free Complete Python - A step towards Data Science
Free Complete Python - A step towards Data ScienceFree Complete Python - A step towards Data Science
Free Complete Python - A step towards Data Science
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
 
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdf
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfSAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdf
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdf
 
Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?
 
PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)
 

PHIDIAS HPC – Building a prototype for Earth Science Data and HPC Services

  • 1. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854. PHIDIAS HPC - Building a prototype for Earth Science Data and HPC services Webinar | February 13, 2020, 10:00 AM CEST
  • 2. PHIDIAS HPC - Building a prototype for Earth Science Data and HPC services 10:00 – 10:10 – PHIDIAS offering to HPC community – Boris Dintrans, PHIDIAS Project Coordinator, CINES 10:10 – 10:20 – Improving the efficiency of the intelligent screening of environmental satellite data – Pascal Prunet, WP4 Leader, SPASCIA 10:20 – 10:30 – On demand image processing for environmental monitoring : challenges and uses cases coming from earth observation data – Jean- Christophe Desconnets, WP5 Leader, IRD 10:30 – 10:35 – Q&A 10:35 – 10:45 – Boosting the use of cloud services for marine data studies – Cecile Nys, WP6 member, IFREMER 10:45 – 10:55 – AI Services targeting user communities – Aleksi Kallio, Development Manager of data analytics, CSC – IT Centre of Science Ltd 10:55 – 11:00 – Q&A 11:00 – 11:05 – Final remarks 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc 2
  • 3. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854. PHIDIAS offering to HPC Community Boris DINTRANS, PHIDIAS coordinator and CINES director Centre Informatique National de l’Enseignement Supérieur (CINES), Montpellier, France
  • 4. CINES: one of the three national supercomputing centers in France 4 TGCC: Joliot-Curie IDRIS: Jean Zay HPC (Occigen) IT hosting Archiving 60 people Budget ~10M€/y National & EU scope 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 5. Projet overview Scope and objectives of the action: the Action’s overall objective is to build a prototype for Data/High Performance Computing (HPC) services based on Earth sciences cases. In this respect, the consortium will develop and provide new services to discover, manage and process spatial and environmental data produced by research communities tackling scientific challenges such as atmospheric, marine and earth observation issues. Duration: 01/09/2019 to 01/09/2022 (3 years) Budget: 3,519,476€ CEF Grant: 2,639,607€ (75%) Project Officer: Mark Vella Muskat from INEA (Innovation & Network Executive Agency) Project number: Action n° 2018-EU-IA-0089 513.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 6. Consortium with 13 partners (8 FR, 2 FI, 1 NL, 1 IT, 1 BE) CINES: Centre Informatique National de l’Enseignement Supérieur (France, Coordinator) CERFACS: Centre Européen de Recherche et de Formation Avancée en Calcul Scientifique (France) CNRS: Centre National de la Recherche Scientifique (France) CSC: IT Centre for Science Ltd. (Finland) Geomatys (France) IRD: Institut de Recherche pour le Développement (France) IFREMER: Institut Français de Recherche pour l‘Exploitation de la Mer (France) MARIS: Mariene Informatie Service MARIS BV (Netherlands) Néovia: Néovia Innovation (France) Spascia (France) SYKE: Finnish Environment Institute (Finland) Trust-IT Services (Italy) ULIEGE: Université de Liège (Belgium) 613.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 7. Four main challenges that must be addressed [Challenge #1] Handling the diversity of data coming from the Earth System Research Infrastructure. [Challenge #2] Developing and testing transversal methods and tools that can be applied to data coming from other scientific domains such as health and environment data. [Challenge #3] Scalability of Data processing tools. [Challenge #4] Industrialization and strength development of HPC/HPDA/AI workflows. 713.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 8. The Data Terra research infrastructure brings the data to PHIDIAS 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 9. Data Terra = 3 scientific use cases for PHIDIAS [Use case #1] Intelligent screening of large amount of satellite data for detection and identification of anomalous atmospheric composition events (WP4, leader: SPASCIA)  AERIS [Use case #2] Big data Earth Observations: processing on-demand for environmental monitoring (WP5, leader: IRD)  THEIA [Use case #3] Ocean (WP6, leader: IFREMER)  ODATIS 913.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 10. The generic and fully-integrated PHIDIAS workflow 1013.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 11. Main project organization 11 Management Board: acting a as the decision-making body of the consortium Technical Board: acting as the supervisory body for the execution of the Action Scientific and users Committee: gather various stakeholders of the HPC community as well as users and citizen communities, and act as an advisory entity for PHIDIAS Security Committee: address transversal security issues and identify Information Technology (IT) risks in the workflows developed during the Action 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 12. 24/04/2020 footer 12 WP1 - Management WP2 – Compute & storage workflow WP3 – Technical coordination WP4 – Satellite data WP5 – Environmental monitoring WP6 – Ocean use case WP7 – Dissemination T1.1: Project contractual management T1.2: Project operational day-to-day management T1.3 – Scientific and Users Committee T2.1: HPC, HPDA and storage services T2.2: Storage backend and archiving services T2.3: On-the-fly computing T3.1: Technical coordination T3.2: Common Metadata Repository T3.3: End-users web Common interactive Processing Services T3.4: Common Portal for data discovery, access and processing T4.1: Implementation of PCA filtering techniques for detection of exceptional atmospheric event T4.2: Adaptation and preparation of tools, data and environment for the data screening processing of Sentinel 5 precursor data T4.3: Processing of 1 year of global S5P data for detection of extreme atmospheric events T4.4.1: UC1 - Exploitation for monitoring and alert service development T4.4.2: UC2 - Scientific exploitation of the product T5.1: EO data processing chains for massive and on-demand execution T5.2: UI web environment dedicated for on-demand execution T5.3: Data workflows for discovery, access of EO raw data and products T6.1: Improvement of long-term stewardship of data T6.2: Improvement of data storage for services to users T6.3: Marine data processing workflows for on-demand processing T6.4: Data inter-comparison, collection and visualization T7.1: Communication and Outreach T7.2: Engagement with user and stakeholder communities T7.3: EU-wide cross-dissemination & concertation T7.4: Sustainability Path & Funding models CINES CINES IRST SPACIA IRD IFREMER TRUST-IT NEOVI A NEOVIA CINES Géomaty s Ulieg e 9 x Delivrable CINES PMs T4.5: Update the service, and prepare for the exploitation Of Copernicus Atmosphere satellite data MARI S 9 3 6 1 8 1 8 Task Leader
  • 13. Thank-you Boris Dintrans, CINES & PHIDIAS Coordination, phidias-hpc.eu boris.dintrans@cines.fr 13.02.2020 PHIDIAS Webinar | 13.02.2020 | www.phidias-hpc.eu | @PhidiasHpc 13
  • 14. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854. Intelligent screening of satellite data for air quality and climate Pascal Prunet, SPASCIA CEO & WP4 Coordinator Webinar | February 13, 2020
  • 15. 15 Independent SME : R&D in space science and environment Processing, analysis and exploitation of remote sensing observations of the Earth atmosphere from space :  For supporting the definition, development, calibration & validation of satellite missions/systems/instruments.  For environmental applications (air quality, atmospheric composition, climate, meteorology). Created in March 2016, located near Toulouse, France 11 staff members : PhDs with complementary skills in Space and Earth Sciences Working for space agencies, research institutes, industries In strong collaboration with European research and scientific institutes SPASCIA is acting for a better use of Earth observations from space THE COMPANY SPASCIA Space Science Algorithmics Satellite mission/systems IASI, IASI-NG, MTG-IRS, MicroCarb, S5P, CO2M Atmosphere and environment Atmsopheric chemistry, GHG, air quality, Cllimate Level 1 data simulation, processing, cal/val Level 2 geophysical products retrieval, caracterisation, analysis Level 3 & 4 atmospheric fields innovation, exploitation 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 16. 16 Earth and environment challenges at local, regional and global scales : continuous increase of human activities modify the atmospheric composition Impact on environment (climate; soil, water and air quality; biodiversity) Impact on health, on economy European opportunities for better adressing these challenges : Europe is entering in the era of operational measurement of the air composition from space, for the analysis and forecasting of chemical weather and climate monitoring PHIDIAS FOR ATMOSPHERE : CONTEXT First measurements already available : The European operational atmospheric composition observation from space currently benefits from data provided by IASI and GOME onboard the Metop satellites, as well as Sentinel 5 precursor (S5P/TROPOMI) 4 Operational satellites from 2021 : Metop-NG, MeteoSat Third Generation, Sentinel 4 and 5 Development of the European strategy for the measurement of GHG for monitoring human emissions Copernicus Operational forecasting services for atmospheric chemistry (CAMS) and Climate (C3S) 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 17. 17 From 2021, European atmospheric sounding missions will deliver each day several TB (terabytes) of raw datacubes at high spatial/temporal/spectral resolutions. This represents an unprecedented amount of atmospheric data, with improved quality and coverage.  Increasing difficulties for properly dealing with all available information  key challenge : provide the capacity of identifying and focusing on useful data, e.g., by targeting scenes of interest in view of their dedicated processing or exploitation. PHIDIAS addresses those needs by using HPC and HPDA capacities : intelligent screening approaches for the exploitation of large amounts of satellite atmospheric data in an operational context, for detection and identification of atmospheric composition events. . CHALLENGES AND OBJECTIVES 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 18. 18 PHIDIAS ADDED VALUE Earth observation from space is underexploited Huge quantity of data, increasing information content and accuracy, complex processes from data to information Operational forecasting systems Efficient, real-time data assimilation in models, but could not use « extreme » data related with « extreme events » Reduced added value of the data Research groups, scientific institutes Innovative and well adapted to event studies but case by case exploitation of the data Not exhaustive exploitation, no real-time analysis Screening (detection and filtering) on the fly of relevant data Prototype service Realtime Detection & analyse of extreme events, pollutant plumes, other targeted events Dedicated services Early warning, monitoring, decision support Added-value : Robust detection, objective filtering Added-value : Verification, improvement Innovation 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 19. 19 DATA FOR USE CASE DEMONSTRATION This use case propose to Develop, test and prototype the approach with Sentinel 5 Precursor (S5P) data/products S5P : first atmospheric Sentinel for air quality and climate Launched Oct. 13 2017, 7 years lifetime UV-Vis-NIR-SWIR naidr view spectrometer, with enhanced radiometric performances, spatial resolution (7x3.5 km2) and temporal revisit (Global daily coverage) 500 GB 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 20. 20 Objective : Test, implement and demonstrate intelligent screening of large amount of satellite data for detection and identification of anomalous atmospheric composition events Two processing prototypes would be proposed for development and test : 1. PCA-based screening of L1 data (SWIR) for detection of extreme events. Based on experience and methods developed for IASI, implementation and consolidation of algorithms and tools for generic processing of atmospheric spectra recorded by S5P. 2. New AI methods for objective/automatic detection of plumes from L2 products (CO, others ? (CH4 NO2, SO2) : SPASCIA experience with physical methods analyzing significant signal enhancements WP4 OBJECTIVES 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 21. 21 Objective : Test, implement and demonstrate intelligent screening of large amount of satellite data for detection and identification of anomalous atmospheric composition events Two processing prototypes would be proposed for development and test : 1. PCA-based screening of L1 data (SWIR) for detection of extreme events. Based on experience and methods developed for IASI, implementation and consolidation of algorithms and tools for generic processing of atmospheric spectra recorded by S5P. Product : Real Time, operational detection, qualification and monitoring of rare or extremely strong events Potential users : Atmospheric research, Operational air chemistry services (CAMS) for forecast validation, quality control, improvements, future services of alerts and decision support 1. New AI methods for objective/automatic detection of plumes from L2 products (CO, others ? (CH4 NO2, SO2) : SPASCIA experience with physical methods analyzing significant signal enhancements WP4 OBJECTIVES 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 22. 22 PCA-Based filtering on L1 data – illustration with IASI data Results from work by SPASCIA/HYGEOS/LATMOS, funded by CNES Nominal case: denoising filter Extreme case: identification of the residual signal Reconstruction of "scores" maps: Rapid detection of anomalous situations Analysis of « residuals" : Interpretation of the anomalous event 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 23. 23 PCA-Based filtering on L1 data – illustration with IASI data Results from work by SPASCIA/HYGEOS/LATMOS, funded by CNES Big fires in Indonésie, 2015 PCA-based detection of the fire event Cloud fraction Operational CO product 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 24. 24 Objective : Test, implement and demonstrate intelligent screening of large amount of satellite data for detection and identification of anomalous atmospheric composition events Two processing prototypes would be proposed for development and test : 1. PCA-based screening of L1 data (SWIR) for detection of extreme events. Based on experience and methods developed for IASI, implementation and consolidation of algorithms and tools for generic processing of atmospheric spectra recorded by S5P. 2. New AI methods for objective/automatic detection of plumes from L2 products (CO, others ? (CH4 NO2, SO2) : SPASCIA experience with physical methods analyzing significant signal enhancements Product : automatic Identification of pollutant plumes and sources in satellite maps Potential users : Necessary input for processing and services of pollutant emission quantification, pollution monitoring : Atmospheric research, Operational system of GHG and pollutant emission monitoring (e.g., CO2 emission survey for supporting Paris Climate Agreement), end users services WP4 OBJECTIVES 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 25. 25 Plume detection for pollutant emission quantification TNO anthr. CO2 emission at 6x6 km2 (WP2) From atmospheric gradients … … to anthropogenic emissions XCO2 COSMO-GHG simulations Provided by EMPA (from H2020 ECMWF CHE project) 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 26. 26 Methods for objective/automatic detection of plumes from L2 products from S5P (CO, CH4 NO2, SO2) : Experience with physical methods analyzing significant signal enhancements Innovative AI methods : efficient detection of plumes and associated source(s) S5P products Plume detection for pollutant emission quantification 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 27. 27 Space-based Earth observation will provide amazing amount of complex, informative data : How to extract and use comprehensively this increasing information ? Operational Models and systems fail to exploit the non-understood data Research face to strong limits for analyzing all the underlined information and science PHIDIAS will provide new tools and approaches to deal with the measurements and the information, allowing better and more efficient use of the data, and pave the ways for finding new paradigms. GOAL XCO2 IFS model simulation provided by ECMWF (from H2020 ECMWF CHE project) : 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 28. Thank-you Pascal Prunet, SPASCIA Pascal.prunet@spascia.fr 13.02.2020 PHIDIAS Webinar | 13.02.2020 | www.phidias-hpc.eu | @PhidiasHpc 28
  • 29. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854. On demand image processing for environmental monitoring: challenges and uses cases coming from earth observation data WP 5 leader : Jean-Christophe Desconnets (IRD, ESPACE-DEV)
  • 30. 30 Working primarily in partnership with Mediterranean and inter- tropical countries on the science of global development issues. Multidisciplinary research: health and society, climate change, humanitarian and political crises, agriculture and biodiversity The French National Research Institute for Sustainable Development Publications : 1300/Year 65 Research Unit 52 % co-publications with South Countries 2048 Agents Budget : 230 M€/Year 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 31. Data and HPC Challenges for land surface community • Late 2000s, shared diagnosis among scientific community and public authorities : Under-utilization of satellite imagery to monitor environment Challenges Facilitate selection and image analysis activities for end-users Issues Technological (Big data and scalability) Openess and reusability (FAIR data) 31 Accessing relevant images Having the skills Finding support Getting appropriate equipments 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 32. State of the art : data land surface interoperable catalogs Catalog GEOSUD of VHR images SPOT6, 14 000 raw images 32 Catalog PEPS of Radar & optical Sentinel products >> 10 Millions of raw images 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 33. State of the art : on demand-processing services Data pre-processing services (GEOSUD IDS V2) : extraction, subsetting, segmentation, classification … 3313.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 34. State of the art : land surface processing chains Coming from scientific remote sensing community THEIA Scientific Expertise Center (SEC) working around ten themes : Agriculture, Forest, Urban, Coastline, Health, Water… Outputs : scientifically validated processing chains for environmental monitoring 34 Land cover IOTA Chain Soil moisture with VR spatial resolution 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 35. Objectives of WP5 Technical Sub-objectives Dedicated environment adapted to the target users Ability to produce maps over large areas in a systematic manner Open the dissemination of processsing chains outputs in FAIR way Improvement of data reusability in perspective of EOSC Leveraging AI techniques to provide alternative image classification methodologies Merging scientific experimental algorithms and catalogs with community user-needs 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 36. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854. Two examples of uses cases coming from environmental monitoring community
  • 37. Sentinel-1/Sentinel-2-derived Soil Moisture product at Plot scale (S2MP) over agricultural areas Importance of monitoring of the soil moisture in agricultural areas Target Users PHIDIAS issues Rationale Free access to Sentinel-1 (SAR satellite) and Sentinel-2 (optical High resolution 10x10 m) with high revisit HPC : Interactive and on-demand processing on specific zone and a large temporal depth Data : Big data access and data outputs FAIRness Scientific : water cycle modelisation purposes Farming sector : mapping of irrigation activities 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 38. Remote sensing images processing with artificial intelligence: application to land cover mapping and super-resolution Taking advantage of IA libraries maturity and GPU architecture to apply deep network for classification images purposes Target Users PHIDIAS issues Rationale Super-resolution of Sentinel-2 images 10m  1.5m HPC : GPU architecture to test scalability of IA approach over national territory temporal depth Data : Big data access and data outputs FAIRness Scientific : experimental mode (notebook) Public authority : very high resolution LU/LC monitoring Processing of multi modal imagery for land cover mapping using semantic segmentation. (SPOT & Sentinel-2) 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 39. Architecture targeted (logical view) user services HPDA /HPC facilities Discovery Data services On demand processing IA RF Data Terra Web Portal (land surface focus) Data and computing layer Data, services, process catalog Notebooks Interactive Web IHM Data sources Processing libraries & framework 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 40. Take away message WP5 Challenges Taking advantage of HPC architecture and big data infrastructure to facilitate selection and image analysis activities for environmental monitoring Use cases • Targeted to land surface community: scientific and public authorities • Capibility to produce on-demand new mapping products at very temporal and spatial resolution (soil moisture, LU/LC, …) • Provide dedicated and interactive user environment to select, configure and execute processing chains • Open and FAIR diffusion of outputs data in reusable manner 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 41. Thank-you Jean-Christophe Desconnets, IRD, ESPACE-DEV PHIDIAS WP 5 leader 13.02.2020 PHIDIAS Webinar | 13.02.2020 | www.phidias-hpc.eu | @PhidiasHpc 41
  • 42. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854. PHIDIAS HPC - Building a prototype for Earth Science Data and HPC services Webinar | February 13, 2020, 10:00 AM CEST
  • 43. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854. Boosting the use of Cloud Services for marine data studies Cécile NYS, IFREMER Assistant Manager Ocean Data Cluster – ODATIS Phidias WP6 member Webinar | February 13, 2020
  • 44. WP6 “Use-case 3 – Ocean” overview Boosting the use of Cloud Services for marine data studies Combine and collocate data from several data sources (in situ & satellite) Enhancing data archiving (most observation cannot be reproduced)  facilitate data reuse Facilitate and speed up co-localisation of data from different sources Adopting new data structures (based on big-data technologies) DataCubes NoSQL databases (numerical data) : Cassandra, MongoDB, etc. Semantic Web (text data) Providing on demand data browsing and processing facilities 4413.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 45. WP6 “Use-case 3 – Ocean” overview Focus on two geographical areas North Atlantic Ocean Baltic Sea 4513.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 46. Ocean observation Ocean  difficult ecosystem to observe Constantly evolving Seasonally Specific phenomenon evolving quickly : waves, currents, plankton blooms, etc. Not easily accessible, especially the deep seas (costs of equipment deployment)  Requires several complementary systems (from satellite to underwater vehicles, but also buoys, etc.)  Importance of inter-comparisons and co-processing of all produced data 4613.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 47. Input Data Initial tests on specific variables Temperature and Salinity  drive the ocean circulation (geostrophic currents, turbulence, water masses) Chlorophyll  indicator of primary production and eutrophication Aggregate and process data (in situ & satellite) from several infrastructures Copernicus Marine Environmental Monitoring Services (marine.copernicus.eu) & associated WEkEO DIAS (www.wekeo.eu) SeaDataNet European Research Infrastructure (www.seadatanet.org) EMODnet (DG-Mare) : European Marine Observation and Data Network (www.emodnet.eu) 4713.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 48. Input Data 4813.02.2020 PHIDIAS Webinar | 13.02.2020 | CMEMS – North Atlantic CMEMS – Baltic Sea SeaDataNet & EMODnet In situ data Atlantic Iberian Biscay Irish Ocean- In- Situ Near Real Time Observations Baltic Sea- In Situ Near Real Time Observations Satellite data North Atlantic Surface Chlorophyll Concentration from Satellite observations (monthly) Baltic Sea, Ocean Colour Chlorophyll (daily observation)
  • 49. Improvement of Data Storage Present data structures (e.g. collections of NetCDF files) of in-situ marine data are not very efficient, due to the large number of files and/or the heterogeneity of data observations Challenges Access quickly to a few number of observations within a large number of observations  data visualization and data selections on web portals Access to large number of observations within a large number of observations  data processing or parallel processing (e.g. machine learning algorithms, interpolation software such as DIVA, etc.) Solutions Data visualization  test of No-SQL databases (e.g. Cassandra - cassandra.apache.org) Data processing  test of “data cubes” structures such as Parquet (columnar storage format from the Haddoop ecosystem - parquet.apache.org) 4913.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 50. On demand Data Processing “Virtual Research Environment” for users, allowing Jupyter Notebook as a basis to develop workflows Annotate, compare, conserve and share expertise Access to different data structures in one environment (access to specified data on premise or remotely) Scripting in various languages (Python, R, Julia, etc.) Access to Pangeo components Use of Diva software Software for gridding data, using a finite-element method Developed in Julia programming language Extension for satellite images GIS features Visualisation of inputs (study area), outputs (compare results of different processes) and images Existing tools : Diva online, Sextant, Geomatys, etc. 5013.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 51. Data Visualisation and inter-comparison 2 case-studies Surface Salinity in North Atlantic CTD (SeaDataNet), Argo Floats (CMEMS), SMOS satellite, Chlorophyll in North-East Atlantic and Balitic Sea CTD and bottles (SeaDataNet) BGC Argo floats (ARGO GDAC) Ferrybox Sentinel 2 images (DIAS WEkEO) Visual comparisons (input and output) using GIS features Software comparisons Using Pangeo components Using DIVA software “Declouding” processing for satellite images (visible and infra-red) 5113.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 52. Thank-you Cécile NYS & Gilbert MAUDIRE, IFREMER PHIDIAS WP6 leader Phidias@Ifremer.fr / Cecile.Nys@Ifremer.fr / Gilbert.Maudire@Ifremer.fr 13.02.2020 PHIDIAS Webinar | 13.02.2020 | www.phidias-hpc.eu | @PhidiasHpc 52
  • 53. AI service targeting user communities Aleksi Kallio PHIDIAS webinar 13.2.2020 53
  • 54. 54 Non-profit state organization with special tasks Headquarters in Espoo, datacenter in Kajaani Owned by state (70%) and all Finnish higher education institutions (30%) Turn over in 2018 44,9M€ Circa employees in 2018 350
  • 55. CSC’s solutions 5513.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc 55 Computing and software Data management and analytics for research Support and training for research Research administration Solutions for managing and organizing education Solutions for learners and teachers Solutions for educational and teaching cooperation Hosting services tailored to customers’ needs Identity and authorisation Management and use of data ICT platforms, Funet network and data center functions are the base for our solutions
  • 56. CSC in PHIDIAS 56 WP2: 12 PM Intelligent data workflows on HPC/HPDA environment Optimisation of distributed large scale AI/ML workloads WP6: 3 PM Supporting Ocean use case “The other computing center partner together with CINES” 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 57. Our AI user communities 57 Researchers in capacity driven fields Researchers in established data driven fields Researchers in emerging data driven fields Research IT management Research eInfrastructures Public institutions looking for data driven solutions57 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 58. Services and solutions for artificial intelligence 58 • Mapping of needs, possibilities and competences • Defining and implementing AI projects • Data science, machine learning, data engineering, AI and BI expertise AI & BI consultation • Practical courses • Methods support for AI and data analytics Training and user support (ministry funded) • Cloud services for building intelligent systems • HPC computing for ambitious machine learning • Software services for displaying data (Notebooks) • Sensitive data environment for personal data Computing services (ministry funded)58 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 59. NeIC iOBS (Improved Observation Usage in Numerical Weather Prediction) 5913.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 60. Training and user support 60 • Data analytics with R • Data visualisation • Big data analytics with Apache Spark Data science courses • Practical machine learning • Practical machine learning for spatial data • Practical deep learning Machine learning courses • Supporting researchers on using our environment and choosing right methods • Data science, machine learning, data engineering and AI expertise Methods support from service desk 60 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 61. Computing services 61 • Rahti container cloud (Kubernetes / OpenShift) • Apache Spark for big data, Apache Kafka for streaming data, GPU capacity coming • cPouta IaaS cloud • GPU capacity (Nvidia P100), IO acceleration (SSD) • Allas object storage system Cloud services • Puhti and Puhti-AI • Nvidia V100 GPU’s with fast interconnect • Coming next: Mahti supercomputer and Lumi EuroHPC supercomputer HPC computing services • Notebooks provides easy-to-use environments for working with data and programming Software services • ePouta sensitive data IaaS cloud • GPU capacity (Nvidia V100) Sensitive data services 61 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 62. 13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
  • 64. LUMI supercomputer One of the world’s fastest computer systems Performance more than tenfold compared to Europe's fastest supercomputer today 450 m2 PINTA-ALA Computing power equivalent to Peak Performance Data reading speed from disk to memory corresponds to the simultaneous operation of MacBook Pro computers 600 000 Size of a basketball court 200+ Pflop/s 1 system Blu-ray Players 18 500 450m2
  • 66. Thank-you for joining us! 13.02.2020 PHIDIAS Webinar | 13.02.2020 | www.phidias-hpc.eu | @PhidiasHpc 66
  • 67. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854. PHIDIAS HPC - Building a prototype for Earth Science Data and HPC services Webinar | February 13, 2020, 10:00 AM CEST