PHIDIAS organised it's third and final PHIDIAS Webinar of the series, this time dedicated to Use Case 2: Big Data Earth Observations (EO), took place on 18 February 2021 at 15:00 CET, showcasing how PHIDIAS is taking advantage of HPC architecture to facilitate selection and image analysis activities for land surface monitoring.
Challenges and Issues of Next Cloud Computing PlatformsFrederic Desprez
Cloud computing has now crossed the frontiers of research to reach industry. It is used every day , whether to exchange emails or make
reservations on web sites. However, many research works remain to be done to improve the performance and functionality of these platforms of tomorrow. In this talk, I will do an overview of some these theoretical and appliead researches done at INRIA and particularly around Clouds distribution, energy monitoring and management, massive data processing and exchange, and resource management.
g-Social - Enhancing e-Science Tools with Social Networking FunctionalityNicholas Loulloudes
Presentation of "g-Social - Enhancing e-Science Tools with Social Networking Functionality" given at the Workshop on Analyzing and Improving Collaborative eScience with Social Networks, Chicago October 8th, 2012. Co-located with IEEE eScience 2012.
A Campus-Scale High Performance Cyberinfrastructure is Required for Data-Int...Larry Smarr
11.12.12
Seminar Presentation
Princeton Institute for Computational Science and Engineering (PICSciE)
Princeton University
Title: A Campus-Scale High Performance Cyberinfrastructure is Required for Data-Intensive Research
Princeton, NJ
GaruaGeo: Global Scale Data Aggregation in Hybrid Edge and Cloud Computing En...Otávio Carvalho
Research work published on the 9th International Conference on Cloud Computing and Services Science (CLOSER 2019) held at Heraklion, Crete.
The combination of Edge Computing devices and Cloud Computing resources brings the best of both worlds: Data aggregation closer to the source and scalable resources to grow the network on demand. However, the ability to leverage each time more powerful edge nodes to decentralize data processing and aggregation is still a significant challenge for both industry and academia. In this work, we extend the Garua platform to analyze the impact of a model for data aggregation in a global scale smart grid application dataset. The platform is extended to support global data aggregators that are placed nearly to the Edge nodes where data is being collected. This way, it is possible to aggregate data not only at the edge of the network but also pre-process data at nearby geographic areas, before sending data to be aggregated globally by global centralization nodes. The results of this work show that the implemented testbed application, through the usage of edge node aggregation, data aggregators geographically distributed and messaging windows, can achieve collection rates above 400 million measurements per second.
Challenges and Issues of Next Cloud Computing PlatformsFrederic Desprez
Cloud computing has now crossed the frontiers of research to reach industry. It is used every day , whether to exchange emails or make
reservations on web sites. However, many research works remain to be done to improve the performance and functionality of these platforms of tomorrow. In this talk, I will do an overview of some these theoretical and appliead researches done at INRIA and particularly around Clouds distribution, energy monitoring and management, massive data processing and exchange, and resource management.
g-Social - Enhancing e-Science Tools with Social Networking FunctionalityNicholas Loulloudes
Presentation of "g-Social - Enhancing e-Science Tools with Social Networking Functionality" given at the Workshop on Analyzing and Improving Collaborative eScience with Social Networks, Chicago October 8th, 2012. Co-located with IEEE eScience 2012.
A Campus-Scale High Performance Cyberinfrastructure is Required for Data-Int...Larry Smarr
11.12.12
Seminar Presentation
Princeton Institute for Computational Science and Engineering (PICSciE)
Princeton University
Title: A Campus-Scale High Performance Cyberinfrastructure is Required for Data-Intensive Research
Princeton, NJ
GaruaGeo: Global Scale Data Aggregation in Hybrid Edge and Cloud Computing En...Otávio Carvalho
Research work published on the 9th International Conference on Cloud Computing and Services Science (CLOSER 2019) held at Heraklion, Crete.
The combination of Edge Computing devices and Cloud Computing resources brings the best of both worlds: Data aggregation closer to the source and scalable resources to grow the network on demand. However, the ability to leverage each time more powerful edge nodes to decentralize data processing and aggregation is still a significant challenge for both industry and academia. In this work, we extend the Garua platform to analyze the impact of a model for data aggregation in a global scale smart grid application dataset. The platform is extended to support global data aggregators that are placed nearly to the Edge nodes where data is being collected. This way, it is possible to aggregate data not only at the edge of the network but also pre-process data at nearby geographic areas, before sending data to be aggregated globally by global centralization nodes. The results of this work show that the implemented testbed application, through the usage of edge node aggregation, data aggregators geographically distributed and messaging windows, can achieve collection rates above 400 million measurements per second.
Gergely Sipos (EGI): Exploiting scientific data in the international context ...Gergely Sipos
Keynote presentation given at "The Emerging Technology Forum – Data Creates Universe - Scientific Data Innovation Conference" of the "Pujiang Innovation Forum 2021" event.
Phidias: Steps forward in detection and identification of anomalous atmospher...Phidias
PHIDIAS is organised a webinar entitled "Steps forward in detection and identification of anomalous atmospheric events" held on 13 October 2020 at 15:00 CEST in collaboration with ESCAPE project. The webinar aimed at showcasing how PHIDIAS is going to improve the usage of HPC and high performance data management services for the development of intelligent screening approaches for the exploitation of large amounts of satellite atmospheric data in an operational context.
Environmental monitoring research requires access to heterogeneous information collected from different sensor sources, based on satellites, UAVs, and in-situ measurements. The wide availability of data collected by Copernicus missions generates a huge amount of data (Big Data) that needs to be organized in an adequate way and requires integration and analysis tools. Moreover, with the possibility of having large areas covered by low-cost sensors, the amount of point data can also grow enormously. Data are typically organized in different data structures, and it is often challenging to understand where the data are collected, how to use them, and how to work with different data sources simultaneously.
The DPS4ESLAB project proposed the creation of an innovative research infrastructure by implementing an IT platform capable of handling the data workflows generated by heterogeneous sources and equipped with web tools and applications to find, use, analyze and share the data stored therein.
The data platform is accessible through the web application at https://edp-portal.eurac.edu/home from which the user can reach the data catalogue, the analysis tool, and the data and mapping collaboration and sharing portal. A fourth element gathers documentation and example code in order to guide the user to use the platform at its best.
All collaborators of Eurac Research have access to the contents of the platform, and thanks to a user management system, access to the platform components is enabled for research partners and private organizations that want to benefit from the available data and tools.
The EDP is build by means of open source components and implements well established interoperability standards, therefore is federated with international organizations such as GEOSS and EOSC, as well as with the OpenDataHub.
Open Science and GEOSS: the Cloud Sandbox enablersterradue
As part of the European project GEOWOW, Terradue was invited to present views at the GEO-X event on future endeavors to serve data democracy & science literacy in GEOSS (http://www.earthobservations.org/geoss.shtml)
PHIDIAS HPC – Building a prototype for Earth Science Data and HPC ServicesPhidias
High-Performance Computing (HPC) technology is becoming increasingly important as a key driver to push European economic growth and Scientific Research. A comprehensive tool that can support the development of a wide array of scientific domains (like Big Data, earth observation and ocean study) and impact societal challenges as well.
The Webinar aims at introducing the Phidias HPC initiative to the European HPC and Research community, including main features, expected impact and advantages for Research & HPC ecosphere. The project is paving the way to increase the HPC and Data capacities of the European Data Infrastructure by pursuing the following objectives:
- Building a prototype for earth scientific data
- Enabling Open Access to HPC Services
- Strengthening FAIRisation
- Creating a framework combining computing, dissemination and archiving resources.
PHIDIAS - Boosting the use of cloud services for marine data management, serv...Phidias
Description and scope of the Project
Phidias HPC is aimed at developing a consolidated and shared HPC and Data service by building on pre-existing and emerging infrastructure in order to create a federation of "user to infrastructure" services.
To achieve its purpose and to gain a comprehensive picture of the European infrastructure landscape, three data area tests will develop and provide new services to discover, manage and process spatial and environmental data produced by research communities tackling scientific challenges such as atmospheric, marine and earth observation issues.
Webinar: How to improve the cloud services for marine data
Observing the ocean is challenging: missions at sea are costly, different scales of processes interact, and the conditions are constantly changing, which is why scientists say that "a measurement not made today is lost forever". For these reasons, it is fundamental to properly store both the data and metadata, so that their access can be guaranteed for the widest community, in line with the FAIR principles: Findable, Accessible, Inter-operable and Reusable.
PHIDIAS HPC has organised a webinar entitled "PHIDIAS: Boosting the use of cloud services for marine management, services and processing" to be held on 4th June 2020 at 11 AM CEST. The webinar aims to introduce the Phidias HPC initiative, in collaboration with the Blue-Cloud project, to the European HPC and Research community, specifically in the Blue economy, to improve the use of (1) cloud services for marine data management, (2) data services to the user in a FAIR perspective, and (3) data processing on demand.
These objectives will be pursued in coherence with the development of the European Open Science Cloud (EOSC) and the Copernicus Data and Information Access Services (DIAS).
Demonstration of the functionality an capabilities of Virtual Hubs developed for ENERGIC-OD project. Presentation given in GEO Business 2017 trade show in London, UK (May 2017).
Data management plans – EUDAT Best practices and case study | www.eudat.euEUDAT
| www.eudat.eu | Presentation given by Stéphane Coutin during the PRACE 2017 Spring School joint training event with the EU H2020 VI-SEEM project (https://vi-seem.eu/) organised by CaSToRC at The Cyprus Institute. Science and more specifically projects using HPC is facing a digital data explosion. Instruments and simulations are producing more and more volume; data can be shared, mined, cited, preserved… They are a great asset, but they are facing risks: we can miss storage, we can lose them, they can be misused,… To start this session, we will review why it is important to manage research data and how to do this by maintaining a Data Management Plan. This will be based on the best practices from EUDAT H2020 project and European Commission recommendation. During the second part we will interactively draft a DMP for a given use case.
Application packaging and systematic processing in earth observation exploita...terradue
An overview of Terradue's solutions supporting Earth Observations (EO) Exploitation Platforms across multiple domains.
Presentation done as part of the Open Geospatial Consortium (OGC) Technical Committee ad-hoc meeting for the setup of a new domain working group on EO Exploitation Platforms.
The EGI Federation of clusters and research clouds are components of the European Open Science Cloud, and they offer technical solutions and an infrastructure to support the EuroGEOSS pilots, GEOSS and EO data exploitation platforms.
Learn how, by looking at the collaboration of EGI with NextGEOSS, the production support of the Geohazards TEP of Terradue and the EOSC-hub collaboration with GEOSS.
Coupling HPC and Data Resources and services together - EUDAT Workshop at exd...EUDAT
Giuseppe Fiameni (CINECA)
The goal of this EUDAT workshop is to present the EUDAT services, the results of the collaboration activity achieved so far and deliver a hands-on on how to write a Data Management Plan or DMP. The DMP is a useful instrument for researchers to reflect on and communicate about the way they will deal with their data as it prompts them to think about how they will generate, analyse and share data during their research project and afterwards.
Gergely Sipos (EGI): Exploiting scientific data in the international context ...Gergely Sipos
Keynote presentation given at "The Emerging Technology Forum – Data Creates Universe - Scientific Data Innovation Conference" of the "Pujiang Innovation Forum 2021" event.
Phidias: Steps forward in detection and identification of anomalous atmospher...Phidias
PHIDIAS is organised a webinar entitled "Steps forward in detection and identification of anomalous atmospheric events" held on 13 October 2020 at 15:00 CEST in collaboration with ESCAPE project. The webinar aimed at showcasing how PHIDIAS is going to improve the usage of HPC and high performance data management services for the development of intelligent screening approaches for the exploitation of large amounts of satellite atmospheric data in an operational context.
Environmental monitoring research requires access to heterogeneous information collected from different sensor sources, based on satellites, UAVs, and in-situ measurements. The wide availability of data collected by Copernicus missions generates a huge amount of data (Big Data) that needs to be organized in an adequate way and requires integration and analysis tools. Moreover, with the possibility of having large areas covered by low-cost sensors, the amount of point data can also grow enormously. Data are typically organized in different data structures, and it is often challenging to understand where the data are collected, how to use them, and how to work with different data sources simultaneously.
The DPS4ESLAB project proposed the creation of an innovative research infrastructure by implementing an IT platform capable of handling the data workflows generated by heterogeneous sources and equipped with web tools and applications to find, use, analyze and share the data stored therein.
The data platform is accessible through the web application at https://edp-portal.eurac.edu/home from which the user can reach the data catalogue, the analysis tool, and the data and mapping collaboration and sharing portal. A fourth element gathers documentation and example code in order to guide the user to use the platform at its best.
All collaborators of Eurac Research have access to the contents of the platform, and thanks to a user management system, access to the platform components is enabled for research partners and private organizations that want to benefit from the available data and tools.
The EDP is build by means of open source components and implements well established interoperability standards, therefore is federated with international organizations such as GEOSS and EOSC, as well as with the OpenDataHub.
Open Science and GEOSS: the Cloud Sandbox enablersterradue
As part of the European project GEOWOW, Terradue was invited to present views at the GEO-X event on future endeavors to serve data democracy & science literacy in GEOSS (http://www.earthobservations.org/geoss.shtml)
PHIDIAS HPC – Building a prototype for Earth Science Data and HPC ServicesPhidias
High-Performance Computing (HPC) technology is becoming increasingly important as a key driver to push European economic growth and Scientific Research. A comprehensive tool that can support the development of a wide array of scientific domains (like Big Data, earth observation and ocean study) and impact societal challenges as well.
The Webinar aims at introducing the Phidias HPC initiative to the European HPC and Research community, including main features, expected impact and advantages for Research & HPC ecosphere. The project is paving the way to increase the HPC and Data capacities of the European Data Infrastructure by pursuing the following objectives:
- Building a prototype for earth scientific data
- Enabling Open Access to HPC Services
- Strengthening FAIRisation
- Creating a framework combining computing, dissemination and archiving resources.
PHIDIAS - Boosting the use of cloud services for marine data management, serv...Phidias
Description and scope of the Project
Phidias HPC is aimed at developing a consolidated and shared HPC and Data service by building on pre-existing and emerging infrastructure in order to create a federation of "user to infrastructure" services.
To achieve its purpose and to gain a comprehensive picture of the European infrastructure landscape, three data area tests will develop and provide new services to discover, manage and process spatial and environmental data produced by research communities tackling scientific challenges such as atmospheric, marine and earth observation issues.
Webinar: How to improve the cloud services for marine data
Observing the ocean is challenging: missions at sea are costly, different scales of processes interact, and the conditions are constantly changing, which is why scientists say that "a measurement not made today is lost forever". For these reasons, it is fundamental to properly store both the data and metadata, so that their access can be guaranteed for the widest community, in line with the FAIR principles: Findable, Accessible, Inter-operable and Reusable.
PHIDIAS HPC has organised a webinar entitled "PHIDIAS: Boosting the use of cloud services for marine management, services and processing" to be held on 4th June 2020 at 11 AM CEST. The webinar aims to introduce the Phidias HPC initiative, in collaboration with the Blue-Cloud project, to the European HPC and Research community, specifically in the Blue economy, to improve the use of (1) cloud services for marine data management, (2) data services to the user in a FAIR perspective, and (3) data processing on demand.
These objectives will be pursued in coherence with the development of the European Open Science Cloud (EOSC) and the Copernicus Data and Information Access Services (DIAS).
Demonstration of the functionality an capabilities of Virtual Hubs developed for ENERGIC-OD project. Presentation given in GEO Business 2017 trade show in London, UK (May 2017).
Data management plans – EUDAT Best practices and case study | www.eudat.euEUDAT
| www.eudat.eu | Presentation given by Stéphane Coutin during the PRACE 2017 Spring School joint training event with the EU H2020 VI-SEEM project (https://vi-seem.eu/) organised by CaSToRC at The Cyprus Institute. Science and more specifically projects using HPC is facing a digital data explosion. Instruments and simulations are producing more and more volume; data can be shared, mined, cited, preserved… They are a great asset, but they are facing risks: we can miss storage, we can lose them, they can be misused,… To start this session, we will review why it is important to manage research data and how to do this by maintaining a Data Management Plan. This will be based on the best practices from EUDAT H2020 project and European Commission recommendation. During the second part we will interactively draft a DMP for a given use case.
Application packaging and systematic processing in earth observation exploita...terradue
An overview of Terradue's solutions supporting Earth Observations (EO) Exploitation Platforms across multiple domains.
Presentation done as part of the Open Geospatial Consortium (OGC) Technical Committee ad-hoc meeting for the setup of a new domain working group on EO Exploitation Platforms.
The EGI Federation of clusters and research clouds are components of the European Open Science Cloud, and they offer technical solutions and an infrastructure to support the EuroGEOSS pilots, GEOSS and EO data exploitation platforms.
Learn how, by looking at the collaboration of EGI with NextGEOSS, the production support of the Geohazards TEP of Terradue and the EOSC-hub collaboration with GEOSS.
Coupling HPC and Data Resources and services together - EUDAT Workshop at exd...EUDAT
Giuseppe Fiameni (CINECA)
The goal of this EUDAT workshop is to present the EUDAT services, the results of the collaboration activity achieved so far and deliver a hands-on on how to write a Data Management Plan or DMP. The DMP is a useful instrument for researchers to reflect on and communicate about the way they will deal with their data as it prompts them to think about how they will generate, analyse and share data during their research project and afterwards.
Remote Sensing and Computational, Evolutionary, Supercomputing, and Intellige...University of Maribor
Slides from talk:
Aleš Zamuda: Remote Sensing and Computational, Evolutionary, Supercomputing, and Intelligent Systems.
11th International Conference on Electrical, Electronics and Computer Engineering (IcETRAN), Niš, 3-6 June 2024
Inter-Society Networking Panel GRSS/MTT-S/CIS Panel Session: Promoting Connection and Cooperation
https://www.etran.rs/2024/en/home-english/
Phenomics assisted breeding in crop improvementIshaGoswami9
As the population is increasing and will reach about 9 billion upto 2050. Also due to climate change, it is difficult to meet the food requirement of such a large population. Facing the challenges presented by resource shortages, climate
change, and increasing global population, crop yield and quality need to be improved in a sustainable way over the coming decades. Genetic improvement by breeding is the best way to increase crop productivity. With the rapid progression of functional
genomics, an increasing number of crop genomes have been sequenced and dozens of genes influencing key agronomic traits have been identified. However, current genome sequence information has not been adequately exploited for understanding
the complex characteristics of multiple gene, owing to a lack of crop phenotypic data. Efficient, automatic, and accurate technologies and platforms that can capture phenotypic data that can
be linked to genomics information for crop improvement at all growth stages have become as important as genotyping. Thus,
high-throughput phenotyping has become the major bottleneck restricting crop breeding. Plant phenomics has been defined as the high-throughput, accurate acquisition and analysis of multi-dimensional phenotypes
during crop growing stages at the organism level, including the cell, tissue, organ, individual plant, plot, and field levels. With the rapid development of novel sensors, imaging technology,
and analysis methods, numerous infrastructure platforms have been developed for phenotyping.
Comparing Evolved Extractive Text Summary Scores of Bidirectional Encoder Rep...University of Maribor
Slides from:
11th International Conference on Electrical, Electronics and Computer Engineering (IcETRAN), Niš, 3-6 June 2024
Track: Artificial Intelligence
https://www.etran.rs/2024/en/home-english/
What is greenhouse gasses and how many gasses are there to affect the Earth.moosaasad1975
What are greenhouse gasses how they affect the earth and its environment what is the future of the environment and earth how the weather and the climate effects.
Toxic effects of heavy metals : Lead and Arsenicsanjana502982
Heavy metals are naturally occuring metallic chemical elements that have relatively high density, and are toxic at even low concentrations. All toxic metals are termed as heavy metals irrespective of their atomic mass and density, eg. arsenic, lead, mercury, cadmium, thallium, chromium, etc.
This presentation explores a brief idea about the structural and functional attributes of nucleotides, the structure and function of genetic materials along with the impact of UV rays and pH upon them.
Bridging the gap to facilitate selection and image analysis activities for land surface monitoring
1.
2. Objectives
Develop a catalogue that will allow users to discover and access data, open-source
software, public Application Programming Interfaces (APIs) and interactive processing
services.
Optimise and industrialize workflows to allow the largest degree of reusability of data as
possible.
Implement an end-user web common interactive processing service based on notebook and
datacube technologies allowing new users to easily have open access to HPC capacities
and develop new algorithms.
Improve the FAIRisation of satellite and environmental datasets and preserve FAIR
(Findable Accessible Interoperable Reusable) datasets into a Remote Data Access (RDA)
certified repository.
Develop new data pre-processing models coupled with HPC capabilities, by building a
new and innovative on-the-fly computing service for smart processing of data, addressing the
problem of efficient filtering and on-the-fly first diagnostics on incoming in-situ data.
Deploy data post-processing methods as a service for several end-users, including
scientific communities, public authorities, private entities and citizen scientists.
19/02/2021 2
Consortium of 13 committed partners from 5 different EU countries, led by CINES (French national
Computing Centre for Higher Education)
3. Webinar series on PHIDIAS Use Cases
19/02/2021 WEBINAR: Bridging the gap to facilitate selection and image analysis activities for land surface monitoring 3
All recorded Webinars are available on Phidias HPC website under the Webinar page
https://www.phidias-hpc.eu/events/webinars/
4. 19/02/2021 4
Webinar agenda
15:00 - 15:05 Introduction
Francesco Osimanti, Trust-IT Services
15:05 - 15:15 Challenges and barriers to the availability of on-demand
processing services for the land surface community
Jean Christophe Desconnets, IRD
15:15 - 15:25 From processing chains prototyping to HPC-based on-demand
services: the example of Soil Moisture Service
Cyprien Alexandre, IRD
15:25 - 15:30 Q&A Session
15:30 15:30 Online and on-demand processing of satellite images: Web
Processing Service, the missing link in the chain
Gérald Fenoy, Geolabs
15:40 15:50 Notebook and Datacube: an original approach for exploiting and
analysing environmental big data within HPC/HPDA environments
Dorian Ginane, Geomatys
15:50 - 16:00 Q&A and Closing Remarks
WEBINAR: Bridging the gap to facilitate selection and image analysis activities for land surface monitoring
6. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
Challenges and barriers to the
availability of on-demand processing
services for the land surface community
Jean-Christophe Desconnets, IRD, WP5 leader
Webinar | February 18, 2021
7. Barriers to use Earth Observation images
7
Accessing relevant images
Having the skills
Finding support
Getting appropriate equipments
Researchers Land managers
Government
agencies
End-users
8. A very large panel and amount of images
8
Increasing diversity of supply
(spectral, spatial, temporal
resolution)
Need for another paradigm :
From « bring the data to the
user » to « bring the user to the
data »
9. Increasing interoperable catalogs
Catalog GEOSUD of VHR images
SPOT6, SPOT7 : findable,
accessible and reusable through
standard web services and
comprehensive metadata
14 000 raw images
9
Catalog PEPS of Sentinel 1 &
Sentinel 2 images, findable
accessible and reusable
>> 10 Millions of raw images
10. Technical and scientific skills
Large storage and processing resources
Ensure the reproductibility of data processing
10
But … produce new knowledge with EO
images requires
11. Implementing scientific processing chains
Coming from scientific remote sensing community
THEIA Scientific Expertise Center (SEC) working around ten themes :
Agriculture, Forest, Urban, Coastline, Health, Water…
Outputs : scientifically validated processing chains for environmental monitoring
11
Land cover IOTA Chain Soil moisture with VR spatial resolution
12. Bridging the gap between academic production
and end users usages
Remote sensing
specialist
Scientific data
processing
End-users
Accessible, scalable
and reproducible data
processing
TRL 3 - 6
TRL 6 - 7
TRL : Technical Readness Level
13. WP5 objectives and users scenarios
Propose architecture, tools and
data models to provide
systematic or on-demand map
production accessible for data
scientist, researcher or land manager
13
Bridging the gap for facilitate selection and image
analysis activities for land surface monitoring
Three THEIA processing chains
Soil Moisture at Plot scale
Images processing with artificial
intelligence: application to land cover
mapping and super-resolution
Very high resolution land use
mapping (Moringa)
14. Bring data and HPC resource to user …
14
→ Define models, protocol and tools to deploy and
link end-users analysing environment and execution environment on HPC center
Data processing
chains
Data processing
chains
WPS server
WPS server
Image and product
Discovery
Image and product
Discovery
HPC/HPDA
Center
HPC/HPDA
Center
Analysing environment
Analysing environment
16. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
From processing chains
prototyping to HPC-based on-
demand services, example of
Soil Moisture Service
Webinar | 18 February 2021, 15:00 CET
Cyprien Alexandre, research engineer (IRD)
17. Introduction
17 18.02.2021 PHIDIAS Webinar | 18.02.2021 |
How to allow a simplified access to high performance computing for researchers and non-
researchers?
How to transfer a processing chain developed for research to a “non-code-friendly” public?
What are the engineering issues for developing such as processing chain?
How to allow access to HPC capacities “for all”?
18. Soil Moisture Service (SMS)
CES Soil humidity at high spatial resolution
Nicolas Baghdadi
Loïc Lozac’h
https://gitlab.irstea.fr/loic.lozach/soilmoistureservices
Model inversion to retrieve soil moisture value from
Sentinel1 SAR images at the plot scale.
Sentinel2 NDVI are used to adjust the impact of vegetation
on moisture estimation
18 18.02.2021 PHIDIAS Webinar | 18.02.2021 |
Sentinel-1
Sentinel-1
Sentinel-2
Sentinel-2 DEM
DEM Plots
Plots
Soil Moisture Map
Soil Moisture Map
19. Soil Moisture Service (SMS)
Input data from different data provider
Sentinel-1 images CNES (PEPS)
Sentinel-2 images THEIA or PEPS
SRTM ESA
Official inventory of plots France, users
vector files
Output data
Raster and csv
Vector
Python, OTB, TensorFlow (Neuronal
network)
19 18.02.2021 PHIDIAS Webinar | 18.02.2021 |
Download
package
Sentinel-2 (freq 6 days) Sentinel-1 (freq 6 days)
Calibration
package
S2 monthly
synthesis
Agricultural Plot
Vector
{sigma0VV,incidence}
TS
DEM
Pre-process
package
NDVI Time Synthesis
(15 days)
Agric Plot Index
Raster
Slope
Model Inversion Package
Soil Moisture
Time Serie
Frequency
Systematic
products: 6 days
On-demand
products
20. Containerisation
What?
Application encapsulation method
Complete environment and all dependencies
Why in HPC?
Easily deployable, at massive scale
Can be easily maintained
No compatibility issues between hosted applications
Two main solutions Docker vs Singularity
20 18.02.2021 PHIDIAS Webinar | 18.02.2021 |
- Largely used for common usage
- Difficulty for partition/manage access rights
between users
- Developed for HPC, adapted security level
- Singularity can run Docker images
- Open source, BSD licensed, free
21. Porting to HPC
Code adaptation to fit security rules
Ex: no downloads allowed on
computing nodes
France mainland:
S2: 3 To /15d
S1: 316 Go /15d
SRTM: 575 Go
21 18.02.2021 PHIDIAS Webinar | 18.02.2021 |
Data storing
• Size limitation
• Duplicated data
On-demand download
• Execution of SMS container on front machine
• Not the fastest
Virtual file system (iRods)
• Direct link with data providers
• Use of metadata link
• Faster and secured
Complexe organisation
Specific data access protocol
Storage
Access rights administration
Data migration etc…
22. Computing capacities access
22 18.02.2021 PHIDIAS Webinar | 18.02.2021 |
Soil Moisture Service
Soil Moisture Service
Heterogeneous community
Non comfortable with code
Comfortable with code and want to go further in the product
analysis
23. Computing capacities access
Two profiles and two interfaces
Standard web carto interface with virtual
products to compute (GEOSUD portal)
Access Jupyter Notebook interface to run
OTB commands on proposed products
23 18.02.2021 PHIDIAS Webinar | 18.02.2021 |
Dorian Ginane - GEOMATYS
24. Soil Moisture Service
Soil Moisture Service
Computing capacities access
ZOO-Kernel as processing orchestrator
Communication between web server and HPC’s front
Web Processing Service (WPS Open Geospatial
Consortium)
24 18.02.2021 PHIDIAS Webinar | 18.02.2021 |
WPS Server
WPS Server
WPS
WPS
SSH
SSH
Gérald FENOY - GeoLabs
Soil Moisture Service
Soil Moisture Service
25. Conclusion
Making HPC capacities accessible to a wide public implies taking into account several parameters…
…at the processing chain level
Code adaptation
Containerisation
…at the HPC architecture level
Security
Data storage and transfer
…for access to processing
All these development have a common thread : FAIR
Findable, Accessible, Interoperable, Reusable
Use of standardised applications (Python, Singularity …)
Use of standardised protocols (WPS - Open Geospatial Consortium)
Link with WP3 in charge of metadata
SMS processing chain is the first chain to be included in Phidias, more to come.
25 18.02.2021 PHIDIAS Webinar | 18.02.2021 |
26. Thank-you
C. Alexandre, research engineer, UMR Espace-Dev (IRD)
cyprien.alexandre@ird.fr
13.02.2020 PHIDIAS Webinar | 13.02.2020 | www.phidias-hpc.eu | @PhidiasHpc
26
27. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
Online and on-demand
processing of satellite images:
Web Processing Service, the
missing link in the chain
Webinar | 18 February 2021, 15:00 CET
Gérald Fenoy, GeoLabs CEO
28. Schedule
Introduction
Web Processing Service
OpenAPI
OGC and OpenAPI
The ZOO-Project implementation
Overview
OGC API – Processes
Remote service execution
18.02.2021 PHIDIAS Webinar | 18.02.2021 | https://www.phidias-hpc.eu/ | @PhidiasHpc
29
29. Introduction
18.02.2021 PHIDIAS Webinar | 18.02.2021 | https://www.phidias-hpc.eu/ | @PhidiasHpc
30
XML / KVP
Data diffusion Application diffusion
HPC
CSW
WMS WFS WCS
WPS Server
WPS Server
Execute asynchronous services running on HPC and available as Singularity containers
30. Web Processing Service
18.02.2021 PHIDIAS Webinar | 18.02.2021 | https://www.phidias-hpc.eu/ | @PhidiasHpc
31
Locked-in applications
Client and Server application
strongly linked together
The Open Geospatial Consotium published the Web Processing Service standard:
WPS provides generic interfaces
Multiple servers can be used
Multiple clients can be implemented
(Server / Desktop / Mobile)
Automatic capability addition
Reproduceability …
…
31. Web Processing Service
18.02.2021 PHIDIAS Webinar | 18.02.2021 | https://www.phidias-hpc.eu/ | @PhidiasHpc
32
What is available on the servers What are the parameters for a service
GetCapabilties DescribeProcess
Available services List
Detailled description of service
including parameters
Kebab menu
Options:
Meat: Beef, Porc, Chiken
Vegetables: Salad leaves, Tomatoes, ..
Type of bread: thin, normal, none
Run the service with
Desired parameters
Execute Sync Async Execute
Execution Result as
Response / Raw / Stored
Place your order
Result:
. plate
. storage for delivery,
or take away
32. OpenAPI
18.02.2021 PHIDIAS Webinar | 18.02.2021 | https://www.phidias-hpc.eu/ | @PhidiasHpc
33
The Linux Foundation sponsored the OpenAPI initiative
Swagger specification 1.0 in 2011
Swagger specification 2.0 in 2014
OpenAPI specification (OAS) 3.0.0 in 2017
OAS offers a simple format (JSON/YAML) for writing REST contract
JSON
33. OGC and OpenAPI
18.02.2021 PHIDIAS Webinar | 18.02.2021 | https://www.phidias-hpc.eu/ | @PhidiasHpc
34
Starting in 2016, the OGC recognized that their actual
Web Services standards were Web API
34. The ZOO-Project implementation
18.02.2021 PHIDIAS Webinar | 18.02.2021 | https://www.phidias-hpc.eu/ | @PhidiasHpc
35
WPS 1.0.0 / 2.0
OGC API - Processes
ZOO-API
ZOO-Services
HPC
service
ZOO-Client
OrfeoToolbox
SAGA-GIS
MapServer
Highly flexible
build new services
upon existing ones.
Open Source (MIT/X11 License) server implementation
of the OGC WPS standard
36. Remote service Execution
18.02.2021 PHIDIAS Webinar | 18.02.2021 | https://www.phidias-hpc.eu/ | @PhidiasHpc
37
GeoSUD Project (WPS 2.0)
Security management
Use asynchronous callbacks invocations for
execution history recording
Automatic publication of execution history,
inputs/utputs publication as OGC Web Services
Use SSH and slurm for remote execution
schedulling
Rely on Mail sent from the HPC server
37. From monolithic to container
18.02.2021 PHIDIAS Webinar | 18.02.2021 | https://www.phidias-hpc.eu/ | @PhidiasHpc
38
Request Handler
Process Manager
…
WPS 1.0.0 / 2.0
OGC API - Processes
Produce
Route
Consume
Exchange
Queue
PG DB redis
38. Conclusion
18.02.2021 PHIDIAS Webinar | 18.02.2021 | https://www.phidias-hpc.eu/ | @PhidiasHpc
39
WPS facilitates the diffusion of scientists knowledge by defining a protocol to discover,
access to, interact with and reproduce service execution.
The addition of the OpenAPI support, through the OGC API – Processes, eases the diffusion
of the knowledge by providing out-of-box tools to document, discover, access to, interact
with, and reproduce service execution.
Reduce additional developments required to be integrated in: Notebooks and Web client
applications.
Ensure scalability by using advanced message queuing protocol
Ease of maintenance, and deployment by using containers
History management to produce provenance metadata of the resulting product
40. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
DataCube and Notebook : another
approach for exploiting images
Dorian Ginane, Project's manager, Geomatys
Webinar | February 18, 2021
41. Virtual products for bridging the gap for the
user
18.02.2021 PHIDIAS Webinar | 18.02.2021 | https://www.phidias-hpc.eu/ | @PhidiasHpc 42
Often, user does not need to access to a collection
of datasets but to processed data for a specific
spatio-temporal area.
Due to the volume of raw data accessible. All the
raw data can not be pre-processed.
The idea : Shows to user synthetic products (for
example a single soil moisture product) and
process raw data on the fly when needed
(Analysis Ready Data approach
https://ceos.org/ard/)
42. Virtual products
43
Source : Geomatys
A virtual product is an arbitrary
aggregation of raw data files that can
be manipulated as a unique cube of
data. (for example, an optical layer
mixing Sentinel2 and another optical
satellite…)
Examind DataCube helps user to
directly obtain and manipulate a
subsample of the virtual product
https://datacube.examind.com
18.02.2021 PHIDIAS Webinar | 18.02.2021 | https://www.phidias-hpc.eu/ | @PhidiasHpc
43. Examind Datacube : How it works
44
The solution catalogues and indexes all
the spatio-temporal and observations’
dimensions for all the datasets available
on the Data Lake :
time range,
Geographic extent,
Resolutions
Unit of measure
Min, max values
… Source : Geomatys
18.02.2021 PHIDIAS Webinar | 18.02.2021 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Data Lake
Request
Tensor
44. 45
DataCube provides a cube of
data corresponing at what users
or software have asked for,
independently of the raw data
structure :
Numpy or Xarray structure, Java
structure, Dask Array…
With the wished resolution and
coordinated reference system
Export format : COG…
Examind Datacube : How it works
18.02.2021 PHIDIAS Webinar | 18.02.2021 | https://www.phidias-hpc.eu/ | @PhidiasHpc
45. 46
The virtual products can be used on different
way
On a classic catalog
Through a Web Map Service
as a classical spatio-temporal layer
Trough a Notebook
18.02.2021 PHIDIAS Webinar | 18.02.2021 | https://www.phidias-hpc.eu/ | @PhidiasHpc
46. 47
Examind Datacube, Notebook and HPC
infrastructure
Notebook are an easy way for
DataScientists to collaborate for
the conception of new algorithm
or data analysis
DataCube provides an easy way
for asking data
Process can be executed on the
HPC cluster using WPS
Standards
18.02.2021 PHIDIAS Webinar | 18.02.2021 | https://www.phidias-hpc.eu/ | @PhidiasHpc