SlideShare a Scribd company logo
1 of 15
Download to read offline
ExtremeEarth
From Copernicus Big Data
to Extreme Earth Analytics
This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement No 825258.
December 9, 2021
ExtremeEarth Workshop
Manolis Koubarakis (Coordinator, UoA)
ExtremeEarth – Overview and Achievements
3
ExtremeEarth Main Objective
• The main objective of ExtremeEarth is to go beyond the state-of-the-art and develop Artificial
Intelligence and Big Data techniques and technologies that scale to the PBs of big Copernicus data,
information and knowledge, and apply these technologies in two of the ESA TEPs: Food Security and
Polar.
• The technologies to be developed will extend the European Hopsworks data intensive AI platform of
partner Logical Clocks to offer unprecedented scalability to extreme data volumes and scale-out
distributed deep learning for Copernicus data.
• The extended Hopsworks platform will run on CREODIAS and will be available as open source to enable
its adoption by the strong European Earth Observation downstream services industry.
• The technologies to be developed will also extend the linked geospatial data systems GeoTriples,
JedAI, Strabon and SemaGrow pioneered by project partners UoA and NCSR in the past, so that they
scale to the extreme volumes of Copernicus data.
4
ExtremeEarth Consortium
1. National and Kapodistrian University of Athens (UoA)
2. VISTA
3. The Arctic University of Norway (UiT)
4. University of Trento (UNITN)
5. The Royal Institute of Technology (KTH)
6. National Center for Scientific Research – Demokritos (NCSR-D)
7. German Aerospace Center (DLR)
8. Polar View
9. Norwegian Meteorological Institute (METNO)
10.LogicalClocks
11.British Antarctic Survey (UKRI-BAS)
5
Key Technical Achievements
• The Food Security use case: we have produced irrigation
recommendations for farmers in agricultural areas in the Danube and
Duero catchments allowing a new level of detail for wide-scale irrigation
support.
6
Key Technical Achievements (cont’d)
• The Polar use case: we have produced high resolution sea-ice charts from massive volumes of
heterogeneous Copernicus data.
7
Key Technical Achievements (cont’d)
• We have developed scalable deep learning techniques for Copernicus big data.
• In more detail:
o Developed an LSTM deep neural network architecture for crop type mapping from Sentinel 2
data. This has been implemented on Hopsworks and it is being used in the Food Security use case.
o Developed multiple deep neural network architectures (LDA, CNN, variational auto-encoders,
GAN) for sea-ice classification from Sentinel 1 data. Most of these have been implemented on
Hopsworks and are being used in the Polar use case.
8
Key Technical Achievements (cont’d)
• We have developed very large training datasets for deep learning architectures targeting the
classification of Sentinel images.
• In more detail:
o Developed a training dataset consisting of ~1M pixels of 16 Sentinel 2 images located in Austria
where each pixel is labelled with one of 13 crop types. This dataset was developed using existing
crop type maps and Sentinel 2 data and it was used to train the LSTM network for the Food Security
use case.
o This dataset and the following ones are available publicly: http://earthanalytics.eu/datasets.html and on
Zenodo.
9
Key Technical Achievements (cont’d)
o Developed a training dataset consisting of 63048 patches of 30 Sentinel 1 images located in the
European Arctic where each patch is labelled with one of 6 ice types. This dataset was developed by
expert photo-interpretation and it was used to train three of the CNN networks for the Polar use case.
10
Key Technical Achievements (cont’d)
o Developed two training datasets based on 24 Sentinel 1 images located in the Belgica Bank of the
Greenland Sea. The first dataset consists of ~62M patches of size 4x4 pixels where each patch is
labelled with one of 11 topics using Latent Dirichlet Allocation. The second dataset consists of 153,600
patches of size 256x256 pixels classified into 8 classes using active learning based on support
vector machines with relevance feedback.
11
Key Technical Objectives and Progress (cont’d)
o Developed a training dataset consisting of 18,000 patches of 12 Sentinel 1 images located in the
Danmarkshavn (East coast of Greenland). In version 1, each patch is labelled with one of 2 classes
(ice or water), while the updated version 2 contains additional information on sea ice concentrations,
ice type and form. These datasets were developed by expert photo-interpretation and used to train
some of the CNN networks for the Polar use case.
12
Key Technical Achievements (cont’d)
• We have developed big linked geospatial data systems that scale to big Copernicus data,
information and knowledge.
• In more detail:
o Developed the system GeoTriples-Spark for transforming big geospatial data from their legacy formats
into RDF. GeoTriples-Spark can transform 2TBs of geospatial data into RDF in 50 minutes.
o Developed the system JedAI-spatial for interlinking big linked geospatial data. JedAI-spatial has been
tested with ~100 GB of geospatial data (~400M geometries) and has been shown to scale almost
linearly.
o Developed the system Strabo 2 for querying big linked geospatial data using the OGC standard
GeoSPARQL. Strabo 2 can process queries of the Geographica benchmark for a dataset of 450GB
with an average execution time of 98 seconds using 128 worker nodes in Hopsworks.
o Developed the version 3 of system Semagrow for federating big linked geospatial data sources.
Semagrow can process queries from the Food Security use case over 4.5GB of data in <5 seconds
and can process queries from the Geographica benchmark in federations with 100 endpoints.
13
Key Technical Achievements (cont’d)
• We have integrated the AI and Big Data technologies presented above in the Hopsworks data
platform and deployed them in CREODIAS and the two TEPs.
• In more detail:
o We developed the ExtremeEarth platform software architecture using Hopsworks.
o The deep learning architectures for the two use cases have been implemented on the platform.
o GeoTriples-Spark, JedAI-spatial, Strabo 2 and Semagrow have been implemented on the platform.
14
Key Technical Achievements (cont’d)
• We have extended the Hopsworks data intensive AI platform with new deep learning functionalities
for Copernicus data.
• Progress:
o We extended the file system HopsFS and the resource scheduler HopsYARN of Hopsworks for
managing EO data.
o We extended the Hopsworks metadata and security model with APIs for EO metadata.
o We carried out the following extensions to the Hopsworks platform that enable large-scale distributed
data processing and building ML/DL pipelines: EO Data Management, Feature Store, Experiment
API, Maggy framework for asynchronous parallel execution of trials for machine learning
experiments, distribution oblivious training functions, Maggy support for hyperparameter
tuning and parallel ablation studies.
Thank you!
Visit our Web site: http://earthanalytics.eu/
Follow us on Twitter:
@ExtremeEarth_EU @mkoubarakis

More Related Content

What's hot

Artificial Intelligence and Big Data Technologies for Copernicus Data: the Ex...
Artificial Intelligence and Big Data Technologies for Copernicus Data: the Ex...Artificial Intelligence and Big Data Technologies for Copernicus Data: the Ex...
Artificial Intelligence and Big Data Technologies for Copernicus Data: the Ex...ExtremeEarth
 
ExtremeEarth Data Science Pipeline for Linked Earth Observation Data
ExtremeEarth Data Science Pipeline for Linked Earth Observation DataExtremeEarth Data Science Pipeline for Linked Earth Observation Data
ExtremeEarth Data Science Pipeline for Linked Earth Observation DataExtremeEarth
 
Geo Analytics Canada Overview - May 2020
Geo Analytics Canada Overview - May 2020Geo Analytics Canada Overview - May 2020
Geo Analytics Canada Overview - May 2020GEO Analytics Canada
 
STAC, ZARR, COG, K8S and Data Cubes: The brave new world of satellite EO anal...
STAC, ZARR, COG, K8S and Data Cubes: The brave new world of satellite EO anal...STAC, ZARR, COG, K8S and Data Cubes: The brave new world of satellite EO anal...
STAC, ZARR, COG, K8S and Data Cubes: The brave new world of satellite EO anal...GEO Analytics Canada
 
BigDataEurope 1st SC5 Workshop, Project Teleios & LEO, by M. Koubarakis, Univ...
BigDataEurope 1st SC5 Workshop, Project Teleios & LEO, by M. Koubarakis, Univ...BigDataEurope 1st SC5 Workshop, Project Teleios & LEO, by M. Koubarakis, Univ...
BigDataEurope 1st SC5 Workshop, Project Teleios & LEO, by M. Koubarakis, Univ...BigData_Europe
 
Enabling Access to Big Geospatial Data with LocationTech and Apache projects
Enabling Access to Big Geospatial Data with LocationTech and Apache projectsEnabling Access to Big Geospatial Data with LocationTech and Apache projects
Enabling Access to Big Geospatial Data with LocationTech and Apache projectsRob Emanuele
 
NASA Advanced Computing Environment for Science & Engineering
NASA Advanced Computing Environment for Science & EngineeringNASA Advanced Computing Environment for Science & Engineering
NASA Advanced Computing Environment for Science & Engineeringinside-BigData.com
 
Processing Geospatial at Scale at LocationTech
Processing Geospatial at Scale at LocationTechProcessing Geospatial at Scale at LocationTech
Processing Geospatial at Scale at LocationTechRob Emanuele
 
Opening the Path to Technical Excellence
Opening the Path to Technical ExcellenceOpening the Path to Technical Excellence
Opening the Path to Technical ExcellenceNETWAYS
 
Processing Geospatial Data At Scale @locationtech
Processing Geospatial Data At Scale @locationtechProcessing Geospatial Data At Scale @locationtech
Processing Geospatial Data At Scale @locationtechRob Emanuele
 
GeoSpatially enabling your Spark and Accumulo clusters with LocationTech
GeoSpatially enabling your Spark and Accumulo clusters with LocationTechGeoSpatially enabling your Spark and Accumulo clusters with LocationTech
GeoSpatially enabling your Spark and Accumulo clusters with LocationTechRob Emanuele
 
Using the Data Cube vocabulary for Publishing Environmental Linked Data on la...
Using the Data Cube vocabulary for Publishing Environmental Linked Data on la...Using the Data Cube vocabulary for Publishing Environmental Linked Data on la...
Using the Data Cube vocabulary for Publishing Environmental Linked Data on la...Laurent Lefort
 
State of the Map US 2018: Analytic Support to Mapping Contributors
State of the Map US 2018: Analytic Support to Mapping ContributorsState of the Map US 2018: Analytic Support to Mapping Contributors
State of the Map US 2018: Analytic Support to Mapping Contributorsrlewis48
 
OGC Interfaces in Thematic Exploitation Platforms
OGC Interfaces in Thematic Exploitation PlatformsOGC Interfaces in Thematic Exploitation Platforms
OGC Interfaces in Thematic Exploitation Platformsterradue
 
Linked Sensor Data cube
Linked Sensor Data cubeLinked Sensor Data cube
Linked Sensor Data cubeLaurent Lefort
 
Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...
Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...
Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...Laurent Lefort
 
Analysis Ready Data workshop - OGC presentation
Analysis Ready Data workshop - OGC presentation Analysis Ready Data workshop - OGC presentation
Analysis Ready Data workshop - OGC presentation George Percivall
 

What's hot (20)

Artificial Intelligence and Big Data Technologies for Copernicus Data: the Ex...
Artificial Intelligence and Big Data Technologies for Copernicus Data: the Ex...Artificial Intelligence and Big Data Technologies for Copernicus Data: the Ex...
Artificial Intelligence and Big Data Technologies for Copernicus Data: the Ex...
 
ExtremeEarth Data Science Pipeline for Linked Earth Observation Data
ExtremeEarth Data Science Pipeline for Linked Earth Observation DataExtremeEarth Data Science Pipeline for Linked Earth Observation Data
ExtremeEarth Data Science Pipeline for Linked Earth Observation Data
 
Extreme earth overview
Extreme earth overviewExtreme earth overview
Extreme earth overview
 
Geo Analytics Canada Overview - May 2020
Geo Analytics Canada Overview - May 2020Geo Analytics Canada Overview - May 2020
Geo Analytics Canada Overview - May 2020
 
STAC, ZARR, COG, K8S and Data Cubes: The brave new world of satellite EO anal...
STAC, ZARR, COG, K8S and Data Cubes: The brave new world of satellite EO anal...STAC, ZARR, COG, K8S and Data Cubes: The brave new world of satellite EO anal...
STAC, ZARR, COG, K8S and Data Cubes: The brave new world of satellite EO anal...
 
BigDataEurope 1st SC5 Workshop, Project Teleios & LEO, by M. Koubarakis, Univ...
BigDataEurope 1st SC5 Workshop, Project Teleios & LEO, by M. Koubarakis, Univ...BigDataEurope 1st SC5 Workshop, Project Teleios & LEO, by M. Koubarakis, Univ...
BigDataEurope 1st SC5 Workshop, Project Teleios & LEO, by M. Koubarakis, Univ...
 
EOSC-hub & Geohazards TEP
EOSC-hub & Geohazards TEPEOSC-hub & Geohazards TEP
EOSC-hub & Geohazards TEP
 
Enabling Access to Big Geospatial Data with LocationTech and Apache projects
Enabling Access to Big Geospatial Data with LocationTech and Apache projectsEnabling Access to Big Geospatial Data with LocationTech and Apache projects
Enabling Access to Big Geospatial Data with LocationTech and Apache projects
 
NASA Advanced Computing Environment for Science & Engineering
NASA Advanced Computing Environment for Science & EngineeringNASA Advanced Computing Environment for Science & Engineering
NASA Advanced Computing Environment for Science & Engineering
 
Processing Geospatial at Scale at LocationTech
Processing Geospatial at Scale at LocationTechProcessing Geospatial at Scale at LocationTech
Processing Geospatial at Scale at LocationTech
 
Opening the Path to Technical Excellence
Opening the Path to Technical ExcellenceOpening the Path to Technical Excellence
Opening the Path to Technical Excellence
 
Processing Geospatial Data At Scale @locationtech
Processing Geospatial Data At Scale @locationtechProcessing Geospatial Data At Scale @locationtech
Processing Geospatial Data At Scale @locationtech
 
GeoSpatially enabling your Spark and Accumulo clusters with LocationTech
GeoSpatially enabling your Spark and Accumulo clusters with LocationTechGeoSpatially enabling your Spark and Accumulo clusters with LocationTech
GeoSpatially enabling your Spark and Accumulo clusters with LocationTech
 
Using the Data Cube vocabulary for Publishing Environmental Linked Data on la...
Using the Data Cube vocabulary for Publishing Environmental Linked Data on la...Using the Data Cube vocabulary for Publishing Environmental Linked Data on la...
Using the Data Cube vocabulary for Publishing Environmental Linked Data on la...
 
CLIM Program: Remote Sensing Workshop, The Earth System Grid Federation as a ...
CLIM Program: Remote Sensing Workshop, The Earth System Grid Federation as a ...CLIM Program: Remote Sensing Workshop, The Earth System Grid Federation as a ...
CLIM Program: Remote Sensing Workshop, The Earth System Grid Federation as a ...
 
State of the Map US 2018: Analytic Support to Mapping Contributors
State of the Map US 2018: Analytic Support to Mapping ContributorsState of the Map US 2018: Analytic Support to Mapping Contributors
State of the Map US 2018: Analytic Support to Mapping Contributors
 
OGC Interfaces in Thematic Exploitation Platforms
OGC Interfaces in Thematic Exploitation PlatformsOGC Interfaces in Thematic Exploitation Platforms
OGC Interfaces in Thematic Exploitation Platforms
 
Linked Sensor Data cube
Linked Sensor Data cubeLinked Sensor Data cube
Linked Sensor Data cube
 
Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...
Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...
Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...
 
Analysis Ready Data workshop - OGC presentation
Analysis Ready Data workshop - OGC presentation Analysis Ready Data workshop - OGC presentation
Analysis Ready Data workshop - OGC presentation
 

Similar to ExtremeEarth Open Workshop - Overview and Achievements

Scalable Deep Learning in ExtremeEarth-phiweek19
Scalable Deep Learning in ExtremeEarth-phiweek19Scalable Deep Learning in ExtremeEarth-phiweek19
Scalable Deep Learning in ExtremeEarth-phiweek19ExtremeEarth
 
Accelerating Research and Enterprise Solutions by Bridging HPC and AI
Accelerating Research and Enterprise Solutions by Bridging HPC and AIAccelerating Research and Enterprise Solutions by Bridging HPC and AI
Accelerating Research and Enterprise Solutions by Bridging HPC and AIinside-BigData.com
 
Dati satellitari e prodotti derivati in modalità open and free del programma ...
Dati satellitari e prodotti derivati in modalità open and free del programma ...Dati satellitari e prodotti derivati in modalità open and free del programma ...
Dati satellitari e prodotti derivati in modalità open and free del programma ...giovannibiallo
 
big_data_casestudies_2.ppt
big_data_casestudies_2.pptbig_data_casestudies_2.ppt
big_data_casestudies_2.pptvishal choudhary
 
NextGEOSS: The Next Generation European Data Hub and Cloud Platform for Earth...
NextGEOSS: The Next Generation European Data Hub and Cloud Platform for Earth...NextGEOSS: The Next Generation European Data Hub and Cloud Platform for Earth...
NextGEOSS: The Next Generation European Data Hub and Cloud Platform for Earth...Wolfgang Ksoll
 
Design phase kick-off event and Ceremony
Design phase kick-off event and CeremonyDesign phase kick-off event and Ceremony
Design phase kick-off event and CeremonyArchiver
 
Gergely Sipos (EGI): Exploiting scientific data in the international context ...
Gergely Sipos (EGI): Exploiting scientific data in the international context ...Gergely Sipos (EGI): Exploiting scientific data in the international context ...
Gergely Sipos (EGI): Exploiting scientific data in the international context ...Gergely Sipos
 
LCG project description
LCG project descriptionLCG project description
LCG project descriptionlouisponcet
 
DSD-INT 2015 - RSS Sentinel Toolbox - J. Manuel Delgado Blasco
DSD-INT 2015 - RSS Sentinel Toolbox - J. Manuel Delgado BlascoDSD-INT 2015 - RSS Sentinel Toolbox - J. Manuel Delgado Blasco
DSD-INT 2015 - RSS Sentinel Toolbox - J. Manuel Delgado BlascoDeltares
 
Big linked geospatial data tools in ExtremeEarth-phiweek19
Big linked geospatial data tools in ExtremeEarth-phiweek19Big linked geospatial data tools in ExtremeEarth-phiweek19
Big linked geospatial data tools in ExtremeEarth-phiweek19ExtremeEarth
 
GEO Analytics Canada Overview April 2020
GEO Analytics Canada Overview April 2020GEO Analytics Canada Overview April 2020
GEO Analytics Canada Overview April 2020GEO Analytics Canada
 
LPS19 ExtremeEarth Project
LPS19 ExtremeEarth ProjectLPS19 ExtremeEarth Project
LPS19 ExtremeEarth ProjectExtremeEarth
 
Hybrid Cloud storage deployment models: ARCHIVER presentation at the CS3 Work...
Hybrid Cloud storage deployment models: ARCHIVER presentation at the CS3 Work...Hybrid Cloud storage deployment models: ARCHIVER presentation at the CS3 Work...
Hybrid Cloud storage deployment models: ARCHIVER presentation at the CS3 Work...Archiver
 
How HPC and large-scale data analytics are transforming experimental science
How HPC and large-scale data analytics are transforming experimental scienceHow HPC and large-scale data analytics are transforming experimental science
How HPC and large-scale data analytics are transforming experimental scienceinside-BigData.com
 
Archiver at CS3 - Cloud Storage Synchronization and Sharing Services
Archiver at CS3 - Cloud Storage Synchronization and Sharing ServicesArchiver at CS3 - Cloud Storage Synchronization and Sharing Services
Archiver at CS3 - Cloud Storage Synchronization and Sharing ServicesArchiver
 
Data-intensive bioinformatics on HPC and Cloud
Data-intensive bioinformatics on HPC and CloudData-intensive bioinformatics on HPC and Cloud
Data-intensive bioinformatics on HPC and CloudOla Spjuth
 
Well identification and environmental management
Well identification and environmental managementWell identification and environmental management
Well identification and environmental managementJean-Michel Bergeon
 

Similar to ExtremeEarth Open Workshop - Overview and Achievements (20)

Scalable Deep Learning in ExtremeEarth-phiweek19
Scalable Deep Learning in ExtremeEarth-phiweek19Scalable Deep Learning in ExtremeEarth-phiweek19
Scalable Deep Learning in ExtremeEarth-phiweek19
 
Accelerating Research and Enterprise Solutions by Bridging HPC and AI
Accelerating Research and Enterprise Solutions by Bridging HPC and AIAccelerating Research and Enterprise Solutions by Bridging HPC and AI
Accelerating Research and Enterprise Solutions by Bridging HPC and AI
 
Dati satellitari e prodotti derivati in modalità open and free del programma ...
Dati satellitari e prodotti derivati in modalità open and free del programma ...Dati satellitari e prodotti derivati in modalità open and free del programma ...
Dati satellitari e prodotti derivati in modalità open and free del programma ...
 
big_data_casestudies_2.ppt
big_data_casestudies_2.pptbig_data_casestudies_2.ppt
big_data_casestudies_2.ppt
 
OpenPOWER System Marconi100
OpenPOWER System Marconi100OpenPOWER System Marconi100
OpenPOWER System Marconi100
 
NextGEOSS: The Next Generation European Data Hub and Cloud Platform for Earth...
NextGEOSS: The Next Generation European Data Hub and Cloud Platform for Earth...NextGEOSS: The Next Generation European Data Hub and Cloud Platform for Earth...
NextGEOSS: The Next Generation European Data Hub and Cloud Platform for Earth...
 
Design phase kick-off event and Ceremony
Design phase kick-off event and CeremonyDesign phase kick-off event and Ceremony
Design phase kick-off event and Ceremony
 
Gergely Sipos (EGI): Exploiting scientific data in the international context ...
Gergely Sipos (EGI): Exploiting scientific data in the international context ...Gergely Sipos (EGI): Exploiting scientific data in the international context ...
Gergely Sipos (EGI): Exploiting scientific data in the international context ...
 
Cool Tools Esri ArcGIS
Cool Tools Esri ArcGISCool Tools Esri ArcGIS
Cool Tools Esri ArcGIS
 
LCG project description
LCG project descriptionLCG project description
LCG project description
 
DSD-INT 2015 - RSS Sentinel Toolbox - J. Manuel Delgado Blasco
DSD-INT 2015 - RSS Sentinel Toolbox - J. Manuel Delgado BlascoDSD-INT 2015 - RSS Sentinel Toolbox - J. Manuel Delgado Blasco
DSD-INT 2015 - RSS Sentinel Toolbox - J. Manuel Delgado Blasco
 
Big linked geospatial data tools in ExtremeEarth-phiweek19
Big linked geospatial data tools in ExtremeEarth-phiweek19Big linked geospatial data tools in ExtremeEarth-phiweek19
Big linked geospatial data tools in ExtremeEarth-phiweek19
 
EPCC MSc industry projects
EPCC MSc industry projectsEPCC MSc industry projects
EPCC MSc industry projects
 
GEO Analytics Canada Overview April 2020
GEO Analytics Canada Overview April 2020GEO Analytics Canada Overview April 2020
GEO Analytics Canada Overview April 2020
 
LPS19 ExtremeEarth Project
LPS19 ExtremeEarth ProjectLPS19 ExtremeEarth Project
LPS19 ExtremeEarth Project
 
Hybrid Cloud storage deployment models: ARCHIVER presentation at the CS3 Work...
Hybrid Cloud storage deployment models: ARCHIVER presentation at the CS3 Work...Hybrid Cloud storage deployment models: ARCHIVER presentation at the CS3 Work...
Hybrid Cloud storage deployment models: ARCHIVER presentation at the CS3 Work...
 
How HPC and large-scale data analytics are transforming experimental science
How HPC and large-scale data analytics are transforming experimental scienceHow HPC and large-scale data analytics are transforming experimental science
How HPC and large-scale data analytics are transforming experimental science
 
Archiver at CS3 - Cloud Storage Synchronization and Sharing Services
Archiver at CS3 - Cloud Storage Synchronization and Sharing ServicesArchiver at CS3 - Cloud Storage Synchronization and Sharing Services
Archiver at CS3 - Cloud Storage Synchronization and Sharing Services
 
Data-intensive bioinformatics on HPC and Cloud
Data-intensive bioinformatics on HPC and CloudData-intensive bioinformatics on HPC and Cloud
Data-intensive bioinformatics on HPC and Cloud
 
Well identification and environmental management
Well identification and environmental managementWell identification and environmental management
Well identification and environmental management
 

More from ExtremeEarth

ExtremeEarth Open Workshop - Introduction
ExtremeEarth Open Workshop - IntroductionExtremeEarth Open Workshop - Introduction
ExtremeEarth Open Workshop - IntroductionExtremeEarth
 
Snow Monitoring for Water Availability and Irrigation
Snow Monitoring for Water Availability and IrrigationSnow Monitoring for Water Availability and Irrigation
Snow Monitoring for Water Availability and IrrigationExtremeEarth
 
Polar Use Case in ExtremeEarth-phiweek19
Polar Use Case in ExtremeEarth-phiweek19Polar Use Case in ExtremeEarth-phiweek19
Polar Use Case in ExtremeEarth-phiweek19ExtremeEarth
 
The ExtremeEarth infrastructure-phiweek19
The ExtremeEarth infrastructure-phiweek19The ExtremeEarth infrastructure-phiweek19
The ExtremeEarth infrastructure-phiweek19ExtremeEarth
 
Food security use case in ExtremeEarth-phiweek19
Food security use case in ExtremeEarth-phiweek19Food security use case in ExtremeEarth-phiweek19
Food security use case in ExtremeEarth-phiweek19ExtremeEarth
 
Copernicus and AI workshop 2020
Copernicus and AI workshop 2020Copernicus and AI workshop 2020
Copernicus and AI workshop 2020ExtremeEarth
 

More from ExtremeEarth (6)

ExtremeEarth Open Workshop - Introduction
ExtremeEarth Open Workshop - IntroductionExtremeEarth Open Workshop - Introduction
ExtremeEarth Open Workshop - Introduction
 
Snow Monitoring for Water Availability and Irrigation
Snow Monitoring for Water Availability and IrrigationSnow Monitoring for Water Availability and Irrigation
Snow Monitoring for Water Availability and Irrigation
 
Polar Use Case in ExtremeEarth-phiweek19
Polar Use Case in ExtremeEarth-phiweek19Polar Use Case in ExtremeEarth-phiweek19
Polar Use Case in ExtremeEarth-phiweek19
 
The ExtremeEarth infrastructure-phiweek19
The ExtremeEarth infrastructure-phiweek19The ExtremeEarth infrastructure-phiweek19
The ExtremeEarth infrastructure-phiweek19
 
Food security use case in ExtremeEarth-phiweek19
Food security use case in ExtremeEarth-phiweek19Food security use case in ExtremeEarth-phiweek19
Food security use case in ExtremeEarth-phiweek19
 
Copernicus and AI workshop 2020
Copernicus and AI workshop 2020Copernicus and AI workshop 2020
Copernicus and AI workshop 2020
 

Recently uploaded

History of Morena Moshoeshoe birth death
History of Morena Moshoeshoe birth deathHistory of Morena Moshoeshoe birth death
History of Morena Moshoeshoe birth deathphntsoaki
 
Using AI to boost productivity for developers
Using AI to boost productivity for developersUsing AI to boost productivity for developers
Using AI to boost productivity for developersTeri Eyenike
 
STM valmiusseminaari 26-04-2024 PUUMALAINEN Ajankohtaista kansainvälisestä yh...
STM valmiusseminaari 26-04-2024 PUUMALAINEN Ajankohtaista kansainvälisestä yh...STM valmiusseminaari 26-04-2024 PUUMALAINEN Ajankohtaista kansainvälisestä yh...
STM valmiusseminaari 26-04-2024 PUUMALAINEN Ajankohtaista kansainvälisestä yh...Sosiaali- ja terveysministeriö / yleiset
 
Introduction to Artificial intelligence.
Introduction to Artificial intelligence.Introduction to Artificial intelligence.
Introduction to Artificial intelligence.thamaeteboho94
 
BIG DEVELOPMENTS IN LESOTHO(DAMS & MINES
BIG DEVELOPMENTS IN LESOTHO(DAMS & MINESBIG DEVELOPMENTS IN LESOTHO(DAMS & MINES
BIG DEVELOPMENTS IN LESOTHO(DAMS & MINESfuthumetsaneliswa
 
Molecular docking- Laxman I. Nimbale.pptx
Molecular docking- Laxman  I. Nimbale.pptxMolecular docking- Laxman  I. Nimbale.pptx
Molecular docking- Laxman I. Nimbale.pptxluckynimbale8892
 
Ready Set Go Children Sermon about Mark 16:15-20
Ready Set Go Children Sermon about Mark 16:15-20Ready Set Go Children Sermon about Mark 16:15-20
Ready Set Go Children Sermon about Mark 16:15-20rejz122017
 
Proofreading- Basics to Artificial Intelligence Integration - Presentation:Sl...
Proofreading- Basics to Artificial Intelligence Integration - Presentation:Sl...Proofreading- Basics to Artificial Intelligence Integration - Presentation:Sl...
Proofreading- Basics to Artificial Intelligence Integration - Presentation:Sl...David Celestin
 
"I hear you": Moving beyond empathy in UXR
"I hear you": Moving beyond empathy in UXR"I hear you": Moving beyond empathy in UXR
"I hear you": Moving beyond empathy in UXRMegan Campos
 
Databricks Machine Learning Associate Exam Dumps 2024.pdf
Databricks Machine Learning Associate Exam Dumps 2024.pdfDatabricks Machine Learning Associate Exam Dumps 2024.pdf
Databricks Machine Learning Associate Exam Dumps 2024.pdfSkillCertProExams
 
Modernizing The Transport System:Dhaka Metro Rail
Modernizing The Transport System:Dhaka Metro RailModernizing The Transport System:Dhaka Metro Rail
Modernizing The Transport System:Dhaka Metro RailKhanMdReahnAftab
 
2024 mega trends for the digital workplace - FINAL.pdf
2024 mega trends for the digital workplace - FINAL.pdf2024 mega trends for the digital workplace - FINAL.pdf
2024 mega trends for the digital workplace - FINAL.pdfNancy Goebel
 
2024-05-15-Surat Meetup-Hyperautomation.pptx
2024-05-15-Surat Meetup-Hyperautomation.pptx2024-05-15-Surat Meetup-Hyperautomation.pptx
2024-05-15-Surat Meetup-Hyperautomation.pptxnitishjain2015
 
BEAUTIFUL PLACES TO VISIT IN LESOTHO.pptx
BEAUTIFUL PLACES TO VISIT IN LESOTHO.pptxBEAUTIFUL PLACES TO VISIT IN LESOTHO.pptx
BEAUTIFUL PLACES TO VISIT IN LESOTHO.pptxthusosetemere
 
The Concession of Asaba International Airport: Balancing Politics and Policy ...
The Concession of Asaba International Airport: Balancing Politics and Policy ...The Concession of Asaba International Airport: Balancing Politics and Policy ...
The Concession of Asaba International Airport: Balancing Politics and Policy ...Kayode Fayemi
 
globalisation project report displayed overview
globalisation project report displayed overviewglobalisation project report displayed overview
globalisation project report displayed overviewasadzafar8520
 
LITTLE ABOUT LESOTHO FROM THE TIME MOSHOESHOE THE FIRST WAS BORN
LITTLE ABOUT LESOTHO FROM THE TIME MOSHOESHOE THE FIRST WAS BORNLITTLE ABOUT LESOTHO FROM THE TIME MOSHOESHOE THE FIRST WAS BORN
LITTLE ABOUT LESOTHO FROM THE TIME MOSHOESHOE THE FIRST WAS BORNtntlai16
 
SOLID WASTE MANAGEMENT SYSTEM OF FENI PAURASHAVA, BANGLADESH.pdf
SOLID WASTE MANAGEMENT SYSTEM OF FENI PAURASHAVA, BANGLADESH.pdfSOLID WASTE MANAGEMENT SYSTEM OF FENI PAURASHAVA, BANGLADESH.pdf
SOLID WASTE MANAGEMENT SYSTEM OF FENI PAURASHAVA, BANGLADESH.pdfMahamudul Hasan
 

Recently uploaded (20)

History of Morena Moshoeshoe birth death
History of Morena Moshoeshoe birth deathHistory of Morena Moshoeshoe birth death
History of Morena Moshoeshoe birth death
 
Using AI to boost productivity for developers
Using AI to boost productivity for developersUsing AI to boost productivity for developers
Using AI to boost productivity for developers
 
STM valmiusseminaari 26-04-2024 PUUMALAINEN Ajankohtaista kansainvälisestä yh...
STM valmiusseminaari 26-04-2024 PUUMALAINEN Ajankohtaista kansainvälisestä yh...STM valmiusseminaari 26-04-2024 PUUMALAINEN Ajankohtaista kansainvälisestä yh...
STM valmiusseminaari 26-04-2024 PUUMALAINEN Ajankohtaista kansainvälisestä yh...
 
Introduction to Artificial intelligence.
Introduction to Artificial intelligence.Introduction to Artificial intelligence.
Introduction to Artificial intelligence.
 
BIG DEVELOPMENTS IN LESOTHO(DAMS & MINES
BIG DEVELOPMENTS IN LESOTHO(DAMS & MINESBIG DEVELOPMENTS IN LESOTHO(DAMS & MINES
BIG DEVELOPMENTS IN LESOTHO(DAMS & MINES
 
Molecular docking- Laxman I. Nimbale.pptx
Molecular docking- Laxman  I. Nimbale.pptxMolecular docking- Laxman  I. Nimbale.pptx
Molecular docking- Laxman I. Nimbale.pptx
 
in kuwait௹+918133066128....) @abortion pills for sale in Kuwait City
in kuwait௹+918133066128....) @abortion pills for sale in Kuwait Cityin kuwait௹+918133066128....) @abortion pills for sale in Kuwait City
in kuwait௹+918133066128....) @abortion pills for sale in Kuwait City
 
Ready Set Go Children Sermon about Mark 16:15-20
Ready Set Go Children Sermon about Mark 16:15-20Ready Set Go Children Sermon about Mark 16:15-20
Ready Set Go Children Sermon about Mark 16:15-20
 
Proofreading- Basics to Artificial Intelligence Integration - Presentation:Sl...
Proofreading- Basics to Artificial Intelligence Integration - Presentation:Sl...Proofreading- Basics to Artificial Intelligence Integration - Presentation:Sl...
Proofreading- Basics to Artificial Intelligence Integration - Presentation:Sl...
 
"I hear you": Moving beyond empathy in UXR
"I hear you": Moving beyond empathy in UXR"I hear you": Moving beyond empathy in UXR
"I hear you": Moving beyond empathy in UXR
 
Databricks Machine Learning Associate Exam Dumps 2024.pdf
Databricks Machine Learning Associate Exam Dumps 2024.pdfDatabricks Machine Learning Associate Exam Dumps 2024.pdf
Databricks Machine Learning Associate Exam Dumps 2024.pdf
 
Modernizing The Transport System:Dhaka Metro Rail
Modernizing The Transport System:Dhaka Metro RailModernizing The Transport System:Dhaka Metro Rail
Modernizing The Transport System:Dhaka Metro Rail
 
2024 mega trends for the digital workplace - FINAL.pdf
2024 mega trends for the digital workplace - FINAL.pdf2024 mega trends for the digital workplace - FINAL.pdf
2024 mega trends for the digital workplace - FINAL.pdf
 
2024-05-15-Surat Meetup-Hyperautomation.pptx
2024-05-15-Surat Meetup-Hyperautomation.pptx2024-05-15-Surat Meetup-Hyperautomation.pptx
2024-05-15-Surat Meetup-Hyperautomation.pptx
 
BEAUTIFUL PLACES TO VISIT IN LESOTHO.pptx
BEAUTIFUL PLACES TO VISIT IN LESOTHO.pptxBEAUTIFUL PLACES TO VISIT IN LESOTHO.pptx
BEAUTIFUL PLACES TO VISIT IN LESOTHO.pptx
 
The Concession of Asaba International Airport: Balancing Politics and Policy ...
The Concession of Asaba International Airport: Balancing Politics and Policy ...The Concession of Asaba International Airport: Balancing Politics and Policy ...
The Concession of Asaba International Airport: Balancing Politics and Policy ...
 
globalisation project report displayed overview
globalisation project report displayed overviewglobalisation project report displayed overview
globalisation project report displayed overview
 
LITTLE ABOUT LESOTHO FROM THE TIME MOSHOESHOE THE FIRST WAS BORN
LITTLE ABOUT LESOTHO FROM THE TIME MOSHOESHOE THE FIRST WAS BORNLITTLE ABOUT LESOTHO FROM THE TIME MOSHOESHOE THE FIRST WAS BORN
LITTLE ABOUT LESOTHO FROM THE TIME MOSHOESHOE THE FIRST WAS BORN
 
SOLID WASTE MANAGEMENT SYSTEM OF FENI PAURASHAVA, BANGLADESH.pdf
SOLID WASTE MANAGEMENT SYSTEM OF FENI PAURASHAVA, BANGLADESH.pdfSOLID WASTE MANAGEMENT SYSTEM OF FENI PAURASHAVA, BANGLADESH.pdf
SOLID WASTE MANAGEMENT SYSTEM OF FENI PAURASHAVA, BANGLADESH.pdf
 
Abortion Pills Fahaheel ௹+918133066128💬@ Safe and Effective Mifepristion and ...
Abortion Pills Fahaheel ௹+918133066128💬@ Safe and Effective Mifepristion and ...Abortion Pills Fahaheel ௹+918133066128💬@ Safe and Effective Mifepristion and ...
Abortion Pills Fahaheel ௹+918133066128💬@ Safe and Effective Mifepristion and ...
 

ExtremeEarth Open Workshop - Overview and Achievements

  • 1. ExtremeEarth From Copernicus Big Data to Extreme Earth Analytics This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 825258.
  • 2. December 9, 2021 ExtremeEarth Workshop Manolis Koubarakis (Coordinator, UoA) ExtremeEarth – Overview and Achievements
  • 3. 3 ExtremeEarth Main Objective • The main objective of ExtremeEarth is to go beyond the state-of-the-art and develop Artificial Intelligence and Big Data techniques and technologies that scale to the PBs of big Copernicus data, information and knowledge, and apply these technologies in two of the ESA TEPs: Food Security and Polar. • The technologies to be developed will extend the European Hopsworks data intensive AI platform of partner Logical Clocks to offer unprecedented scalability to extreme data volumes and scale-out distributed deep learning for Copernicus data. • The extended Hopsworks platform will run on CREODIAS and will be available as open source to enable its adoption by the strong European Earth Observation downstream services industry. • The technologies to be developed will also extend the linked geospatial data systems GeoTriples, JedAI, Strabon and SemaGrow pioneered by project partners UoA and NCSR in the past, so that they scale to the extreme volumes of Copernicus data.
  • 4. 4 ExtremeEarth Consortium 1. National and Kapodistrian University of Athens (UoA) 2. VISTA 3. The Arctic University of Norway (UiT) 4. University of Trento (UNITN) 5. The Royal Institute of Technology (KTH) 6. National Center for Scientific Research – Demokritos (NCSR-D) 7. German Aerospace Center (DLR) 8. Polar View 9. Norwegian Meteorological Institute (METNO) 10.LogicalClocks 11.British Antarctic Survey (UKRI-BAS)
  • 5. 5 Key Technical Achievements • The Food Security use case: we have produced irrigation recommendations for farmers in agricultural areas in the Danube and Duero catchments allowing a new level of detail for wide-scale irrigation support.
  • 6. 6 Key Technical Achievements (cont’d) • The Polar use case: we have produced high resolution sea-ice charts from massive volumes of heterogeneous Copernicus data.
  • 7. 7 Key Technical Achievements (cont’d) • We have developed scalable deep learning techniques for Copernicus big data. • In more detail: o Developed an LSTM deep neural network architecture for crop type mapping from Sentinel 2 data. This has been implemented on Hopsworks and it is being used in the Food Security use case. o Developed multiple deep neural network architectures (LDA, CNN, variational auto-encoders, GAN) for sea-ice classification from Sentinel 1 data. Most of these have been implemented on Hopsworks and are being used in the Polar use case.
  • 8. 8 Key Technical Achievements (cont’d) • We have developed very large training datasets for deep learning architectures targeting the classification of Sentinel images. • In more detail: o Developed a training dataset consisting of ~1M pixels of 16 Sentinel 2 images located in Austria where each pixel is labelled with one of 13 crop types. This dataset was developed using existing crop type maps and Sentinel 2 data and it was used to train the LSTM network for the Food Security use case. o This dataset and the following ones are available publicly: http://earthanalytics.eu/datasets.html and on Zenodo.
  • 9. 9 Key Technical Achievements (cont’d) o Developed a training dataset consisting of 63048 patches of 30 Sentinel 1 images located in the European Arctic where each patch is labelled with one of 6 ice types. This dataset was developed by expert photo-interpretation and it was used to train three of the CNN networks for the Polar use case.
  • 10. 10 Key Technical Achievements (cont’d) o Developed two training datasets based on 24 Sentinel 1 images located in the Belgica Bank of the Greenland Sea. The first dataset consists of ~62M patches of size 4x4 pixels where each patch is labelled with one of 11 topics using Latent Dirichlet Allocation. The second dataset consists of 153,600 patches of size 256x256 pixels classified into 8 classes using active learning based on support vector machines with relevance feedback.
  • 11. 11 Key Technical Objectives and Progress (cont’d) o Developed a training dataset consisting of 18,000 patches of 12 Sentinel 1 images located in the Danmarkshavn (East coast of Greenland). In version 1, each patch is labelled with one of 2 classes (ice or water), while the updated version 2 contains additional information on sea ice concentrations, ice type and form. These datasets were developed by expert photo-interpretation and used to train some of the CNN networks for the Polar use case.
  • 12. 12 Key Technical Achievements (cont’d) • We have developed big linked geospatial data systems that scale to big Copernicus data, information and knowledge. • In more detail: o Developed the system GeoTriples-Spark for transforming big geospatial data from their legacy formats into RDF. GeoTriples-Spark can transform 2TBs of geospatial data into RDF in 50 minutes. o Developed the system JedAI-spatial for interlinking big linked geospatial data. JedAI-spatial has been tested with ~100 GB of geospatial data (~400M geometries) and has been shown to scale almost linearly. o Developed the system Strabo 2 for querying big linked geospatial data using the OGC standard GeoSPARQL. Strabo 2 can process queries of the Geographica benchmark for a dataset of 450GB with an average execution time of 98 seconds using 128 worker nodes in Hopsworks. o Developed the version 3 of system Semagrow for federating big linked geospatial data sources. Semagrow can process queries from the Food Security use case over 4.5GB of data in <5 seconds and can process queries from the Geographica benchmark in federations with 100 endpoints.
  • 13. 13 Key Technical Achievements (cont’d) • We have integrated the AI and Big Data technologies presented above in the Hopsworks data platform and deployed them in CREODIAS and the two TEPs. • In more detail: o We developed the ExtremeEarth platform software architecture using Hopsworks. o The deep learning architectures for the two use cases have been implemented on the platform. o GeoTriples-Spark, JedAI-spatial, Strabo 2 and Semagrow have been implemented on the platform.
  • 14. 14 Key Technical Achievements (cont’d) • We have extended the Hopsworks data intensive AI platform with new deep learning functionalities for Copernicus data. • Progress: o We extended the file system HopsFS and the resource scheduler HopsYARN of Hopsworks for managing EO data. o We extended the Hopsworks metadata and security model with APIs for EO metadata. o We carried out the following extensions to the Hopsworks platform that enable large-scale distributed data processing and building ML/DL pipelines: EO Data Management, Feature Store, Experiment API, Maggy framework for asynchronous parallel execution of trials for machine learning experiments, distribution oblivious training functions, Maggy support for hyperparameter tuning and parallel ablation studies.
  • 15. Thank you! Visit our Web site: http://earthanalytics.eu/ Follow us on Twitter: @ExtremeEarth_EU @mkoubarakis