High-Performance Computing (HPC) technology is becoming increasingly important as a key driver to push European economic growth and Scientific Research. A comprehensive tool that can support the development of a wide array of scientific domains (like Big Data, earth observation and ocean study) and impact societal challenges as well.
The Webinar aims at introducing the Phidias HPC initiative to the European HPC and Research community, including main features, expected impact and advantages for Research & HPC ecosphere. The project is paving the way to increase the HPC and Data capacities of the European Data Infrastructure by pursuing the following objectives:
- Building a prototype for earth scientific data
- Enabling Open Access to HPC Services
- Strengthening FAIRisation
- Creating a framework combining computing, dissemination and archiving resources.
Cloud for Research and Innovation - UK USA HPC workshop, Oxford, July 205Martin Hamilton
How can public cloud and technologies like Docker and OpenStack help to deliver next generation scientific computing infrastructure? My talk for the UK/USA HPC workshop in July 2015, organized by HPC-SIG (UK) and CASC (USA).
Cloud for Research and Innovation - UK USA HPC workshop, Oxford, July 205Martin Hamilton
How can public cloud and technologies like Docker and OpenStack help to deliver next generation scientific computing infrastructure? My talk for the UK/USA HPC workshop in July 2015, organized by HPC-SIG (UK) and CASC (USA).
RINPAS Data for Decisions - Big Data and Data CommunitiesHelen Thompson
The Research Innovation Network for Precision Agriculture systems held a workshop in Sydney from 31 May to 1 June 2016 where the focus was on Data for Decisions - Big Data and Data Communities. This presentation profiles research being undertaken by Federation University Australia's Centre for eResearch and Digital Innovation which includes knowledge management, spatial research and decision tools including for agriculture.
Linking EUDAT services to the EGI Fed-Cloud - EUDAT Summer School (Hans van P...EUDAT
The main goal of the EGI-EUDAT collaboration is to harmonise the two eInfrastructures, including technical interoperability, authentication, authorisation and identity management, policy and operations. As main objective, this work is to provide end-users with a seamless access to an integrated infrastructure offering both EGI and EUDAT services and then, pairing data and high-throughput computing resources together. Selected user communities are able to bring requirements and help assign the right priorities to each of them. In this way, the integration activity has been driven by the end users from the start. The use case permits a user of either e-infrastructure to instantiate a VM on the EGI Cloud Federation for the execution of a computational job consuming data preserved onto EUDAT resources. The results of such analysis can be staged back to EUDAT storages, and if needed, allocated with Persistent identifiers (PIDs) for future use. To implement all the steps of this use case the following integration activities between the two infrastructures has to be fulfilled: (1) harmonisation between the authentication and authorisation model, (2) definition and implementation of the interfaces between the involved EGI and EUDAT services.
Visit: https://www.eudat.eu/eudat-summer-school
The INSPIRE Implementing Rules (IRs) on interoperability of spatial data sets and services and for network services include requirements for setting up a Spatial Data Infrastructure in Europe for supporting environmental policy making as well as policies with impact on the environment. To help Data provider with technical aspects of the IRs as well as with its correct implementation, INSPIRE Technical Guidelines (TG) were developed for each 34 data themes (INSPIRE data specifications) and for the different types of INSPIRE network services (discovery, view, download and transformation).
Spatial objects are mapped, digitalized and stored in a GIS data sets or (spatial) database. Normally, the structure of the data will depend on the specific needs for which the data are collected and used. In order to provide them in compliance with INSPIRE, these source data sets have to be transformed to match the data model prescribed by INSPIRE and have to be provided through INSPIRE download services.
This training will show and illustrate through "hands on" exercises how data sets can be transformed and provided through INSPIRE-compliant services by covering the following topics:
1) Data transformation: This session gives an introduction and explanations about encoding rules, mapping original attributes into the INSPIRE data models and vocabularies and extending data models and vocabularies.
2) Download services: This session will explore the procedure of providing transformed dataset into through an INSPIRE network service, e.g. through an WMS (for view services) or WFS or ATOM feeds (download services).
3) "Hands on" session: This session will give an overview of different architectural approaches (e.g. on-the-fly transformation and stand-alone offline transformation) and concrete software solutions for transforming spatial data and creating INSPIRE-compliant services.
High-level Meeting & Workshop on Environmental and Scientific Open Data for Sustainable Development Goals in Developing Countries. Madagascar, 4-6 December 2017
Artificial Intelligence and Big Data Techniques for Copernicus Data: the ExtremeEarth project
Manolis Koubarakis (Professor at the National and Kapodistrian University of Athens and Adjunct Researcher at the Institute of the Management of Information Systems)
The Value Added Element (VAE) and Data User Element (DUE) are two programmatic components of the Earth Observation Envelope Programme (EOEP), an optional programme of the European Space Agency, currently subscribed by 20 ESA Member States. While the DUE mission focuses on the establishment of a long-term relationship between the User communities and Earth Observation in support of major environmental conventions, the VAE focuses on developing the European and Canadian EO Services Industry in growing the prospects of sustainable use of EO-based information the operations of non-EO businesses and organisations.
Phidias: Steps forward in detection and identification of anomalous atmospher...Phidias
PHIDIAS is organised a webinar entitled "Steps forward in detection and identification of anomalous atmospheric events" held on 13 October 2020 at 15:00 CEST in collaboration with ESCAPE project. The webinar aimed at showcasing how PHIDIAS is going to improve the usage of HPC and high performance data management services for the development of intelligent screening approaches for the exploitation of large amounts of satellite atmospheric data in an operational context.
BioDT for the UiO Science section meeting 2023-03-24Dag Endresen
Presentation of the Biodiversity Digital Twin (BioDT) project for the University of Oslo (UiO) Natural History Museum (NHMO) Science department on 2023-03-24.
RINPAS Data for Decisions - Big Data and Data CommunitiesHelen Thompson
The Research Innovation Network for Precision Agriculture systems held a workshop in Sydney from 31 May to 1 June 2016 where the focus was on Data for Decisions - Big Data and Data Communities. This presentation profiles research being undertaken by Federation University Australia's Centre for eResearch and Digital Innovation which includes knowledge management, spatial research and decision tools including for agriculture.
Linking EUDAT services to the EGI Fed-Cloud - EUDAT Summer School (Hans van P...EUDAT
The main goal of the EGI-EUDAT collaboration is to harmonise the two eInfrastructures, including technical interoperability, authentication, authorisation and identity management, policy and operations. As main objective, this work is to provide end-users with a seamless access to an integrated infrastructure offering both EGI and EUDAT services and then, pairing data and high-throughput computing resources together. Selected user communities are able to bring requirements and help assign the right priorities to each of them. In this way, the integration activity has been driven by the end users from the start. The use case permits a user of either e-infrastructure to instantiate a VM on the EGI Cloud Federation for the execution of a computational job consuming data preserved onto EUDAT resources. The results of such analysis can be staged back to EUDAT storages, and if needed, allocated with Persistent identifiers (PIDs) for future use. To implement all the steps of this use case the following integration activities between the two infrastructures has to be fulfilled: (1) harmonisation between the authentication and authorisation model, (2) definition and implementation of the interfaces between the involved EGI and EUDAT services.
Visit: https://www.eudat.eu/eudat-summer-school
The INSPIRE Implementing Rules (IRs) on interoperability of spatial data sets and services and for network services include requirements for setting up a Spatial Data Infrastructure in Europe for supporting environmental policy making as well as policies with impact on the environment. To help Data provider with technical aspects of the IRs as well as with its correct implementation, INSPIRE Technical Guidelines (TG) were developed for each 34 data themes (INSPIRE data specifications) and for the different types of INSPIRE network services (discovery, view, download and transformation).
Spatial objects are mapped, digitalized and stored in a GIS data sets or (spatial) database. Normally, the structure of the data will depend on the specific needs for which the data are collected and used. In order to provide them in compliance with INSPIRE, these source data sets have to be transformed to match the data model prescribed by INSPIRE and have to be provided through INSPIRE download services.
This training will show and illustrate through "hands on" exercises how data sets can be transformed and provided through INSPIRE-compliant services by covering the following topics:
1) Data transformation: This session gives an introduction and explanations about encoding rules, mapping original attributes into the INSPIRE data models and vocabularies and extending data models and vocabularies.
2) Download services: This session will explore the procedure of providing transformed dataset into through an INSPIRE network service, e.g. through an WMS (for view services) or WFS or ATOM feeds (download services).
3) "Hands on" session: This session will give an overview of different architectural approaches (e.g. on-the-fly transformation and stand-alone offline transformation) and concrete software solutions for transforming spatial data and creating INSPIRE-compliant services.
High-level Meeting & Workshop on Environmental and Scientific Open Data for Sustainable Development Goals in Developing Countries. Madagascar, 4-6 December 2017
Artificial Intelligence and Big Data Techniques for Copernicus Data: the ExtremeEarth project
Manolis Koubarakis (Professor at the National and Kapodistrian University of Athens and Adjunct Researcher at the Institute of the Management of Information Systems)
The Value Added Element (VAE) and Data User Element (DUE) are two programmatic components of the Earth Observation Envelope Programme (EOEP), an optional programme of the European Space Agency, currently subscribed by 20 ESA Member States. While the DUE mission focuses on the establishment of a long-term relationship between the User communities and Earth Observation in support of major environmental conventions, the VAE focuses on developing the European and Canadian EO Services Industry in growing the prospects of sustainable use of EO-based information the operations of non-EO businesses and organisations.
Phidias: Steps forward in detection and identification of anomalous atmospher...Phidias
PHIDIAS is organised a webinar entitled "Steps forward in detection and identification of anomalous atmospheric events" held on 13 October 2020 at 15:00 CEST in collaboration with ESCAPE project. The webinar aimed at showcasing how PHIDIAS is going to improve the usage of HPC and high performance data management services for the development of intelligent screening approaches for the exploitation of large amounts of satellite atmospheric data in an operational context.
BioDT for the UiO Science section meeting 2023-03-24Dag Endresen
Presentation of the Biodiversity Digital Twin (BioDT) project for the University of Oslo (UiO) Natural History Museum (NHMO) Science department on 2023-03-24.
Big Data Europe at eHealth Week 2017: Linking Big Data in HealthBigData_Europe
Of the four V's of big data – Volume, Velocity, Variety and Veracity – the most challenging for the health sector is Variety. Health data comes from many sources, formats and standards – how can we bring these together to reap the benefits of big data technologies?
Big Data Europe is tackling this challenge head-on, building a big data infrastructure flexible enough to tackle all seven Societal Challenges identified by Horizon 2020. Here we demonstrate our pilot implementation of Open PHACTS, which integrates life science data for drug discovery.
12 May 2017
Available data sources & Real-time data collectionCLEEN_Ltd
The amount of environmental data is increasing, and the data would be valuable to the society if they are delivered to the right processes at the right time. In the seminar, we show examples of available data, how they are produced and processed, and how the data can be used in new innovative applications.
This presentation is part of the Environmental Data for Applications Seminar held on the 23rd of September 2015. The seminar was organised by the MMEA (Measurement, Measuring and Environmental Assessment) research programme under the Cleen Ltd (SHOK). The presentations are based on the research results related to environmental data interoperability. The participants included key players and partners in the field of environmental monitoring in Finland.
More info at www.mmea.fi
CeRDI Research RUN Vietnam Agriculture GroupHelen Thompson
Federation University's Centre for eResearch and Digital Innovation (CeRDI) is participating in the Regional University Network (RUN) Vietnam Agriculture Group. This presentation provides some background on CeRDI initiatives in eResearch.
Areas of focus include capacity building and engagement, research collaborations around soil management, water resources, land use, crop productivity, climate change and adaption, biodiversity, participatory GIS and citizen science.
Major technology and research trends link to ubiquitous high-speed broadband, the petabyte age, open data policies and the opportunities for Universities and particularly regional universities to play a significant role in generating insight from data.
Mobile technologies… App development and responsive design – for student and staff recruitment, engagement, knowledge transfer
3d and visualisation technologies… Massive innovation and research opportunities
Bridging the gap to facilitate selection and image analysis activities for la...Phidias
PHIDIAS organised it's third and final PHIDIAS Webinar of the series, this time dedicated to Use Case 2: Big Data Earth Observations (EO), took place on 18 February 2021 at 15:00 CET, showcasing how PHIDIAS is taking advantage of HPC architecture to facilitate selection and image analysis activities for land surface monitoring.
PHIDIAS - Boosting the use of cloud services for marine data management, serv...Phidias
Description and scope of the Project
Phidias HPC is aimed at developing a consolidated and shared HPC and Data service by building on pre-existing and emerging infrastructure in order to create a federation of "user to infrastructure" services.
To achieve its purpose and to gain a comprehensive picture of the European infrastructure landscape, three data area tests will develop and provide new services to discover, manage and process spatial and environmental data produced by research communities tackling scientific challenges such as atmospheric, marine and earth observation issues.
Webinar: How to improve the cloud services for marine data
Observing the ocean is challenging: missions at sea are costly, different scales of processes interact, and the conditions are constantly changing, which is why scientists say that "a measurement not made today is lost forever". For these reasons, it is fundamental to properly store both the data and metadata, so that their access can be guaranteed for the widest community, in line with the FAIR principles: Findable, Accessible, Inter-operable and Reusable.
PHIDIAS HPC has organised a webinar entitled "PHIDIAS: Boosting the use of cloud services for marine management, services and processing" to be held on 4th June 2020 at 11 AM CEST. The webinar aims to introduce the Phidias HPC initiative, in collaboration with the Blue-Cloud project, to the European HPC and Research community, specifically in the Blue economy, to improve the use of (1) cloud services for marine data management, (2) data services to the user in a FAIR perspective, and (3) data processing on demand.
These objectives will be pursued in coherence with the development of the European Open Science Cloud (EOSC) and the Copernicus Data and Information Access Services (DIAS).
Data management plans – EUDAT Best practices and case study | www.eudat.euEUDAT
| www.eudat.eu | Presentation given by Stéphane Coutin during the PRACE 2017 Spring School joint training event with the EU H2020 VI-SEEM project (https://vi-seem.eu/) organised by CaSToRC at The Cyprus Institute. Science and more specifically projects using HPC is facing a digital data explosion. Instruments and simulations are producing more and more volume; data can be shared, mined, cited, preserved… They are a great asset, but they are facing risks: we can miss storage, we can lose them, they can be misused,… To start this session, we will review why it is important to manage research data and how to do this by maintaining a Data Management Plan. This will be based on the best practices from EUDAT H2020 project and European Commission recommendation. During the second part we will interactively draft a DMP for a given use case.
BDE-SC1 Webinar: OpenPHACTS Re-engineered with Big Data EuropeBigData_Europe
Watch this webinar on YouTube: https://youtu.be/MwG0yhrctDs
Slides for the latest update on our Big Data Europe pilot in Societal Challenge 1: Health, Demographic Change and Wellbeing.
Last year we successfully completed the first phase of this pilot, replicating the functionality of the Open PHACTS Discovery Platform on the BDE infrastructure. The Open PHACTS Discovery Platform brings together pharmacological data resources in an integrated, interoperable infrastructure, and has been developed to reduce barriers to drug discovery for industry, academia, and small businesses.
Learn more about the progress we’ve made, and what’s coming next.
1. General overview of the Big Data Europe project and Societal Challenges it addresses (Ronald Siebes, VU Amsterdam)
2. The Big Data Europe infrastructure, generic components that are being developed, and their flexibility for different applications (Hajira Jabeen, University of Bonn)
3. Latest details of the current state of the Open PHACTS architecture in BDE, and ongoing work (Nick Lynch, CTO, Open PHACTS Foundation)
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024Neo4j
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
PHP Frameworks: I want to break free (IPC Berlin 2024)
PHIDIAS HPC – Building a prototype for Earth Science Data and HPC Services
1. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
PHIDIAS HPC - Building a prototype
for Earth Science Data and HPC
services
Webinar | February 13, 2020, 10:00 AM CEST
2. PHIDIAS HPC - Building a prototype for
Earth Science Data and HPC services
10:00 – 10:10 – PHIDIAS offering to HPC community – Boris Dintrans,
PHIDIAS Project Coordinator, CINES
10:10 – 10:20 – Improving the efficiency of the intelligent screening of
environmental satellite data – Pascal Prunet, WP4 Leader, SPASCIA
10:20 – 10:30 – On demand image processing for environmental monitoring :
challenges and uses cases coming from earth observation data – Jean-
Christophe Desconnets, WP5 Leader, IRD
10:30 – 10:35 – Q&A
10:35 – 10:45 – Boosting the use of cloud services for marine data studies –
Cecile Nys, WP6 member, IFREMER
10:45 – 10:55 – AI Services targeting user communities – Aleksi Kallio,
Development Manager of data analytics, CSC – IT Centre of Science Ltd
10:55 – 11:00 – Q&A
11:00 – 11:05 – Final remarks
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc 2
3. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
PHIDIAS offering to HPC
Community
Boris DINTRANS, PHIDIAS coordinator and CINES director
Centre Informatique National de l’Enseignement Supérieur
(CINES), Montpellier, France
4. CINES: one of the three national supercomputing
centers in France
4
TGCC: Joliot-Curie
IDRIS: Jean Zay
HPC (Occigen) IT hosting Archiving
60 people
Budget ~10M€/y
National & EU scope
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
5. Projet overview
Scope and objectives of the action: the Action’s overall objective is to build a prototype for
Data/High Performance Computing (HPC) services based on Earth sciences cases. In
this respect, the consortium will develop and provide new services to discover, manage and
process spatial and environmental data produced by research communities tackling scientific
challenges such as atmospheric, marine and earth observation issues.
Duration: 01/09/2019 to 01/09/2022 (3 years)
Budget: 3,519,476€
CEF Grant: 2,639,607€ (75%)
Project Officer: Mark Vella Muskat from INEA (Innovation & Network Executive Agency)
Project number: Action n° 2018-EU-IA-0089
513.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
6. Consortium with 13 partners (8 FR, 2 FI, 1 NL, 1 IT, 1 BE)
CINES: Centre Informatique National de l’Enseignement Supérieur (France, Coordinator)
CERFACS: Centre Européen de Recherche et de Formation Avancée en Calcul Scientifique (France)
CNRS: Centre National de la Recherche Scientifique (France)
CSC: IT Centre for Science Ltd. (Finland)
Geomatys (France)
IRD: Institut de Recherche pour le Développement (France)
IFREMER: Institut Français de Recherche pour l‘Exploitation de la Mer (France)
MARIS: Mariene Informatie Service MARIS BV (Netherlands)
Néovia: Néovia Innovation (France)
Spascia (France)
SYKE: Finnish Environment Institute (Finland)
Trust-IT Services (Italy)
ULIEGE: Université de Liège (Belgium)
613.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
7. Four main challenges that must be addressed
[Challenge #1] Handling the diversity of data coming from the
Earth System Research Infrastructure.
[Challenge #2] Developing and testing transversal methods and
tools that can be applied to data coming from other scientific
domains such as health and environment data.
[Challenge #3] Scalability of Data processing tools.
[Challenge #4] Industrialization and strength development of
HPC/HPDA/AI workflows.
713.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
8. The Data Terra research infrastructure brings the data to
PHIDIAS
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
9. Data Terra = 3 scientific use cases for PHIDIAS
[Use case #1] Intelligent screening of large amount of satellite data
for detection and identification of anomalous atmospheric
composition events (WP4, leader: SPASCIA) AERIS
[Use case #2] Big data Earth Observations: processing on-demand
for environmental monitoring (WP5, leader: IRD) THEIA
[Use case #3] Ocean (WP6, leader: IFREMER) ODATIS
913.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
10. The generic and fully-integrated PHIDIAS workflow
1013.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
11. Main project organization
11
Management Board: acting a as the decision-making body of the consortium
Technical Board: acting as the supervisory body for the execution of the Action
Scientific and users Committee: gather various stakeholders of the HPC community as well as users and
citizen communities, and act as an advisory entity for PHIDIAS
Security Committee: address transversal security issues and identify Information Technology (IT) risks in the
workflows developed during the Action
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
12. 24/04/2020 footer 12
WP1 - Management WP2 – Compute & storage workflow WP3 – Technical coordination
WP4 – Satellite data WP5 – Environmental monitoring WP6 – Ocean use case
WP7 – Dissemination
T1.1: Project contractual management
T1.2: Project operational day-to-day management
T1.3 – Scientific and Users Committee
T2.1: HPC, HPDA and storage services
T2.2: Storage backend and archiving services
T2.3: On-the-fly computing
T3.1: Technical coordination
T3.2: Common Metadata Repository
T3.3: End-users web Common interactive Processing
Services
T3.4: Common Portal for data discovery, access and
processing
T4.1: Implementation of PCA filtering techniques for
detection of exceptional atmospheric event
T4.2: Adaptation and preparation of tools, data and environment for
the data screening processing of Sentinel 5 precursor data
T4.3: Processing of 1 year of global S5P data for detection of
extreme atmospheric events
T4.4.1: UC1 - Exploitation for monitoring and alert service
development
T4.4.2: UC2 - Scientific exploitation of the product
T5.1: EO data processing chains for massive and on-demand
execution
T5.2: UI web environment dedicated for on-demand
execution
T5.3: Data workflows for discovery, access of EO
raw data and products
T6.1: Improvement of long-term stewardship of data
T6.2: Improvement of data storage for services to users
T6.3: Marine data processing workflows for on-demand
processing
T6.4: Data inter-comparison, collection and visualization
T7.1: Communication and Outreach
T7.2: Engagement with user and stakeholder communities
T7.3: EU-wide cross-dissemination & concertation
T7.4: Sustainability Path & Funding models
CINES CINES IRST
SPACIA IRD IFREMER
TRUST-IT
NEOVI
A
NEOVIA
CINES
Géomaty
s
Ulieg
e
9
x
Delivrable
CINES PMs
T4.5: Update the service, and prepare for the exploitation
Of Copernicus Atmosphere satellite data
MARI
S
9 3
6
1
8
1
8
Task Leader
14. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
Intelligent screening of satellite
data for air quality and climate
Pascal Prunet, SPASCIA CEO & WP4 Coordinator
Webinar | February 13, 2020
15. 15
Independent SME : R&D in space science and environment
Processing, analysis and exploitation of remote sensing observations of the Earth
atmosphere from space :
For supporting the definition, development, calibration & validation of satellite
missions/systems/instruments.
For environmental applications (air quality, atmospheric composition, climate, meteorology).
Created in March 2016, located near Toulouse, France
11 staff members : PhDs with complementary skills in Space and Earth Sciences
Working for space agencies, research institutes, industries
In strong collaboration with European research and scientific institutes
SPASCIA is acting for a better use of Earth observations from space
THE COMPANY
SPASCIA
Space Science Algorithmics
Satellite mission/systems
IASI, IASI-NG, MTG-IRS, MicroCarb, S5P, CO2M
Atmosphere and environment
Atmsopheric chemistry, GHG, air quality, Cllimate
Level 1 data simulation,
processing, cal/val
Level 2 geophysical
products retrieval,
caracterisation,
analysis
Level 3 & 4
atmospheric fields
innovation, exploitation
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
16. 16
Earth and environment challenges at local, regional and global scales : continuous increase of
human activities modify the atmospheric composition
Impact on environment (climate; soil, water and air quality; biodiversity)
Impact on health, on economy
European opportunities for better adressing these challenges : Europe is entering in the era of
operational measurement of the air composition from space, for the analysis and forecasting of chemical
weather and climate monitoring
PHIDIAS FOR ATMOSPHERE : CONTEXT
First measurements already available : The European operational atmospheric composition
observation from space currently benefits from data provided by IASI and GOME onboard the
Metop satellites, as well as Sentinel 5 precursor (S5P/TROPOMI)
4 Operational satellites from 2021 : Metop-NG, MeteoSat Third Generation, Sentinel 4 and 5
Development of the European strategy for the measurement of GHG for monitoring human emissions
Copernicus Operational forecasting services for atmospheric chemistry (CAMS) and Climate (C3S)
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
17. 17
From 2021, European atmospheric sounding missions will deliver each day several TB (terabytes) of raw
datacubes at high spatial/temporal/spectral resolutions. This represents an unprecedented amount of
atmospheric data, with improved quality and coverage.
Increasing difficulties for properly dealing with all available information
key challenge : provide the capacity of identifying and focusing on useful data, e.g., by targeting scenes of interest in
view of their dedicated processing or exploitation.
PHIDIAS addresses those needs by using HPC and HPDA capacities : intelligent screening approaches for
the exploitation of large amounts of satellite atmospheric data in an operational context, for detection and
identification of atmospheric composition events.
.
CHALLENGES AND OBJECTIVES
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
18. 18
PHIDIAS ADDED VALUE
Earth observation from space is underexploited
Huge quantity of data, increasing information content and
accuracy, complex processes from data to information
Operational forecasting systems
Efficient, real-time data assimilation in models,
but could not use « extreme » data related with
« extreme events »
Reduced added value of the data
Research groups, scientific institutes
Innovative and well adapted to event studies
but case by case exploitation of the data
Not exhaustive exploitation,
no real-time analysis
Screening (detection
and filtering) on the
fly of relevant data
Prototype service
Realtime Detection &
analyse of
extreme events, pollutant
plumes, other targeted
events
Dedicated services
Early warning, monitoring, decision support
Added-value :
Robust detection,
objective filtering
Added-value :
Verification,
improvement
Innovation
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
19. 19
DATA FOR USE CASE DEMONSTRATION
This use case propose to Develop, test and prototype the approach with Sentinel 5 Precursor (S5P) data/products
S5P : first atmospheric Sentinel for air
quality and climate
Launched Oct. 13 2017, 7 years lifetime
UV-Vis-NIR-SWIR naidr view spectrometer,
with enhanced radiometric performances,
spatial resolution (7x3.5 km2) and temporal
revisit (Global daily coverage)
500 GB
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
20. 20
Objective : Test, implement and demonstrate intelligent screening of large amount of satellite data for detection
and identification of anomalous atmospheric composition events
Two processing prototypes would be proposed for development and test :
1. PCA-based screening of L1 data (SWIR) for detection of extreme events. Based on experience and methods
developed for IASI, implementation and consolidation of algorithms and tools for generic processing of atmospheric
spectra recorded by S5P.
2. New AI methods for objective/automatic detection of plumes from L2 products (CO, others ? (CH4 NO2, SO2) :
SPASCIA experience with physical methods analyzing significant signal enhancements
WP4 OBJECTIVES
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
21. 21
Objective : Test, implement and demonstrate intelligent screening of large amount of satellite data for detection
and identification of anomalous atmospheric composition events
Two processing prototypes would be proposed for development and test :
1. PCA-based screening of L1 data (SWIR) for detection of extreme events. Based on experience and methods
developed for IASI, implementation and consolidation of algorithms and tools for generic processing of atmospheric
spectra recorded by S5P.
Product : Real Time, operational detection, qualification and monitoring of rare or extremely strong events
Potential users : Atmospheric research, Operational air chemistry services (CAMS) for forecast validation, quality control,
improvements, future services of alerts and decision support
1. New AI methods for objective/automatic detection of plumes from L2 products (CO, others ? (CH4 NO2, SO2) :
SPASCIA experience with physical methods analyzing significant signal enhancements
WP4 OBJECTIVES
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
22. 22
PCA-Based filtering on L1 data – illustration with IASI data
Results from work by SPASCIA/HYGEOS/LATMOS, funded by CNES
Nominal
case:
denoising
filter
Extreme
case:
identification
of the
residual
signal
Reconstruction of
"scores" maps:
Rapid detection of
anomalous situations
Analysis of « residuals" :
Interpretation of the
anomalous event
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
23. 23
PCA-Based filtering on L1 data – illustration with IASI data
Results from work by SPASCIA/HYGEOS/LATMOS, funded by CNES
Big fires in Indonésie,
2015
PCA-based detection of the fire event
Cloud fraction
Operational CO product
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
24. 24
Objective : Test, implement and demonstrate intelligent screening of large amount of satellite data for detection
and identification of anomalous atmospheric composition events
Two processing prototypes would be proposed for development and test :
1. PCA-based screening of L1 data (SWIR) for detection of extreme events. Based on experience and methods
developed for IASI, implementation and consolidation of algorithms and tools for generic processing of atmospheric
spectra recorded by S5P.
2. New AI methods for objective/automatic detection of plumes from L2 products (CO, others ? (CH4 NO2, SO2) :
SPASCIA experience with physical methods analyzing significant signal enhancements
Product : automatic Identification of pollutant plumes and sources in satellite maps
Potential users : Necessary input for processing and services of pollutant emission quantification, pollution monitoring : Atmospheric
research, Operational system of GHG and pollutant emission monitoring (e.g., CO2 emission survey for supporting Paris Climate
Agreement), end users services
WP4 OBJECTIVES
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
25. 25
Plume detection for pollutant emission
quantification
TNO anthr. CO2 emission at 6x6 km2
(WP2)
From atmospheric gradients
…
… to anthropogenic
emissions
XCO2 COSMO-GHG simulations
Provided by EMPA (from H2020 ECMWF CHE project)
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
26. 26
Methods for objective/automatic detection of plumes from L2 products from S5P (CO, CH4 NO2, SO2) :
Experience with physical methods analyzing significant signal enhancements
Innovative AI methods : efficient detection of plumes and associated source(s)
S5P products
Plume detection for pollutant emission
quantification
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
27. 27
Space-based Earth observation will provide amazing amount of complex, informative data :
How to extract and use comprehensively this increasing information ?
Operational Models and systems fail to exploit the non-understood data
Research face to strong limits for analyzing all the underlined information and science
PHIDIAS will provide new tools and approaches to deal with the measurements and the information, allowing better and
more efficient use of the data, and pave the ways for finding new paradigms.
GOAL
XCO2 IFS model simulation
provided by ECMWF
(from H2020 ECMWF CHE project)
:
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
29. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
On demand image processing for
environmental monitoring: challenges
and uses cases coming from earth
observation data
WP 5 leader : Jean-Christophe Desconnets
(IRD, ESPACE-DEV)
30. 30
Working primarily in partnership with Mediterranean and inter-
tropical countries on the science of global development issues.
Multidisciplinary research: health and society, climate change,
humanitarian and political crises, agriculture and biodiversity
The French National Research Institute for
Sustainable Development
Publications : 1300/Year
65 Research Unit
52 % co-publications with
South Countries
2048 Agents
Budget : 230 M€/Year
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
31. Data and HPC Challenges for land surface community
• Late 2000s, shared diagnosis among scientific community and public
authorities : Under-utilization of satellite imagery to monitor environment
Challenges
Facilitate selection and image analysis activities for end-users
Issues
Technological (Big data and scalability)
Openess and reusability (FAIR data)
31
Accessing relevant images
Having the skills
Finding support
Getting appropriate equipments
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
32. State of the art : data land surface interoperable
catalogs
Catalog GEOSUD of VHR images
SPOT6, 14 000 raw images
32
Catalog PEPS of Radar & optical
Sentinel products
>> 10 Millions of raw images
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
33. State of the art : on demand-processing services
Data pre-processing services (GEOSUD IDS V2) : extraction,
subsetting, segmentation, classification …
3313.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
34. State of the art : land surface processing chains
Coming from scientific remote sensing community
THEIA Scientific Expertise Center (SEC) working around ten themes :
Agriculture, Forest, Urban, Coastline, Health, Water…
Outputs : scientifically validated processing chains for environmental monitoring
34
Land cover IOTA Chain Soil moisture with VR spatial resolution
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
35. Objectives of WP5
Technical Sub-objectives
Dedicated environment adapted to the target users
Ability to produce maps over large areas in a systematic manner
Open the dissemination of processsing chains outputs in FAIR way
Improvement of data reusability in perspective of EOSC
Leveraging AI techniques to provide alternative image classification
methodologies
Merging scientific experimental algorithms and catalogs
with community user-needs
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
36. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
Two examples of uses cases coming from
environmental monitoring community
37. Sentinel-1/Sentinel-2-derived Soil Moisture
product at Plot scale (S2MP) over agricultural
areas
Importance of monitoring of the soil moisture in
agricultural areas
Target Users
PHIDIAS issues
Rationale
Free access to Sentinel-1 (SAR satellite) and
Sentinel-2 (optical High resolution 10x10 m) with
high revisit
HPC : Interactive and on-demand
processing on specific zone and a large
temporal depth
Data : Big data access and data outputs
FAIRness
Scientific : water cycle modelisation
purposes
Farming sector : mapping of irrigation
activities
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
38. Remote sensing images processing with artificial
intelligence: application to land cover mapping
and super-resolution
Taking advantage of IA libraries maturity and
GPU architecture to apply deep network for
classification images purposes
Target Users
PHIDIAS issues
Rationale
Super-resolution of Sentinel-2 images 10m 1.5m
HPC : GPU architecture to test scalability of IA
approach over national territory
temporal depth
Data : Big data access and data outputs
FAIRness
Scientific : experimental mode (notebook)
Public authority : very high resolution
LU/LC monitoring
Processing of multi modal imagery for land cover
mapping using semantic segmentation. (SPOT &
Sentinel-2)
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
39. Architecture targeted (logical view)
user
services
HPDA /HPC facilities
Discovery Data services
On demand
processing
IA RF
Data Terra Web Portal (land surface focus)
Data and
computing layer
Data,
services,
process
catalog
Notebooks
Interactive Web IHM
Data sources Processing libraries & framework
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
40. Take away message
WP5 Challenges
Taking advantage of HPC architecture and big data infrastructure to
facilitate selection and image analysis activities for environmental monitoring
Use cases
• Targeted to land surface community: scientific and public authorities
• Capibility to produce on-demand new mapping products at very temporal and spatial
resolution (soil moisture, LU/LC, …)
• Provide dedicated and interactive user environment to select, configure and execute
processing chains
• Open and FAIR diffusion of outputs data in reusable manner
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
42. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
PHIDIAS HPC - Building a prototype
for Earth Science Data and HPC
services
Webinar | February 13, 2020, 10:00 AM CEST
43. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
Boosting the use of Cloud
Services for marine data studies
Cécile NYS, IFREMER
Assistant Manager Ocean Data Cluster – ODATIS
Phidias WP6 member
Webinar | February 13, 2020
44. WP6 “Use-case 3 – Ocean” overview
Boosting the use of Cloud Services for marine data studies
Combine and collocate data from several data sources (in situ &
satellite)
Enhancing data archiving (most observation cannot be reproduced)
facilitate data reuse
Facilitate and speed up co-localisation of data from different sources
Adopting new data structures (based on big-data technologies)
DataCubes
NoSQL databases (numerical data) : Cassandra, MongoDB, etc.
Semantic Web (text data)
Providing on demand data browsing and processing facilities
4413.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
45. WP6 “Use-case 3 – Ocean” overview
Focus on two geographical areas
North Atlantic Ocean Baltic Sea
4513.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
46. Ocean observation
Ocean difficult ecosystem to observe
Constantly evolving
Seasonally
Specific phenomenon evolving quickly : waves, currents, plankton blooms, etc.
Not easily accessible, especially the deep seas (costs of equipment
deployment)
Requires several complementary systems (from satellite to
underwater vehicles, but also buoys, etc.)
Importance of inter-comparisons and co-processing of all
produced data
4613.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
47. Input Data
Initial tests on specific variables
Temperature and Salinity drive the ocean circulation (geostrophic
currents, turbulence, water masses)
Chlorophyll indicator of primary production and eutrophication
Aggregate and process data (in situ & satellite) from several
infrastructures
Copernicus Marine Environmental Monitoring Services
(marine.copernicus.eu) & associated WEkEO DIAS (www.wekeo.eu)
SeaDataNet European Research Infrastructure (www.seadatanet.org)
EMODnet (DG-Mare) : European Marine Observation and Data Network
(www.emodnet.eu)
4713.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
48. Input Data
4813.02.2020 PHIDIAS Webinar | 13.02.2020 |
CMEMS –
North Atlantic
CMEMS –
Baltic Sea
SeaDataNet & EMODnet
In situ data Atlantic Iberian Biscay Irish Ocean- In-
Situ Near Real Time Observations
Baltic Sea- In Situ Near Real Time
Observations
Satellite data North Atlantic Surface Chlorophyll
Concentration from Satellite observations
(monthly)
Baltic Sea, Ocean Colour Chlorophyll (daily
observation)
49. Improvement of Data Storage
Present data structures (e.g. collections of NetCDF files) of in-situ marine
data are not very efficient, due to
the large number of files
and/or
the heterogeneity of data observations
Challenges
Access quickly to a few number of observations within a large number of observations
data visualization and data selections on web portals
Access to large number of observations within a large number of observations data
processing or parallel processing (e.g. machine learning algorithms, interpolation
software such as DIVA, etc.)
Solutions
Data visualization test of No-SQL databases (e.g. Cassandra - cassandra.apache.org)
Data processing test of “data cubes” structures such as Parquet (columnar storage
format from the Haddoop ecosystem - parquet.apache.org)
4913.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
50. On demand Data Processing
“Virtual Research Environment” for users, allowing
Jupyter Notebook as a basis to develop workflows
Annotate, compare, conserve and share expertise
Access to different data structures in one environment (access to specified data on premise or remotely)
Scripting in various languages (Python, R, Julia, etc.)
Access to Pangeo components
Use of Diva software
Software for gridding data, using a finite-element method
Developed in Julia programming language
Extension for satellite images
GIS features
Visualisation of inputs (study area), outputs (compare results of different processes) and images
Existing tools : Diva online, Sextant, Geomatys, etc.
5013.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
51. Data Visualisation and inter-comparison
2 case-studies
Surface Salinity in North Atlantic
CTD (SeaDataNet),
Argo Floats (CMEMS),
SMOS satellite,
Chlorophyll in North-East Atlantic and Balitic Sea
CTD and bottles (SeaDataNet)
BGC Argo floats (ARGO GDAC)
Ferrybox
Sentinel 2 images (DIAS WEkEO)
Visual comparisons (input and output) using GIS features
Software comparisons
Using Pangeo components
Using DIVA software
“Declouding” processing for satellite images (visible and infra-red)
5113.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
53. AI service targeting user communities
Aleksi Kallio
PHIDIAS webinar 13.2.2020
53
54. 54
Non-profit state
organization with
special tasks
Headquarters in
Espoo,
datacenter in
Kajaani
Owned by state (70%)
and all Finnish higher education institutions
(30%)
Turn over
in 2018
44,9M€
Circa
employees
in 2018
350
55. CSC’s solutions
5513.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
55
Computing and software
Data management and analytics for
research
Support and training for research
Research administration
Solutions for managing and
organizing education
Solutions for learners and teachers
Solutions for educational and
teaching cooperation
Hosting services tailored to
customers’ needs
Identity and authorisation
Management and use of data
ICT platforms, Funet network and
data center functions are the base
for our solutions
56. CSC in PHIDIAS
56
WP2: 12 PM
Intelligent data workflows on HPC/HPDA environment
Optimisation of distributed large scale AI/ML workloads
WP6: 3 PM
Supporting Ocean use case
“The other computing center partner together with CINES”
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
57. Our AI user communities
57
Researchers in capacity driven fields
Researchers in established data driven fields
Researchers in emerging data driven fields
Research IT management
Research eInfrastructures
Public institutions looking for
data driven solutions57
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
58. Services and solutions for artificial intelligence
58
• Mapping of needs, possibilities and competences
• Defining and implementing AI projects
• Data science, machine learning, data engineering, AI and BI expertise
AI & BI consultation
• Practical courses
• Methods support for AI and data analytics
Training and user support (ministry funded)
• Cloud services for building intelligent systems
• HPC computing for ambitious machine learning
• Software services for displaying data (Notebooks)
• Sensitive data environment for personal data
Computing services (ministry funded)58
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
60. Training and user support
60
• Data analytics with R
• Data visualisation
• Big data analytics with Apache Spark
Data science courses
• Practical machine learning
• Practical machine learning for spatial data
• Practical deep learning
Machine learning courses
• Supporting researchers on using our environment and choosing right methods
• Data science, machine learning, data engineering and AI expertise
Methods support from service desk
60
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
61. Computing services
61
• Rahti container cloud (Kubernetes / OpenShift)
• Apache Spark for big data, Apache Kafka for streaming data, GPU capacity coming
• cPouta IaaS cloud
• GPU capacity (Nvidia P100), IO acceleration (SSD)
• Allas object storage system
Cloud services
• Puhti and Puhti-AI
• Nvidia V100 GPU’s with fast interconnect
• Coming next: Mahti supercomputer and Lumi EuroHPC supercomputer
HPC computing services
• Notebooks provides easy-to-use environments for working with data and programming
Software services
• ePouta sensitive data IaaS cloud
• GPU capacity (Nvidia V100)
Sensitive data services
61
13.02.2020 PHIDIAS Webinar | 13.02.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
64. LUMI supercomputer
One of the world’s fastest computer systems
Performance more than tenfold compared to Europe's fastest
supercomputer today
450 m2
PINTA-ALA
Computing power
equivalent to
Peak Performance
Data reading speed from
disk to memory
corresponds to the
simultaneous operation of
MacBook Pro computers
600 000
Size of a basketball court
200+
Pflop/s
1 system
Blu-ray Players
18 500
450m2
67. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
PHIDIAS HPC - Building a prototype
for Earth Science Data and HPC
services
Webinar | February 13, 2020, 10:00 AM CEST