This presentation was given during the Japan Geosciences Union 2019. Session details can be found at http://www.jpgu.org/meeting_e2019/SessionList_en/detail/M-GI31.htm
European Open Science Cloud: Concept, status and opportunitiesEOSC-hub project
European Open Science Cloud: Concept, status and opportunities.
Presentation given by Gergely Sipos at the International Symposium on Grids and Clouds 2019 event in Taiwan.
EOSC support to scientific computing needs in to Earth Observation with the EGI Federated Cloud
The European Open Science Cloud (EOSC) supports multi-disciplinary science, and Earth Observation is one of the major use cases.
EOSC will provide capacity and capabilities for the fostering the exploitation of EO data, this can be achieved by federating cloud providers of EGI, DIAS, and data analytics tools. In this presentation, we show how EOSC can rely on a public-private cloud federation for delivering its compute platform for EO.
The EOSC Compute Platform with the EGI-ACE project EGI Federation
EGI-ACE’s main goal is to implement the compute platform of the European Open Science Cloud and contribute to the EOSC Data Commons by delivering integrated computing platforms, data spaces and tools as an integrated solution that is aligned with major European cloud federation projects and HPC initiatives.
This presentation introduces you to the architecture and composition of the EOSC Compute Platform, which delivers capabilities at the IaaS, PaaS and SaaS level.
EOSC-hub contribution to the EOSC implementation, the Hub concept and engagem...EOSC-hub project
EOSC-hub contribution to the EOSC implementation, the Hub concept and engagement with stakeholders, Tiziana Ferrari, Technical Director, EGI & EOSC-hub Project Coordinator; Per Öster, Director, CSC & EOSC-hub Project Director (EOSC hub week, Malaga, 16 - 20 April 2018)
European Open Science Cloud: Concept, status and opportunitiesEOSC-hub project
European Open Science Cloud: Concept, status and opportunities.
Presentation given by Gergely Sipos at the International Symposium on Grids and Clouds 2019 event in Taiwan.
EOSC support to scientific computing needs in to Earth Observation with the EGI Federated Cloud
The European Open Science Cloud (EOSC) supports multi-disciplinary science, and Earth Observation is one of the major use cases.
EOSC will provide capacity and capabilities for the fostering the exploitation of EO data, this can be achieved by federating cloud providers of EGI, DIAS, and data analytics tools. In this presentation, we show how EOSC can rely on a public-private cloud federation for delivering its compute platform for EO.
The EOSC Compute Platform with the EGI-ACE project EGI Federation
EGI-ACE’s main goal is to implement the compute platform of the European Open Science Cloud and contribute to the EOSC Data Commons by delivering integrated computing platforms, data spaces and tools as an integrated solution that is aligned with major European cloud federation projects and HPC initiatives.
This presentation introduces you to the architecture and composition of the EOSC Compute Platform, which delivers capabilities at the IaaS, PaaS and SaaS level.
EOSC-hub contribution to the EOSC implementation, the Hub concept and engagem...EOSC-hub project
EOSC-hub contribution to the EOSC implementation, the Hub concept and engagement with stakeholders, Tiziana Ferrari, Technical Director, EGI & EOSC-hub Project Coordinator; Per Öster, Director, CSC & EOSC-hub Project Director (EOSC hub week, Malaga, 16 - 20 April 2018)
Presented during the Research Data Alliance's 11th Plenary in Berlin, Germany, the EOSC-hub project, through this presentation, gave an overview on the project and how it will contribute to the development of the European Open Science Cloud. Moreover, it also gives a more comprehensive rundown of services that will be made available through EOSC-hub
Presented during the Research Data Alliance's 11th Plenary in Berlin, Germany, the EOSC-hub project, through this presentation, gave an overview on the project and how it will contribute to the development of the European Open Science Cloud.
Open Data management is still not trivial nor sustainable - COMSODE results are here to bring automation to publication and management of Open Data in public institutions and companies. Presentation includes Open Data Ready standard proposal, three use cases and invitation for Horizon 2020 projects 2016.
PHIDIAS - Boosting the use of cloud services for marine data management, serv...Phidias
Description and scope of the Project
Phidias HPC is aimed at developing a consolidated and shared HPC and Data service by building on pre-existing and emerging infrastructure in order to create a federation of "user to infrastructure" services.
To achieve its purpose and to gain a comprehensive picture of the European infrastructure landscape, three data area tests will develop and provide new services to discover, manage and process spatial and environmental data produced by research communities tackling scientific challenges such as atmospheric, marine and earth observation issues.
Webinar: How to improve the cloud services for marine data
Observing the ocean is challenging: missions at sea are costly, different scales of processes interact, and the conditions are constantly changing, which is why scientists say that "a measurement not made today is lost forever". For these reasons, it is fundamental to properly store both the data and metadata, so that their access can be guaranteed for the widest community, in line with the FAIR principles: Findable, Accessible, Inter-operable and Reusable.
PHIDIAS HPC has organised a webinar entitled "PHIDIAS: Boosting the use of cloud services for marine management, services and processing" to be held on 4th June 2020 at 11 AM CEST. The webinar aims to introduce the Phidias HPC initiative, in collaboration with the Blue-Cloud project, to the European HPC and Research community, specifically in the Blue economy, to improve the use of (1) cloud services for marine data management, (2) data services to the user in a FAIR perspective, and (3) data processing on demand.
These objectives will be pursued in coherence with the development of the European Open Science Cloud (EOSC) and the Copernicus Data and Information Access Services (DIAS).
Bob Jones, CERN & HNSciCloud Coordinator gives an update on the HNSciCloud Pre-Commercial Procurement which is now in its Solution Prototyping phase. The presentation includes also an overview of the prototypes under development.
Experience in managing service portfolio by Pasquale PaganoBlue BRIDGE
Pasquale Pagano discusses managing a service portfolio including the challenges and the future challenges in managing a service portfolio.
This work is licensed under the Creative Commons CC-BY 4.0 licence.
Spatineo Webinar: Shedding Light on INSPIRE ConformityIlkka Rinne
These are the slides from Spatineo Webinar held online on 26th March 2015. For the video recording of the webinar, see https://www.youtube.com/watch?v=0-1Ni3i4M-s
Past, present and future of advanced computing for data-driven scienceEGI Federation
The EGI Federation celebrates 15 years of distributed computing in 2019. Many milestones were achieved to bring distributed computing from a vision to a real-life international production platform that today enables data-intensive processing at an unprecedented scale, supporting some of the greatest groundbreaking scientific discoveries of the XXI century.
This presentation, given by Bob Jones, CERN & HNSciCloud Coordinator, at the ESA-ESPI Workshop on “Space Data & Cloud Computing Infrastructures: Policies and Regulations”, describes what are the challenges and needs of the cloud users and explains how an hybrid cloud model can support them.
Distributed scientific computing for open science, eResearch Africa 2019EGI Federation
The presentation provides a perspective on how distributed computing has been instrumental to make ground breaking scientific discoveries possible, and how the opening of computing infrastructures at international level has been effective in delivering unprecedented compute capacity and advance data analytics tools to international research collaborations.
The presentation provides examples of the enormous scientific impact produced by the international collaboration of cyber infrastructures in Europe, Africa and other continents, and will explain the federated organizational model adopted by European countries to leverage national ICT investments and mobilize them.
The presentation offers an overview of the present and future technical and organisational challenges of data-driven research in various scientific domains. The European Open Science Cloud initiative of the European Commission will be explained and opportunities of collaboration will be discussed with the audience.
Conference website: http://www.eresearch-africa.uct.ac.za/
EOSC-hub brings together multiple service providers to create the Hub: a single contact point for European researchers and innovators to discover, access, use and reuse a broad spectrum of resources for advanced data-driven research.
This presentation introduces the services on offer to scientists of all disciplines
This a RECAP project overview slide deck prepared by Thang Le Duc (UMU), P-O Östberg (UMU) and Tomas Brännström (Tieto). It starts with an introduction and continues with a section on challenges for a self-orchestrated, self-remediated cloud system. It then presents the RECAP vision and use cases and finishes with a conclusion.
Presented during the Research Data Alliance's 11th Plenary in Berlin, Germany, the EOSC-hub project, through this presentation, gave an overview on the project and how it will contribute to the development of the European Open Science Cloud. Moreover, it also gives a more comprehensive rundown of services that will be made available through EOSC-hub
Presented during the Research Data Alliance's 11th Plenary in Berlin, Germany, the EOSC-hub project, through this presentation, gave an overview on the project and how it will contribute to the development of the European Open Science Cloud.
Open Data management is still not trivial nor sustainable - COMSODE results are here to bring automation to publication and management of Open Data in public institutions and companies. Presentation includes Open Data Ready standard proposal, three use cases and invitation for Horizon 2020 projects 2016.
PHIDIAS - Boosting the use of cloud services for marine data management, serv...Phidias
Description and scope of the Project
Phidias HPC is aimed at developing a consolidated and shared HPC and Data service by building on pre-existing and emerging infrastructure in order to create a federation of "user to infrastructure" services.
To achieve its purpose and to gain a comprehensive picture of the European infrastructure landscape, three data area tests will develop and provide new services to discover, manage and process spatial and environmental data produced by research communities tackling scientific challenges such as atmospheric, marine and earth observation issues.
Webinar: How to improve the cloud services for marine data
Observing the ocean is challenging: missions at sea are costly, different scales of processes interact, and the conditions are constantly changing, which is why scientists say that "a measurement not made today is lost forever". For these reasons, it is fundamental to properly store both the data and metadata, so that their access can be guaranteed for the widest community, in line with the FAIR principles: Findable, Accessible, Inter-operable and Reusable.
PHIDIAS HPC has organised a webinar entitled "PHIDIAS: Boosting the use of cloud services for marine management, services and processing" to be held on 4th June 2020 at 11 AM CEST. The webinar aims to introduce the Phidias HPC initiative, in collaboration with the Blue-Cloud project, to the European HPC and Research community, specifically in the Blue economy, to improve the use of (1) cloud services for marine data management, (2) data services to the user in a FAIR perspective, and (3) data processing on demand.
These objectives will be pursued in coherence with the development of the European Open Science Cloud (EOSC) and the Copernicus Data and Information Access Services (DIAS).
Bob Jones, CERN & HNSciCloud Coordinator gives an update on the HNSciCloud Pre-Commercial Procurement which is now in its Solution Prototyping phase. The presentation includes also an overview of the prototypes under development.
Experience in managing service portfolio by Pasquale PaganoBlue BRIDGE
Pasquale Pagano discusses managing a service portfolio including the challenges and the future challenges in managing a service portfolio.
This work is licensed under the Creative Commons CC-BY 4.0 licence.
Spatineo Webinar: Shedding Light on INSPIRE ConformityIlkka Rinne
These are the slides from Spatineo Webinar held online on 26th March 2015. For the video recording of the webinar, see https://www.youtube.com/watch?v=0-1Ni3i4M-s
Past, present and future of advanced computing for data-driven scienceEGI Federation
The EGI Federation celebrates 15 years of distributed computing in 2019. Many milestones were achieved to bring distributed computing from a vision to a real-life international production platform that today enables data-intensive processing at an unprecedented scale, supporting some of the greatest groundbreaking scientific discoveries of the XXI century.
This presentation, given by Bob Jones, CERN & HNSciCloud Coordinator, at the ESA-ESPI Workshop on “Space Data & Cloud Computing Infrastructures: Policies and Regulations”, describes what are the challenges and needs of the cloud users and explains how an hybrid cloud model can support them.
Distributed scientific computing for open science, eResearch Africa 2019EGI Federation
The presentation provides a perspective on how distributed computing has been instrumental to make ground breaking scientific discoveries possible, and how the opening of computing infrastructures at international level has been effective in delivering unprecedented compute capacity and advance data analytics tools to international research collaborations.
The presentation provides examples of the enormous scientific impact produced by the international collaboration of cyber infrastructures in Europe, Africa and other continents, and will explain the federated organizational model adopted by European countries to leverage national ICT investments and mobilize them.
The presentation offers an overview of the present and future technical and organisational challenges of data-driven research in various scientific domains. The European Open Science Cloud initiative of the European Commission will be explained and opportunities of collaboration will be discussed with the audience.
Conference website: http://www.eresearch-africa.uct.ac.za/
EOSC-hub brings together multiple service providers to create the Hub: a single contact point for European researchers and innovators to discover, access, use and reuse a broad spectrum of resources for advanced data-driven research.
This presentation introduces the services on offer to scientists of all disciplines
This a RECAP project overview slide deck prepared by Thang Le Duc (UMU), P-O Östberg (UMU) and Tomas Brännström (Tieto). It starts with an introduction and continues with a section on challenges for a self-orchestrated, self-remediated cloud system. It then presents the RECAP vision and use cases and finishes with a conclusion.
Gergely Sipos (EGI): Exploiting scientific data in the international context ...Gergely Sipos
Keynote presentation given at "The Emerging Technology Forum – Data Creates Universe - Scientific Data Innovation Conference" of the "Pujiang Innovation Forum 2021" event.
Integrating and managing services for the European Open Science CloudOpenAIRE
Integrating and managing services for the European Open Science Cloud - Abdulrahman Azab (EOSC-Hub, University of Oslo).
Presented : at OpenAIRE - EOSC-hub webinar “Data Privacy and Sensitive Data Services” https://www.openaire.eu/item/openaire-eosc-hub-webinar-data-privacy-and-sensitive-data-services
The first workshop of the series "Services to support FAIR data" took place in Prague during the EOSC-hub week (on April 12, 2019).
Speaker: Baptiste Grenier
Using a Widely Distributed Federated Cloud System to Support Multiple Dispara...inside-BigData.com
In this deck from the 2014 ISC Cloud Conference, David Wallom from the University of Oxford presents:
Using a Widely Distributed Federated Cloud System to Support Multiple Disparate User Communities.
"The EGI federated cloud, which has been in development for the past 3 years has now entered production. Building on the tried and trusted EGI core services we have added federated IaS compute and storage services, utilising open standards to support more than 10 pilot communities. We will discuss the model of federation, and the different application design models that the users use and why cloud will be a success when compared with grid due to this inherent flexibility."
Learn more: http://www.isc-events.com/cloud14/schedule.html
Watch the video presentation: http://wp.me/p3RLHQ-daY
Using the EGI Fed-Cloud for Data Analysis - EUDAT Summer School (Giuseppe La ...EUDAT
During this talk, Giuseppe will introduce the EGI Federated Cloud Infrastructure, a federation of private and public clouds, offering a scalable and flexible e-Infrastructure to the European research community. The service is implemented as a hybrid 'Infrastructure as a Service' (IaaS) cloud, composed of multiple clouds that are federated into a scalable compute and storage platform using EGI core infrastructure services. The Federated Cloud serves scientific applications, long-running services and data- and compute-intensive workloads worldwide. The federated cloud also serves as a reference infrastructure for structured scientific communities who want to build their own, cloud federations from partner sites and with open source federation software and standards. The talk and the following demonstration will explain how research workloads can be spread between EGI and EUDAT services, integrating storage, compute and PID solutions from these two network of providers
Visit: https://www.eudat.eu/eudat-summer-school
| www.eudat.eu | The EGI-EUDAT collaboration started in March 2016 with the main goal to harmonise the two e-Infrastructures, including technical interoperability, authentication, authorisation and identity management, policy and operations. The main objective of this work is to provide end-users with a seamless access to an integrated infrastructure offering both EGI and EUDAT services and then, pairing data and high-throughput computing resources together.
To define the roadmap of this collaboration, EGI and EUDAT selected a set of relevant user communities who are already collaborating with both infrastructures. These user communities are able to bring requirements and help assign the right priorities to each of them. In this way, the integration activity has been driven by the end users from the start. The identified user communities are relevant European Research infrastructure in the field of Earth Science (EPOS and ICOS), Bioinformatics (BBMRI and ELIXIR) and Space Physics (EISCAT-3D).
The first outcome of this activity has been the definition of a universal use case that covers the user needs with respect the integration of the two infrastructures previously identified. This use case permits a user of either e-infrastructure to instantiate a VM on the EGI Cloud Federation for the execution of a computational job consuming data preserved onto EUDAT resources. The results of such analysis can be staged back to EUDAT storages, and if needed, allocated with Permanent identifiers (PIDs) for future use. To implement all the steps of this use case the following integration activities between the two infrastructures has to be fulfilled: (1) harmonisation between the authentication and authorisation model, (2) definition and implementation of the interfaces between the involved EGI and EUDAT services.
The first phase of the implementation of this use case has been demonstrated at the EGI Community Forum 2015 (Bari, IT). In addition, two pilot use cases (EPOS and ICOS) have been selected to drive the implementation and validate the results.
RDMkit, a Research Data Management Toolkit. Built by the Community for the ...Carole Goble
https://datascience.nih.gov/news/march-data-sharing-and-reuse-seminar 11 March 2022
Starting in 2023, the US National Institutes of Health (NIH) will require institutes and researchers receiving funding to include a Data Management Plan (DMP) in their grant applications, including the making their data publicly available. Similar mandates are already in place in Europe, for example a DMP is mandatory in Horizon Europe projects involving data.
Policy is one thing - practice is quite another. How do we provide the necessary information, guidance and advice for our bioscientists, researchers, data stewards and project managers? There are numerous repositories and standards. Which is best? What are the challenges at each step of the data lifecycle? How should different types of data? What tools are available? Research Data Management advice is often too general to be useful and specific information is fragmented and hard to find.
ELIXIR, the pan-national European Research Infrastructure for Life Science data, aims to enable research projects to operate “FAIR data first”. ELIXIR supports researchers across their whole RDM lifecycle, navigating the complexity of a data ecosystem that bridges from local cyberinfrastructures to pan-national archives and across bio-domains.
The ELIXIR RDMkit (https://rdmkit.elixir-europe.org (link is external)) is a toolkit built by the biosciences community, for the biosciences community to provide the RDM information they need. It is a framework for advice and best practice for RDM and acts as a hub of RDM information, with links to tool registries, training materials, standards, and databases, and to services that offer deeper knowledge for DMP planning and FAIR-ification practices.
Launched in March 2021, over 120 contributors have provided nearly 100 pages of content and links to more than 300 tools. Content covers the data lifecycle and specialized domains in biology, national considerations and examples of “tool assemblies” developed to support RDM. It has been accessed by over 123 countries, and the top of the access list is … the United States.
The RDMkit is already a recommended resource of the European Commission. The platform, editorial, and contributor methods helped build a specialized sister toolkit for infectious diseases as part of the recently launched BY-COVID project. The toolkit’s platform is the simplest we could manage - built on plain GitHub - and the whole development and contribution approach tailored to be as lightweight and sustainable as possible.
In this talk, Carole and Frederik will present the RDMkit; aims and context, content, community management, how folks can contribute, and our future plans and potential prospects for trans-Atlantic cooperation.
Data policy must be partnered with data practice. Our researchers need to be the best informed in order to meet these new data management and data sharing mandates.
The Ascent of Open Science and the European Open Science CloudTiziana Ferrari
Open science is becoming more and more part of the daily practice in conducting science. Around the world, researchers are increasingly aware of the value and importance of open science. As scientific research becomes highly data-driven and dependent on computing, scientists are conscious of the growing need to share data, software and infrastructure to reduce wasteful duplication and increase economies of scale. In an ideal world, every step of the research process would be public and transparent – the full methodology and all the tools used, as well as the data, would be accessible to the public and all groups without restriction, enabling reproducibility and refinement by other scientists.
This presentation will show case a number of success stories indicating how federated digital infrastructure, that have been sustained by the member states and the European Commission, have become an indispensable tool to enable collaboration ad sharing.
The European Open Science Cloud was launched by the European Commission in 2016 aiming to (1) increase the ability to exploit research data across scientific disciplines and between the public and private sector, (2) interconnect existing and new digital infrastructures in Europe and (3) support open science.
The presentation showcases how open data, open data analytics and open e-Infrastructures like EGI (https://www.egi.eu/) have been key enables of scientific discoveries from the discovery of gravitational waves with LIGO-VIRGO to drug design with the molecular modelling tools of WeNMR.
EOSC-hub (https://www.eosc-hub.eu/) - the first and the largest of the EOSC implementation projects of the H2020 funding programme, has succeeded in delivering some of the building blocks like the EOSC portal and Marketplace, tools and processes for federating data and services providers, harmonized policies, a federated AAI infrastructure, Competence Centres to support research infrastructures in their complex digital needs, interoperability guidelines and the Early Adopter Programme to provide expert support and service capacity to research projects.
Presentation about EGI's Cloud Container Compute Service at the CompBioMed Containerisation Meeting (https://www.compbiomed.eu/events-2/compbiomed-containerisation-meeting/)
Learn SQL from basic queries to Advance queriesmanishkhaire30
Dive into the world of data analysis with our comprehensive guide on mastering SQL! This presentation offers a practical approach to learning SQL, focusing on real-world applications and hands-on practice. Whether you're a beginner or looking to sharpen your skills, this guide provides the tools you need to extract, analyze, and interpret data effectively.
Key Highlights:
Foundations of SQL: Understand the basics of SQL, including data retrieval, filtering, and aggregation.
Advanced Queries: Learn to craft complex queries to uncover deep insights from your data.
Data Trends and Patterns: Discover how to identify and interpret trends and patterns in your datasets.
Practical Examples: Follow step-by-step examples to apply SQL techniques in real-world scenarios.
Actionable Insights: Gain the skills to derive actionable insights that drive informed decision-making.
Join us on this journey to enhance your data analysis capabilities and unlock the full potential of SQL. Perfect for data enthusiasts, analysts, and anyone eager to harness the power of data!
#DataAnalysis #SQL #LearningSQL #DataInsights #DataScience #Analytics
The Building Blocks of QuestDB, a Time Series Databasejavier ramirez
Talk Delivered at Valencia Codes Meetup 2024-06.
Traditionally, databases have treated timestamps just as another data type. However, when performing real-time analytics, timestamps should be first class citizens and we need rich time semantics to get the most out of our data. We also need to deal with ever growing datasets while keeping performant, which is as fun as it sounds.
It is no wonder time-series databases are now more popular than ever before. Join me in this session to learn about the internal architecture and building blocks of QuestDB, an open source time-series database designed for speed. We will also review a history of some of the changes we have gone over the past two years to deal with late and unordered data, non-blocking writes, read-replicas, or faster batch ingestion.
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI
Round table discussion of vector databases, unstructured data, ai, big data, real-time, robots and Milvus.
A lively discussion with NJ Gen AI Meetup Lead, Prasad and Procure.FYI's Co-Found
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...John Andrews
SlideShare Description for "Chatty Kathy - UNC Bootcamp Final Project Presentation"
Title: Chatty Kathy: Enhancing Physical Activity Among Older Adults
Description:
Discover how Chatty Kathy, an innovative project developed at the UNC Bootcamp, aims to tackle the challenge of low physical activity among older adults. Our AI-driven solution uses peer interaction to boost and sustain exercise levels, significantly improving health outcomes. This presentation covers our problem statement, the rationale behind Chatty Kathy, synthetic data and persona creation, model performance metrics, a visual demonstration of the project, and potential future developments. Join us for an insightful Q&A session to explore the potential of this groundbreaking project.
Project Team: Jay Requarth, Jana Avery, John Andrews, Dr. Dick Davis II, Nee Buntoum, Nam Yeongjin & Mat Nicholas
Unleashing the Power of Data_ Choosing a Trusted Analytics Platform.pdfEnterprise Wired
In this guide, we'll explore the key considerations and features to look for when choosing a Trusted analytics platform that meets your organization's needs and delivers actionable intelligence you can trust.
Global Situational Awareness of A.I. and where its headedvikram sood
You can see the future first in San Francisco.
Over the past year, the talk of the town has shifted from $10 billion compute clusters to $100 billion clusters to trillion-dollar clusters. Every six months another zero is added to the boardroom plans. Behind the scenes, there’s a fierce scramble to secure every power contract still available for the rest of the decade, every voltage transformer that can possibly be procured. American big business is gearing up to pour trillions of dollars into a long-unseen mobilization of American industrial might. By the end of the decade, American electricity production will have grown tens of percent; from the shale fields of Pennsylvania to the solar farms of Nevada, hundreds of millions of GPUs will hum.
The AGI race has begun. We are building machines that can think and reason. By 2025/26, these machines will outpace college graduates. By the end of the decade, they will be smarter than you or I; we will have superintelligence, in the true sense of the word. Along the way, national security forces not seen in half a century will be un-leashed, and before long, The Project will be on. If we’re lucky, we’ll be in an all-out race with the CCP; if we’re unlucky, an all-out war.
Everyone is now talking about AI, but few have the faintest glimmer of what is about to hit them. Nvidia analysts still think 2024 might be close to the peak. Mainstream pundits are stuck on the wilful blindness of “it’s just predicting the next word”. They see only hype and business-as-usual; at most they entertain another internet-scale technological change.
Before long, the world will wake up. But right now, there are perhaps a few hundred people, most of them in San Francisco and the AI labs, that have situational awareness. Through whatever peculiar forces of fate, I have found myself amongst them. A few years ago, these people were derided as crazy—but they trusted the trendlines, which allowed them to correctly predict the AI advances of the past few years. Whether these people are also right about the next few years remains to be seen. But these are very smart people—the smartest people I have ever met—and they are the ones building this technology. Perhaps they will be an odd footnote in history, or perhaps they will go down in history like Szilard and Oppenheimer and Teller. If they are seeing the future even close to correctly, we are in for a wild ride.
Let me tell you what we see.
Analysis insight about a Flyball dog competition team's performanceroli9797
Insight of my analysis about a Flyball dog competition team's last year performance. Find more: https://github.com/rolandnagy-ds/flyball_race_analysis/tree/main
Cloud Computing Needs for Earth Observation Data Analysis: EGI and EOSC-hub
1. Björn Backeberg
Senior User Community Support Officer
EGI Founda?on – advanced compu?ng for research
bjorn.backeberg@egi.eu
Japan Geoscience Union 2019, Chiba, Japan
Cloud Computing Needs for Earth Observation Data
Analysis: EGI and EOSC-hub
2. 2
European Open Science Cloud
European Cloud Initiative by
the European Commission (April 2016)
Problem Statement
1. How to maximise the incentives for sharing data and to
increase the capacity to exploit them?
2. How to ensure that data can be used as widely as possible,
across scientific disciplines and between the public and the
private sector?
3. How better to interconnect the existing and the new data
infrastructures across Europe?
4. How best to coordinate the support available to European
data infrastructures as they move towards exascale computing?
“…a trusted, open environment for the scientific community for storing, sharing and re-
using scientific data and results…by 2020…”
4. Vision
100
partners
36
months
33
million
Euro
Researchers from all disciplines
have easy, integrated and open access to the
advanced digital services, scien6fic instruments,
data, knowledge and exper6se they need
to collaborate to achieve excellence
in science, research and innova6on
5. Mission
EGI
Federation EUDAT
INDIGO-
DataCloud
Research
Infrastruct
ures
The EOSC-hub project mobilises providers of
European relevance offering services, software and
data for advanced data-driven research and
innovation.
These resources are offered via the Hub – the
integration and management system of the
European Open Science Cloud, acting as a
European-level entry point for all stakeholders.
6. 6
The project established ‘The Hub’
• Data
• Applications &
tools
• Baseline services
(storage, compute,
connectivity)…
• Training,
consultants
• Marketplace
• AAI
• Accounting
• Monitoring
• …
Usage according to
Rules of
Participation
From the
consortium AND
from external
contributors
• Lightweight
certification of
providers
• SLA negotiation
• Customer Relationship
Management
• …
Based on
FitSM
• Security regula?ons,
• Compliance to standards,
• Terms of use,
• FAIR implementa?on
guidelines
• …
• A system with: Federation & collaboration services; Processes & policies; Business models
& procurement experience; Strategy & Technical Service Roadmap, etc.
• Simplify access to a broad portfolio of products, resources and service provided by the
major pan-European and international organizations through an open and integrated
service catalogue
11 Feb 2019
https://marketplace.eosc-portal.eu
7. 7
Generic services:
Support to the Research Data Lifecycle
Processing & Analysis
Data Management, Curation &
Preservation
Access, DeposiZon & Sharing
Federation Services
● B2FIND (data)
● Marketplace (Services)
● Applications on Demand
● Federated HTC & Cloud Compute IaaS & PaaS
● Processing of sensitive data
● Jupyter Notebook
● Application DB (software & VM)
● B2DROP (data)
● B2Note (data)
● B2SHARE (data)
● DataHub
● Federated AAI. monitoring,
accounting
● SLA and order Management
● Security incident response and
policies
● Technical support & Training
● B2HANDLE
● B2SAFE
● European Certified Trusted
Repository
● Thematic data analytics
● Scientific Workflow Management,
Orchestration (DIRAC, PaaS Orchestrator)
1
2
3
4
Discover & Reuse
11 Feb 2019
9. 9
The Compute Platform: EGI
EGI is a federation of over 200 computing and data centres spread across
Europe and the rest of the world.
47 Countries
12 Integrated
e-Infrastructures
61,000 users
31 large-scale research
collaborations
1,700 Open Access
Publications in 2018
EGI delivers services for computing and data-intensive science for EOSC-hub.
10. 10
The EGI Federation (as of May 2019)
4.4 Billion
CPU core
hours (2018)
> 1 Million
computing
cores in 2019
> 740 PB
disk & tape
2,915 service
end-points
11. 11
2014: EGI Federated Cloud is launched
in Helsinki
Multi-cloud IaaS with Single Sign-On
Federation features:
• Common VM image catalogue
• Discovery, accounting, SLO monitoring
• Unified GUI dashboard
Cloud Compute
Cloud Container
Compute BETA
Training Infrastructure
Online Storage
Applications on
Demand BETA
Notebooks BETA
EGI Services powered by the Cloud Federation
https://marketplace.egi.eu
15. JupyterHub hosted in the EGI Cloud
• Offers Jupyter notebooks ‘as Service’
• One-click solution: login and start
using
Extra EGI Features:
• Login with the EGI AAI Check-In
service
• Persistent storage for notebooks
• Bring your own environments/kernels
• Use EGI computing and storage
resources from your notebooks
15
EGI Notebooks as a Service
16. 16
EGI Notebooks Target Scenario
GitHub
Your
repository
EGI Notebooks
& Binder
services
Zenodo
Your
laptop
Download ipynb file
Create repository
Upload ipynb file
Add requirements.txt
Specify GitHub repo
Generate DOI
Execute
Data repository
Re-execute
Obtain GitHub project reference
Provide GitHub project reference
Discover Notebook
(use DOI)
Fellow
researchers
Journal
paper
DOI
Distributed big
data
DataHub
B2DROP
++
GenerateDOI
17. 17
EGI assets for the implementation of EOSC
Interoperability
guidelines,
standards
Policies,
processes
tools for
federated
service
management
Competence
Centres Training
Compute and
storage facilities
(national and
European
infrastructures)
Validated
Middleware for
federation-level
interoperability
Centrally
managed services
(AAI, workload
orchestration, data
management,
marketplace)
Thematic services
(Data analytics
and scientific
Gateways)
Federating services, activities, know-how
Technical infrastructure
18. EOSC-hub support for Earth Observation R&D
• Bring together services and research products towards
implementing Open Innovation and Open Science
• Harmonize services for accessing information and
knowledge derived from Earth Observations
• Support informed decision making for sustainable
development and policy making
• Provide the large computing and storage capabilities to
support Big Data Analytics
• Host the online Sentinel Long-Term Archive (in
discussion with European Commission)
EOSC Mission for
Earth Observation
Research
18
19. EO Pillar
Thematic
Services
IaaS, EO Finder, Data
Collections Catalogue,
EO Browser
JupyterHub for global
Copernicus data & Data
Catalogue Service
Data Cube
Data
Exploitation
Platform
Earthquake Response and
Landslides Analysis 19
EOSC-hub Thematic Services
Thematic Services are scientific services (incl. data)
that provide discipline-specific capabilities for
researchers.
Examples include:
• Browse and download
data and apps,
• Develop workflows
• Execute workflows,
• Online analytics,
• Visualise results,
• Share results and
publications.
https://marketplace.eosc-portal.eu/
20. A Hybrid Cloud to support exploitation of EO data
End-user access point
EOSC
OpenStack API
powered by libcloud
OpenStack API
powered by libcloud
.EC2
Powered by jclouds
PaaS
Terradue Ellip PaaS
PaaS layer
Develop workflows
interactively
Design scalable
processing chains
Deploy on
processing chains
Connect to custom
Geobrowser
Hybrid IaaS
layer
>18,000 Data Products generated/month
21. • The European Open Science Cloud will provide capacity and capabilities,
fostering the exploitation of research data
• The EGI federated cloud IaaS delivers the compute platform for EOSC
• Generic and (science discipline) thematic services are made discoverable
and accessible through the EOSC Marketplace
• EOSC supports multi-disciplinary science and Earth Observation is one of
the major use cases
• Early Adopter Programme (15 June - deadline) and Webinar (29 May)
http://bit.ly/EOSC-hub_EAP_webinar
21
Conclusions
22. Acknowledgements
• Co-authors & collaborators:
- Yin Chen (EGI/EOSC-hub)
- Tiziana Ferrari (EGI/EOSC-hub)
- Diego Scardaci (EGI/EOSC-hub)
- Gergely Sipos (EGI/EOSC-hub)
- Pedro Gonçalves (Terradue)
- Paolo Mazzeq (CNR-IIA)
- Anabela Oliveira (LNEC)
• EOSC-hub for providing travel support
22
Thank you for your attention!
23. EOSC-hub receives funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 777536.
eosc-hub.eu
@EOSC_eu
23
Back-up slides
24. 24
EGI in numbers
NumberofinstalledCPUs/year
From 300,000 installed CPUs in
2011 to over 1 million in 2019
The EGI Federation offers online
storage (disk) and archive storage
(tape) to meet requirements for Data
Sharing - the capacity in 2019 is 696
Petabytes
25. EGI FedCloud Architecture
25
EGI Federation services:
Accounting, Monitoring, Configuration Database, Information Discovery, VM Marketplace
EGI Check-in
IaaS Federated Access Tools
Community PlatformsAppDB VMOps
Cloud Management
Framework
IaaS API
Cloud Management
Framework
IaaS API
The EGI Federated Cloud is not just AppDB
Providers have their APIs that
can be used with EGI Check-in
accounts, opening the door to
automation of cloud-native
applications.
IaaS Federated Access Tools layer helps
users of the cloud to deal with the
heterogeneity in the IaaS API and EGI
Federation services
EGI Federated Cloud no longer mandates a single API for
every provider. OCCI still widely supported but sites are
moving to native APIs (mainly OpenStack!)
26. 26
EGI FedCloud in action
Since 2015 1.9 million
Virtual Machines consumed
over 10,000 years of
compute time
28. On-demand operational coastal circulation forecast service
28
OPENCoastS Service
Configure Forecast Systems
Manage Forecast Systems
Visualise outputs