The EGI Federation, delivering HTC, HPC and Cloud computing services for scientific communities in Europe and the world, for the first time in 2019 exceeded 71,500 users and delivered more that 5 Billion CPU hours in the last 12 months. Natural Sciences and Medical and Health sciences are the largest user communities.

1. www.egi.eu
@EGI_eInfra
The work of the EGI Foundation
is partly funded by the European Commission
under H2020 Framework Programme
The ascent of scientific computing:
the EGI role and contribution
towards EOSC
IBERGRID 2019
Tiziana Ferrari, EGI Foundation

2. @EGI_eInfrawww.egi.eu 24/09/2019 2
• 15 years of scientific computing in the EGI Federation – figures and trends
• EGI for open science
• EGI and the European Open Science Cloud
Outline

3. @EGI_eInfrawww.egi.eu 24/09/2019 3
EGI is a federation of > 250 computing and data centres spread
across Europe and the rest of the world.
47 Countries
12 Integrated
e-Infrastructures
> 71,500 users
31 large-scale
research
collaborations
3,000 Open Access
Publications in 2018

4. @EGI_eInfrawww.egi.eu 24/09/2019 4
2001: From Design to Implementation
In 2000 the next generation High Energy Physics experiments at CERN
established the design of the first large-scale scientific computing
infrastructure in the Europe.
The EC-funded “DataGrid” project proposal was launched.

5. @EGI_eInfrawww.egi.eu 24/09/2019 5
EGI User Statistics (Sep 2019)
 BBMRI, ELIXIR
 Bioinformatics
 Neuroimaging
 Structural Biology
 Hematology
 Physical Sciences
LHC, BELLE, ICECUBE,
LOFAR, VIRGO, CTA, XENON,
LSST, SKA, KM3NET, ..
 Biological Sciences
LifeWatch
 Earth Science
EISCAT, EMSO
 Chemical sciences

6. @EGI_eInfrawww.egi.eu 24/09/2019 6
Data-centric Scientific Computing
IaaS
providers
Federation Services
Orchestration
Platforms
Check-in : Common AuthN and AuthZ across all layers
Research Platforms
Operators
Research Communities
Research Communities
Federated Access
Federated Applications
Federated Data & Compute

7. @EGI_eInfrawww.egi.eu 24/09/2019 7
Installed Compute Capacity 2011-2019

8. @EGI_eInfrawww.egi.eu 24/09/2019 8
The EGI Federation (Sep 2019)
5.0 Billion
CPU core
wall time /
year
> 1 Million
computing
cores in
2019
> 740 PB
disk & tape
2,915
service end-
points

9. @EGI_eInfrawww.egi.eu 24/09/2019 9
EGI Participants (2019)

10. @EGI_eInfrawww.egi.eu 24/09/2019 10
International Partners
Africa and Arabia
Council for Scientific and
Industrial Research, South Africa
India Centre for
Development of
Advanced Comp.
China
Inst. Of HEP
Chinese Academy
of Sciences
Shanghai Science
and Technology
Innov. Resources
Center
Latin America
Universida de Federal do
Rio de Janeiro
USA
Canada
Asia Pacific Region
Academia Sinica
at Taiwan
> 20 SLAs, 240 Research projects

11. @EGI_eInfrawww.egi.eu 24/09/2019 11
Federated Operations

12. @EGI_eInfrawww.egi.eu 24/09/2019 12
EGI Federation Services
Making
services
discoverable
and
orderable
Making services
accessible
Making
services
robust
Making problems
resolvable
Check-in
Helpdesk
Marketplace
Accounting
Monitoring
Applications
DB
Configuration DB
Manage
federated
assets
Federated data
services &
solutions

13. @EGI_eInfrawww.egi.eu 24/09/2019 13
CPU wall time consumed 2009-2018
+20% annual increase (2018)

14. @EGI_eInfrawww.egi.eu 24/09/2019 14
• Multi-cloud IaaS with Single Sign-On
• Federation features:
 Common VM image catalogue
 Discovery, accounting, SLO monitoring
 Unified GUI dashboard
May 21, 2014: EGI Federated Cloud is launched
Cloud Compute
Cloud Container
Compute BETA Training Infrastructure
Online Storage
Applications on
Demand BETA
Notebooks BETA
EGI Services powered by the Cloud Federation

15. @EGI_eInfrawww.egi.eu 24/09/2019 15
Scientific Disciplines/Cloud
Humanities: CLARIN (2018), DARIAH (+28%, 2017),
Life Sciences: ELIXIR (+104%, 2017)
Environment: EMSO (2018), LifeWatch (+45%, 2015)
BIOISI (+45%, 2016), BIOMED (+575%, 2016), CHIPSTER (+256%, 2016),
GEOHAZARDS (+19%, 2016), OpenCOAST (2018), EXTRAS (2018),

16. @EGI_eInfrawww.egi.eu 24/09/2019 16
Cloud Accounting Jan 2017 – June 2019
The IBERGRID Contribution

17. @EGI_eInfrawww.egi.eu 24/09/2019 17
Scientific Computing for Open Science
Open data, open software, open applications

18. @EGI_eInfrawww.egi.eu 24/09/2019 18
Sharing of data across research groups
LIGO-VIRGO CollaborationLHC Collaboration
3000 open access publications / year

19. @EGI_eInfrawww.egi.eu 24/09/2019 19
Open Data for the UN Sustainable
Development Goals
Innovative Pilots Business Pilots

20. @EGI_eInfrawww.egi.eu 24/09/2019 20
Open software and solutions
• ESRF (France)
• ILL (France)
• ESS (Sweden)
• ELI-DC (Belgium)
• XFEL (Germany)
• CERIC-ERIC (Italy)
• EGI Foundation (Netherlands)
Notebooks
Federated Cloud

21. @EGI_eInfrawww.egi.eu 24/09/2019 21
• Proteins and biomolecules interact in a
complex network
• Glitches in this network can cause diseases
like cancer
• Researchers use 3D models to study how
protein and biomolecules interact
• This structural information is key to
understand the origins of disease and to
develop new drugs
Sharing of scientific applications
Understanding the pathways to disease:
the WeNMR community of practice

22. @EGI_eInfrawww.egi.eu 24/09/2019 22
The WeNMR service portfolio
60
Million
CPU
Hours
+120%
Q1-Q2
2019
250 TB
storage

23. @EGI_eInfrawww.egi.eu 24/09/2019 23
Geo Distribution of the WeNMR Users
https://wenmr.science.uu.nl/user_map

24. @EGI_eInfrawww.egi.eu 24/09/2019 24
EGI and the European Open Science Cloud
Data, applications and computing infrastructure as integrated solution

25. @EGI_eInfrawww.egi.eu 24/09/2019 25
• Growing demand of Cloud access to HTC, GPU and HPC resources
 Machine learning adopted by many applications running in the EGI Federation
 Hybrid provisioning is key
• End-users engaged at the SaaS level
 Easy to use solutions bring together
o Federated AAI
o Federated data discovery and management
o Data analytics capabilities
• Data-centric approach to Cloud
 Zero download access to large research data holdings e.g Copernicus
Towards a data-centric compute
infrastructure

26. 26
Climate (big) data challenges and scientific workflow
Big data from simulations
The volume of worldwide climate
data is expanding rapidly, creating
challenges for both physical
archiving and sharing, as well as for
ease of access and finding what’s
needed, particularly if you are not a
climate scientist.
Scientific workflow
• Still mostly based on the workflow:
search, locate, download, analyze,
visualize
• client-side and sequential nature
of the current approach
• Strong need for server-side
appoaches with advanced parallel
capabilities and no more data
download
Credits: Hannes Thiemann/DKRZ

27. @EGI_eInfrawww.egi.eu 24/09/2019 27
Federating data in a multi-cloud with
the EGI DataHub
JOHN’S SPACES
SENTINEL 2
DEEP LEARNING
PUBLICATIONSSARA’S SPACES
SENTINEL 2
SKY MAPS
MY DATA
A cloud provider:
Physical storage for
files
A space: Organisation of
files exposed to the user
(Directory structure)

28. @EGI_eInfrawww.egi.eu 24/09/2019 28
Example: EGI Notebooks
EGI notebooks
server
Federated files
appear as a
directory structure
for the user

29. @EGI_eInfrawww.egi.eu 24/09/2019 29

30. @EGI_eInfrawww.egi.eu 24/09/2019 30

31. @EGI_eInfrawww.egi.eu 24/09/2019 31
8
Key Exploitable Results and Innovation Roles
Services in the
EOSC service
portfolio
EOSC Portal and
Marketplace
Training Courses
and Materials
Interoperability &
Integration
Guidelines
Digital Innovation
Hub
Rules of
Participation
Service
Management
System
Internal Services in
the Hub Portfolio
Business and
Sustainability
Models
Service
Providers
EOSC Hub
Operators
Researche
rs and
Research
Communit
ies
Enterprise
Education
Training &
Support
EOSC Portal/Marketplace
development and operations
Compute and
Data analytics
HTC & Cloud
IaaS/PaaS/SaaS
Coordination &
Infrastructure
Security policy
coordination
Coordination
Helpdesk/
Accounting/
Monitoring/
AAI
Coordination
Compute and Data Analytics

32. @EGI_eInfrawww.egi.eu 24/09/2019 32
EGI project proposal for INFRAEOSC-07 A1

33. @EGI_eInfrawww.egi.eu 24/09/2019 33
EGI tiered architecture

34. @EGI_eInfrawww.egi.eu 24/09/2019 34
• Scientific computing infrastructures are a necessary instrument of
scientific excellence and open science in Europe
• EGI: from HTC infrastructure, to hybrid HTC & HPC infrastructure
supporting cloud IaaS, PaaS, SaaS access
 Federated identity provisioning and access
 Federated data management towards exabyte scale data processing
• European Open Science Cloud
 Supporting the entire research data lifecycle from production to zero
download data exploitation
Conclusions

35. This work by the EGI Foundation
is licensed under a Creative Commons
Attribution 4.0 International License.
Questions?
Thank you
for your attention.
www.egi.eu
@EGI_eInfra
EGI: Advanced Computing for Research
The work of the EGI Foundation
is partly funded by the European Commission
under H2020 Framework Programme