EuroHPC Joint Undertaking. Accelerating the convergence between Big Data and High performance Computing Jean Pierre Panziera (Chairman, EuroHPC)

1. EuroHPC JointUndertaking
Acceleratingthe convergencebetweenBig
Data andHigh performanceComputing
Jean-Pierre Panziera
ETP4HPC chair

2. ETP4HPC: Who are we?
We are an
industry-led
think-tank
promoting
European HPC
research and
innovation to
support Europe’s
competitiveness
2
SABRI
PLLANA

3. SME
35%
Association
3%
Research
Organisation
41%
Large company
20%
Individual
1%
Organisation type
Our members
Our members have
diverse profiles:
● providers
● users
● research
● academics
● industry
● large companies
● SMEs…
ETP4HPC is open to new
members sharing its
vision and objectives
3

4. Weather Forecast HPC and (big) data
● weather forecast is the typical HPC application
● large computing resources
● + large datasets assimilation
“When it comes to events like Hurricane Dorian,
accurately predicting the weather can save lives.
But the US' system has lagged behind Europe's for
years”
… “data assimilation is the crucial reason why
ECMWF has better forecasts.”
https://www.wired.co.uk/article/hurricane-dorian-weather-predictions-us-europe
4
best forecast
40 Million observations/day, from 85 satellites (50 instruments)
+ many ground-based and airborne measurement systems.

5. HPC, Big Data and Deep Learningstacks
5
HPC Big Data Deep Learning
Infiniband &
OPA fabrics
Storage & I/O
nodes, NAS
GP* CPU nodes,
GPUs, FPGAs
Linux OS Variant
Containers
PFS
(Lustre etc.)
MPI
OpenMP,
threading
Accelerator
APIs
Numerical
libraries
Performance
& debugging
Domain-specific libraries
Conventional compiled
languages (C, C++, FORTRAN)
Scripting
(Python, …)
IDEs & Frameworks
(PETSc, …)
Compiled in-house, commercial & OSS applications
Cluster management
(OpenHPC)
Batch scheduling
(SLURM …)
Linux OS Variant (some Windows)
Ethernet
fabrics
Local
storage
GP* CPU hyper-
convergent nodes
Virtualization: hypervisor or containers
(Dockers, Kubernetes, …)
VMM and container management
I/O libraries
(HDF5, …)
Orchestration and RMS
Cloud service I/F
Storage systems
(DFS, Key/value, …)
Map-Reduce Processing
(Hadoop, Spark)
Data stream processing
(Storm, …)
Distributed coordination
(Zookeeper, …)
Workflows combining many application elements
Compiled languages
(C++)
Traditional ML
(Mahout)
Scripting & WF languages
(R, Python, Java, Scala, …)
Linux OS Variant (Windows?)
Ethernet‡ GP* CPU +
GPU/FPGA, TPU
Local storage or
NAS/SAN
Virtualization: hypervisor or containers
(Dockers, Kubernetes, …)
VMM and container management
Orchestration and RMS
Neural network frameworks
(Caffe, Torch, Theano, … )
Load distribution layer
Scripting languages
(Python, …)
Inference engines
(low precision)
Defined and instantiated/trained neural networks
Can be part of
Applications
Middleware
& Mgmt.
System
SW
Hardware
* GP: general purpose
User-space
fabric access
Direct
accelerator
use
Numerical libraries (dense
LA)
Accelerator APIs
Cloud service I/F
Storage systems
(DFS, Key/value, …)
Red boxes: data components ‡ need for faster fabrics for training scale-out

6. HPC and Big Data testbeds (Horizon2020/ICT-11)
6
Fostering precision agriculture and livestock
farming through secure access to large-scale
HPC-enabled virtual industrial experiment-
ation environment empowering scalable big
data analytics
Deep-Learning and HPC to Boost
Biomedical Applications for Health
HPC and Cloud-enhanced Testbed for
Extracting Value from Diverse Data at
Large Scale
Large-scale EXecution for Industry &
Society

7. The European HPC ecosystemis based on 3 pillars
Excelling in HPC
applications, and
widening HPC use
Securing our own
independent HPC
system supply
Acquiring
leadership-class
supercomputers
Technologies
7

8. The European HPC ecosystemis based on 3 pillars
Excelling in HPC
applications, and
widening HPC use
Securing our own
independent HPC
system supply
Acquiring
leadership-class
supercomputers
Technologies
8

9. 9

10. ETP4HPC and BDVAare the privatemembersofEuroHPC
● ETP4HPC and BDVA are part of the Research and Innovation
Advisory Group (RIAG)
● RIAG’s recommendations are the basis for the Work Programme
definition
10

11. HPC and data centricenvironmentsand applicationplatforms
● Strategic Research and Innovation Agenda 2019 (05/04/2019)
https://eurohpc-ju.europa.eu/documents/EuroHPC_RIAG_Strategic_Agenda_2019.pdf
Recommendation 2: HPC and data-driven application-oriented platforms
High Performance Computing (HPC) and data-driven application-oriented platforms should be developed, driven by complex application
workflows (for instance, High Performance Data Analytics (HPDA), combining artificial intelligence and simulation modelling, exploiting
underlying hardware architectures’ heterogeneity/modularity, integrating cloud-based solutions etc.) and offer solutions to key application
areas, including industrial use cases. The use of HPC solutions to generate innovation and value creation should be clearly demonstrated (for
instance in manufacturing, farming, health, mobility, natural hazards, climate or cybersecurity), aiming at providing secure and simple access
and service provisioning to relevant stakeholders based on such HPC solutions. The focus should be on the development of energy-efficient
HPC solutions supporting the adoption of applications with industrial and societal relevance on evolving HPC hardware and system
software/programming environments. It should include co-design in close cooperation with the scientific disciplines to explore and
demonstrate the technical feasibility and value of advanced workflows, e.g. mixed/integrated simulation, HPDA & AI, and ensuring wide
adoption in production use.
● EuroHPC-02-2019: HPC and data centric environments and application platforms
https://eurohpc-ju.europa.eu/documents/EuroHPC_Work_Plan_2019_12july.pdf
≥ 4x projects, each with 4M€ EU funding + 4M€ from participating states.
11

12. HPC, Big Data andAI
● HPC, Big Data and AI convergence in underway
… and is here to stay beyond Research projects and PoCs
● Big Data technologies → HPC
● AI frameworks → HPC
● Digital continuum Edge/Cloud/Datacenter
● multi-disciplinary HPC/BigData/AI → trans-disciplinary HPC-BigData-AI
ETP4HPC12