Living Heart Project:
Using High Performance Computing in the
Cloud to Save Lives
Burak Yenier & Francisco Sahli
HPC Advisory Council – Stanford Conference
Stanford University, February 2018
2012 2013 2014 2016
Start Cloud
Experiments
(200 so far)
2015
HPC Container
Development
CAE software
containers
Online
Marketplace
(45+ stores)
How did we get here?
2017
Hybrid HPC
Living Heart
Project
Intel/UberCloud/LLNL
LLNL multi-container
multi-node project
Lessons We’ve Learned
What we learnt from 200 cloud experiments and 80 case studies
Security
PerformanceUsability
Software
Hybrid HPC Approach
Right-Sized
Infrastructure
1
End User in
Control
3
Enables New
Capabilities
4
Faster Time
to Solution
2
Example: UberCloud ANSYS Container
UberCloud ANSYS Container
Bare Metal Performance
Hybrid Cloud
Micro Focus Cloud Service Automation
Micro Focus Operations Orchestrator
HPE CMUHPC Automated Life Cycle
Management Layer
HPE UberCloud Stack
Hybrid HPC Workflow
• Service Signup
• Contact HPE and sign up for the Hybrid HPC service
• An HPE Partner creates administrator account for Cloud Service
Automation and define service levels, security parameters
• User and Offer Setup
• PointNext creates user accounts for Cloud Service Automation
• PointNext and UberCloud create the various HPC software offers
• Self-Service Hybrid HPC
• Users gain access to on-demand Hybrid HPC offers through Cloud
Service Automation and manage the life cycle.
HPE UberCloud Security Layers
HTTPS/VPN Access*
OS Firewall
OS PKI Login
Network & Storage
Segmentation
Dedicated Servers in
High Security Data
Centers
Single tenant, bare metal servers (not shared between customers). These
servers reside in professionally managed, highly secured data centers.
Network and Storage tiers are segmented automatically. Each customer
gets a private HPE CMU, a private IP range, and a private storage volume
Admin access is protected by Public Key Encryption (vs passwords, which
can be guessed)
Servers are protected with firewalls. Only necessary ports are turned on to
reduce attacks
Connection to our servers are protected by strong encryption techniques such
as HTTPS and VPN (HTTPS/VPN access and Disk Encryption are optional)
Case Study: The Living Heart Project
Studying Drug-induced Arrhythmias of a
Human Heart with Abaqus 2017 in the Cloud
The Living Heart Project
•Leading cardiovascular researchers, educators, medical
device developers, regulatory agencies, and practicing
cardiologists
•Shared mission to develop and validate highly accurate
personalized digital human heart models (DHHM).
•Models establish a unified foundation for cardiovascular in
silico medicine
•Models serve as a common technology base for
education and training, medical device design, testing,
clinical diagnosis and regulatory science
•Rapidly translating current and future cutting-edge
innovations directly into improved patient care.
Motivation
Cisapride in
the market Stockbridge et. al (2013)
New regulation
introduced
• Gastrointestinal purposes
• 80 fatalities associated with
Cisapride-induced arrhythmias
• The average cost to develop a new
drug is $2.5 billion
• Only 12% of all the drugs that make
it into a clinical trial will eventually get
approval
• The time to market for a new drug is
more than 10 years
• Many drugs, not just cardiac drugs
have serious side effects on the
heart, which could result in sudden
cardiac death
• 300,000 sudden cardiac deaths
occur each year in the US, but it is
unclear how many of those are
caused by drugs
Multi-scale model to simulate drug-induced arrhythmias
~7,000,000 elements, 250,000,000 internal variables, 1,000,000 time steps
HPCaaS Environment and Simulations
Advania / HPE / Intel / UberCloud
• Advania’s HPC as a Service (HPCaaS)
hardware configuration
• Built upon HPE ProLiant servers XL230 Gen9
• Each with 2x Intel Broadwell E5-2683 v4 with
Intel OmniPath interconnect
• UberCloud HPC software containers hosting
Stanford’s workflow
• Dassault SIMULIA for advanced electro-
physiological modelling
• LHP model scaled well up to 240 compute cores
• 42 simulations each 40 hours on 5-node (160-
core) subsystem
0
5
10
15
20
25
0
50
100
150
200
250
300
0 50 100 150 200 250 300 350 400 450 500
speedup
walltime[hours]
number of CPUs
Scalability of drug simulations using Abaqus 2017
wall time (hours) linear speed up speed up
Sahli Costabal F, Yao J, Kuhl E. Predicting drug-induced arrhythmias by
multiscale modeling. Int J Num Meth Biomed Eng. 2018; doi:10.1002/cnm.2964
Multi-scale model - baseline
Multi-scale model - low risk drug
Multi-scale model - high risk drug
Take Aways
• UberCloud is part of the HPE’s HPCaaS
offering “Hybrid HPC”
• UberCloud Containers give us a way to
solve software management problems
without performance issues
• Able to manage and run the most complex
engineering workflows
• Providing SaaS-like user experience and
desktop level ease of use

Living Heart Project: Using High Performance Computing in the Cloud to Save Lives

  • 1.
    Living Heart Project: UsingHigh Performance Computing in the Cloud to Save Lives Burak Yenier & Francisco Sahli HPC Advisory Council – Stanford Conference Stanford University, February 2018
  • 3.
    2012 2013 20142016 Start Cloud Experiments (200 so far) 2015 HPC Container Development CAE software containers Online Marketplace (45+ stores) How did we get here? 2017 Hybrid HPC Living Heart Project Intel/UberCloud/LLNL LLNL multi-container multi-node project
  • 4.
    Lessons We’ve Learned Whatwe learnt from 200 cloud experiments and 80 case studies Security PerformanceUsability Software
  • 5.
    Hybrid HPC Approach Right-Sized Infrastructure 1 EndUser in Control 3 Enables New Capabilities 4 Faster Time to Solution 2
  • 6.
  • 7.
  • 8.
  • 9.
    Hybrid Cloud Micro FocusCloud Service Automation Micro Focus Operations Orchestrator HPE CMUHPC Automated Life Cycle Management Layer HPE UberCloud Stack
  • 10.
    Hybrid HPC Workflow •Service Signup • Contact HPE and sign up for the Hybrid HPC service • An HPE Partner creates administrator account for Cloud Service Automation and define service levels, security parameters • User and Offer Setup • PointNext creates user accounts for Cloud Service Automation • PointNext and UberCloud create the various HPC software offers • Self-Service Hybrid HPC • Users gain access to on-demand Hybrid HPC offers through Cloud Service Automation and manage the life cycle.
  • 11.
    HPE UberCloud SecurityLayers HTTPS/VPN Access* OS Firewall OS PKI Login Network & Storage Segmentation Dedicated Servers in High Security Data Centers Single tenant, bare metal servers (not shared between customers). These servers reside in professionally managed, highly secured data centers. Network and Storage tiers are segmented automatically. Each customer gets a private HPE CMU, a private IP range, and a private storage volume Admin access is protected by Public Key Encryption (vs passwords, which can be guessed) Servers are protected with firewalls. Only necessary ports are turned on to reduce attacks Connection to our servers are protected by strong encryption techniques such as HTTPS and VPN (HTTPS/VPN access and Disk Encryption are optional)
  • 12.
    Case Study: TheLiving Heart Project Studying Drug-induced Arrhythmias of a Human Heart with Abaqus 2017 in the Cloud
  • 13.
    The Living HeartProject •Leading cardiovascular researchers, educators, medical device developers, regulatory agencies, and practicing cardiologists •Shared mission to develop and validate highly accurate personalized digital human heart models (DHHM). •Models establish a unified foundation for cardiovascular in silico medicine •Models serve as a common technology base for education and training, medical device design, testing, clinical diagnosis and regulatory science •Rapidly translating current and future cutting-edge innovations directly into improved patient care.
  • 14.
    Motivation Cisapride in the marketStockbridge et. al (2013) New regulation introduced • Gastrointestinal purposes • 80 fatalities associated with Cisapride-induced arrhythmias • The average cost to develop a new drug is $2.5 billion • Only 12% of all the drugs that make it into a clinical trial will eventually get approval • The time to market for a new drug is more than 10 years • Many drugs, not just cardiac drugs have serious side effects on the heart, which could result in sudden cardiac death • 300,000 sudden cardiac deaths occur each year in the US, but it is unclear how many of those are caused by drugs
  • 15.
    Multi-scale model tosimulate drug-induced arrhythmias ~7,000,000 elements, 250,000,000 internal variables, 1,000,000 time steps
  • 16.
    HPCaaS Environment andSimulations Advania / HPE / Intel / UberCloud • Advania’s HPC as a Service (HPCaaS) hardware configuration • Built upon HPE ProLiant servers XL230 Gen9 • Each with 2x Intel Broadwell E5-2683 v4 with Intel OmniPath interconnect • UberCloud HPC software containers hosting Stanford’s workflow • Dassault SIMULIA for advanced electro- physiological modelling • LHP model scaled well up to 240 compute cores • 42 simulations each 40 hours on 5-node (160- core) subsystem 0 5 10 15 20 25 0 50 100 150 200 250 300 0 50 100 150 200 250 300 350 400 450 500 speedup walltime[hours] number of CPUs Scalability of drug simulations using Abaqus 2017 wall time (hours) linear speed up speed up Sahli Costabal F, Yao J, Kuhl E. Predicting drug-induced arrhythmias by multiscale modeling. Int J Num Meth Biomed Eng. 2018; doi:10.1002/cnm.2964
  • 17.
  • 18.
    Multi-scale model -low risk drug
  • 19.
    Multi-scale model -high risk drug
  • 20.
    Take Aways • UberCloudis part of the HPE’s HPCaaS offering “Hybrid HPC” • UberCloud Containers give us a way to solve software management problems without performance issues • Able to manage and run the most complex engineering workflows • Providing SaaS-like user experience and desktop level ease of use

Editor's Notes

  • #4 UberCloud history and how we got here Realized that many issues need to be solved to get cloud ready for HPC Conducted lots of experiments. Intel sponsorship. Based on the learning, started work on building HPC containers in 2013 Launched the online marketplace in 2014 By 2016 have lots of CAE codes containerized and ready to deploy on public and private cloud. HPE sponsorship in 2016/2017 The HPE Cloud CoE in Iceland is live. A similar cloud CoE is under development in Bengaluru and uses UberCloud container technology
  • #5 Lessons Learned After 190 cloud experiments, these are the most commonly asked questions Security – how does my data stay secure after it leaves my data center? Performance – what sort of hardware is this? What interconnects? Data transfer? network latency? Software – If I have node locked licenses of my simulation software, how can I use this cloud system Usability - how can I switch seamlessly between my workstation, servers, private cloud, and between public cloud? Do I need to learn anything new? Data management? Application Container Approach Containers were originally developed for the micro services use case (which is a design pattern for developing web facing applications) Based on Docker, enhanced for engineering & scientific application software such as ANSYS Software packages designed to deliver the tools that an engineer needs Our technology, allow us to launch SaaS offers from otherwise traditional technical computing software. We mastered the art and science of getting high performance out of Cloud services and removed barriers such as remote desktop access, licensing, data security and monitoring. Ready to execute, in an instant. No need to install software, deal with complex OS commands, or configure. Software is pre-installed, configured, and tested, and running on bare metal, without loss of performance. Security – Layered approach. Encrypted connections, server FW, PKI, data encryption, single tenant. Automated monitoring Performance – bare metal equivalent. Specialized hardware. MPI. Dedicated infrastructure Software – dozens of titles to choose from. Usability – desktop look and feel. Nothing to install or configure. Production ready. solutions for remote NICE DCV and VCollab. Data – dropbox. Pre and post processing in the cloud
  • #6 UberCloud Approach to HPC as a Service Right sized Infrastructure - Fitting the infrastructure to the end users needs. We are not in the h/w business, we rely on our partners like HPE and Advania for the h/w. We support hybrid and public cloud deployments. Faster Time to Solution – dramatically lower turnaround time. Encourages experimentation. Examples – 128 core cluster in 1 day. Puts the user in control – We put user experience – that of the end user, the engineer - in the center. Usability of the service, portability of the engineers code and data are our strong point. An already large and growing ecosystem of software partners is our strength. Enables new use cases – Ability to do new methods. For workstation users - DOE, Optimization. For HPC cluster users - collaboration and access. Short term access to the software.
  • #10 Putting it all together This is the cloud stack that HPE is building to deliver hybrid HPC to customers. HPC as a Service needs to be thought of as part of an overall computation approach which includes various models. There are a number of factors to be considered including data management, orchestration, provisioning, workflow automation etc. Applications partners ISVs are critical to this and a large number are already partnering at different levels : ANSYS, STAR-CCM+ from CD-adapco, Siemens PLM, Simulia, Opensource CFD tools such as OpenFOAM, Univa GE for workload management; HPE Software Suite Cloud orchestration : special version of HPE’s Cloud Service Automation. Integrates the HPC specifics. Its flexible enough to integrate HPC workload managers, cluster managers, portals and applications ; the provisioning engine HP OO is the interface with the HPC world ; we can also re use workflows already developed to tap in the type other resources like public clouds - Manage Legacy, Private Cloud and Public Cloud