This document discusses lessons learned from operating the AIST Super Green Cloud (ASGC), a fully virtualized high-performance computing (HPC) cloud system. It summarizes key findings from the first six months of operation, including performance evaluations of SR-IOV virtualization and HPC applications. It also outlines conclusions and future work, such as improving data movement efficiency across hybrid cloud environments.

1. Ryousei Takano, Yusuke Tanimura, Akihiko Oota,!
Hiroki Oohashi, Keiichi Yusa, Yoshio Tanaka
National Institute of Advanced Industrial Science and Technology, Japan
ISGC 2015@Taipei, 20 March. 2015
AIST Super Green Cloud: !
Lessons Learned from the Operation and !
the Performance Evaluation of HPC Cloud

2. AIST Super Green Cloud
This talk is about…

3. Introduction
• HPC Cloud is promising HPC platform.
• Virtualization is a key technology.
– Pro: a customized software environment, elasticity, etc
– Con: a large overhead, spoiling I/O performance.
• VMM-bypass I/O technologies, e.g., PCI passthrough and !
SR-IOV, can significantly mitigate the overhead.
• “99% of HPC jobs running on US NSF computing
centers fit into one rack.” -- M. Norman, UCSD
• Current virtualization technologies are feasible
enough for supporting such a scale.

4. LINPACK on ASGC
4
0
10
20
30
40
50
60
0 32 64 96 128
Performance(TFLOPS)
Number of Nodes
Physical Cluster
Virtual Cluster
Performance degradation:!
5.4 - 6.6%
Efficiency* on 128 nodes
Physical cluster: 90%
Virtual cluster: 84%
*) Rmax / Rpeak
IEEE CloudCom 2014

5. Introduction (cont’d)
• HPC Clouds are heading for hybrid-cloud and multi-
cloud systems, where the user can execute their
application anytime and anywhere he/she wants.
• Vision of AIST HPC Cloud:
“Build once, run everywhere”
• AIST Super Green Cloud (ASGC): !
a fully virtualized HPC system

6. Outline
• AIST Super Green Cloud (ASGC) and!
HPC Cloud service
• Lessons learned from the first six months of operation
• Experiments
• Conclusion
6

7. Vision of AIST HPC Cloud!
“Build Once, Run Everywhere”
7
Academic Cloud
Private Cloud
Commercial Cloud
Virtual cluster!
templates
Deploy a Virtual Cluster
Feature 1: Create a customized
virtual cluster easily
Feature 2: Build a virtual
cluster once, and run it
everywhere on clouds

8. Usage Model of AIST Cloud
8
Allow users to
customize their
virtual clusters
0 21
Web appsBigDataHPC
1. Select a template of!
a virtual machine
2. Install required software!
package
VM template files
HPC
+
Ease of use
Virtual cluster
deploy
take snapshots
Launch a virtual
machine when
necessary
3. Save a user-customized!
template in the repository

9. 9
Elastic Virtual Cluster
Cloud
controller
Login node
sgc-tools
Image
repository
Virtual cluster!
template
Cloud
controller
Frontend node
cmp!
node
cmp
node
cmp
node Scale in/!
scale out
NFSdJob scheduler
Virtual Cluster on ASGC
InfiniBand/Ethernet
Image
repository
Import/
export
Create a virtual
cluster
Virtual Cluster!
on Public Cloud
Frontend node
cmp!
node
cmp!
node
Ethernet
Submit a job
Submit a job
In operationUnder development

10. ASGC Hardware Spec.
10
Compute Node
CPU Intel Xeon E5-2680v2/2.8GHz !
(10 core) x 2CPU
Memory 128 GB DDR3-1866
InfiniBand Mellanox ConnectX-3 (FDR)
Ethernet Intel X520-DA2 (10 GbE)
Disk Intel SSD DC S3500 600 GB
• 155 node-cluster consists of Cray H2312 blade server
• The theoretical peak performance is 69.44 TFLOPS
Network switch
InfiniBand Mellanox SX6025
Ethernet Extreme BlackDiamond X8

11. ASGC Software Stack
Management Stack
– CentOS 6.5 (QEMU/KVM 0.12.1.2)
– Apache CloudStack 4.3 + our extensions
• PCI passthrough/SR-IOV support (KVM only)
• sgc-tools: Virtual cluster construction utility
– RADOS cluster storage
HPC Stack (Virtual Cluster)
– Intel Cluster Studio SP1 1.2.144
– Mellanox OFED 2.1
– TORQUE job scheduler 4.2.8
11

12. Storage Architecture
User%a'ached+
storage
x+N
VLAN
Compute+nodes
Compute+network+
(Infiniband+FDR)
Management+network+
(10/1GbE)
x+155
x+155
BDX8
x+155
x+5
VMDI+public+SW
RGWRADOS
VMDI+cluster+SW
x+10
x+10
VMDI+storage
Data+network+(10GbE)
NFS
x+2 x+2
VM
• No shared storage/!
filesystem
• VMDI (Virtual Machine
Disk Image) storage
– RADOS storage cluster
– RADOS gateway
– NFS secondary staging
server
• User-attached storage
primary storage!
(local SSD)
secondary
storage
user data

13. Zabbix Monitoring System
CPU Usage
Power Usage

14. Outline
• AIST Super Green Cloud (ASGC) and!
HPC Cloud service
• Lessons learned from the first six months of operation
– CloudStack on a Supercomputer
– Cloud Service for HPC users
– Utilization
• Experiments
• Conclusion
14

15. Overview of ASGC Operation
• The operation started from July 2014.
• Accounts: 30+
– Main users are material scientists and genome scientists.
• Utilization: < 70%
• 95% of the total usage time is consumed for running
HPC VM instances.
• Hardware failures: 19 (memory, M/B, power supply)

16. CloudStack on Supercomputer
• Supercomputer is not designed for cloud computing.
– Cluster management software is troublesome.
• We can launch a highly productive system in a short
development time by leveraging open source system
software.
• Software maturity of CloudStack
– Our storage architecture is slightly uncommon, that is we
use local SSD disk as primary storage, and S3-compatible
object store as secondary storage.
– We discovered and resolved several serious bugs.

17. Software Maturily
CloudStack Issue Our action Status
cloudstack-agent jsvc gets too large virtual memory space Patch Fixed
listUsageRecords generates NullPointerExceptions for
expunging instances
Patch Fixed
Duplicate usage records when listing large number of records /
Small page sizes return duplicate results
Backporting Fixed
Public key content is overridden by template's meta data when
you create a instance
Bug report Fixed
Migration of aVM with volumes in local storage to another
host in the same cluster is failing
Backporting Fixed
Negative ref_cnt of template(snapshot/volume)_store_ref
results in out-of-range error in MySQL
Patch (not merged) Fixed
[S3] Parallel deployment makes reference count of a cache in
NFS secondary staging store negative(-1)
Patch (not merged) Unresolved
Can't create proper template fromVM on S3 secondary
storage environment
Patch Fixed
Fails to attach a volume (is made from a snapshot) to aVM
with using local storage as primary storage
Bug report Unresolved

18. Cloud service for HPC users
• SaaS is the best if the target application is clear.
• IaaS is quite flexible. However, it is difficult to
manage an HPC environment from scratch for
application users.
• To bridge this gap, sgc-tools is introduced on top of
an IaaS service.
• We believe it works well, although some minor
problems are remained.
• To improve the ability to maintain VM template, the
idea of “Infrastructure as code” can help.

19. Utilization
• Efficient use of limited resources is required.
• A virtual cluster dedicates resources whether the
user fully utilizes them or not.
• sgc-tools do not support queuing at system wide,
therefore, the users need to check the availability.
• Introducing a global scheduler, e.g., Condor VM
universe, can be a solution for this problem.

20. Outline
• AIST Super Green Cloud (ASGC) and!
HPC Cloud service
• Lessons learned from the first six months of operation
• Experiments
– Deployment time
– Performance evaluation of SR-IOV
• Conclusion
20

21. 0 5 10 15 20 25 30 35
0
200
400
600
800
1000
Number of nodes
Deploytime[second]
●
●
●
●
●
● no cache
cache on NFS/SS
cache on local node
Virtual Cluster Deployment
Breakdown (second)
Device attach!
(before OS boot)
90
OS boot 90
FS creation (mkfs) 90
transfer from !
RADOS to SS
transfer from SS to !
local node
VM
VM
RADOS
NFS/SS
Compute!
node
VM

22. Benchmark Programs
Micro benchmark
– Intel Micro Benchmark (IMB) version 3.2.4
• Point-to-point
• Collectives: Allgather, Allreduce, Alltoall, Bcast, Reduce, Barrier
Application-level benchmark
– LAMMPS Molecular Dynamics Simulator version 28 June
2014
• EAM benchmark, 100x100x100 atoms

23. MPI Point-to-point
communication
23
6.00GB/s
5.72GB/s
5.73GB/s
IMB
Message size [Byte]
Bandwidth[MB/sec]
101
102
103
104
105
10610−1
101
102
103
104
BM
PCI passthrough
SR−IOV
The overhead is less than 5% with large message,
though it is up to 30% with small message.

24. Message size [Byte]
Executiontime[microsecond]
101
102
103
104
105
106101
102
103
104
105
BM
PCI passthrough
SR−IOV
MPI Collectives
Time [microsecond]
BM 6.87 (1.00)
PCI passthrough 8.07 (1.17)
SR-IOV 9.36 (1.36)
Message size [Byte]
Executiontime[microsecond]
101
102
103
104
105
106101
102
103
104
105
BM
PCI passthrough
SR−IOV
Message size [Byte]
Executiontime[microsecond]
101
102
103
104
105
106101
102
103
104
105
BM
PCI passthrough
SR−IOV
Message size [Byte]
Executiontime[microsecond]
101
102
103
104
105
106101
102
103
104
BM
PCI passthrough
SR−IOV
Message size [Byte]
Executiontime[microsecond]
101
102
103
104
105
106101
102
103
104
BM
PCI passthrough
SR−IOV
ReuceBcast
AllreduceAllgather Alltoall
The performance of SR-IOV is comparable to
that of PCI passthrough while unexpected
performance degradation is often observed
Barrier

25. LAMMPS: MD simulator
• EAM benchmark:
– Fixed problem size (1M
atoms)
– #proc: 20 - 160
• VCPU pinning reduces
performance fluctuation.
• Performance overhead
of PCI passthrough and
SR-IOV is about 13 %.
20 40 60 80 100 120 140 160
0
10
20
30
40
50
60
Number of processes
Executiontime[second]
●
●
●
●
●
● BM
PCI passthrough
PCI passthrough w/ vCPU pin
SR−IOV
SR−IOV w/ vCPU pin

26. Findings
• The performance of SR-IOV is comparable to that of
PCI passthrough while unexpected performance
degradation is often observed.
• VCPU pinning improves the performance for HPC
applications.

27. Outline
• AIST Super Green Cloud (ASGC) and!
HPC Cloud service
• Lessons learned from the first six months of operation
• Experiments
• Conclusion
27

28. Conclusion and Future work
• ASGC is a fully virtualized HPC system.
• We can launch a highly productive system in a short
development time by leveraging start-of-the-art open
source system software.
– Extension: PCI passthrough/SR-IOV support, sgc-tools
– Bug fixes…
• Future research direction: data movement is key.
– Efficient data management and transfer methods
– Federated identity management
28

29. Question?
Thank you for your attention!
29
Acknowledgments:
This work was partly supported by JSPS KAKENHI Grant Number 24700040.