The document discusses CSC's high performance computing capabilities and developments from 2012-2014. It summarizes that CSC upgraded its Sisu and Taito systems in 2014 with new Intel Haswell CPUs, increasing cores by 50% and reducing energy usage. The upgrades boosted Sisu's performance to 1700 TFlops, making it the 37th most powerful system in the world. CSC now provides a total of 2.54 PFlops of computing power and is the most powerful academic computing facility in the Nordic countries.

1. CSC Computing Services
Olli-Pekka Lehto
Development Manager
Computing Platforms
olli-pekka.lehto@csc.fi
@ople
December 11th 2014

2. CSC Computing Capacity 1989–2014
2

3. CSC Computing Capacity 2012-2014
244
240
1700
180
600
3000
2500
2000
1500
1000
500
0
2012 2013 2014
Bull
Taito
Sisu
Vuori
Louhi
Phase 1 installed
Louhi retired
Phase 2 installed
Vuori retired
3,4x
5,6x
19,2x

4. Total performance:
2.54 PFlop/s
CSC is the most powerful academic
computing facility in the Nordics

5. CSC Computing Services
Performance Capacity Accelerated Cloud Hosting
Sisu
Massive
parallelism
Fast
interconnect
Taito
General use
Large memory
>100
applications
Taito
extension
Visualization
Special codes
Nvidia GPU
Intel Xeon Phi
cPouta
Build your
own
Openstack
IaaS
Kajaani
Espoo
Efficient and
secure
datacenters
Virtual and
physical
servers
Fast parallel storage
Backup Archiving
Storage Services

6. CSC Computing Services
Performance Capacity Accelerated Cloud Hosting
Sisu
40512
cores
1700 TFlops
Taito
18880
cores
600 TFlops
Taito
extension
76 Nvidia
K40 GPU
90 Intel Xeon
Phi 7120X
240 TFlops
cPouta
Dynamically
provisioned
from Taito
Kajaani
Espoo
>4PB, ~ 100GB/s
Storage Services

7. New in 2014: Xeon Haswell E5 CPUs
Intel Xeon E5-2690v3 2,6GHz
– 12 cores/CPU (+50%)
– AVX2 instructions (2x max flops/GHz)
– DDR4 memory
– “Energy-to-solution” at best 1/3 vs.
Sandy Bridge
We are one of earliest adopters
– Sisu upgraded 7/2014
– Taito upgraded 12/2014

8. Sisu Cray XC päivitys
8 new cabinets
Haswell CPUs
More memory per node
>7x performance
~2x energy consumption
40512 cores
108TB memory
680kW
15384kg

9. Sisu – Cray XC40
Designed for computationally intensive tasks
40512 cores, 64GB RAM / node
#37 on the Nov 14 Top500 -list
Aries-interconnect with 7TB/s bisection BW
Cray development tools
Comprehensive set of scalable applications

10. Cray XC blade
4 dual CPU nodes (96 cores)
64GB RAM per node
Aries Router
Net
Power
(500GB/s switching capability)

11. Blade in XC Rack
48 blades
384 CPUs
4608 cores

12. Aries Interconnect Cabling

13. Aries Interconnect Topology
All-to-all network
between groups
2 dimensional
all-to-all network
in a group
Source:
Robert Alverson, Cray
Hot Interconnects 2012 keynote
Optical uplinks to
inter-group net
CSC presentation 13

14. Aries Bisection Bandwidth
9 000 000 x 1,75 x
Average European
consumer IP traffic in 2013
OR
1080p Netflix streams
~7 TB/s
=

15. Taito
HP cluster for general use
– 576 dual-CPU 8 core E5-2670 (Sandy Bridge)
64GB RAM per node
– 400 dual-CPU 12 core E5-2690v3 (Haswell)
128GB RAM per node
New HP Apollo 6000 chassis and blades
– Big memory nodes
10 x 256GB ; 2 x 1,5TB
56Gbit/s FDR InfiniBand –interconnect
Large selection of applications
taito-shell for instant interactive use

16. Taito Extension
Bull DLC 715
– Direct warm-water cooling
Special processors for computing
– 72 Nvidia Tesla K40 GPGPU
– 90 Intel Xeon Phi 7120X
High-performance and energy efficient
– Porting and optimization of applications needed
GPUs can be used for visualization
– For example by using VirtualGL
Connected to the Taito cluster

17. Taito Extension

18. Energy-efficiency of Systems
3.5
3
2.5
2
1.5
1
0.5
0
Vuori Sisu P1 Taito P1 Sisu P2 Taito P2 Bull
GFlop/W

19. Taito ja Sisu user interfaces
Unix shell & X-forwarding
NoMachine NX –virtual desktop
(NX Web-client in beta testing)
Scientists’ User Interface
https://sui.csc.fi

20. cPouta Cloud
IaaS cloud service for HPC
Use cases
– Running HTTP and other servers
– Non-CentOS Linux or Windows OS needed
– Superuser access needed
– Agile service development (“DevOps”)
Tuned for HPC
– Provisioned from Taito nodes
– Powerful CPUs, interconnect, storage
Simple web interface, CLI, REST API
https://research.csc.fi/pouta-iaas-cloud

21. Creating an Instance in cPouta

22. Creating a Virtual Volume in cPouta

23. Storage Services
HPC Storage (~4PB, ~100GB/s)
– Lustre parallel filesystem
– DDN SFA10k and SFA12k storage arrays
– Capacity and performance scalable
Cloud storage (in 2015)
– Ceph filesystem
– Software-defined storage (SDS)
– Block and object storage
Archive (iRODS), tape backup

24. Kajaani Datacenter
30MW power
PUE 1.05-1.2
3000m2 floorspace
99% free cooling

25. Modular Datacenter (MDC):
Easily Expandable, Highly Efficient

26. Interior view of the MDC

27. Cooling Technologies
Water-air hybrid (Sisu)
Air (Taito)
Direct warm-water
(Bull Taito extension)
PUE: 1,2
PUE: 1,05 PUE: 1,02

28. Why CSC?
High-performance, latest technologies
Secure environment (ISO27001:2013)
Finnish, non-profit organization
Ecologically sustainable infrastructure
Competitive and simple pricing model
Excellent network connectivity
Everything under one roof
– Cloud, traditional HPC, visualization, accelerators
– Various storage solutions, EUDAT, RDA
– Consulting, training, porting, optimization

29. ISO27001:2013 certification
Reflects our commitment to security
– Risk management
– Leadership
– Technical solutions and documentation
– Continual improvement
– Recovery planning
– Security as part of company culture
Covers nearly all ICT services and
datacenters
– Cloud services to be certified in early 2015

30. Leveraging Best-of-breed Open
Source Software
Deployment & Clustering
Cloud
Operating Systems
Monitoring
Storage Queuing
Logstash
Graphite

31. Near future development
cPouta improvements, including:
– Oversubscribed instances (WWW-servers etc)
– Docker-support
– ISO27001:2013 certification
New ePouta service
– Productization of Biomedinfra
– Secure computing for organizations
– No direct visibility to Internet (VPN / OPN)
– Possible to extend your local resources
seamlessly
Services for data-intensive computing
– Hadoop/MapReduce optimized systems
– SSD storage

33. Backup slides
Intro to HPC architecture

34. Supercomputers in Olden Days

35. Supercomputers Today
Commodity technologies
– Server clusters
– Linux
– Ethernet, InfiniBand
– x86
Proprietary solutions in very high-end
– BlueGene, Cray, NEC
Cloud services on the rise
– Especially for modest compute needs
Use is constantly spreading to new fields
– Skilled people needed!

36. Basic Supercomputer Architecture
Edustasolmu
Frontend nodes
Interconnect network
Compute nodes
Storage servers
Storage system
Internet
Management network
Hallintasolmut
Management nodes
Management nodes
§