This document summarizes a presentation on scaling academic clouds with Ceph. It discusses the software-defined datacenter and how Ceph can manage cloud resources and work with OpenStack. Two university Ceph deployments are described: a collaboration between 4 UK universities using Ceph for a microbial bioinformatics cloud, with 6.9PB of total capacity across the sites; and a deployment at the University of Zurich with 4.2PB of high-capacity tier storage and 112TB of high-performance SSD storage to provide block access for scientific applications. The presentation covers considerations for infrastructure like storage nodes, networks, and datacenters, and references a Dell/Inktank Ceph reference architecture providing hardware, software, and services.

1. Scaling an Academic Cloud
with Ceph
28.04.2015 | Berlin, Germany
Ceph Day Berlin
Christian Spindeldreher
Enterprise Technologist
Dell EMEA

2. The Cloud
2

3. The
Software-Defined
Datacenter
3

4. Defining “software-defined”
The capabilities
• Compute
• Storage/availability
• Networking/
security & management
The benefits
• Automated &
simplified
• Unlimited agility
• Maximum efficiency
SDN
SDS
SDC
SDE
4
Data
plane
Control
plane
Traditional
system
Purpose-built
hardware & software
General-purpose hardware
Software-
defined
Open standard,
e.g., OpenFlow
Next-gen compute block
Purpose-built function virtualized
in general-purpose hardware
delivered as a service
The basics

5. 5
The Cloud Operating System
Manage the Resources…

6. 6
Ceph and OpenStack

7. Ceph in
Academia & Research
7

8. CLIMB project
8
picture from http://westcampus.yale.edu
• Collaboration between 4 Universities:
Birmingham, Cardiff, Swansea & Warwick
• Ceph environment across the 4 sites
– part of a HPC Cloud to deploy virtual
resources for microbial bioinformatics
(e.g. DNA sequencer output,…)
– shared data across the sites
– robust solution with low €/TB ratio for
mid/long term storage
– Ceph Solution by OCF, Inktank* & Dell
– more information:
http://www.climb.ac.uk
* now Red Hat

9. CLIMB project
• 4 Ceph Clusters
– 6.9PB raw capacity (total)
– 3 replicas – at least 1 remote:
2.3PB useable capacity
– server infrastructure (per site)
› 5 MON nodes
› 2 Gateway nodes
– R420, 4x 10GbE
› 27 OSD nodes
– R730xd, 16x 4TB, 2 SSDs, 2x 10GbE
– network infrastructure
› Brocade VDX6740T switches
– 48x 10GbE, 4x 40GbE
9

10. S3IT − Central IT, University of Zurich (UZH)
• UZH – some interesting facts
– 26.000 enrolled students – Switzlerland‘s
largest university
– member of the “League European Research
Universities” (LERU)
– international renown in medicine, immunology,
genetics, neuroscience, structural biology,
economics,…
› 12 UZH scholars have been awarded the Nobel Prize
• Scale-Out Storage for Scientific Cloud (based on OpenStack)
– based on Ceph
– commodity components
– ethernet network
– good balance between performance, capacity & cost
10
picture: http://www.hausarztmedizin.uzh.ch/index.html

11. S3IT − Central IT, University of Zurich (UZH)
• Requirements for High-Capacity Tier
– 4.2PB raw capacity (1st batch)
› cinder volumes, glance images, ephemeral disks of VMs,
radosgw (S3-like object storage)
› replication, erasure coding & cache tiering
– R630 + 2x MD1400 JBOD
› 24x 4TB nSAS
› 6x 800GB SSD (in R630)
• Requirements for High-Performance Tier
– 112TB raw capacity (1st batch)
› block access
› SSD pool, replicated
– R630
› 8x 1.6TB SSD
• Network
– scale-out 40GbE back-bone:
2x Z9500 (132x 40GbE in 3RU)
– ToR: S4810 (48x 10GbE, 4x 40GbE)
11

12. Requirements in Academia, Science & Research today
What we see…
• Ceph Stand-Alone vs. OpenStack-related
• Large Scale Environments
– 5PB / 20PB / 100PB target capacity
– usually object
• Multi-Site Environments
– cross-site replication
– unified object space
– searchable meta data
› out-of-scope for Ceph?!
12

13. Design Considerations
13

14. Infrastructure Considerations – Storage Nodes
• Form Factors
– Small Nodes vs. Big Nodes
vs. Super-Nodes
– Node Count
– Ethernet-based Drives
• Use of SSDs
– Journaling
– Cache Tiering
– SSD-only Pools
– Check new SSD Types
› PCIe, form factors (1.8“ size),
write endurance,…
14

15. Infrastructure Considerations – Storage Node Example
• Storage Node: R730xd
– 2 RU
– 1 or 2 CPUs
– local drives
› 16x 3.5“ HDD slots (+ 2x 2.5“ for boot)
– up to 6TB per drive today (96TB total)
› 24x 2.5“ HDD slots (+ 2x 2.5“ for boot)
› 8x 3.5“ HDD slots + 18x 1.8“ SSDs
(+ 2x 2.5“ for boot)
– highly flexible system
– JBOD expansion optional
15

16. Infrastructure Considerations – Storage Node Example
• Head Node: R630
– 1 RU
– 1 or 2 CPUs
– local drives
› 10x 2.5“ HDD slots or
› 24x 1.8“ SSDs
› could host Write Journaling, Cache Tiering or
SSD-only pools (then without a JBOD)
• JBOD: MD3060e
– 4 RUs
– SAS attach
– 60x 3.5“ HDD slost
› up to 6TB per drive today (360TB total)
• VoC (example)
– “Write Journal on SSD has no real impact
with 60 HDDs“
16
SAS

17. Infrastructure Considerations – Network
• Client-facing vs. Cluster-internal IO
– be aware of replication traffic
• ToR
– 1x or 2x 10GbE Switch
› failure domain?!
– 40GbE Uplinks
• Distributed Core
– Scale-Out Core-Switch Design
– 40/50/100GbE Mesh
– Virtual Link Trunking (VLT) for HA/Load-
Balancing
17

18. Infrastructure Considerations – the Site/DC…
• Power & Cooling
– high density has some impacts
– example for 1 rack (42 RUs)
› R630 & MD3060e building block / 8 units
› input power:
› weight:
› raw capacity:
• Fresh Air Technology
– use higher air temperature for cooling
– 25°C vs. 30°C vs. 40°C
18
High Density: TACC Stampede Cluster
› 21kW
› ~ 1000kg
› 2.9PB
Dell Fresh Air Hot House,
Round Rock TX

19. 19
Dell|Inktank (now RH) Ceph Reference Architecture
HW + SW + Services
Hardware
HW Reference
Architecture
• R730xd Servers
• Storage and compute
• Dell S/Z-Series Switches
Configuration
• Min of 6 nodes:
3x MON + 3x Data
Software
Software
• Inktank ICE platform
• optional OpenStack cloud
software
Operating System
• RHEL
• SUSE, Ubuntu,…
Access • Object & Block (today)
Services
Deployment
• Onsite HW Install
• Onsite SW Install
• Whiteboard session & training
Support
• HW: Dell ProSupport
• SW: OpenStack support
Solution based on (e.g.):
• Server nodes:
• R730xd,…
• Fully populated drives
• Dell F10 10/40GbE switches
• Modules are flexible

20. Dell Solution Centers
• 30-90 minute briefings
• 1-4 hour Design Workshops
• 5-10 days Proofs-of-Concept for
hands-on “prove-it”
20

21. Thank You!
Christian_Spindeldreher@Dell.com