Ceph Day Portland Gary Lee, Quanta/QCT http://ceph.com/cephdays/ceph-day-portland/

1. QCT Ceph
Solution – Design
Consideration and
Reference
Architecture
Gary Lee
AVP, QCT

2. • Industry Trend and Customer Needs
• Ceph Architecture
• Technology
• Ceph Reference Architecture and QCT Solution
• Test Result
• QCT/Red Hat Ceph Whitepaper
2
AGENDA

3. 3
Industry Trend
and
Customer Needs

4. 4

5. • Structured Data -> Unstructured/Structured Data
• Data -> Big Data, Fast Data
• Data Processing -> Data Modeling -> Data Science
• IT -> DT
• Monolithic -> Microservice
5
Industry Trend

6. • Scalable Size
• Variable Type
• Longivity Time
• Distributed Location
• Versatile Workload
6
• Affordable Price
• Available Service
• Continuous Innovation
• Consistent Management
• Neutral Vendor
Customer Needs

7. 7
Ceph
Architecture

8. Ceph Storage Cluster
8
Cluster Network
Ceph
Linux
CPU
Memory
SSD
HDD
NIC
Ceph
Linux
CPU
Memory
SSD
HDD
NIC
Ceph
Linux
CPU
Memory
SSD
HDD
NIC
Ceph
Linux
CPU
Memory
SSD
HDD
NIC
Object Block File
Unified
Storage
Scale-out
Cluster
Open
Source
Software
Open
Commodity
Hardware
…..

9. 9
Block
I/O
Ceph
Client
RBD
RADO
SGW
Ceph
FS
Object
I/O
File I/O
RADOS/Cluster Network
OSD
File
System
I/O
Disk I/O
PublicNetwork
End-to-end Data Path
App
Service

10. 10
Ceph Software Architecture

11. Public Network (ex. 10GbE or 40GbE)
Cluster Network (ex. 10GbE or 40GbE)
Ceph Monitor
…...
RCT or RCC
Nx Ceph OSD Nodes
Ceph OSD Node
Clients
Ceph OSD Node Ceph OSD Node
Ceph Hardware Architecture

12. 12
Technology

13. 13
• 2x Intel E5-2600 CPU
• 16x DDR4 Memory
• 12x 3.5” SAS/SATA HDD
• 4x SATA SSD + PCIe M.2
• 1x SATADOM
• 1x 1G/10G NIC
• BMC with 1G NIC
• 1x PCIe x8 Mezz Card
• 1x PCIe x8 SAS Controller
• 1U
QCT Ceph Storage Server
D51PH-1ULH

14. 14
• Mono/Dual Node
• 2x Intel E5-2600 CPU
• 16x DDR4 Memory
• 78x or 2x 35x SSD/HDD
• 1x 1G/10G NIC
• BMC with 1G NIC
• 1x PCIe x8 SAS Controller
• 1x PCIe x8 HHLH Card
• 1x PCIe x16 FHHL Card
• 4U
QCT Ceph Storage Server
T21P-4U

15. 15
• 1x Intel Xeon D SoC CPU
• 4x DDR4 Memory
• 12x SAS/SATA HDD
• 4x SATA SSD
• 2x SATA SSD for OS
• 1x 1G/10G NIC
• BMC with 1G NIC
• 1x PCIe x8 Mezz Card
• 1x PCIe x8 SAS Controller
• 1U
QCT Ceph Storage Server
SD1Q-1ULH

16. • Standalone, without EC
• Standalone, with EC
• Hyper-converged, without EC
• High Core vs. High Frequency
• 1x OSD ~ (0.3-0.5)x Core + 2G RAM
16
CPU/Memory

17. • SSD:
– Journal
– Tier
– File System Cache
– Client Cache
• Journal
– HDD: SSD (SATA/SAS): 4~5
– HDD: NVMe: 12~18
17
SSD/NVMe

18. • 2x NVMe ~40Gb
• 4x NVMe ~100Gb
• 2x SATA SSD ~10Gb
• 1x SAS SSD ~10Gb
• (20~25)x HDD ~10Gb
• ~100x HDD ~40Gb
18
NIC
10G/40G -> 25G/100G

19. • CPU Offload through RDMA/iWARP
• Erasure Coding Offload
• Allocate computing on different silicon areas
19
NIC
I/O Offloading

20. • Object Replication
– 1 Primary + 2 Replica (or more)
– CRUSH Allocation Ruleset
• Erasure Coding
– [k+m], e.g. 4+2, 8+3
– Better Data Efficiency
• k/(k+m) vs. 1/(1+replication)
20
Erasure Coding vs. Replication

21. Size/
Workload
Small Medium Large
Throughput
Transfer Bandwidth
Sequential R/W
Capacity
Cost/capacity
Scalability
IOPS
IOPS/ per 4k Block
Random R/W
Hyper-converged ?
Desktop
Virtualization
Latency
Random R/W
Hadoop ?
21
Workload and Configuration

22. 22
Red Hat Ceph

23. • Intel ISA-L
• Intel SPDK
• Intel CAS
• Mellanox Accelio Library
23
Vendor-specific Value-added Software

24. 24
Ceph Reference
Architecture and
QCT Solution

25. • Trade-off among Technologies
• Scalable in Architecture
• Optimized for Workload
• Affordable as Expected
Design Principle

26. 1. Needs for scale-out storage
2. Target workload
3. Access method
4. Storage capacity
5. Data protection methods
6. Fault domain risk tolerance
26
Design Considerations

27. 27
Transactio
n
Data
Warehous
e
Big
Data
Scientific
Block
Transfe
r
Audio Video
IOPS
MB/sec
OLTP
OLAP
HPC
Streaming
DB
Storage Workload

28. SMALL (500TB*) MEDIUM (>1PB*) LARGE (>2PB*)
Throughput
optimized
QxStor RCT-200
16x D51PH-1ULH (16U)
• 12x 8TB HDDs
• 3x SSDs
• 1x dual port 10GbE
• 3x replica
QxStor RCT-400
6x T21P-4U/Dual (24U)
• 2x 35x 8TB HDDs
• 2x 2x PCIe SSDs
• 2x single port 40GbE
• 3x replica
QxStor RCT-400
11x T21P-4U/Dual (44U)
• 2x 35x 8TB HDDs
• 2x 2x PCIe SSDs
• 2x single port 40GbE
• 3x replica
Cost/Capacity
optimized
IOPS optimized Future direction Future direction NA
* Usable storage capacity
QxStor RCC-400
Nx T21P-4U/Dual
• 2x 35x 8TB HDDs
• 0x SSDs
• 2x dual port 10GbE
• Erasure Coding 4:2
QCT QxStor Red Hat Ceph Storage Edition Portfolio
Workload-driven Integrated Software/Hardware Solution

29. • Densest 1U Ceph building block
• Best reliability with smaller
failure domain
• Scale at high scale 2x 280TB
• At once obtain best throughput
and density
• Block or object storage
• 3x replication
• Video, audio, image repositories, and streaming media
• Highest density 560TB raw
capacity per chassis with greatest
price/performance
• Typically object storage
• Erasure coding common
for maximizing usable capacity
• Object archive
Throughput-Optimized
RCC-400RCT-200 RCT-400
Cost/Capacity-Optimized
USECASE
QCT QxStor Red Hat Ceph Storage Edition
Co-engineered with Red Hat Storage team to provide Optimized Ceph Solution

30. 30
Ceph Solution Deployment
Using QCT QPT Bare Metal Privision Tool

31. 31
Ceph Solution Deployment
Using QCT QPT Bare Metal Privision Tool

32. 32
QCT Solution Value Proposition
• Workload-driven
• Hardware/software pre-validated, pre-optimized and
pre-integrated
• Up and running in minutes
• Balance between production (stable) and innovation
(up-streaming)

33. 33
Test Result

34. Client 1
S2B
Client 2
S2B
Client 3
S2B
Ceph 1
S2PH
Ceph 2
S2PH
Ceph 3
S2PH
Ceph 5
S2PH
Ceph 4
S2PH
Client 8
S2B
Client 9
S2B
Client 10
S2B
10Gb
10Gb
Public Network
Cluster Network
General Configuration
• 5 Ceph nodes (S2PH) with each 2 x 10Gb link.
• 10 Client nodes (S2B) with each 2 x 10Gb link.
• Public network : Balanced bandwidth between Client nodes and Ceph nodes.
• Cluster network : Offload the traffic from public network to improve performance.
Option 1 (w/o SSD)
a. 12 OSD per Ceph storage node
b. S2PH (E5-2660) x2
c. RAM : 128 GB
Option 2 : (w/ SSD)
a. 12 OSD / 3 SSD per Ceph storage node
b. S2PH (E5-2660) x2
c. RAM : 12 (OSD) x 2GB = 24 GB
Testing Configuration (Throughput-Optimized)

35. Client 1
S2S
Client 2
S2S
Client 3
S2S
Ceph 1
S2P
Ceph 2
S2P
Client 6
S2S
Client 7
S2S
Client 8
S2S
10Gb
Public Network
40Gb 40Gb
General Configuration
• 2 Ceph nodes (S2P) with each 2 x 10Gb link.
• 8 Client nodes (S2S) with each 2 x 10Gb link.
• Public network : Balanced bandwidth between Client nodes and Ceph nodes.
• Cluster network : Offload the traffic from public network to improve performance.
Option 1 (w/o SSD)
a. 35 OSD per Ceph storage node
b. S2P (E5-2660) x2
c. RAM : 128 GB
Option 2 : (w/ SSD)
a. 35 OSD / 2 PCI-SSD per Ceph storage node
b. S2P (E5-2660) x2
c. RAM : 128 GB
Testing Configuration (Capacity-Optimized)

36. Level Component Test Suite
Raw I/O Disk FIO
Network I/O Network iperf
Object API I/O librados radosbench
Object I/O RGW Cosbench
Block I/O RBD librbdfio
36
CBT (Ceph Benchmarking Tool)

37. 37
Linear Scale Out

38. 38
Linear Scale Up

39. 39
Price, in terms of Performance

40. 40
Price, in terms of Capacity

41. 41
Protection Scheme

42. 42
Cluster Network

43. 43
QCT/Red Hat
Ceph
Whitepaper

44. 44
http://www.qct.io/account/d
ownload/download?order_
download_id=1022&dtype=
Reference%20Architecture
QCT/Red Hat Ceph Solution Brief

45. https://www.redhat.com/en/
files/resources/st-
performance-sizing-guide-
ceph-qct-inc0347490.pdf
http://www.qct.io/Solution/
Software-Defined-
Infrastructure/Storage-
Virtualization/QCT-and-
Red-Hat-Ceph-Storage-
p365c225c226c230
QCT/Red Hat Ceph Reference Architecture

46. • The Red Hat Ceph Storage Test Drive lab in QCT Solution Center
provides you a free hands-on experience. You'll be able to
explore the features and simplicity of the product in real-time.
• Concepts:
Ceph feature and functional test
• Lab Exercises:
Ceph Basics
Ceph Management - Calamari/CLI
Ceph Object/Block Access
46
QCT Offer TryCeph (Test Drive) Later

47. 47
Remote access
to QCT cloud solution centers
• Easy to test. Anytime and anywhere.
• No facilities and logistic needed
• Configurations
• RCT-200 and newest QCT solutions
QCT Offer TryCeph (Test Drive) Later

48. • Ceph is Open Architecture
• QCT, Red Hat and Intel collaborate to provide
– Workload-driven,
– Pre-integrated,
– Comprehensive-tested and
– Well-optimized solution
• Red Hat – Open Software/Support Pioneer
Intel – Open Silicon/Technology Innovator
QCT – Open System/Solution Provider
• Together We Provide the Best
48
CONCLUSION

49. www.QuantaQCT.com
Thank you!

50. www.QCT.io
QCT CONFIDENTIAL50
Looking for
innovative cloud solution?
Come to QCT,
who else?