This document discusses reference architectures for Ceph storage solutions. It provides guidance on key design considerations for Ceph clusters, including workload profiling, storage access methods, capacity planning, fault tolerance, and data protection schemes. Example hardware configurations are also presented for different performance and cost optimization targets.

1. Reference
Architectures:
Architec/ng
Ceph
Storage
Solu/ons
Brent
Compton
Director
Storage
Solu/ons,
Red
Hat
Kyle
Bader
Senior
Solu/on
Architect,
Red
Hat

2. RefArch
Building
Blocks
Servers
and
Media
(HDD,
SSD,
PCIe)
Network
Bare
metal
OpenStack
virt
Container
virt
Other
virt
Defined
Workloads
OS/Virt
PlaMorm
Network
Ceph

3. RefArch
Flavors
• How-­‐to
integra/on
guides
(Ceph+OS/Virt,
or
Ceph+OS/Virt+Workloads)
hQp://www.dell.com/learn/us/en/04/shared-­‐content~data-­‐sheets~en/documents~dell-­‐red-­‐hat-­‐cloud-­‐
solu/ons.pdf
• Performance
and
sizing
guides
(Network+Server+Ceph+[OS/Virt])
hQp://www.redhat.com/en/resources/red-­‐hat-­‐ceph-­‐storage-­‐clusters-­‐supermicro-­‐storage-­‐servers
hQps://www.redhat.com/en/resources/cisco-­‐ucs-­‐c3160-­‐rack-­‐server-­‐red-­‐hat-­‐ceph-­‐storage
hQps://www.scalableinforma/cs.com/assets/documents/Unison-­‐Ceph-­‐Performance.pdf

4. 1. Qualify
need
for
scale-­‐out
storage
2. Design
for
target
workload
IO
profile(s)
3. Choose
storage
access
method(s)
4. Iden/fy
capacity
5. Determine
fault-­‐domain
risk
tolerance
6. Select
data
protec/on
method
• Target
server
and
network
hardware
architecture
(performance
and
sizing)
Design
Considera/ons

5. 1.
Qualify
Need
for
Scale-­‐out
• Elas/c
provisioning
across
storage
server
cluster
• Data
HA
across
‘islands’
of
scale-­‐up
storage
servers
• Standardized
servers
and
networking
• Performance
and
capacity
scaled
independently
• Incremental
v.
forklie
upgrades

7. PAST:
SCALE
UP
FUTURE:
SCALE
OUT
Designed
for
Agility

8. 2.
Design
for
Workload
IO
• Performance
v.
‘cheap-­‐and-­‐deep’?
• Performance:
throughput
v.
IOPS
intensive?
• Sequen/al
v.
random?
• Small
block
v.
large
block?
• Read
v.
write
mix?
• Latency:
absolute
v.
consistent
targets?
• Sync
v.
async?

9. Generalized
Workload
IO
Categories
IOPS
Op/mized
Throughput
Op/mized
Cost-­‐Capacity
Op/mized

10. • Highest
performance
(MB/sec
or
IOPS)
• CapEx:
Lowest
$/performance-­‐unit
• OpEx:
Highest
performance/BTU
• OpEx:
Highest
performance/waQ
• Meets
minimum
server-­‐fault
domain
recommenda)on
(1
server
<=
10%
cluster)
Performance-­‐Op/mized

11. • CapEx:
Lowest
$/TB
• OpEx:
Lowest
BTU/TB
• OpEx:
Lowest
waQ/TB
• OpEx:
Highest
TB/Rack-­‐unit
• Meets
minimum
server-­‐fault
domain
recommenda)on
(1
server
<=
15%
cluster)
Cost/Capacity-­‐Op/mized

12. 3.
Storage
Access
Methods
distributed
file*
object
block**
software-defined storage cluster
*CephFS
not
yet
supported
by
RHCS
**
RBD
supported
with
replicated
data
protec/on

13. 4.
Iden/fy
Capacity
OpenStack
Starter
100TB
S
500TB
M
1PTB
L
2PB+

14. 5.
Fault-­‐Domain
Risk
Tolerance
• What
%
of
cluster
capacity
does
you
want
on
a
single
node?
When
a
server
fails:
• More
workload
performance
impairment
during
backfill/recovery
with
fewer
nodes
in
the
cluster
(each
node
has
greater
%
of
its
compute/IO
u/liza/on
devoted
to
recovery).
• Larger
%
of
cluster’s
reserve
storage
capacity
u/lized
during
backfill/recovery
with
fewer
nodes
in
the
cluster
(must
reserve
larger
%
of
capacity
for
recovery
with
fewer
nodes).
• Guidelines:
– Minimum
supported
(RHCS):
3
OSD
nodes
per
Ceph
cluster.
– Minimum
recommended
(performance
cluster):
10
OSD
nodes
cluster
(1
node
represents
<10%
of
total
cluster
capacity)
– Minimum
recommended
(cost/capacity
cluster):
7
OSD
nodes
per
cluster
(1
node
represents
<15%
of
total
cluster
capacity)

15. 6.
Data
Protec/on
Schemes
• Replica/on
• Erasure
Coding
(analogous
to
network
RAID)
One
of
the
biggest
choices
affec)ng
TCO
in
the
en)re
solu)on!

16. • Replica/on
– 3x
rep
over
JBOD
=
33%
usable:raw
capacity
ra/o
• Erasure
Coding
(analogous
to
network
RAID)
– 8+3
over
JBOD
=
73%
usable:raw
Data
Protec/on
Schemes

17. • Replica/on
– Ceph
block
storage
default:
3x
rep
over
JBOD
disks.
– Gluster
file
storage
default:
2x
rep
over
RAID6
bricks.
• Erasure
Coding
(analogous
to
network
RAID)
– Data
encoded
into
k
chunks
with
m
parity
chunks
and
spread
onto
different
disks
(frequently
on
different
servers).
Can
tolerate
m
disk
failures
without
data
loss.
8+3
popular.
Data
Protec/on
Schemes

18. Target
Cluster
Hardware
OSP
Starter
100TB
S
500TB
M
1PTB
L
2PB
IOPS
Op/mized
Throughput
Op/mized
Cost-­‐Capacity
Op/mized

19. • Following
are
extracts
from
the
recently
published
Ceph
on
Supermicro
RefArch
• Based
on
lab
benchmarking
results
from
many
different
configura/ons
RefArch
Examples

20. Sequen/al
Throughput
(R)
(per
server)
!"
#!!"
$!!!"
$#!!"
%!!!"
%#!!"
&!'(" '!!" (&" &"
!"#$%&'
()*%&+',-.%'/0"1'
23',%45%6789':%8;'<=>?5@=A5+'BC:',%>D%>'
EF':%AG'9-)>8;?$'
$%)$"
$!*)$!*"
$+)$"
$!*)$!*"
$+)!"
$!*)$!*"
,()%"
$!*)$!*"
,()!"
$!*)$!*"
,()%"
&!*"-./01234"
!)%"
&!*"-./01234"
(!)$%"
&!*"-./01234"
5%)!"
&!*"-./01234"

21. Sequen/al
Throughput
(R)
(per
OSD/HDD)
!"
#!"
$!"
%!"
&!"
'!"
(!"
)!"
*!"
&!+(" +!!" (&" &"
!"#$%&'
()*%&+',-.%'/0"1'
23',%45%6789':%8;'<=>?5@=A5+'BC:'(,D'
EF':%AG'9-)>8;?$'
#$,#"
#!-,#!-"
#*,#"
#!-,#!-"
#*,!"
#!-,#!-"
%(,$"
#!-,#!-"
%(,!"
#!-,#!-"
%(,$"
&!-"./012345"
(!,#$"
61/37893"
(!,#$"
&!-"./012345"
)$,!"
&!-"./012345"

22. Sequen/al
Throughput
(W)
(per
OSD/HDD,
3xRep)
!"
#"
$!"
$#"
%!"
%#"
&!"
'!()" (!!" )'" '"
!"#$%&'
()*%&+',-.%'/0"1'
234',%56%789:';<-+%'=><?6@>A6+'BCD'(,E'
FG'D%A'/H6:8A:I')I'F'+?'@%+'A>I$-&9:'J<-+%'+><?6@>A6+1K':-)<9L?$'
$%*$"
$!+*$!+"
$,*$"
$!+*$!+"
$,*!"
$!+*$!+"
&)*%"
$!+*$!+"
&)*!"
$!+*$!+"
&)*%"
'!+"-./01234"
)!*$%"
50.26782"
)!*$%"
'!+"-./01234"
9%*!"
'!+"-./01234"

23. !"
#"
$!"
$#"
%!"
%#"
&!"
&#"
'!"
'#"
'!()" (!!" )'" '"
!"#$%&'
()*%&+',-.%'/0"1'
234',%56%789:';<-+%'=><?6@>A6+'BCD'(,E'
CF'GH3'/I6:8A:J')J'/2H/K#/KHI11'+?'@%+'A>J$-&9:'L<-+%'+><?6@>A6+1M':-)<9N?$'
$%*$"
$!+*$!+"
$,*$"
$!+*$!+"
$,*!"
$!+*$!+"
&)*%"
$!+*$!+"
&)*!"
$!+*$!+"
&)*%"
'!+"-./01234"
)!*$%"
50.26782"
$%*!"
$!+*$!+"
)!*$%"
'!+"-./01234"
9%*!"
'!+"-./01234"
Sequen/al
Throughput
(W)
(per
OSD/HDD,
EC)

24. !"#$%"&'()$*"+,&")$-"%$./)0"
12%3452#46"7#89:;.83<=">$./"
?%:*$"#$%"4<:6"3@"A3%B"
+C3A$)6"#%:*$"D"E$)60"
!"#$%&'&())*&
$(#+%&'&,"#))*&
-$#+%&'&"#./01&
-(#+%&'&$#))*&
,"#+%&'&,#./01&
Throughput-­‐Op/mized
(R)

25. Throughput-­‐Op/mized
(W)
!"#$%"&'()$*"+,&")$-"%$./)0"
12%3452#46"7#89:;.83<=">%:6$"
?%:*$"#$%"4<:6"3@"A3%B"
+C3A$)6"#%:*$"D"E$)60"
!"#$%&'&())*&
$(#+%&'&,"#))*&
-$#+%&'&"#./01&
-(#+%&'&$#))*&
,"#+%&'&,#./01&

26. Capacity-­‐Op/mized
!"#$%"&'"
()#)*+,-".#/0+1)/23"
4%+*$"#$%"&'"
5627$8,"#%+*$"9":$8,;"
!"#$%&'&())*&
+$#$%&'&())*&
,"#$%&'&())*&

27. !"#$%&'()*)+,-.
!"#$%&'()*
%&'+&#+
,-./ %*0*,--./ 10*23/ 4*0*53/
6!3%*!"&7879#:
/0'12*3,+)-24'4)-)
.;+<=>;"=&*!"&7879#:
/0'12*3,+)-24'4)-)
5'6742
89:;).'<8=9!>
5=?'@)+A:!6,-B
890'5C$'DEE
80'?FFG'""EHI(J2
/9'6742
89:;).'<8=9!>
/9=K5'@)+A:!6,-B
890'5C$'DEE
80'?FFG'""EHI(J2
K/'6742
89:;).'<8=9!>
K/=89K'@)+A:!6,-B
890'5C$'DEE
80'?FFG'""EHI(J2
89L'6742
89:;).'<8=9!>
89L=9LF'@)+A:!6,-B
890'5C$'DEE
80'?FFG'""EHI(J2
?'"'(7&@*!"&7879#:
KM9'2+'*17-2+-24'4)-)
8F'6742
89:;).'<8=9!>
8F=9F'@)+A:!6,-B
890'KC$'DEE
F0'""E
N'6742
/K:;).
9?'@)+A:!6,-B
/K0'KC$'DEE
F0'""E
N'6742
N9:;).
9?'@)+A:!6,-B
N90'KC$'DEE
F0'""E
I3)6624'O71'"PQQ21'9F8L
Ceph
Op/mized
Configs

28. • Server
chassis
size
• CPU
• Memory
• Disk
• SSD
Write
Journals
(Ceph
only)
• Network
Add’l
Subsystem
Guidelines

29. • See
Ceph
on
Supermicro
PorMolio
RefArch
hQp://www.redhat.com/en/resources/red-­‐hat-­‐ceph-­‐storage-­‐clusters-­‐supermicro-­‐storage-­‐servers
• See
Ceph
on
Cisco
UCS
C3160
Whitepaper
hQp://www.cisco.com/c/en/us/products/collateral/servers-­‐unified-­‐compu/ng/ucs-­‐c-­‐series-­‐rack-­‐servers/whitepaper-­‐C11-­‐735004.html
• See
Ceph
on
Scalable
Informa/cs
Whitepaper
hQps://www.scalableinforma/cs.com/assets/documents/Unison-­‐Ceph-­‐Performance.pdf
RefArchs
&
Whitepapers

30. THANK
YOU