The document discusses Ceph, an open-source distributed storage system. It provides an overview of Ceph's architecture and components, how it works, and considerations for setting up a Ceph cluster. Key points include: Ceph provides unified block, file and object storage interfaces and can scale exponentially. It uses CRUSH to deterministically map data across a cluster for redundancy. Setup choices like network, storage nodes, disks, caching and placement groups impact performance and must be tuned for the workload.

1. SUSE® Storage
Sizing and Performance for Ceph
Lars Marowsky-Brée
Distinguished Engineer
lmb@suse.com

2. What is Ceph?

3. 3
From 10,000 Meters
• Open Source Distributed Storage solution
• Most popular choice of distributed storage for
[1]
OpenStack
• Lots of goodies
‒ Distributed Object Storage
‒ Redundancy
‒ Efficient Scale-Out
‒ Can be built on commodity hardware
‒ Lower operational cost
[1] http://www.openstack.org/blog/2013/11/openstack-user-survey-statistics-november-2013/

4. 4
From 1,000 Meters
• Three interfaces rolled into one
‒ Object Access (like Amazon S3)
‒ Block Access
‒ (Distributed File System)
• Sitting on top of a Storage Cluster
‒ Self Healing
‒ Self Managed
‒ No Bottlenecks

5. 5
From 1,000 Meters
Unified Data Handling for 3 Purposes
Object Storage
(Like Amazon S3) Block Device File System
●RESTful Interface
●S3 and SWIFT APIs
●Block devices
●Up to 16 EiB
●Thin Provisioning
●Snapshots
●POSIX Compliant
●Separate Data and
Metadata
●For use e.g. with
Hadoop
Autonomous, Redundant Storage Cluster

6. 6
Component Names
radosgw
Object Storage
RBD
Block Device
Ceph FS
File System
RADOS
librados
Direct
Application
Access to
RADOS

7. 7
SUSE Storage architectural benefits
● Integration with SUSE Cloud and SUSE Linux Enterprise Server
●Exabyte scalability
● No bottlenecks or single points of failure
● Industry-leading functionality
● Remote replication, erasure coding
● Cache tiering
● Unified block, file and object interface
● Thin provisioning, copy on write
●100% software based; can use commodity hardware
●Automated management
● Self-managing, self-healing

8. How Does Ceph Work?

9. 9
For a Moment, Zooming to Atom Level
OSD Object Storage Daemon
FS
Disk
File System (btrfs, xfs)
Physical Disk
● OSDs serve storage objects to clients
● Peer to perform replication and recovery

10. 10
Put Several of These in One Node
OSD
FS
Disk
OSD
FS
Disk
OSD
FS
Disk
OSD
FS
Disk
OSD
FS
Disk
OSD
FS
Disk

11. 11
Mix In a Few Monitor Nodes
M • Monitors are the brain cells of the cluster
‒ Cluster Membership
‒ Consensus for Distributed Decision Making
• Do not serve stored objects to clients

12. 12
Voilà, a Small RADOS Cluster
M M
M

13. 13
Different Access Modes
• radosgw:
‒ An additional gateway in
front of your RADOS cluster
‒ Little impact on throughput,
but it does affect latency
• User-space RADOS
access:
‒ More feature rich than in-kernel
rbd.ko module
‒ Typically provides higher
performance

14. 14
Several Ingredients
• Basic Idea
‒ Coarse grained partitioning of storage supports policy based
mapping (don't put all copies of my data in one rack)
‒ Topology map and Rules allow clients to “compute” the exact
location of any storage object
• Three conceptual components
‒ Pools
‒ Placement groups
‒ CRUSH: deterministic decentralized placement algorithm

15. 15
Pools
• A pool is a logical container for storage objects
• A pool has a set of parameters
‒ a name
‒ a numerical ID (internal to RADOS)
‒ number of replicas OR erasure encoding settings
‒ number of placement groups
‒ placement rule set
‒ owner
• Pools support certain operations
‒ create/remove/read/write entire objects
‒ snapshot of the entire pool

16. 16
Placement Groups
• Placement groups help balance data across OSDs
• Consider a pool named “swimmingpool”
‒ with a pool ID of 38 and 8192 placement groups (PGs)
• Consider object “rubberduck” in “swimmingpool”
‒ hash(“rubberduck”) % 8192 = 0xb0b
‒ The resulting PG is 38.b0b
• One PG typically exists on several OSDs
‒ for replication
• One OSD typically serves many PGs

17. 17
CRUSH
• CRUSH uses a map of all OSDs in your
cluster
‒ includes physical topology, like row, rack, host
‒ includes rules describing which OSDs to
consider for what type of pool/PG
• This map is maintained by the monitor
nodes
‒ Monitor nodes use standard cluster algorithms
for consensus building, etc

18. 18
swimmingpool/rubberduck

19. 19
CRUSH in Action: Reading
swimmingpool/rubberduck
M
M
M
38.b0b
Reads could be
serviced by any
of the replicas
(parallel reads
improve thruput)

20. 20
CRUSH in Action: Writing
swimmingpool/rubberduck
M
M
M
38.b0b
Writes go to one
OSD, which then
propagates the
changes to other
replicas

21. Software Defined Storage

22. 22
Legacy Storage Arrays
• Limits:
‒ Tightly controlled
environment
‒ Limited scalability
‒ Few options
‒ Only certain approved drives
‒ Constrained number of disk
slots
‒ Few memory variations
‒ Only very few networking
choices
‒ Typically fixed controller and
CPU
• Benefits:
‒ Reasonably easy to
understand
‒ Long-term experience and
“gut instincts”
‒ Somewhat deterministic
behavior and pricing

23. 23
Software Defined Storage (SDS)
• Limits:
‒ ?
• Benefits:
‒ Infinite scalability
‒ Infinite adaptability
‒ Infinite choices
‒ Infinite flexibility
‒ ... right.

24. 24
Properties of a SDS System
• Throughput
• Latency
• IOPS
• Availability
• Reliability
• Capacity • Cost
• Density

25. 25
Architecting a SDS system
• These goals often conflict:
‒ Availability versus Density
‒ IOPS versus Density
‒ Everything versus Cost
• Many hardware options
• Software topology offers many configuration choices
• There is no one size fits all

26. Setup Choices

27. 27
Network
• Choose the fastest network you can afford
• Switches should be low latency with fully meshed
backplane
• Separate public and cluster network
• Cluster network should typically be twice the public
bandwidth
‒ Incoming writes are replicated over the cluster network
‒ Re-balancing and re-mirroring utilize the cluster network

28. 28
Networking (Public and Internal)
• Ethernet (1, 10, 40 GbE)
‒ Reasonably inexpensive (except for 40 GbE)
‒ Can easily be bonded for availability
‒ Use jumbo frames
• Infiniband
‒ High bandwidth
‒ Low latency
‒ Typically more expensive
‒ No support for RDMA yet in Ceph, need to use IPoIB

29. 29
Storage Node
• CPU
‒ Number and speed of cores
• Memory
• Storage controller
‒ Bandwidth, performance, cache size
• SSDs for OSD journal
‒ SSD to HDD ratio
• HDDs
‒ Count, capacity, performance

30. 30
Adding More Nodes
• Capacity increases
• Total throughput
increases
• IOPS increase
• Redundancy increases
• Latency unchanged
• Eventually: network
topology limitations
• Temporary impact during
re-balancing

31. 31
Adding More Disks to a Node
• Capacity increases
• Redundancy increases
• Throughput might
increase
• IOPS might increase
• Internal node bandwidth
is consumed
• Higher CPU and memory
load
• Cache contention
• Latency unchanged

32. 32
OSD File System
• btrfs
‒ Typically better write
throughput performance
‒ Higher CPU utilization
‒ Feature rich
‒ Compression, checksums, copy
on write
‒ The choice for the future!
• XFS
‒ Good all around choice
‒ Very mature for data
partitions
‒ Typically lower CPU
utilization
‒ The choice for today!

33. 33
Impact of Caches
• Cache on the client side
‒ Typically, biggest impact on performance
‒ Does not help with write performance
• Server OS cache
‒ Low impact: reads have already been cached on the client
‒ Still, helps with readahead
• Caching controller, battery backed:
‒ Significant impact for writes

34. 34
Impact of SSD Journals
• SSD journals accelerate bursts and random write IO
• For sustained writes that overflow the journal,
performance degrades to HDD levels
• SSDs help very little with read performance
• SSDs are very costly
‒ ... and consume storage slots -> lower density
• A large battery-backed cache on the storage controller
is highly recommended if not using SSD journals

35. 35
Hard Disk Parameters
• Capacity matters
‒ Often, highest density is not
most cost effective
‒ On-disk cache matters less
• Reliability advantage of
Enterprise drives typically
marginal compared to
cost
‒ Buy more drives instead
‒ Consider validation matrices
for small/medium NAS
servers as a guide
• RPM:
‒ Increase IOPS & throughput
‒ Increases power
consumption
‒ 15k drives quite expensive
still

36. 36
Impact of Redundancy Choices
• Replication:
‒ n number of exact, full-size
copies
‒ Potentially increased read
performance due to striping
‒ More copies lower
throughput, increase latency
‒ Increased cluster network
utilization for writes
‒ Rebuilds can leverage
multiple sources
‒ Significant capacity impact
• Erasure coding:
‒ Data split into k parts plus m
redundancy codes
‒ Better space efficiency
‒ Higher CPU overhead
‒ Significant CPU and cluster
network impact, especially
during rebuild
‒ Cannot directly be used with
block devices (see next
slide)

37. 37
Cache Tiering
• Multi-tier storage architecture:
‒ Pool acts as a transparent write-back overlay for another
‒ e.g., SSD 3-way replication over HDDs with erasure coding
‒ Can flush either on relative or absolute dirty levels, or age
‒ Additional configuration complexity and requires workload-specific
tuning
‒ Also available: read-only mode (no write acceleration)
‒ Some downsides (no snapshots), memory consumption for
HitSet
• A good way to combine the advantages of replication
and erasure coding

38. 38
Number of placement groups
●Number of hash buckets
per pool
● Data is chunked &
distributed across nodes
● Typically approx. 100 per
OSD/TB
• Too many:
‒ More peering
‒ More resources used
• Too few:
‒ Large amounts of data per
group
‒ More hotspots, less striping
‒ Slower recovery from failure
‒ Slower re-balancing

39. 39
Measuring Ceph performance
• rados bench
‒ Measures backend performance of the RADOS store
‒ Simple options
• rados load-gen
‒ Generate configurable load on the cluster
• fio rbd backend
‒ Swiss army knife of IO benchmarking on Linux
‒ Can also compare in-kernel rbd with user-space librados
• rest-bench
‒ Measures S3/radosgw performance

40. Conclusion

41. 41
How to size a Ceph cluster?
• Understand your workload
• Make a best guess, based on the desirable properties
and factors
• Build a 10% pilot / proof of concept
‒ Preferably using loaner hardware from a vendor to avoid early
commitment
‒ The outlook of selling a few PB of storage and compute nodes
makes vendors very cooperative
• Refine this until desired performance is achieved
• Scale up
‒ Ceph retains most characteristics at scale or even improves

42. Thank you.
42
Questions and Answers?

43. Corporate Headquarters
Maxfeldstrasse 5
90409 Nuremberg
Germany
+49 911 740 53 0 (Worldwide)
www.suse.com
Join us on:
www.opensuse.org
43

44. Unpublished Work of SUSE LLC. All Rights Reserved.
This work is an unpublished work and contains confidential, proprietary and trade secret information of SUSE LLC.
Access to this work is restricted to SUSE employees who have a need to know to perform tasks within the scope of
their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated,
abridged, condensed, expanded, collected, or adapted without the prior written consent of SUSE.
Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.
General Disclaimer
This document is not to be construed as a promise by any participating company to develop, deliver, or market a
product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making
purchasing decisions. SUSE makes no representations or warranties with respect to the contents of this document,
and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The
development, release, and timing of features or functionality described for SUSE products remains at the sole
discretion of SUSE. Further, SUSE reserves the right to revise this document and to make changes to its content, at
any time, without obligation to notify any person or entity of such revisions or changes. All SUSE marks referenced in
this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All
third-party trademarks are the property of their respective owners.