The document discusses Wido den Hollander's experience deploying Ceph storage clusters in various organizations. It describes two example deployments: one using Ceph block storage with CloudStack for 1000 VMs and S3 object storage, using SSDs and HDDs optimized for IOPS; the other using Ceph block storage with OCFS2 shared filesystem between web servers, using all SSDs and 10GbE networking for low latency. The document emphasizes designing for high IOPS rather than just capacity, and discusses best practices like regular hardware updates and testing recovery scenarios.

1. Deploying Ceph in the wild

2. Who am I?
● Wido den Hollander (1986)
● Co-owner and CTO of a PCextreme B.V., a
dutch hosting company
● Ceph trainer and consultant at 42on B.V.
● Part of the Ceph community since late 2009
– Wrote the Apache CloudStack integration
– libvirt RBD storage pool support
– PHP and Java bindings for librados

3. What is 42on?
● Consultancy company focused on Ceph and
it's eco-system
● Founded in 2012
● Based in the Netherlands
● I'm the only employee
– My consultancy company

4. Deploying Ceph
● As a consultant I see a lot of different organizations
– From small companies to large governments
– I see Ceph being used in all kinds of deployments
● It starts with gathering information about the use-case
– Deployment application: RBD? Objects?
– Storage requirements: TBs or PBs?
– I/O requirements

5. I/O is EXPENSIVE
● Everybody talks about storage capacity,
almost nobody talks about Iops
● Think about IOps first and then about
TerraBytes
Storage type € per I/O Remark
HDD € 1,60 Seagate 3TB drive for €150 with 90 IOps
SSD € 0,01 Intel S3500 480GB with 25k iops for €410

6. Design for I/O
● Use more, but smaller disks
– More spindles means more I/O
– Can go for consumer drives, cheaper
● Maybe deploy SSD-only
– Intel S3500 or S3700 SSDs are reliable and fast
● You really want I/O during recovery operations
– OSDs replay PGLogs and scan directories
– Recovery operations require a lot of I/O

7. Deployments
● I've done numerous Ceph deployments
– From tiny to large
● Want to showcase two of the deployments
– Use cases
– Design principles

8. Ceph with CloudStack
● Location: Belgium
● Organization: Government
● Use case:
– RBD for CloudStack
– S3 compatible storage
● Requirements:
– Storage for ~1000 Virtual Machines
● Including PostgreSQL databases
– TBs of S3 storage
● Actual data is unknown to me

9. Ceph with CloudStack
● Cluster:
– 16 nodes with 24 drives
● 19 1TB 7200RPM 2.5”
● 2 Intel S3700 200GB SSDs for journaling
● 2 Intel S3700 480GB SSDs for SSD-only storage
● 64GB of memory
● Xeon E5-2609 2.5Ghz CPU
– 3x replication and 80% rounding provides:
● 81TB HDD storage
● 8TB SSD storage
– 3 small nodes as monitors
● SSD for Operating System and monitor data

10. Ceph with CloudStack

11. Ceph with CloudStack
ROTATIONAL=$(cat /sys/block/$DEV/queue/rotational)
if [ $ROTATIONAL -eq 1 ]; then
echo "root=hdd rack=${RACK}-hdd host=${HOST}-hdd"
else
echo "root=ssd rack=${RACK}-ssd host=${HOST}-ssd"
fi
● If we detect the OSD is running on a SSD it goes into
a different 'host' in the CRUSH Map
– Rack is encoded in hostname (dc2-rk01)
-48 2.88 rack rk01-ssd
-33 0.72 host dc2-rk01-osd01-ssd
252 0.36 osd.252 up 1
253 0.36 osd.253 up 1
-41 69.16 rack rk01-hdd
-10 17.29 host dc2-rk01-osd01-hdd
20 0.91 osd.20 up 1
19 0.91 osd.19 up 1
17 0.91 osd.17 up 1

12. Ceph with CloudStack
● Download the script on my Github page:
– Url: https://gist.github.com/wido
– Place it in /usr/local/bin
● Configure it in your ceph.conf
– Push the config to your nodes using Puppet, Chef,
Ansible, ceph-deploy, etc
[osd]
osd_crush_location_hook = /usr/local/bin/crush-location-looukp

13. Ceph with CloudStack
● Highlights:
– Automatic assignment of OSDs to right type
– Designed for IOps. More, smaller drives
● SSD for the real high I/O applications
– RADOS Gateway for object storage
● Trying to push developers towards objects instead of
shared filesystems. A challenge!
● Future:
– Double cluster size within 6 months

14. RBD with OCFS2
● Location: Netherlands
● Organization: ISP
● Use case:
– RBD for OCFS2
● Requirements:
– Shared filesystem between webservers
● Until CephFS is stable

15. RBD with OCFS2
● Cluster:
– 9 nodes with 8 drives
● 1 SSD for Operating System
● 7 Samsung 840 Pro 512GB SSDs
● 10Gbit network (20Gbit LACP)
– At 3x replication and 80% filling it provides 8.6TB
of storage
– 3 small nodes as monitors

16. RBD with OCFS2

17. RBD with OCFS2
● “OCFS2 is a general-purpose shared-disk
cluster file system for Linux capable of
providing both high performance and high
availability.”
– RBD disks are shared
– ext4 or XFS can't be mounted on multiple
locations at the same time

18. RBD with OCFS2
● All the challenges were in OCFS2, not in Ceph
nor RBD
– Running 3.14.17 kernel due to OCFS2 issues
– Limited OCFS2 volumes to 200GB to minimize
impact in case of volume corruption
– Done multiple hardware upgrades without any
service interruption
● Runs smoothly while waiting for CephFS to
mature

19. RBD with OCFS2
● 10Gbit network for lower latency:
– Lower network latency provides more performance
– Lower latency means more IOps
● Design for I/O!
● 16k packet roundtrip times:
– 1GbE: 0.8 ~ 1.1ms
– 10GbE: 0.3 ~ 0.4ms
● It's not about the bandwidth, it's about latency!

20. RBD with OCFS2
● Highlights:
– Full SSD cluster
– 10GbE network for lower latency
– Replaced all hardware since cluster was build
● From 8 to 16 bays machines
● Future:
– Expand when required. No concrete planning

21. DO and DON'T
● DO
– Design for I/O, not raw TerraBytes
– Think about network latency
● 1GbE vs 10GbE
– Use small(er) machines
– Test recovery situations
● Pull the plug out of those machines!
– Reboot your machines regularly to verify it all works
● So do update those machines!
– Use dedicated hardware for your monitors
● With a SSD for storage

22. DO and DON'T

23. DO and DON'T
● DON'T
– Create to many Placement Groups
● It might overload your CPUs during recovery situations
– Fill your cluster over 80%
– Try to be smarter then Ceph
● It's auto-healing. Give it some time.
– Buy the most expensive machines
● Better to have two cheap(er) ones
– Use RAID-1 for journaling SSDs
● Spread your OSDS over them

24. DO and DON'T

25. REMEMBER
● Hardware failure is the rule, not the exception!
● Consistency goes over availability
● Ceph is designed to run on commodity
hardware
● There is no more need for RAID
– forget it ever existed

26. Questions?
● Twitter: @widodh
● Skype: @widodh
● E-Mail: wido@42on.com
● Github: github.com/wido
● Blog: http://blog.widodh.nl/