DigitalOcean uses Ceph for block and object storage backing for their cloud services. They operate 37 production Ceph clusters running Nautilus and one on Luminous, storing over 54 PB of data across 21,500 OSDs. They deploy and manage Ceph clusters using Ansible playbooks and containerized Ceph packages, and monitor cluster health using Prometheus and Grafana dashboards. Upgrades can be challenging due to potential issues uncovered and slow performance on HDD backends.

1. Ceph at DigitalOcean
1
Alex Marangone - Sr Engineer

2. Agenda
○ What’s DigitalOcean
○ Ceph at DigitalOcean
○ Operating Ceph
○ Ceph gripes
○ Questions
2

3. 3
What’s DigitalOcean
A brief history

4. DigitalOcean
○ Cloud provider founded in 2012
○ Core concept: simplicity
○ First product: Droplets (i.e. VM)
○ Main attractivity was SSD based VM
○ Second product: Volumes (Block) in 2016
○ Many more products have been added since, including Spaces (Object)
○ Datacenters in 8 regions
○ IPO in 2021
4

5. 5
Ceph at DigitalOcean

6. Ceph at DigitalOcean
○ Block and Object storage are backed by Ceph
○ Images backups/BYOI isn’t but we’re looking into it
○ 37 production clusters on Ceph Nautilus
○ 1 on Ceph Luminous
○ >54 PB of raw Ceph storage
○ ~21,500 OSDs
○ Block storage is all flash
○ Object storage is HDD/Flash
○ Index nodes are flash
6

7. Why Ceph
● Scalability
○ Scales horizontally
○ And vertically
● Reliability
○ Self-healing
○ Strongly consistent
● Performance
○ Generally acceptable performance
○ Performance scales ~linearly~ with disk/node count
● Works with off-the-shelf components
● Usable for multiple features/products
7

8. 8
Operating Ceph

9. Operating Ceph
○ Our team should scale independently from the amount of clusters we
have. Goal is to automate as much as possible
○ Ceph operations are mostly done via Ansible and AWX
○ Some standalone tooling that we want to move to services in the future
9

10. Deploying Ceph
○ In-house ansible playbooks deploying Ceph in containers
○ Build our own Ceph packages
○ ceph/daemon-base image is used with a minimal script to mount OSDs at boot
○ OSD in sequential order are cool but that’s slow…
○ … pre-create all OSDs in CRUSH and launch OSD creation in parallel on all hosts
○ Pushes all the infos needed by services to our secrets management
platform
○ Future: automatic inventory discovery for 0 intervention deployment
10

11. Monitoring Ceph
○ ceph_exporter to monitor overall cluster health get useful data
○ https://github.com/digitalocean/ceph_exporter
○ storexporter and node_exporter used to gather data from the admin
socket of each daemon on every host
○ Canary process running regularly to check the performance and
availability of every cluster
11

12. Augmenting Ceph
○ Two ways to augment Ceph today
○ Block
○ Create all OSDs in a dummy root tree
○ Move hosts to default tree and slowly upweight them to avoid latency issues
■ Archimedes for upweights: https://github.com/digitalocean/archimedes
○ Object
○ Set nobackfill/norebalance
○ Create all OSDs
○ Cancel all backfills via upmaps (pgremapper)
○ Unset nobackfill/norebelance
○ ceph balancer or pgremapper’s undo-upmaps to remove the mappings
12

13. pgremapper
○ Pgremapper is opensource and heavily inspired by CERN’s scripts
○ https://github.com/digitalocean/pgremapper
○ https://github.com/cernceph/ceph-scripts/tree/master/tools/upmap
○ Allows to cancel backfill for augment or to prioritize recovery_wait PGs
○ Undo upmaps
○ Balance a CRUSH bucket
○ Drain an OSD
○ Remap PG
○ export/import mappings
13

14. OSD Lifecycle
○ Standalone tooling to diagnose, recondition, deploy, upgrade firmware
of OSDs
○ Wraps ceph-volume for OSD deployment and recondition
○ Automatically fetch latest firmware on recondition
○ Diagnose tooling looks at SMART data, syslog, to determine if a disk
needs to be physically replaced
○ Needs to be turned into a service
14

15. Automatic remediation service
○ Fixes recurring issues (e.g. ceph-mgr running out of memory)
○ Monitors every cluster in our fleet
○ If a known recurring issue is detected an AWX job is executed to fix it
○ Dramatically reduces alerts and interventions from our Cloud Operations
team
15

16. 16
Ceph gripes

17. Ceph upgrades are hard
○ Process is simple but issues are many
○ Documentation is sparse
○ Can be slow (days) on HDDs with on-disk checks/changes
○ History of the cluster matters
○ A successful upgrade does not make a trend
○ Testing is extremely difficult
○ We’re thinking of ways to improve this!
17

18. RGW defaults
○ Dynamic resharding
○ Can DoS a cluster on very large buckets
○ Makes listing very expensive over time
○ Beast FE (Nautilus)
○ Most librados calls are still synchronous
○ Slow Ops can consume all the threads very quickly
○ Auto shard removal
○ Can DoS a cluster on very large shards
18

19. 19
Questions?
We’re hiring! do.co/jobs - amarangone@digitalocean.com