PLNOG16: Ceph distributed storage in OVH, Paweł Sadowski

Ceph Architecture Pools Client usage CaaS ?
Ceph distributed storage in OVH
Paweł Sadowski
March 1st, 2016
Paweł Sadowski Ceph in OVH March 1st, 2016 1 / 23

1 Ceph distributed storage
2 Architecture
3 Pool types
4 Client usage
5 Ceph in OVH
6 Questions

Main features
distributed, fault tolerant storage
self healing/rebalancing after failures/changes
online adding/removing of nodes
data can be replicated or erasure coded
data balanced according to hardware (disk size)
periodic data consistency veriﬁcation (deep scrub)
online (rolling) upgrades

Ceph stack

Ceph stack
Reliable Autonomic Distributed Object Store

Ceph stack
librados – library for direct RADOS access

Ceph stack
RBD – block device (via librbd or krbd)

Ceph stack
radosGW – S3/Swift compatible REST interface for object storage

Ceph stack
radosGW – S3/Swift compatible REST interface for object storage
CephFS – POSIX compliant ﬁlesystem

Single–node cluster
mon
osd

Single–node cluster
mon
osd
client 1
client 2
client 3
client 4
client 5
client 6

Multi–node cluster
client
mon1 mon2 mon3

Multi–node cluster
client
mon1 mon2 mon3
osd1 osd2 osd3
osd4 osd5 osd6
osd7 osd8 osd9

Data placement – CRUSH

data are stored in objects

objects are stored in pools

pools contains placement groups

object name is hashed and assigned to one placement group

placement groups are assigned to OSDs using CRUSH

crushmap deﬁne way to balance data between nodes

crushmap deﬁne way to balance data between nodes
client calculates target OSD based on crushmap

crushmap – bucket types
mon-01:~ # ceph osd crush dump
...
"types": [{ "type_id": 0, "name": "osd" },
{ "type_id": 1, "name": "host" },
{ "type_id": 2, "name": "chassis" },
{ "type_id": 3, "name": "rack" },
{ "type_id": 4, "name": "row" },
{ "type_id": 5, "name": "pdu" },
{ "type_id": 6, "name": "pod" },
{ "type_id": 7, "name": "room" },
{ "type_id": 8, "name": "datacenter" },
{ "type_id": 9, "name": "region" },
{ "type_id": 10, "name": "root" }],
...

crushmap – bucket hierarchy
...
"buckets": [{"id": -1,
"name": "default",
"type_id": 10,
"type_name": "root",
"weight": 89511660,
"alg": "straw",
"hash": "rjenkins1",
"items": [{ "id": -8, "weight": 22377915, "pos": 0 },
{ "id": -9, "weight": 22377915, "pos": 1 },
{ "id": -10, "weight": 22733120, "pos": 2 },
{ "id": -24, "weight": 22022710, "pos": 3 }]},
{ "id": -8,
"name": "73A20",
"type_id": 3,
"type_name": "rack",
"weight": 22377915,
"alg": "straw",
"hash": "rjenkins1",
"items": [{"id": -2, "weight": 3552050, "pos": 0 },
... Paweł Sadowski Ceph in OVH March 1st, 2016 10 / 23

crushmap – rules
...
"rules": [
{ "rule_id": 0, "rule_name": "replicated_ruleset",
"ruleset": 0, "type": 1,
"min_size": 1, "max_size": 10,
"steps": [
{ "op": "take", "item": -1, "item_name": "default"},
{ "op": "chooseleaf_firstn", "num": 0, "type": "rack"},
{ "op": "emit"}]}],
...

crushmap in action
mon
osd 1
osd 2
osd 3
client 1

crushmap in action
mon
osd 1
osd 2
osd 3
client 1
fetch crushmap

crushmap in action
mon
osd 1
osd 2
osd 3
client 1
fetch crushmap
calculate #1 PG and OSD

crushmap in action
mon
osd 1
osd 2
osd 3
client 1
fetch crushmap
push/fetch
data
#1

crushmap in action
mon
osd 1
osd 2
osd 3
client 1
fetch crushmap
push/fetch
data
#1
push/fetch data #2

ceph osd tree example
mon-01:~ # ceph osd tree
# id weight type name up/down reweight
-1 455.3 root default
-8 151.8 rack ABC20
-2 16.26 host cephhost-123456
0 5.42 osd.0 up 1
5 5.42 osd.5 up 1
6 5.42 osd.6 up 1
-3 65.04 host cephhost-123457
10 5.42 osd.10 up 1
13 5.42 osd.13 up 1
15 5.42 osd.15 up 1
19 5.42 osd.19 up 1
20 5.42 osd.20 up 1
24 5.42 osd.24 up 1
27 5.42 osd.27 up 1
30 5.42 osd.30 up 1
...

Replicated pool
data replicated n-times – pool size
operation on object allowed only when pool min_size replicas exists
can use it’s own CRUSH rule for data placement
deep scrub can detect object inconsistency
reads are served from primary acting OSD
writes are propagated by primary acting OSD

Erasure coded pool
k + m (i.e. 10 + 4 => 40% overhead, 4 can be lost)
space efﬁcient (similar to RAID5/RAID6)
higher CPU requirement
check object consistency on read1
object can’t be modiﬁed, must be written at once
multiple plugins (i.e. from Intel with XEON optimization, from Fujitsu – speed up
recovery)
1
http://tracker.ceph.com/issues/12000

Cache tiering
pool can have another layer (another pool) on top of it
usually top, hot, pool will be stored on fast disks (SSD, NVMe), bottom, cold, pool
will be stored on slow disks (HDD, SMR)
rbd can used with cache tiering when hot pool is replicated – cold pool can be EC
evicting object from hot pool is a costly process so make sure your hot data set ﬁts
in hot pool
possible to conﬁgure number of hits before promoting object

Rados Block Device
provides access to raw block device
rbd images are groups of object with ﬁxed size (speciﬁed at the time of creation)
snapshots, clones, copy-on-write
full/differential import/export
librbd (the fastest one), krbd, fuse
QEMU uses librbd to access Ceph
OpenStack supports Ceph for VM disk, snapshots and volumes

Object Storage – radosGW
S3/Swift compatible API (via radosGW)
store only full objects
can use EC pools
multi region asynchronous replication

CephFS
starting point but not yet production ready
like NFS but with distributed server part
POSIX compatible ﬁlesystem
since RedHat acquired InkTank they are more focused on RBD performance

Problems
quite difﬁcult to setup at the beginning
hard to predict performance level without experience
performance degradation during recovery/rebalance or deep scrub
high CPU usage when doing small IOs – CPU bound on Flash storage – big
improvement in Hammer and then Infernalis
corner case bugs on new features

Why and how we use Ceph?
HA backend for OVH Public Cloud
we are planning to use it for backups (with EC pools)
15 PB of raw storage available in running clusters (and growing)
mixed and Flash–only based clusters
we are using multiple Ceph clusters ...

Ceph-as-a-Service
create cluster in selected DC with one API request
conﬁgure users, pools and network access via API
multiple versions available: Fireﬂy, Hammer, Infernalis
RBD full support, radosGW work in progress
public beta available (ask on ceph@ml.ovh.net)

Questions?
ceph@ml.ovh.net
https://www.ovh.pl/careers/

PLNOG16: Ceph distributed storage in OVH, Paweł Sadowski

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (16)

Similar to PLNOG16: Ceph distributed storage in OVH, Paweł Sadowski

Similar to PLNOG16: Ceph distributed storage in OVH, Paweł Sadowski (20)

Recently uploaded

Recently uploaded (20)