Ceph is an open-source distributed storage system that provides object storage, block storage, and file storage in a single unified system. It uses a CRUSH algorithm to dynamically determine data placement across clusters of storage nodes. Ceph components include RADOS for reliable distributed storage, Librados for direct application access, RADOS Gateway for object storage compatibility with S3 and Swift, and RBD for virtual disk storage in hypervisors and cloud platforms. Ceph aims to provide scalable, open-source storage for private and public clouds.

1. INTRODUCTION TO CEPH
Q2 2014

2. A STORAGE REVOLUTION
SUPPORT &
MAINTENANCE
PROPRIETARY
SOFTWARE
PROPRIETARY
HARDWARE
COMPUTE
R DISK
COMPUTE
R DISK
COMPUTE
R DISK
ENTERPRISE
PRODUCTS &
SERVICES
OPEN SOURCE
SOFTWARE
STANDARD
HARDWARE
COMPUTE
R DISK
COMPUTE
R DISK
COMPUTE
R DISK

3. ARCHITECTURAL COMPONENTS
APP HOST/VM CLIENT
RBD
A reliable, fully-distributed
block
device with cloud
platform integration
A distributed file
system with POSIX
semantics and
scale-out metadata
management
Copyright © 2014 by Inktank | Private and Confidential
3
RGW
A web services
gateway for object
storage, compatible
with S3 and Swift
LIBRADOS
CEPHFS
A library allowing apps to directly access RADOS (C, C++, Java, Python, Ruby,
PHP)
RADOS
A software-based, reliable, autonomous, distributed object store comprised of
self-healing, self-managing, intelligent storage nodes and lightweight monitors

4. ARCHITECTURAL COMPONENTS
APP HOST/VM CLIENT
RBD
A reliable, fully-distributed
block
device with cloud
platform integration
A distributed file
system with POSIX
semantics and
scale-out metadata
management
Copyright © 2014 by Inktank | Private and Confidential
4
RGW
A web services
gateway for object
storage, compatible
with S3 and Swift
LIBRADOS
CEPHFS
A library allowing apps to directly access RADOS (C, C++, Java, Python, Ruby,
PHP)
RADOS
A software-based, reliable, autonomous, distributed object store comprised of
self-healing, self-managing, intelligent storage nodes and lightweight monitors

5. OBJECT STORAGE DAEMONS
5
OSD
FS
DISK
OSD
FS
DISK
OSD
FS
DISK
OSD
FS
DISK
btrfs
xfs
ext4
M
M
M

6. RADOS CLUSTER
6
APPLICATION
M M
M M
M
RADOS CLUSTER

7. RADOS COMPONENTS
7
OSDs:
 10s to 10000s in a cluster
 One per disk (or one per SSD, RAID
group…)
 Serve stored objects to clients
 Intelligently peer for replication & recovery
Monitors:
 Maintain cluster membership and state
 Provide consensus for distributed decision-making
 Small, odd number
 These do not serve stored objects to
clients
M

8. WHERE DO OBJECTS LIVE?
8
??
APPLICATION
M
M
M
OBJECT

9. A METADATA SERVER?
9
1
APPLICATION
M
M
M
2

10. CALCULATED PLACEMENT
10
APPLICATION F
M
M
M
A-G
H-N
O-T
U-Z

11. EVEN BETTER: CRUSH!
11
01 11
11
01
RADOS CLUSTER
OBJECT
10
01
01
10
10
01
11
01
10
01
01
10
10
01
01 10
10 10 01 01

12. CRUSH IS A QUICK
CALCULATION
12
01 11
11
01
RADOS CLUSTER
OBJECT
10
01
01
10
10
01
01 10
10 10 01 01

13. CRUSH: DYNAMIC DATA
PLACEMENT
13
CRUSH:
 Pseudo-random placement algorithm
 Fast calculation, no lookup
 Repeatable, deterministic
 Statistically uniform distribution
 Stable mapping
 Limited data migration on change
 Rule-based configuration
 Infrastructure topology aware
 Adjustable replication
 Weighting

14. ARCHITECTURAL COMPONENTS
APP HOST/VM CLIENT
RBD
A reliable, fully-distributed
block
device with cloud
platform integration
A distributed file
system with POSIX
semantics and
scale-out metadata
management
Copyright © 2014 by Inktank | Private and Confidential
14
RGW
A web services
gateway for object
storage, compatible
with S3 and Swift
LIBRADOS
CEPHFS
A library allowing apps to directly access RADOS (C, C++, Java, Python, Ruby,
PHP)
RADOS
A software-based, reliable, autonomous, distributed object store comprised of
self-healing, self-managing, intelligent storage nodes and lightweight monitors

15. ACCESSING A RADOS CLUSTER
15
APPLICATION
LIBRADOS
OBJECT
socket
M M
M
RADOS CLUSTER

16. LIBRADOS: RADOS ACCESS FOR
APPS
L
16
LIBRADOS:
 Direct access to RADOS for applications
 C, C++, Python, PHP, Java, Erlang
 Direct access to storage nodes
 No HTTP overhead

17. ARCHITECTURAL COMPONENTS
APP HOST/VM CLIENT
RBD
A reliable, fully-distributed
block
device with cloud
platform integration
A distributed file
system with POSIX
semantics and
scale-out metadata
management
Copyright © 2014 by Inktank | Private and Confidential
17
RGW
A web services
gateway for object
storage, compatible
with S3 and Swift
LIBRADOS
CEPHFS
A library allowing apps to directly access RADOS (C, C++, Java, Python, Ruby,
PHP)
RADOS
A software-based, reliable, autonomous, distributed object store comprised of
self-healing, self-managing, intelligent storage nodes and lightweight monitors

18. THE RADOS GATEWAY
18
APPLICATION APPLICATION
REST
RADOSGW
LIBRADOS
M M
M
RADOS CLUSTER
RADOSGW
LIBRADOS
socket

19. RADOSGW MAKES RADOS
WEBBY
19
RADOSGW:
 REST-based object storage proxy
 Uses RADOS to store objects
 API supports buckets, accounts
 Usage accounting for billing
 Compatible with S3 and Swift applications

20. ARCHITECTURAL COMPONENTS
APP HOST/VM CLIENT
RBD
A reliable, fully-distributed
block
device with cloud
platform integration
A distributed file
system with POSIX
semantics and
scale-out metadata
management
Copyright © 2014 by Inktank | Private and Confidential
20
RGW
A web services
gateway for object
storage, compatible
with S3 and Swift
LIBRADOS
CEPHFS
A library allowing apps to directly access RADOS (C, C++, Java, Python, Ruby,
PHP)
RADOS
A software-based, reliable, autonomous, distributed object store comprised of
self-healing, self-managing, intelligent storage nodes and lightweight monitors

21. STORING VIRTUAL DISKS
21
VM
HYPERVISOR
LIBRBD
M M
RADOS CLUSTER

22. SEPARATE COMPUTE FROM
STORAGE
22
M M
RADOS CLUSTER
HYPERVISOR
LIBRB
D
VM HYPERVISOR
LIBRB
D

23. KERNEL MODULE FOR MAX
FLEXIBLE!
23
LINUX HOST
KRBD
M M
RADOS CLUSTER

24. RBD STORES VIRTUAL DISKS
24
RADOS BLOCK DEVICE:
 Storage of disk images in RADOS
 Decouples VMs from host
 Images are striped across the cluster
(pool)
 Snapshots
 Copy-on-write clones
 Support in:
 Mainline Linux Kernel (2.6.39+)
 Qemu/KVM, native Xen coming soon
 OpenStack, CloudStack, Nebula,
Proxmox

25. ARCHITECTURAL COMPONENTS
APP HOST/VM CLIENT
RBD
A reliable, fully-distributed
block
device with cloud
platform integration
A distributed file
system with POSIX
semantics and
scale-out metadata
management
Copyright © 2014 by Inktank | Private and Confidential
25
RGW
A web services
gateway for object
storage, compatible
with S3 and Swift
LIBRADOS
CEPHFS
A library allowing apps to directly access RADOS (C, C++, Java, Python, Ruby,
PHP)
RADOS
A software-based, reliable, autonomous, distributed object store comprised of
self-healing, self-managing, intelligent storage nodes and lightweight monitors

26. SEPARATE METADATA SERVER
26
LINUX HOST
KERNEL
MODULE
metadata 01 data
10
M M
M
RADOS CLUSTER

27. SCALABLE METADATA SERVERS
27
METADATA SERVER
 Manages metadata for a POSIX-compliant
shared filesystem
 Directory hierarchy
 File metadata (owner, timestamps,
mode, etc.)
 Stores metadata in RADOS
 Does not serve file data to clients
 Only required for shared filesystem

28. CEPH AND OPENSTACK
28
OPENSTACK
KEYSTONE CINDER GLAN
SWIFT CE NOVA
RADOSGW
LIBRADOS
LIBRB
D
M M
RADOS CLUSTER
LIBRB
D
HYPE
R-VLIISBROBR
D

29. GETTING STARTED WITH CEPH
 Read about the latest version of Ceph.
 The latest stuff is always at http://ceph.com/get
 Deploy a test cluster using ceph-deploy.
 Read the quick-start guide at http://ceph.com/qsg
 Read the rest of the docs!
 Find docs for the latest release at http://ceph.com/docs
 Ask for help when you get stuck!
 Community volunteers are waiting for you at
http://ceph.com/help
Copyright © 2014 by Inktank | Private and Confidential
29

30. THANK YOU!
Ross Turk
VP Marketing &
Community
ross@inktank.co
m
@rossturk
YOUR
PICTUR
E HERE