This document provides an overview of trends and evolutions in data storage. It discusses storage architecture before and after Amazon S3, including features like snapshots, cloning, and replication. It then covers Ceph storage concepts like RADOS, RBD, CephFS, and Rados Gateway (RGW). Specific Ceph components are explained like MON, CRUSH, OSD. Example commands are shown for setting up and using RBD block storage and CephFS distributed file storage. Lastly, it briefly discusses multi-site object storage replication between Ceph clusters.

1. Tendências e
Evoluções em
Armazenamento de
Dados

2. HELLO!• Jefferson Especialista de Storage no Walmart.com
• Processamento de Dados, Fatec de São Paulo
• Experiência em alta criticidade em sistemas de
armazenamento de dados
• SNIA Certificate .

3. AGENDA
• Storage Architecture overview
• Ceph overview
• Rados Gateway overview
• Rados Gateway architecture
• RBD
• CephFS
• Object Storage Multi-site

4. Storage Overview
BEFORE S3 AFTER S3

5. “ Before S3 ”

6. R5
B1
R6
B
q
R5
C1
R6
C
p
R5
D
p
R6
D1
R5
A1
R6
A1
R5
B2
R6
B1
R5
C
p
R6
C
q
R5
D1
R6
D
p
R5
A2
R5
A2
R5
B
p
R6
B2
R5
C2
R6
C1
R5
D2
R6
D
q
R5
A3
R6
A
p
R5
B3
R6
B
p
R5
C3
R6
C2
R5
D3
R6
D2
R5
A
p
R6
A
q
CONTROLER
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
CONTROLER
Raid Group 0
Raid Group 2
Raid Group 1
Protection
LUNS or Volumes
COMPUTER DISK
COMPUTER DISK
COMPUTER DISK
COMPUTER DISK
COMPUTER DISK
COMPUTER DISK
COMPUTER DISK
COMPUTER DISK
COMPUTER DISK
COMPUTER DISK
COMPUTER DISK
COMPUTER DISK
COMPUTER DISK

7. Storage
Features
• Snapshots
• Map to Any
• Clone
• Volume Copy
• Virtual Storage
• Thin Provisioning
• Dedup
• Replication

8. 2 Tipos de Storages
CONTROLER
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
DI
S
K
CONTROLER
Block
Unified
SAN
FC/ISCSI
/DEV/SDA
OU
DRIVE F:
CIFS IPSHARE
NFS IP:/MOUNT
Ext4
4k 4k 4k
• Ext4
• XFS
• BRFS
• ZFS

9. “ S3 ”

10. O Conceito
March 2006
Simple Storage Service
Escalável
Sem Pontos De Falha
Rápido
Barato
Simples

11. RBD
CephFS
MON
CRUSH
OSD
PG
radosgw
Sage A. Weil

12. Community
ceph-community@ceph.com
ceph-users
ceph-users@ceph.com

13. Ceph Versions
Argonaut – on July 3, 2012
Bobtail (v0.56) – on January 1, 2013
Cuttlefish (v0.61) – on May 7, 2013
Dumpling (v0.67) – on August 14, 2013
Emperor (v0.72) – on November 9, 2013
Firefly (v0.80) – on May 7, 2014
Giant (v0.87) – on October 29, 2014
Hammer (v0.94) – on April 7, 2015
Infernalis (v9.2.0) – on November 6, 2015
Jewel (v10.2.0) – on April 21, 2016

14. Open source (LGPL license)
Software defined storage distributed
No single point of failure
Massively scalable
Self healing
Unified storage: object, block and file
Ceph

15. Ceph architecture
CEPHFS
A distributed file
system with POSIX
semantics and scale-
out metadata
management
RGW
A web services
gateway for object
storage, compatible
with S3 and Swift
RBD
A reliable, fully-
distributed block
device with cloud
platform integration
LIBRADOS
A library allowing apps to directly access RADOS (C, C++, Java, Python, Ruby,
PHP)
RADOS
APP HOST CLIENT

16. Rados
Reliable Distributed Object Storage
Replication
Flat object namespace within each pool
Strong consistency (CP system)
Infrastructure aware, dynamic topology
Hash-based placement (CRUSH)

17. 3 até 10.000 OSDs p/ um Cluster
One per Disk
Server Stored object to client
Intelligently peer to replication
OSD

18. Maintain cluster membership and state
Provide consensus for distributed decision-
making
Small, odd number
These do not serve stored objects to clients
Monitor node

20. Object Placement
Pool
placement group (PG)
CRUSH(pg, cluster state,
rule) = [A, B]
O
B
J
O
B
J O
B
J
O
B
J O
B
J
O
B
J O
B
J
O
B
J O
B
J
O
B
J O
B
J
O
B
J O
B
J
O
B
J O
B
J
O
B
J O
B
J
O
B
J O
B
J
O
B
J
BINARY
ID
METADATA
OBJ

21. Data Center
Rack1
Host1
OSD OSD
Host2
Rack2
Host3
OSD OSD
Host4
Rack3
Host5
OSD OSD
Host6

22. #ceph osd setcrushmap -i crushmap-filename
# begin crush map
# devices
device 1 osd.1
device 2 osd.2
host Host1 {
id -1
alg straw
hash 0 # rjenkins1
item osd.1 weight 3.500
}
host Host2 {
id -2
alg straw
hash 0 # rjenkins1
item osd.2 weight 3.500
}
rack Rack1 {
id -4
alg straw
hash 0 # rjenkins1
item Host1 weight 3.500
}
rack Rack2 {
id -4
alg straw
hash 0 # rjenkins1
item Host2 weight 3.500
}
datacenter DataCenter {
id -5
alg straw
hash 0 # rjenkins1
item Rack1 weight 3.500
item Rack2 weight 3.500
}
rule data {
ruleset 0
type replicated
min_size 1
max_size 3
step take DataCenter
step chooseleaf firstn 0 type rack
step emit
}

23. Rados Gateway overview
RGW
A web services
gateway for object
storage, compatible
with S3 and Swift
LIBRADOS
A library allowing apps to directly access RADOS (C, C++, Java, Python, Ruby,
PHP)
RADOS
APP

24. RADOS CLUSTER
RADOSGW
LIBRADOS
APLICATION
REST
SOCKET

25. RGW Components
• Frontend
• FastCGI - external web servers
• Civetweb– embedded web server
• Rest Dialect S3
• Swift
• Other API
• Execution layer – common layer for all dialects

26. $ radosgw-admin user create --display-name="johnny rotten" --uid=johnny
access_key": "TCICW53D9BQ2VGC46I44”
secret_key": "tfm9aHMI8X76L3UdgE+ZQaJag1vJQmE6HDb5Lbrz”
API :
Java, Python, C++, ruby , Perl, C#
HTTP REST:
DELETE/GET/PUT /{bucket} HTTP/1.1
Host: cname.domain.com
Authorization: AWS {access-key}:{hash-of-header-and-secret}
S3 CLOUD CLIENT:
S3cmd, cyberduck , s3fs
S3cmd ls s3://bucket_name/file.txt

27. RBD
RBD
A reliable, fully-
distributed block
device with cloud
platform integration
LIBRADOS
A library allowing apps to directly access RADOS (C, C++, Java, Python, Ruby,
PHP)
RADOS
HOST

28. RBD
• Thinly provisioned
• Resizable images
• Image import/export
• Image copy or rename
• Read-only snapshots
• Revert to snapshots
• Ability to mount with Linux or
QEMKVM clients!
RDB(module)
Librados
RADOS
VM/H
ost

29. RBD connectors
RADOS
RDB(module)
Libvirt

30. rbd create --size 1024 POOL/IMAGE
rbd resize --size 2048 IMAGE (to increase)
rbd resize --size 2048 IMAGE --allow-shrink (to decrease)
$ sudo apt-get install ceph-common
$ sudo modprobe rbd
ceph-authtool --print-key /etc/ceph/keyring.admin
sudo echo “mon:6789
name=admin,secret=AQDVGc5P0LXzJhAA5C019tbdrgypFNXUpG2cqQ==
rbd IMAGE" | sudo tee /sys/bus/rbd/add
$ sudo mkfs.xfs /dev/rbd0
$ sudo mount /dev/rbd0 /mnt/

31. CephFS
CEPHFS
A distributed file
system with POSIX
semantics and scale-
out metadata
management
LIBRADOS
A library allowing apps to directly access RADOS (C, C++, Java, Python, Ruby,
PHP)
RADOS
CLIENT

32. CephFs Overview

33. Filesystem lookup by inode

34. INO
Metadata
Server
OID
ONO
PGID
OBJ Ceph Crush
O
bj
O
bj
O
bj
File
1
2
3 4 5
6

35. Librados
RADOS
Metadata Servers
• POSIX-compliant file system
• Linux Kernel Clientt
• Mount –t ceph 1.2.3.4:/
• /mnt
• Export (NFS), Samba(CIFS)
• Ceph-fuse
• Recursive Directory Stats
• FileSize
• File and Directory Count
• Modification Time
• Libcephfs.so
• your app
• samba
• Hadoop
• Ganesha(NFS)

36. CephFS na Prática :
ceph-deploy mds create myserver
ceph osd pool create fs_data
ceph osd pool create fs_metadata
ceph fs new myfs fs_metadata fs_data
mount -t cephfs x.x.x.x:6789 /mnt/ceph

37. Object Storage Multi-site
• Replication
• 3 Million Objects
SITE A
CEPH CLUSTER
RADOSGW
SITE B
CEPH CLUSTER
RADOSGW
S3 S3
• 800 Objs/s
• Active/standby

38. THANKS!
Any questions?
You can find me at:
Jefferson · Jefferson22alcantara@gmail.com