Hostvn ceph in production v1.1 dungtq

1. What is Ceph?
How we deploy in production
VietOpenStack Meetup
Hanoi July 1, 2017

2. Agenda
• What is Ceph?
• Key features
• Why we choose Ceph
• How we deploy in production
• Lessons learned
• What next?
• Questions

3. About me
Trinh Quang Dung
• Chief Technology Officer at Hostvn.net
• Been with Hostvn since 2009
• Start deploy OnApp public cloud since 2011
• System lead for all Hostvn services
• Excited about storage, distributed systems, networking, linux, automation and monitoring
• Interested in new technology

4. What is Ceph?
Ceph is a free & open distributed storage platform that
provides unified object, block, and file storage

5. Key Features
• Self managing/healing
• Self balancing (uniform distribution)
• Painless scaling
• Reliable multiple Replicas
• Distributed Data/ replicas spread
across nodes/racks/DCs
• Data placement with CRUSH
• Pseudo-random placement algorithm
• Rule-based configuration

6. Why we choose Ceph
• Free & Open Source software
• Unified distributed storage system
• Horizontal scaling
• No Vendor “lock-in”
• Doesn’t require specified hardware
• Customize as needed
#We call this is a “Heart of system”, sound great?

7. Everything free? Not really!
• Need one guy that really know what’s need to do
• Need a dedicated team to operation
• Need a lot of times to build, test, troubleshoot, solving issues
• Need consider everything that could happen
• Need support from else where when you’re over-control
…and finally it’s need to find out a “Golden ratio” for configuration
Hey! Look like we should stop right here?
Hold on. Move to the next slide please!

8. How we deploy in production
Few things/ideas
• We want to handle ceph cluster as much as possible (command line only)
• Troubleshoot ceph cluster is not easy task so it must be take note
everything we did
• Start with small cluster first
• Using all SSD drive to achieve better read/write performance
• Journal on the same osd-data drive

9. How we deploy in production - cont
Few things/ideas
• Keep all default settings
• LVM with RBD backend
• Using x2 replication pool
• Deployment automation
• Monitoring all metrics, logs of cluster

10. How we deploy in production - cont
Software
• Ceph Jewel 10.2
• CentOS 7.3 x64 minimal
• ansible
• collectd, graphite, grafana, beacon
• filebeat, logstash, elasticsearch, kibana

11. How we deploy in production - cont
Hardware
• 1x Admin node
• 2x Intel Xeon 2.9Ghz
• 32GB memory
• 2x 120GB OS
• 2x 1GbE, 2x 10GbE
• 3x MON nodes
• 2x Intel Xeon 2.9Ghz
• 32GB memory
• 2x 120GB OS
• 2x 1GbE, 2x 10GbE

12. How we deploy in production - cont
Hardware
• 3x OSD nodes
• 2x Intel Xeon 2.9Ghz
• 32GB memory
• 2x 120GB OS
• 4x 2TB SSD
• LSI HBA
• 2x 1GbE, 2x 10GbE
• 1x 10GbE ceph-public switch
• 1x 10GbE ceph-cluster switch
• 1x 1GbE management switch

13. How we deploy in production - cont
• Prepare
• Install ntp, configure and start it on all node (important)
• Set selinux to permissive mode and disable iptables
• Create hostname for each MON, OSD node in /etc/hosts (or using local dns)
• Create a user, add ssh public_key for that user on MON, OSD node (i.e: hostvn)
• Create yum repo for ceph install at /etc/yum.repos.d/ceph-deploy.repo
• Update OS, install epel-release, ceph-deploy
[ceph-noarch]
name=Ceph noarch packages
baseurl=https://download.ceph.com/rpm-jewel/el7/noarch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc

14. How we deploy in production - cont
• Deploy
• Create a directory on admin node for maintaining the
configuration files and keys
• Initial monitor nodes:
• Create config file /etc/ceph/ceph.conf
• Install ceph, add the initial monitor and gather the keys:
mkdir /etc/ceph && cd /etc/ceph
osd pool default size = 2
osd pool default min size = 1
public network = 10.10.0.0/24
cluster network = 172.16.10.0/24
ceph-deploy install admin-node mon0 mon1 mon2 osd0 osd1 osd2
ceph-deploy mon create-initial
ceph-deploy new mon0 mon1 mon2

15. How we deploy in production - cont
• Add disk to cluster (on each OSD node)
• Create a pool and rbd
ceph-deploy osd prepare {node-name}:{data-disk}[:{journal-disk}]
ceph-deploy osd prepare osd0:sda:/dev/sda
ceph-deploy osd activate {node-name}:{data-disk-partition}[:{journal-disk-partition}]
ceph-deploy osd activate osd0:/dev/sda1:/dev/sda2
ceph osd pool create {pool-name} {pg-num}
ceph osd pool set {pool-name} size {size}
ceph osd pool create onapp-vps 1024
ceph osd pool set onapp-vps size 2
rbd create vps0 -p onapp-vps --size 2000000
[root@mon0 ~]# rbd info vps0 -p onapp-vps
rbd image 'vps0':
size 1953 GB in 500000 objects
order 22 (4096 kB objects)
block_name_prefix: rb.0.10a7.238e1f29
format: 1

16. How we deploy in production - cont
• Cluster status
[root@mon0 ~]# ceph df
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
22285G 21848G 436G 1.96
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
onapp-vps 1 221G 1.99 10904G 57977
[root@mon0 ~]# ceph status
cluster a10afbea-f727-430b-93d8-c59d214f0bef
health HEALTH_OK
monmap e3: 3 mons at {mon0=10.10.0.198:6789/0,mon1=10.10.0.199:6789/0,mon2=10.10.0.202:6789/0}
election epoch 8, quorum 0,1,2 mon0,mon1,mon2
osdmap e82: 12 osds: 12 up, 12 in
pgmap v74854: 1024 pgs, 1 pools, 221 GB data, 57977 objects
436 GB used, 21848 GB / 22285 GB avail
1024 active+clean

17. How we deploy in production - cont
• OSD tree
[root@mon0 ~]# ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 21.71997 root default
-2 7.23999 host osd0
0 1.81000 osd.0 up 1.00000 1.00000
1 1.81000 osd.1 up 1.00000 1.00000
2 1.81000 osd.2 up 1.00000 1.00000
3 1.81000 osd.3 up 1.00000 1.00000
-3 7.23999 host osd1
4 1.81000 osd.4 up 1.00000 1.00000
5 1.81000 osd.5 up 1.00000 1.00000
6 1.81000 osd.6 up 1.00000 1.00000
7 1.81000 osd.7 up 1.00000 1.00000
-4 7.23999 host osd2
8 1.81000 osd.8 up 1.00000 1.00000
9 1.81000 osd.9 up 1.00000 1.00000
10 1.81000 osd.10 up 1.00000 1.00000
11 1.81000 osd.11 up 1.00000 1.00000

18. Diagram

19. Lessons learned
• Designing simple as much as possible
• Always use enterprise SSD for journal if using SATA drive
• Take sizing PGs carefully
• Choose the right hardware
• IOPS’s isn’t everything, latency potential bottleneck
• Keep an eye on back-fill, re-balance process
• No one size fits all model so make sure you’re understanding
what you need when deploying ceph

20. What next?
• Extending current SSD ceph cluster with more OSD nodes
• Deploy a new ceph cluster with SATA drive
• Journal on enterprise SSD
• 3x replication pool
• Upgrade 40-60 Gbps networking for ceph-cluster
• Planning to replace current storage
• Integrated with OpenStack, seriously? :D

21. Thanks For Your Time!
Questions?
CÔNG TY CỔ PHẦN CÔNG NGHỆ ÁNH SÁNG VIỆT
Trụ sở: Số 1, ngõ 73 Hoàng Cầu, Đống Đa, Hà Nội, Việt Nam
VP Hồ Chí Minh: 166/3 Hồ Bá Kiện, Q10, TP. Hồ Chí Minh, Việt Nam
Website: www.hostvn.net - www.hostvn.com
Dịch vụ: Domain | Hosting | Cloud VPS | Email | Server | License | Online Solutions