The document discusses using Ceph storage as a PaaS platform and service. It describes PASTA, an in-house PaaS platform that uses Ceph for persistent volumes for containers. Ceph provides strong consistency for block and filesystem volumes and is used for stateful containers in Docker Swarm farms for services like Jenkins, Elasticsearch, and DRUID storage. Operational issues with Ceph discussed include multi-mapped volumes, upgrades, network failures, scrub/deep-scrub performance impacts, recovering RBD images, and monitor failures/recovery. Configuration options and methods for addressing these issues are also provided.

1. Ceph Storage, PaaS 로 서비스 / 운영하기
Hyun Ha @ naver

2. In-House Platform
StorageSecurity DB
MonitoringToolsComputing

3. StorageSecurity DB
MonitoringToolsComputing
In-House PaaS Platform - PASTA

4. Containerized
(stateless / stateful)
Goal:
In-House PaaS Platform - PASTA

5. Mission:
Persistent Volume for (stateful)Container
Data
Ceph
Container
Container

6. Strong Consistency à Block, Filesystem
Open Source Eco-System
Linux Kernel Source
Kernel mount / fuse mount / rbd kernel map
Why Ceph?

7. Docker Swarm Farm(RBD, CephFS)
Jenkins Farm(RBD)
Elastic-Search Farm(RBD)
Deep storage for DRUID (CephFS)
etc …
적용된 PaaS 플랫폼

8. PASTA
Docekr-Swarm
Host
Container Docker-plugin
Keystone Cinder
Ceph
Volume
/dev/rbd0
Client
Docker Registry
Use Case #1: 서비스 생성 Flow
1. Login
2. 인증/권한

9. PASTA
Docekr-Swarm
Host
Container Docker-plugin
Keystone Cinder
Ceph
Volume
/dev/rbd0
Client
Docker Registry
Use Case #1: 서비스 생성 Flow
3. 서비스 생성 요청

10. PASTA
Docekr-Swarm
Host
Container Docker-plugin
Keystone Cinder
Ceph
Volume
/dev/rbd0
Client
Docker Registry
Use Case #1: 서비스 생성 Flow
4. Container 생성

11. PASTA
Docekr-Swarm
Host
Container Docker-plugin
Keystone Cinder
Ceph
Volume
/dev/rbd0
Client
Docker Registry
Use Case #1: 서비스 생성 Flow
5. Image download / Container 생성

12. PASTA
Docekr-Swarm
Host
Container Docker-plugin
Keystone Cinder
Ceph
Volume
/dev/rbd0
Client
Docker Registry
Use Case #1: 서비스 생성 Flow
6. Volume 생성

13. PASTA
Docekr-Swarm
Host
Container Docker-plugin
Keystone Cinder
Ceph
Volume
/dev/rbd0
Client
Docker Registry
Use Case #1: 서비스 생성 Flow
7. 인증/권한

14. PASTA
Docekr-Swarm
Host
Container Docker-plugin
Keystone Cinder
Ceph
Volume
/dev/rbd0
Client
Docker Registry
Use Case #1: 서비스 생성 Flow
8. Volume 생성

15. PASTA
Docekr-Swarm
Host
Container Docker-plugin
Keystone Cinder
Ceph
Volume
/dev/rbd0
Client
Docker Registry
Use Case #1: 서비스 생성 Flow
9. Attach

16. PASTA
Docekr-Swarm
Host
Container Docker-plugin
Keystone Cinder
Ceph
Volume
/dev/rbd0
Client
Docker Registry
Use Case #1: 서비스 생성 Flow
10. mkfs, Mount

17. Use Case #2: Ceph UI

18. $ ceph-cli -h
NAME:
ceph-cli - use ceph volume for your PM/VM!
USAGE:
ceph-cli [global options] command [command options] [arguments...]
COMMANDS:
auth Certify with `TOKEN`
show Gets detailed information about the given `VOLUME ID` or `VOLUME NAME`
list Get volume list
create Create volume
delete Delete Volume by `VOLUME ID` or `VOLUME NAME`
attach Attach volume
detach Detach volume
extend Extend volume
reset Rest volume
GLOBAL OPTIONS:
--debug, -d Enable debug logging
--help, -h show help
Use Case #3: Ceph CLI

19. MON MGR
MDS
OSD OSD
OSD OSD
OSD OSD
OSD OSD
…
MON MGR
MDS
MON
MDS
MON
MDS
MON
MDS
Ceph - Component

20. SSD OSD.0
HDD OSD.2
SSD OSD.1
HDD OSD.3 HDD OSD.5
SSD Pool
HDD Pool
Ceph Cluster
Host #1 Host #2 Host #3
Application
HDD OSD.4
Hybrid Pool
Device Classes 기능 이용: Luminous 이후 버전
Ceph – Pool

21. 운영 / Troubleshooting
• Multi-Mapped Volume
• Upgrade Ceph
• Network 장애
• scrub/deep-scrub
• RBD Image 복구
• Monitor 장애/복구

22. ISSUE#1 : Multi-Mapped Volume

23. Mapped
Host #2 Host #4
Container
Host #3
Host #1
Container
Container
Container Container Container
Container Container Container
Volume
System hang / network 단절 등 비정상 동작
Multi-Mapped Volume

24. Host #2 Host #4
Container
Host #3
Host #1
Container
Container
Container Container Container
Container Container Container
Volume
Mapped
System hang / network 단절 등 비정상 동작
Multi-Mapped Volume
Mapped

25. Host #2 Host #4Host #3
Host #1
Container Container Container
Container Container Container
Container
Container
Container
Volume
Mapped
Mapped
System 정상화
Multi-Mapped Volume
?

26. $ ceph osd blacklist add ${client_ip} 10
blacklisting ${client_ip}:0/0 until 2018-05-02 (10 sec)
$ ceph osd blacklist ls
listed 1 entries
${client_ip}:0/0 2018-05-02 15:42:12.935377
Host blacklist 등록을 통한 multi-map 방지
• Add blacklist
• Automation
Docker
Call action
Lambda Ceph
Add blacklist

27. $ rbd -p volumes status ${volume-uuid}
Watchers:
watcher=${client_host_1_ip}:0/1015181303 client.259635408 cookie=18446462598732840991
watcher=${client_host_2_ip}:0/4152018459 client.201522571 cookie=18446462598732841309
• Multi mapped volumes 확인
• Monitoring / alarm
Multi-Mapped Volume 관리

28. ISSUE#2 : Upgrade Ceph

29. Upgrade Policy:
Newton Ocata Queens rockey
Jewel LTS
(v10.1.0)
Kraken EOL Luminous LTS
(v12.2.0)
Mimic
(v13.2.0)
Nautilus LTS
Openstack
Ceph
Pike
2018. 6.12016. 4 2017. 4 2017. 8
2016. 102016. 4
Mitaka
2017. 2 2017. 8 2018. 2 2018. 8
2018. 4
Luminous
2017. 10
Ocata
2017. 2
NewtonMitaka
2017. 3
Jewel

30. • ceph/src/vstart.sh
$ MON=1 MDS=1 ../src/vstart.sh -d -n -x
Upgrade 준비:
• Ceph-ansible (https://github.com/ceph/ceph-ansible)
• Kolla (https://github.com/openstack/kolla)
(https://review.openstack.org/#/c/566810/)

31. 주의할 점 : Kolla
ü ceph osd set noout
ü ceph osd set norebalance
ü Deploy one by one
ü Health check

32. • osd crush update on start = false (default: true)
• osd class update on start = false (default: true)
• osd_beacon_report_interval = 200 (default: 300)
• mon_osd_report_timeout = 300 (default: 900)
• ceph osd set-full-ratio 0.95
• ceph osd set-backfillfull-ratio 0.90
• ceph osd set-nearfull-ratio 0.70
주의할 점 : Configuration
üCheck : Custom configuration VS Default configuration

33. ISSUE#3 : Network 장애

34. libceph
/dev/rbd0
libceph: mon1 socket closed (con state CONNECTING)
libceph: mon2 socket closed (con state CONNECTING)
libceph: mon2 socket closed (con state CONNECTING)
Network장애 시
Ceph은 network attatced storage!

35. libceph
/dev/rbd0
libceph: mon1 socket closed (con state CONNECTING)
libceph: mon2 socket closed (con state CONNECTING)
libceph: mon2 socket closed (con state CONNECTING)
libceph: mon2 socket closed (con state CONNECTING)
libceph: mon1 socket closed (con state CONNECTING)
libceph: mon1 socket closed (con state CONNECTING)
libceph: mon0 socket closed (con state CONNECTING)
libceph: mon0 socket closed (con state CONNECTING)
Network장애 시

36. libceph
/dev/rbd0
libceph: mon1 socket closed (con state CONNECTING)
libceph: mon2 socket closed (con state CONNECTING)
libceph: mon2 socket closed (con state CONNECTING)
libceph: mon2 socket closed (con state CONNECTING)
libceph: mon1 socket closed (con state CONNECTING)
libceph: mon1 socket closed (con state CONNECTING)
libceph: mon1 socket closed (con state CONNECTING)
libceph: mon0 socket closed (con state CONNECTING)
libceph: mon0 socket closed (con state CONNECTING)
libceph: mon0 socket closed (con state CONNECTING)
libceph: mon1 socket closed (con state CONNECTING)
libceph: mon1 socket closed (con state CONNECTING)
libceph: mon1 socket closed (con state CONNECTING)
libceph: mon2 socket closed (con state CONNECTING)
libceph: mon0 socket closed (con state CONNECTING)
libceph: mon1 socket closed (con state CONNECTING)
libceph: mon1 socket closed (con state CONNECTING)
libceph: mon1 socket closed (con state CONNECTING)
libceph: mon2 socket closed (con state CONNECTING)
Network장애 시

37. http://tracker.ceph.com/issues/20927#change-96952
Network장애 시

38. https://github.com/rook/rook/pull/1179/files#diff-dabdd325e9ee838bb51e4f3f6c5b046cR142
rbd force unmap 만 지원됨. full-force unmap 개발 완료 후 적용 예정
동일 이슈 – Rook(Storage Orchestration for Kubernetes)
# no in-flight I/O only
$ rbd unmap -o force
# in-flight I/O
$ rbd unmap –o full-force

39. 우리가 해결한 방법은?
rbd map 이 아닌 rbd kernel map 사용, “osd_request_timeout” 옵션 적용
http://docs.ceph.com/docs/argonaut/rbd/rbd-ko/
(3600 second)
(Linux kernel : 4.11 이후)

40. rbd kernel map을 사용 시 이슈 1)
: Client 에서 keyring 이 그대로 노출됨
$ cat /sys/bus/rbd/devices/0/config_info
[mon_ip] name=admin,secret=AQDnZHxxAAy4OUSreyDDE6YMwKOT4Bug==
$ cat config_info
[mon_ip] name=admin,key=client.admin volumes vol01 -
Ø 해결 방법: Keyutils 적용

41. rbd kernel map을 사용 시 이슈 2)
: different major number
$ ls -al /dev/rbd*
brw------- 1 root root 252, 0 Dec 5 21:16 /dev/rbd0
brw------- 1 root root 251, 0 Dec 5 21:16 /dev/rbd1
brw------- 1 root root 242, 0 Dec 5 21:17 /dev/rbd10
brw------- 1 root root 241, 0 Dec 7 15:43 /dev/rbd11
brw------- 1 root root 240, 0 Dec 18 11:49 /dev/rbd12
$ ls -la /dev/rbd*
brw-rw----. 1 root disk 252, 0 Feb 8 02:14 /dev/rbd0
brw-rw----. 1 root disk 252, 16 Feb 8 02:13 /dev/rbd1
brw-rw----. 1 root disk 252, 32 Feb 8 02:14 /dev/rbd2
brw-rw----. 1 root disk 252, 48 Feb 8 02:20 /dev/rbd3
brw-rw----. 1 root disk 252, 64 Feb 8 02:29 /dev/rbd4
Ø 해결 방법 : single_major 적용
echo ”${mon_ip} name=admin,secret=*** volumes vol01" > /sys/bus/rbd/add_single_major

42. ISSUE #4 : scrub / deep-scrub

43. Deep Scrub 시 performance impact
“3 slow requests are blocked > 32 sec”

44. Deep-scrub performance impact
동시 deep-scrub 실행되는 PG 수 Disk Read (MB/s)

45. Configuration & Manual schedule
• osd scrub chunk min = 1
• osd scrub chunk max = 1
• Scrub : 전체 pg 가 2일에 1회 실행
• Deep scrub : 전체 pg가 34일에 1회 실행(동시 최대 1개)
$ ceph osd pool set $pool nodeep-scrub
Ø Configuration
Ø Set noscrub/nodeep-scrub
Ø Manual Schedule

46. ISSUE #5 : RBD Image 복구

47. Ceph Object 저장 방식 – Directory 구조
/var/lib/ceph/osd/ceph-4/current/3.72_head/
OSD DIR OSD ID PG ID
rbd_data.2576d643c9869.0000000000000000__head_22269772__3
Block_name_prefix Seq Num Hash Pool ID

48. $ find /var/lib/ceph/osd -type f -name *vol01*
/var/lib/ceph/osd/ceph-4/current/3.7a_head/rbd_id.vol01__head_E10E397A__3
$ hexdump -C /var/lib/ceph/osd/ceph-4/current/3.7a_head/rbduid.vol01__head_E10E397A__3
00000000 0d 00 00 00 32 35 37 36 64 36 34 33 63 39 38 36 |....2576d643c986|
00000010 39 |9|
Block_name_prefix : 2576d643c9869
2. “blocke_name_prefix” 로 모든 object get
1. rbd image의 block_name_prefix 찾기
3. 모은 object 로 image cerate & device 생성
Ceph Data 복구 방법

49. ISSUE #6 : Monitor 장애 / 복구

50. MON MON MONMON
MON
Service : Fine
Mon 장애 시나리오

51. MON MON MONMON
MON
Service : Failure
(단, client I/O는 정상. rbd map/unmap 불가)
Mon 장애 시나리오

52. MON MON MONMON
MON
$ monmaptool --rm ${mon_id} /tmp/monmap
1. 전체 mon stop, monmap 1개로 변경
3. 이후 mon 전체 배포하여 5개 quorum 구성
2. 새로운 monmap 적용
$ ceph-mon --inject-monmap /tmp/oo
Mon 장애 복구

53. MON MON MONMON
MON
다 죽은 경우는?
Mon 장애 복구

54. MON MON MONMON
MON
Remote backup
MON
백업본으로 복구!
Mon 장애 복구

55. MON MON MONMON
MON
Local backup
Remote backup
Local backup
Mon backup

56. MON
/local_backup
backup_pool
백업 정책>
- 백업 주기 : 1일 2회 (새벽 2시, 3시)
- 보관 주기 : 1 주
- 2개 mon에서 교대로 백업 수행,
- 다른 ceph cluster 로 remote backup
/dev/rbd
Ceph
Mon backup
/var/lib/ceph/mon/ceph-{mon_id}

57. 감사합니다.

58. Q & A

59. Thank you
하 현 / Hyun Ha
hyun.ha@navercorp.com