1.
USING ROOK TO MANAGE K8S
STORAGE WITH CEPH
MOH AHMED, RAYMOND MAIKA
JUNE 11TH, 2019

2.
Agenda
• What is Rook?
• Rook Timeline
• Operator Pattern
• Rook Design
• Rook Architecture with Ceph
• Container Storage Interface
• Monitoring Ceph on Kubernetes
• Demo of a Ceph Cluster Upgrade
• Upcoming Rook Features

3.
What is Rook?
• Reliance on external storage
– Not portable
– Requires these services to be accessible
– Deployment burden
• Reliance on cloud provider managed services
– Vendor lock-in
• Day 2 operations - who is managing the storage?

4.
What is Rook?
A Storage Orchestrator for
Kubernetes

5.
What is Rook?
A Storage Orchestrator for
Kubernetes
Also, a framework for many
other storage providers and
solutions

6.
Rook Timeline
v0.7
• Focus on Ceph
storage
v0.8
• CockroachDB
• Minio
v0.9
• Cassandra
• EdgeFS
• NFS
• Ceph independence
v1.0
• Backend maturity and
stability
• Ease of upgrade of
operator and storage
• Experimental CSI-
Ceph driver
2018/02
2018/07
2018/12
2019/05
Incubating

7.
Storage Backends Overview
EdgeFS

8.
Operator Pattern
• A method of packaging, deploying, and managing an application
• Extends Kubernetes API through Custom Resource Definitions (CRDs)
• Reconciliation loops to enforce state of the CRDs
• The Operator will:
– Observe the objects
– Analyze current vs. desired state
– Act on changes
Observe
AnalyzeAct

9.
etcdetcd
Rook Design
Kubernetes
API
kubectl
etcd
Rook Operator
Rook
Agent
Kubelet
Rook Flex Driver
Management &
Health API
New Objects:
Volume
Attachments
New Objects:
Storage
Clusters
Storage Pools
Object Store
File Store
Objects:
Deployments
DaemonSets
Pods
Services
StorageClass / PV / PVC
ClusterRole
Namespace
Config Maps
Daemons
CSI
Driver

10.
Rook Architecture - Ceph
Container Container Container Container
Volume Claim Volume Claim
Rook-agent Rook-agent Rook-agent Rook-agent Rook-agentOperator

11.
Container Storage Interface
• A specification to establish a standard for block and file storage system
• Allows the freedom to develop volume plug-ins externally from the orchestrator
• Similar to how Container Network Interface (CNI) became a standard
• Orchestrator agnostic ensuring compatibility across different platforms

12.
Ceph CSI Driver
• Implements an interface between a CSI Orchestrator (e.g. Kubernetes) and the
Ceph cluster
Ceph CSI Driver
Version
CSI Spec Version
v0.3.0 v0.3
v1.0.0 v1.0.0
Kubernetes CSI Spec
Compatibility
Status
v1.9 v0.1.0 Alpha
v1.10 v0.2.0 Beta
v1.11 v0.3.0 Beta
v1.13 v0.3.0, v1.0.0 GA

13.
Ceph CSI Driver
Feature CephFS Plugin
RADOS Block Device
(RBD) Plugin
Dynamically provision, de-provision
File mode RWO volume
>=v0.3.0 >=v0.3.0
Dynamically provision, de-provision
File mode RWX volume
>=v0.3.0
>=v0.3.0
Creating and deleting snapshot - v1.0.0
Provision volume from snapshot -
>=v0.3.0
Provision volume from another
volume
- v1.0.0
Resize volume - -

14.
Monitoring Ceph on K8s
• Ceph Manager Daemon (ceph-mgr)
– Became required since 12.x
(luminous) release
– Used to provide monitoring
interfaces
– Has a Prometheus plugin
– Built in dashboard exposed with
Rook 0.8 release

15.
Upgrade Workflow
• Since Rook v0.9:
– The operator and its storage can be upgraded independently
– Two separate images for the Rook operator and the Ceph cluster
• Rook 1.0 supports:
– Ceph Luminous (v12)
– Ceph Mimic (v13)
– Ceph Nautilus (v14)
• Upgrading the Ceph cluster is as simple as editing the image in the CephCluster
object

16.
Demo
• Show the CRDs and the pod deployment
• Show monitoring tools
• Upgrade the Ceph cluster to a new version

17.
Future Plans
• Rook v1.1
– Increased stability for other storage backends
– Stable release for Ceph-CSI plugin
– Improved upgrade workflows

18.
Get Involved
• Contribute to Rook
– https://github.com/rook/rook
– https://rook.io/
• Slack - https://rook-io.slack.com/
• Twitter - @rook_io
• Forums - https://groups.google.com/forum/#!forum/rook-dev
• Community Meetings

19.
References
• Thanks to Jared Watts, founder of Upbound and Senior maintainer of the Rook project for his
help and usage of some of his slides materials
• https://rook.io/docs/rook/v1.0/
• https://coreos.com/operators/
• https://www.slideshare.net/Jakobkaralus/the-kubernetes-operator-pattern-containerconf-nov-
2017
• https://kubernetes-csi.github.io/docs/
• https://github.com/kubernetes/community/blob/master/contributors/design-
proposals/storage/container-storage-interface.md
• https://github.com/kubernetes/community/blob/master/contributors/devel/sig-
storage/flexvolume.md
• https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/