Rook turns distributed storage systems into self-managing, self-scaling, self-healing storage services. It automates the tasks of a storage administrator: deployment, bootstrapping, configuration, provisioning, scaling, upgrading, migration, disaster recovery, monitoring, and resource management. Rook uses the power of the Kubernetes platform to deliver its services via a Kubernetes Operator for each storage provider. Oleg Chunikhin, Co-Founder and CTO @ Kublr.com, will present an introduction to storage management on k8s using Rook and Ceph.

1. Rook/Ceph on K8S Introduction
Oleg Chunikhin | CTO, Kublr

2. Oleg Chunikhin
CTO, Kublr
• 25 years in software architecture & development
• Working w/ Kubernetes since its release in 2015
• Software architect behind Kublr—an enterprise
ready container management platform
• @olgch
Introductions

3. Automation
Ingress
Custom
Clusters
Infrastructure
Logging Monitoring
Observability
API
Usage
Reporting
RBAC IAM
Air Gap TLS
Certificate
Rotation
Audit
Storage Networking Container
Registry
CI / CD App Mgmt
Infrastructure
Container Runtime Kubernetes
OPERATIONS SECURITY &
GOVERNANCE

4. Application DevOps Automation
Developers
SRE/Ops/DevOps/
SecOps
• Self-service
• Compatible
• Conformant
• Configurable
• Open & flexible
• Governance
• Org multi-tenancy
• Single pane of glass
• Operations
• Monitoring
• Log collection
• Image management
• Identity management
• Security
• Reliability
• Performance
• Portability

5. Hybrid Architecture
● Hybrid ~ Distributed ~ Complex
● Services
○ Connectivity
○ Discovery
● Data
○ Sharding
○ Mirroring / Replication
● BCDR
○ Failure and recovery scenarios
● Architecture
○ Application
○ Middleware
○ Infrastructure
● PoC
● Hardening and Optimization
● Operations

6. Kubernetes as Container Management
K8s is a solution for:
• Uniform application management
• Uniform resource management
• In-cluster failover, load balancing, traffic management (service mesh)
Challenges:
• Heterogeneous middleware ⇒ distributed data is difficult
• Heterogeneous infrastructure ⇒ distributed load balancing and ingress is difficult
Infrastructure
K8S
Applications
Infrastructure
K8S
Applications
Managed Middleware
(e.g. RDS, EFS, ...)
Managed Middleware
(e.g. Azure SQL, FS, ...)
?

7. Kubernetes as Infrastructure and Platform
Adds:
● Homogenous middleware
○ Ceph/Rook, Portworx, Vitess, Strimzi/Kafka
● Open cross-vendor inter-cluster connectivity
○ Submariner
● Uniform BCDR
○ Velero
● Uniform (operator-based) and portable middleware management
● Flexible and portable infrastructure mapping for middleware
Infrastructure
K8S
Applications
VPN
/
WAN
Self-hosted Middleware
(e.g. Ceph/Rook, Vitess, ...)
Infrastructure
K8S
Applications
Managed Middleware
(e.g. RDS, EFS, ...)
Managed Middleware
(e.g. Azure SQL, FS, ...)
VPN
/
WAN
Self-hosted Middleware
(e.g. Ceph/Rook, Vitess, ...)
BCDR
(e.g.
Velero)
BCDR
(e.g.
Velero)
IPSec, Wireguard, ...
Mirroring, Sharding, ...

8. Demo: Stateful App in Hybrid Environment
● AWS and Azure
● Kublr for Infrastructure and K8S provisioning
● Submariner as reliable VPN
● Ceph / Rook as an example of portable storage middleware
● HA PoC: multi-zone, HA storage
● BCDR PoC: mirroring, failover
● Cost control PoC: spot instances
Infrastructure
K8S
Applications
Submariner
Ceph / Rook storage
Infrastructure
K8S
Applications
AZ, EBS Zones, Azure Disks
Submariner
Ceph / Rook storage
IPSec
Mirroring, Snapshots

9. Kubernetes Operators
● Operator Pattern
● CRD
○ Spec: component definition
○ Status: component status
● Operator
○ Links the component and CRD
● Operator in this demo
○ Submariner
○ Rook
○ ~ Kublr

10. VPN: Submariner
Worker Node
Worker Node
Passive Gateway Node
Active Gateway Node
Gateway Label
Gateway Label
Public Network
Passive Gateway Node
Active Gateway Node
Gateway Label
Gateway Label
Worker Node
Worker Node
Cluster Node
Route Agent
Gateway Engine
VxLAN Traffic
IPSEC Traffic

11. Storage: Rook / Ceph
Data pool
mon
mon
mon
config
data
raw data
osd
raw data
osd
raw data
mds
osd
Data pool
Image Image
Ceph
Filesystem
Components
Abstractions
Ceph
rgw
S3/Swift
Object Store
mgr
Rook
Operator
CSI plugins
osd
osd
ganesha
NFS
CephCluster
Block Pool
Object Store
Filesystem
NFS
Object Store User
Provisioners
rbd-mirror

12. Stack Definition
SOURCE TYPE DESCRIPTION
Infrastructure Specification Virtual Machines, Networks, Disks, etc
Cloud Formation, ARM Templates, Terraform, Kublr
Kubernetes Specification Cluster topology, masters and workers number, groups, K8S components configuration, versions
System/support Software
Specification
Kubernetes system components: e.g. overlay network, DNS, etc
(Self-)managed application services: Cloud Native Storage (Ceph/Rook), DB (Vitess),
Messaging (Strimzi/Kafka, Nats), API Management (Ambassador, Kong), etc
DevOps tools: CI (Jenkins), CD (Spinnaker), Repositories (Nexus, Artifactory) etc
Provisioning scripts Provisioning procedures and processes: shell, Makefile, Jenkinsfile, CircleCI etc

13. kind: Cluster
metadata:
name: demo-hybrid-1-aws
spec:
...
network:
apiServerSecurePort: 6443
locations:
- name: aws1
aws:
...
master:
minNodes: 1
...
locations:
- aws:
...
nodes:
- name: group1
minNodes: 3
...
locations:
- aws:
...
features:
monitoring: { ... }
packages:
my-package: { ... }
Cluster Specification
Kublr metadata for the cluster - name,
space, labels
Cluster-wide non provider specific
configuration - network, cluster-wide
settings, k8s version, etc
Infrastructure provider specific cluster-wide
configuration - account, access creds, AZs
etc
Infrastructure provider specific group
configuration - AZs, image, group type,
zone locking, etc
Group-specific non provider specific
configuration - k8s options, autoscaling,
etc
Kublr-specific built-in K8S components
Additional custom helm packages

14. kind: Cluster
metadata:
name: demo-hybrid-1-aws
spec:
...
network:
clusterCIDR: 100.64.0.0/10
dnsDomain: cluster1.local
stubDomains:
- dns: cluster2.local
servers:
- 100.128.0.10
locations:
- name: aws1
aws:
resourcesCloudFormationExtras:
SgDefaultSubmariner500:
Type: AWS::EC2::SecurityGroupIngress
...
...
master:
minNodes: 1
...
locations:
- aws:
groupType: asg-mip
mixedInstancesPolicyCloudFormationExtras:
...
nodes:
- name: group1
minNodes: 3
...
locations:
- aws:
groupType: asg-mip
mixedInstancesPolicyCloudFormationExtras:
...
pinToZone: pin
availabilityZones:
- us-east-1a
- us-east-1b
- us-east-1c
Infrastructure
Additional ports
for VPN
kind: Cluster
metadata:
name: demo-hybrid-2-azure
spec:
...
network:
clusterCIDR: 100.128.0.0/10
dnsDomain: cluster2.local
stubDomains:
- dns: cluster1.local
servers:
- 100.64.0.10
locations:
- name: azure1
azure:
virtualNetworkSubnetCidrBlock: 172.18.0.0/16
armTemplateExtras:
securityGroup:
...
...
master:
minNodes: 1
...
locations:
- azure:
armTemplateExtras:
...
priority: Spot
nodes:
- name: group1
minNodes: 3
...
locations:
- azure:
armTemplateExtras:
...
priority: Spot
zones:
- '1'
- '2'
- '3'
pinToZone: pin
Non-intersecting
CIDR
Mutual
discoverability
Mixed instance policy
and spot instances
Multi-zone

15. kind: Cluster
metadata:
name: demo-hybrid-1-aws
spec:
...
packages:
submariner-broker: { ... }
rook-ceph: { ... }
rook-ceph-additional-configuration: { ... }
rook-ceph-cluster: { ... }
Middleware
Prepare namespace for Submariner
broker
Ceph cluster definition
Rook operator
Auxiliary preconfigured CRD (e.g.
snapshot class etc)

16. Kubernetes Persistence
Kubernetes Cluster
Namespace
Pod
Container 1 Container 2
Volume
Volume
Mount
Volume
Device
“actual”
storage
Persistent Volume
Volume
Claim
Spec
Spec
2 PVC with SC
5 Pod reference PVC
1 Storage Class
Storage
Class
Provisioner
3
Provision storage
Create PV
4 PVC bound to PV

17. Demo: RBD and CephFS
Data pool
Image
Data pool
mon
mon
mon
config
data
raw data
osd
raw data
osd
raw data
mds
osd
Data pool
Image Image
Ceph Filesystem
Data pool (data)
Data pool (md)
PV
Pod
PVC
PV
Pod
PVC
PV
Pod
PVC
PV
PVC
Pod Pod
Pod
sub-volumes
Rook
Operators
CephBlockPool
CephBlockPool CephFilesystem
StorageClass StorageClass

18. Demo: Mirroring
Data pool
Image
mon
mon
mon
config
data
raw data
osd
raw data
osd
raw data
osd
Data pool
Image
PV
Pod
PVC
rbd-mirror
Rook
Operators
Data pool
Image
mon
mon
mon
config
data
raw data
osd
raw data
osd
raw data
osd
Data pool
Image
PV
Pod
PVC
rbd-mirror
Rook
Operators
primary replay
CephBlockPool
mirroring:
enabled:
true
mode: image
peers: ...
CephBlockPool
mirroring:
enabled:
true
mode: image
peers: ...
VolumeReplicationClass VolumeReplicationClass
VolumeReplication VolumeReplication

19. Snapshot
Demo: Snapshots
PV
Pod
PVC
Rook
Operators
VolumeSnapshotClass
Snapshot PV
PVC

20. Demo: Cloning
PV
Pod
PVC
Rook
Operators
PV
PVC

21. Beyond the Demo
● Optimization: Resources, Throughput, ...
● Management: Quotas, Topology, ...
● Ceph: Object Store, NFS, ...
● BCDR: Velero
● Connectivity: VPN, Perring, Submariner, ...

22. References
@olgch; @kublr
github.com/kublr/hybrid-demo
rook.io/docs/rook/v1.7/
docs.ceph.com/en/pacific/
submariner.io/
docs.kublr.com/
docs.kublr.com/reference/kublr-cluster-spec/

23. Q&A

24. Oleg Chunikhin
CTO
oleg@kublr.com
@olgch
Follow Us
@kublr
Thank You