Intro into Rook and Ceph on Kubernetes

1. Rook/Ceph on K8S Introduction Oleg Chunikhin | CTO, Kublr

2. Oleg Chunikhin CTO, Kublr • 25 years in software architecture & development • Working w/ Kubernetes since its release in 2015 • Software architect behind Kublr—an enterprise ready container management platform • @olgch Introductions

3. Automation Ingress Custom Clusters Infrastructure Logging Monitoring Observability API Usage Reporting RBAC IAM Air Gap TLS Certificate Rotation Audit Storage Networking Container Registry CI / CD App Mgmt Infrastructure Container Runtime Kubernetes OPERATIONS SECURITY & GOVERNANCE

4. Application DevOps Automation Developers SRE/Ops/DevOps/ SecOps • Self-service • Compatible • Conformant • Configurable • Open & flexible • Governance • Org multi-tenancy • Single pane of glass • Operations • Monitoring • Log collection • Image management • Identity management • Security • Reliability • Performance • Portability

5. Hybrid Architecture ● Hybrid ~ Distributed ~ Complex ● Services ○ Connectivity ○ Discovery ● Data ○ Sharding ○ Mirroring / Replication ● BCDR ○ Failure and recovery scenarios ● Architecture ○ Application ○ Middleware ○ Infrastructure ● PoC ● Hardening and Optimization ● Operations

6. Kubernetes as Container Management K8s is a solution for: • Uniform application management • Uniform resource management • In-cluster failover, load balancing, traffic management (service mesh) Challenges: • Heterogeneous middleware ⇒ distributed data is difficult • Heterogeneous infrastructure ⇒ distributed load balancing and ingress is difficult Infrastructure K8S Applications Infrastructure K8S Applications Managed Middleware (e.g. RDS, EFS, ...) Managed Middleware (e.g. Azure SQL, FS, ...) ?

7. Kubernetes as Infrastructure and Platform Adds: ● Homogenous middleware ○ Ceph/Rook, Portworx, Vitess, Strimzi/Kafka ● Open cross-vendor inter-cluster connectivity ○ Submariner ● Uniform BCDR ○ Velero ● Uniform (operator-based) and portable middleware management ● Flexible and portable infrastructure mapping for middleware Infrastructure K8S Applications VPN / WAN Self-hosted Middleware (e.g. Ceph/Rook, Vitess, ...) Infrastructure K8S Applications Managed Middleware (e.g. RDS, EFS, ...) Managed Middleware (e.g. Azure SQL, FS, ...) VPN / WAN Self-hosted Middleware (e.g. Ceph/Rook, Vitess, ...) BCDR (e.g. Velero) BCDR (e.g. Velero) IPSec, Wireguard, ... Mirroring, Sharding, ...

8. Demo: Stateful App in Hybrid Environment ● AWS and Azure ● Kublr for Infrastructure and K8S provisioning ● Submariner as reliable VPN ● Ceph / Rook as an example of portable storage middleware ● HA PoC: multi-zone, HA storage ● BCDR PoC: mirroring, failover ● Cost control PoC: spot instances Infrastructure K8S Applications Submariner Ceph / Rook storage Infrastructure K8S Applications AZ, EBS Zones, Azure Disks Submariner Ceph / Rook storage IPSec Mirroring, Snapshots

9. Kubernetes Operators ● Operator Pattern ● CRD ○ Spec: component definition ○ Status: component status ● Operator ○ Links the component and CRD ● Operator in this demo ○ Submariner ○ Rook ○ ~ Kublr

10. VPN: Submariner Worker Node Worker Node Passive Gateway Node Active Gateway Node Gateway Label Gateway Label Public Network Passive Gateway Node Active Gateway Node Gateway Label Gateway Label Worker Node Worker Node Cluster Node Route Agent Gateway Engine VxLAN Traffic IPSEC Traffic

11. Storage: Rook / Ceph Data pool mon mon mon config data raw data osd raw data osd raw data mds osd Data pool Image Image Ceph Filesystem Components Abstractions Ceph rgw S3/Swift Object Store mgr Rook Operator CSI plugins osd osd ganesha NFS CephCluster Block Pool Object Store Filesystem NFS Object Store User Provisioners rbd-mirror

12. Stack Deﬁnition SOURCE TYPE DESCRIPTION Infrastructure Specification Virtual Machines, Networks, Disks, etc Cloud Formation, ARM Templates, Terraform, Kublr Kubernetes Specification Cluster topology, masters and workers number, groups, K8S components configuration, versions System/support Software Specification Kubernetes system components: e.g. overlay network, DNS, etc (Self-)managed application services: Cloud Native Storage (Ceph/Rook), DB (Vitess), Messaging (Strimzi/Kafka, Nats), API Management (Ambassador, Kong), etc DevOps tools: CI (Jenkins), CD (Spinnaker), Repositories (Nexus, Artifactory) etc Provisioning scripts Provisioning procedures and processes: shell, Makefile, Jenkinsfile, CircleCI etc

13. kind: Cluster metadata: name: demo-hybrid-1-aws spec: ... network: apiServerSecurePort: 6443 locations: - name: aws1 aws: ... master: minNodes: 1 ... locations: - aws: ... nodes: - name: group1 minNodes: 3 ... locations: - aws: ... features: monitoring: { ... } packages: my-package: { ... } Cluster Speciﬁcation Kublr metadata for the cluster - name, space, labels Cluster-wide non provider specific configuration - network, cluster-wide settings, k8s version, etc Infrastructure provider specific cluster-wide configuration - account, access creds, AZs etc Infrastructure provider specific group configuration - AZs, image, group type, zone locking, etc Group-specific non provider specific configuration - k8s options, autoscaling, etc Kublr-specific built-in K8S components Additional custom helm packages

14. kind: Cluster metadata: name: demo-hybrid-1-aws spec: ... network: clusterCIDR: 100.64.0.0/10 dnsDomain: cluster1.local stubDomains: - dns: cluster2.local servers: - 100.128.0.10 locations: - name: aws1 aws: resourcesCloudFormationExtras: SgDefaultSubmariner500: Type: AWS::EC2::SecurityGroupIngress ... ... master: minNodes: 1 ... locations: - aws: groupType: asg-mip mixedInstancesPolicyCloudFormationExtras: ... nodes: - name: group1 minNodes: 3 ... locations: - aws: groupType: asg-mip mixedInstancesPolicyCloudFormationExtras: ... pinToZone: pin availabilityZones: - us-east-1a - us-east-1b - us-east-1c Infrastructure Additional ports for VPN kind: Cluster metadata: name: demo-hybrid-2-azure spec: ... network: clusterCIDR: 100.128.0.0/10 dnsDomain: cluster2.local stubDomains: - dns: cluster1.local servers: - 100.64.0.10 locations: - name: azure1 azure: virtualNetworkSubnetCidrBlock: 172.18.0.0/16 armTemplateExtras: securityGroup: ... ... master: minNodes: 1 ... locations: - azure: armTemplateExtras: ... priority: Spot nodes: - name: group1 minNodes: 3 ... locations: - azure: armTemplateExtras: ... priority: Spot zones: - '1' - '2' - '3' pinToZone: pin Non-intersecting CIDR Mutual discoverability Mixed instance policy and spot instances Multi-zone

15. kind: Cluster metadata: name: demo-hybrid-1-aws spec: ... packages: submariner-broker: { ... } rook-ceph: { ... } rook-ceph-additional-configuration: { ... } rook-ceph-cluster: { ... } Middleware Prepare namespace for Submariner broker Ceph cluster definition Rook operator Auxiliary preconfigured CRD (e.g. snapshot class etc)

16. Kubernetes Persistence Kubernetes Cluster Namespace Pod Container 1 Container 2 Volume Volume Mount Volume Device “actual” storage Persistent Volume Volume Claim Spec Spec 2 PVC with SC 5 Pod reference PVC 1 Storage Class Storage Class Provisioner 3 Provision storage Create PV 4 PVC bound to PV

17. Demo: RBD and CephFS Data pool Image Data pool mon mon mon config data raw data osd raw data osd raw data mds osd Data pool Image Image Ceph Filesystem Data pool (data) Data pool (md) PV Pod PVC PV Pod PVC PV Pod PVC PV PVC Pod Pod Pod sub-volumes Rook Operators CephBlockPool CephBlockPool CephFilesystem StorageClass StorageClass

18. Demo: Mirroring Data pool Image mon mon mon config data raw data osd raw data osd raw data osd Data pool Image PV Pod PVC rbd-mirror Rook Operators Data pool Image mon mon mon config data raw data osd raw data osd raw data osd Data pool Image PV Pod PVC rbd-mirror Rook Operators primary replay CephBlockPool mirroring: enabled: true mode: image peers: ... CephBlockPool mirroring: enabled: true mode: image peers: ... VolumeReplicationClass VolumeReplicationClass VolumeReplication VolumeReplication

19. Snapshot Demo: Snapshots PV Pod PVC Rook Operators VolumeSnapshotClass Snapshot PV PVC

20. Demo: Cloning PV Pod PVC Rook Operators PV PVC

21. Beyond the Demo ● Optimization: Resources, Throughput, ... ● Management: Quotas, Topology, ... ● Ceph: Object Store, NFS, ... ● BCDR: Velero ● Connectivity: VPN, Perring, Submariner, ...

22. References @olgch; @kublr github.com/kublr/hybrid-demo rook.io/docs/rook/v1.7/ docs.ceph.com/en/pacific/ submariner.io/ docs.kublr.com/ docs.kublr.com/reference/kublr-cluster-spec/

23. Q&A

24. Oleg Chunikhin CTO oleg@kublr.com @olgch Follow Us @kublr Thank You

Intro into Rook and Ceph on Kubernetes

You are reading a preview.

Check these out next

Intro into Rook and Ceph on Kubernetes

More Related Content

Slideshows for you (20)

Similar to Intro into Rook and Ceph on Kubernetes (20)

More from Kublr (20)

Recently uploaded (20)

Intro into Rook and Ceph on Kubernetes