Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Kubernetes Stateful Workloads on Legacy Storage
Similar to Kubernetes Stateful Workloads on Legacy Storage (20)
Recently uploaded
Recently uploaded (20)
Kubernetes Stateful Workloads on Legacy Storage
- 1. Kubernetes Stateful Workloads on Legacy Storage DevConf India August 2, 2019 @akhilerm @openebs @mayadata_inc
- 2. Kubernetes Storage Kubernetes deals with stateless apps very well But when comes to stateful , who will manage the storage. Rook, OpenEBS, and some closed source solutions can be used.
- 3. What about the old storage box I bought? How to bring these drives in the cloud ? How to reuse drives from pre-cloud era to reduce cost of transition ? How to manage these Disks / Block Devices ?
- 4. How about using CSI ? CSI Driver can be used to connect to these old storage boxes OpenSDS and EmberCSI are two projects which help to achieve that Limitations: ● Careful planning of Kubernetes Clusters so that node to storage box connectivity is maintained. ● The storage itself was never designed to work with high volume of provisioning and deprovisioning of volumes with small capacities. ● Troubleshooting
- 5. Use Cases Pre-attached Storage. Common Use cases seen in OpenEBS community ● Legacy storage connected to OpenStack. Kubernetes is installed on OpenStack nodes that come with (cinder) attached volumes. The cinder volumes come from the Legacy storage. ● Another popular use case is with VMWare connecting to the Legacy storage. Kubernetes is installed on VMs that are provisioned with VMDK disks coming from Legacy storage. This pattern is Container Attached Storage or Hyper Converged Storage from the Kubernetes perspective.
- 6. How does it Work? Kubernetes Cluster node2node1 Pod Stateful Workload (DB, etc) cStor (iSCSI) Target Rep-1 Rep-2 Setup OpenEBS PV Cluster admin (1) node-disk-manager, provisioner, cstor operator,.. (2) SPC=>StoragePool(s) (3) StorageClass OS Developer Using OpenEBS (4) Pod with OpenEBS PVC (5) PV cStorPool OS cStorPool node3 Rep-3 OS cStorPool Disk Disk Disk Disk Disk Disk PVC
- 7. Node storage Device Manager There is no API to know what storage block devices are attached to each of the nodes, outside of the PVs. ● NDM exposes these block devices as a k8s custom resource ● Fetch attributes if it is a LVM, HDD, SSD, NVMe, GPD, or a partition ● Perform pre-post operations on a Disk - like secure delete before and/or after using a disk.
- 8. Node Components Cluster Level NDM Kube API Server NDM Operator etcd Node NDM Node NDM Node NDM NDM Operator: runs as a manager Pod helps with associating BlockDevice to BlockDeviceClaim NDM: runs as DaemonSet on the Storage Nodes, discovers and manages the underlying Block Devices/Disks. It exposes SMART metrics of the underlying disks.
- 9. Workflow - NDM Daemon OpenSeaChest & SMART Block Device Filtering Custom Resource Create / Update Block Device Monitoring Block Device Discovery The NDM Daemon runs on each storage node and manages block devices, updates the status of devices and exposes SMART metrics about the device
- 10. 10 Workflow - NDM Operator Select Block- Device Watch on BlockDeviceClaim etcd CSI-Plugin (OpenSDS) VendorCeph Cinder NDM Operator watches on BlockDevice Claims and bounds it with an available block device. If a device is not available, a block can be provisioned using OpenSDS and made available to the claim.
- 13. Questions ?
- 14. Demo on using LocalPV with disks
Editor's Notes
- Some questions that come to your mind while thinking about legacy storage There are big enterprises out there which have invested money in legacy storage systems. How can this be brought to the cloud native era
- Explain little about opensds, how they have CSI plugin (hotpot) which interacts with CSI of old storage devices and get the volumes connected. This requires careful planning of Kubernetes Clusters so that node to storage box connectivity is maintained. The storage itself was never designed to work with high volume of provisioning and deprovisioning of volumes with small capacities. A new use case driven by Containers and Kubernetes. Race conditions and scalability issues. Kubernetes pods will involve stepping out of Kubernetes cluster and interacting with Storage Administrators.
- Working of openebs
- Now comes the main part. How we actually manage the block devices.
- This is the github repo. The project is completely open source and written in golang.