Everything you *need* to know about storage with Kubernertes

1. © Copyright 2018 Dell Inc.1

2. Everything You Need to
Know About Kubernetes
Persistent Storage
Kenny Coleman, VMware
Tushar Thole, VMware

3. © Copyright 2018 Dell Inc.3
Who are we?
Kendrick Coleman
• Open Source, Technical Product Manager @ VMware
• A reformed sysadmin and virtualization junkie.
• His free time is spent sharing bourbon industry knowledge hosting
the Bourbon Pursuit Podcast.
• github.com/kacole2
• @kendrickcoleman
Tushar Thole
• Senior R&D Manager @ VMware, Cloud Native Storage
• Been with VMware for 10 years!
• Loves to read, hike & travel.
• github.com/tusharnt
• @TusharThole

4. © Copyright 2018 Dell Inc.4
Trying to run these in Kubernetes?
• Databases
• Postgres, MongoDB, MySQL, MariaDB, Redis, Cassandra
• Search, Analytics, Messaging
• ElasticSearch, LogStash, Kafka, RabbitMQ
• Content Management
• Wordpress, Joomla, Drupal, SugarCRM
• Service Discovery
• Consul, Zookeeper, etcd
• Continuous Integration and Delivery
• Jenkins, GitLab, SonarQube, Selenium, Nexus
• Custom Applications
• That Java app your company built
Stateful and
persistent applications

5. © Copyright 2018 Dell Inc.5
Applications need data
Lots of different types of persistent services to consider
Files Blocks
Documents
Logstreams
Time Series
Media and Streaming
Modern or Traditional
Applications
Storage
Services
Objects
Your use case here…

6. © Copyright 2018 Dell Inc.6
The Award for Most Popular App Goes To…
5 of the top 11
require persistence
05/01/18
hub.docker.com/explore

7. © Copyright 2018 Dell Inc.7
Storage Platform
Persistent Volume Object
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv0003
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Recycle
storageClassName: standard
Persistent Volume Claim Object
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: vol01
spec:
storageClassName: standard
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi
Storage Class Object
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: standard
provisioner: kubernetes.io/{provisioner}
parameters:
x
y
z Pod
Deployment StatefulSet
Control Data

8. © Copyright 2018 Dell Inc.8
Pre-Provisioned or On-the-Fly?
• In-tree vs out-of tree drivers
› Baked into the code vs an externally accessed plugin
› Externally accessed through the FlexVolume or CSI
› No more in-tree
• Driver Maintenance
– In-tree means you are subject to K8s release cycle
– Out-of-tree gives you more flexibility
• Features
– In-tree only has features that are exposed via k8s interface
• Provisioning
– Dynamic is only supported with in-tree drivers
– FlexVolume supports attach/detach, mount/unmount only (pretty much dead)
• Where’s the future?
– Container Storage Interface

9. © Copyright 2018 Dell Inc.9
Direct to Pod
• This is wrong and you know it. apiVersion: v1
kind: Pod
metadata:
name: pod-0
spec:
containers:
- image: gcr.io/google_containers/test-webserver
name: pod-0
volumeMounts:
- mountPath: /test-pd
name: vol0
volumes:
- name: vol0
scaleIO:
gateway: https://192.168.50.12:443/api
system: cluster1
protectionDomain: pdomain
storagePool: pool1
sslEnabled: false
volumeName: k8vol01
secretRef:
name: sio-secret

10. © Copyright 2018 Dell Inc.10
Options to understand
• capacity
– Fixed point integers in Ei, Pi, Ti, Gi, Mi, Ki (ie 32Gi)
– No other resources available such as IOPS, throughput, replication, etc.
• persistentVolumeReclaimPolicy
– Static: independent lifecycle (PV or PVC)
– Dynamic: Retain, Recycle or Delete (default Delete) (PVC)
› Retain: manual reclamation of the resource
› Delete: deletes the object from k8s and from the external infrastructure
• accessModes
– ReadWriteOnce: read-write by a single node
– ReadOnlyMany: read-only by many nodes
– ReadWriteMany: read-write by many nodes
• mountOptions
– Mount options are not validated, so mount will simply fail if one is invalid.
• mountPropagation
– allows for sharing volumes mounted by a Container to other Containers in the same Pod, or even to other Pods on the same node
– HostToContainer: equal to rSlave volume mount will receive all subsequent mounts that are mounted to this volume or any of its subdirectories
– Bidirectional: equal to rShared and same as HostToContainer except an addition that all volume mounts created by the Container will be propagated back to the host
and to all Containers of all Pods that use the same volume. Potentially dangerous to hosts and only available to privileged containers.
• allowVolumeExpansion
– Administrator can allow expanding persistent volume claims
– a user can request larger volume for their PersistentVolumeClaim by simply editing the claim and requesting a larger size. This in turn will trigger expansion of the
volume that is backing the underlying PersistentVolume

11. © Copyright 2018 Dell Inc.11
PersistentVolume Object
• Pre-provisioned storage by an administrator (static)
• Lifecycle is independent if static
– Just because you delete the object, the volume isn’t deleted
• Can be used by higher level controllers directly
• Does not require a provisioner
• This API object captures the details of the implementation of
the storage, be that NFS, iSCSI, or a cloud-provider-specific
storage system
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv0003
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
mountOptions:
- hard
- nfsvers=4.1
nfs:
path: /tmp
server: 172.17.0.2

12. © Copyright 2018 Dell Inc.12
PersistentVolumeClaim Object
• A request for storage by a user
• Can claim an existing PV or will create a new PV
– Based on name or size
– “claim” durable storage (such as a vSphere PersistentDisk or an iSCSI
volume) without knowing the details of the particular cloud environment
• Pods use claims as volumes
• When a PVC is deleted, the PV exists and the reclaim policy
takes effect
– If retained, then the PV object must be deleted and recreated to be
reused since previous claim data remains
• Many of the same PV options are used in the PVC and passed
down to the PV
• The StorageClass holds more attributes and is a reference-able
object for the provisioner however isn’t necessary and can use “”
or not set at all to use a DefaultStorageClass
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: vol01
spec:
storageClassName: standard
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi

13. © Copyright 2018 Dell Inc.13
StorageClass Object
• A StorageClass provides a way for administrators to
describe the “classes” of storage they offer. Different classes
might map to quality-of-service levels, or to backup policies,
or to arbitrary policies determined by the cluster
administrators.
• Each StorageClass contains the fields provisioner,
parameters, and reclaimPolicy, which are used when a
PersistentVolume belonging to the class needs to be
dynamically provisioned
– provisioner can be in-tree or out-of-tree
– parameters are specific to individual storage drivers
– reclaimPolicy is set to delete by default when not specified
• If the admission plugin is turned on, the administrator may
specify a default StorageClass. All PVCs that have no
storageClassName can be bound only to PVs of that default.
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: standard
provisioner: kubernetes.io/vsphere-volume
parameters:
diskformat: thin
fstype: ext3
datastore: vsanDatastore

14. © Copyright 2018 Dell Inc.14
Volumes
• Types of volumes and Plugins

15. © Copyright 2018 Dell Inc.15
Volume Type Outliers
• emptyDir
– Containers in the pod can all read and write the same files in the volume and that volume can be mounted at the same or
different paths in each container. When a Pod is removed from the node, data is deleted forever. volumes are stored on
whatever medium is backing the node. Use for scratch space or temp holding
• hostPath
– mounts a file or directory from the host node’s filesystem into your pod like running a container that needs access to Docker
internals.
• nfs and iscsi
– Unlike emptyDir, which is erased when a Pod is removed, the contents a volume are preserved and the volume is merely
unmounted and can be moved among pods.
• gitRepo
– Mounts an empty directory and clones a git repository into it for your pod to use.
• secret
– Used to pass sensitive information to pods and backed by tmpfs
• downwardAPI
– used to make downward API data available to applications. It mounts a directory and writes the requested data in plain text
files.
• projected
– maps several existing volume sources into the same directory such as secret, downwardAPI, and configMap
• local
– represents a mounted local storage device such as a disk, partition or directory

16. © Copyright 2018 Dell Inc.16
Why local isn’t the answer for all
• Can only be used as a statically created
PersistentVolume (until 1.11)
• Subject to the availability of the underlying
node
• On a positive note, they are better than
HostPath volumes
apiVersion: v1
kind: PersistentVolume
metadata:
name: example-pv
annotations:
"volume.alpha.kubernetes.io/node-affinity": '{
"requiredDuringSchedulingIgnoredDuringExecution": {
"nodeSelectorTerms": [
{ "matchExpressions": [
{ "key": "kubernetes.io/hostname",
"operator": "In",
"values": ["example-node"]
}
]}
]}
}'
spec:
capacity:
storage: 100Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: local-storage
local:
path: /mnt/disks/ssd1

17. © Copyright 2018 Dell Inc.17
Container Storage Interface
• An industry standard “Container Storage Interface” (CSI) that will enable storage vendors (SP) to develop a plugin
once and have it work across a number of container orchestration (CO) systems.
– Continual move to out of tree plugins
– Development is not waiting for internal k8s releases
– Allows drivers to maintain their own special features
– A predictable and consistent end user experience
– Storage providers for clouds and on-premises datacenters must provide two plugins that have associated services. The Node
Plugin must run on worker nodes within a cluster where the provisioned volume will be used, while the other is the Controller
Plugin that can run anywhere. Each of these plugins have specific roles and responsibilities that are defined as “services”.
– https://github.com/container-storage-interface

18. © Copyright 2018 Dell Inc.18
Deployments vs StatefulSets
Deployment
• A controller that provides declarative updates for
Pods and ReplicaSets.
• Describe a desired state in a Deployment object,
and the Deployment controller changes the actual
state to the desired state at a controlled rate. (aka
Stateless)
• Multiple in-flight updates
• Supports rollback to an earlier Deployment revision
• Deployed names are given random unique
identifiers in both ReplicaSets and Pods
• No documented way to use volumes
• Pod names are unique
• Volume names are recognizable
StatefulSet
• Manage the deployment and scaling of a set of Pods,
and provide guarantees about ordering. They do so by
maintaining a unique, sticky identity for each of their
Pods.
• Pods in a StatefulSet are not interchangeable. Each Pod
has a persistent identifier that it maintains across any
rescheduling.
• Require a Headless Service to be responsible for the
network identity of the Pods
• Pod names are in sequential order
• Volume names are unique and non-discernable since
based of StorageClass and not PVC
• Kubernetes will not delete Pods just because a Node
is unreachable. The Pods running on an unreachable
Node enter the ‘Terminating’ or ‘Unknown’ state after
a timeout. Pods may also enter these states when the
user attempts graceful deletion of a Pod on an
unreachable Node.

19. © Copyright 2018 Dell Inc.19
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: web01-deployment
labels:
app: web01
spec:
replicas: 3
selector:
matchLabels:
app: web01
template:
metadata:
labels:
app: web01
spec:
containers:
- name: web01
image: nginx
ports:
- containerPort: 80
Deployment
ReplicaSet
Replicas: 3
Pod Pod Pod
Deployments

20. © Copyright 2018 Dell Inc.20
Deployments
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: postgresdynamic
labels:
app: postgresdynamic
role: master
spec:
replicas: 1
template:
metadata:
labels:
app: postgresdynamic
role: master
spec:
containers:
- name: postgresdynamic
image: postgres
ports:
- containerPort: 5432
env:
- name: POSTGRES_PASSWORD
value: "Password123!"
volumeMounts:
- name: postgres-data
mountPath: /var/lib/postgresql/data
volumes:
- name: postgres-data
persistentVolumeClaim:
claimName: vol01
Deployment
ReplicaSet
Replicas: 1
Pod Pod

21. © Copyright 2018 Dell Inc.21
StatefulSet
apiVersion: v1
kind: Service
metadata:
name: pgnet
labels:
app: pgnet
spec:
ports:
- port: 5432
name: pgport
clusterIP: None
selector:
app: postgres
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: pgdatabase
spec:
serviceName: "postgres"
replicas: 1
template:
metadata:
labels:
app: postgres
spec:
containers:
- name: postgres01
image: postgres
ports:
- containerPort: 5432
name: pgport
env:
- name: POSTGRES_PASSWORD
value: "Password123!"
volumeMounts:
- name: pgvolume
mountPath: /var/lib/postgresql/data
volumeClaimTemplates:
- metadata:
name: pgvolume
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: standard
resources:
requests:
storage: 16Gi
Replicas: 1
Pod
StatefulSet
ReplicaSet
Service Network traffic

22. © Copyright 2018 Dell Inc.22
StatefulSet
apiVersion: v1
kind: Service
metadata:
name: pgnet
labels:
app: pgnet
spec:
ports:
- port: 5432
name: pgport
clusterIP: None
selector:
app: postgres
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: pgdatabase
spec:
serviceName: "postgres"
replicas: 1
template:
metadata:
labels:
app: postgres
spec:
containers:
- name: postgres01
image: postgres
ports:
- containerPort: 5432
name: pgport
env:
- name: POSTGRES_PASSWORD
value: "Password123!"
volumeMounts:
- name: pgvolume
mountPath: /var/lib/postgresql/data
volumeClaimTemplates:
- metadata:
name: pgvolume
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: standard
resources:
requests:
storage: 16Gi
• The only ways in which a Pod in such a state can be removed from the
apiserver are as follows:
– The Node object is deleted (either by you, or by the Node Controller).
– The kubelet on the unresponsive Node starts responding, kills the Pod and
removes the entry from the apiserver.
– Force deletion of the Pod by the user.
• No auto-restarts for network partitions, failed hosts, or deleted nodes
– kubectl delete pods <pod> --grace-period=0 --force

23. © Copyright 2018 Dell Inc.23
DaemonSet
apiVersion: apps/v1beta2
kind: DaemonSet
metadata:
name: fluentd-elasticsearch
namespace: kube-system
labels:
k8s-app: fluentd-logging
spec:
selector:
matchLabels:
name: fluentd-elasticsearch
template:
metadata:
labels:
name: fluentd-elasticsearch
spec:
containers:
- name: fluentd-elasticsearch
image: gcr.io/google-containers/fluentd-elasticsearch:1.20
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
• A DaemonSet ensures that all (or some) Nodes run a
copy of a Pod. As nodes are added to the cluster, Pods
are added to them. As nodes are removed from the
cluster, those Pods are garbage collected. Deleting a
DaemonSet will clean up the Pods it created.
– running a cluster storage daemon, such as glusterd, ceph,
on each node.
– running a logs collection daemon on every node, such as
fluentd or logstash.
– running a node monitoring daemon on every node, such as
Prometheus Node Exporter, collectd, Datadog agent,
New Relic agent, or Ganglia gmond.
• Good use of local storage, varying use cases for
centralizing storage.

24. © Copyright 2018 Dell Inc.24
DaemonSet
DaemonSet
Pod
apiVersion: apps/v1beta2
kind: DaemonSet
metadata:
name: fluentd-elasticsearch
namespace: kube-system
labels:
k8s-app: fluentd-logging
spec:
selector:
matchLabels:
name: fluentd-elasticsearch
template:
metadata:
labels:
name: fluentd-elasticsearch
spec:
containers:
- name: fluentd-elasticsearch
image: gcr.io/google-containers/fluentd-elasticsearch:1.20
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
Pod

25. © Copyright 2018 Dell Inc.25
Project Hatchway : Persistent storage for Containers
29
Developer Ready Infrastructure for stateful containers on vSphere
Docker API Kubernetes API
docker stack deploy
docker service create
kubectl create
kubectl get pods
Docker Swarm/
Datacenter
Kubernetes
Infrastructure of choice:
vSphere with vSAN,
VMFS and NFS
Stateful Containers with
Orchestrator of choice:
Docker Swarm, Kubernetes
VMware ESX VMware ESXVMware ESX
VMFS NFSvSAN
Available Now!
https://vmware.github.io/hatchway

26. © Copyright 2018 Dell Inc.26
https://github.com/vmware/vsphere-storage-for-kubernetes
ESXi
vCenter
vSphere Storage
for Kubernetes
ESXi
K8s Worker
(Container Host)
Datastore1dataVol.vmdk
K8s kubelet
# vi vsphere.conf
Pod
Tools, Libs,
SW
Redis
DB
K8s API
ESXi
<Add Flags & Restart Ctrlr, API,
Kubelets>
--cloud-provider=vsphere
--cloud-config=vsphere.conf
# systemctl restart kubelet.service
kind: PersistentVolume
spec:
capacity: [storage: 16Gi]
storageClassName: slow
K8s Volume
PersistentVolumeClaim
K8s vSphere
Cloud provider

27. © Copyright 2018 Dell Inc.27
Enabling policy driven dynamic provisioning for Containers
29
Describ
e
Admin
Users
Claim
kind: StorageClass
Name: Gold
Provisioner:
vsphere-volume
storagePolicyName:
vsan-default”
…
Gold
Silver
vSAN vSANvSAN
vSphere
• Storage Policy Based
Management(SPBM) for containers
on vSAN
• Kubernetes Storage classes defined by
Admin
• Developers can dynamically self-
provision storage volumes for their
containers
• Full life cycle management without
Admin intervention
• vSAN Data services defined at the
granularity of a container volume

28. © Copyright 2018 Dell Inc.28
Demo