Slides from my talk on "Provisioning and Management of Storage in the Docker Platform"

1. Anusha Ragunathan
Provisioning & Management of
Storage in the Docker platform
Sr Software Engineer, Docker

2. Stateful Apps and Storage
Deploy
Apps and
use the
volume
Planning
for High
availability
Planning
for frequent
Backup and
restore of
the volume
Provisioning
volumes

3. Agenda
Docker Volume Plugins
Docker Enterprise & Kubernetes
Plugins & Ecosystem

4. Docker Volume Plugins

5. ➢ Docker Volume Plugins (aka DVP) are a way to extend the docker
engine's volume subsystem beyond the local host.
➢ Docker provides an API specification for storage providers to plugin into
docker engine.
➢ The plugins are an out-of-tree and out-of-process implementation for:
Create/Remove volumes
Mount/Unmount volumes
Get Volume Capabilities
List volumes
➢ Example plugin: DVP from NetApp
Docker Volume Plugins

6. Life of a volume: Single node
CreateVolume
UnMountVolume (from container)
MountVolume (to container and format if necessary)
DeleteVolume (after use)

7. Life of a volume: Cluster
CreateVolume
AttachVolume (to Node)
Format and partition volume, if necessary
MountVolume (to container)
UnMountVolume (from container)
DetachVolume (from Node)
DeleteVolume (after use)

8. ➢ Docker Swarm is the native container orchestrator from
Docker.
➢ Swarm is not volume aware with respect to scheduling and
orchestration.
➢ Volume Plugins must run independently on each node in a
Swarm cluster. Also Volume requests are simultaneously sent
to each node in a Swarm cluster.
➢ Plugins are expected to be cluster aware
➢ Example of a Swarm volume plugin: Docker’s cloudstor plugin.
It queries the volume state in a cluster using Cloud APIs.
Docker Volume Plugins: Swarm

9. • Around the same time that Docker Swarm was considering
rearchitecting Swarm to be cluster aware, a new community
effort around standardization was evolving.
• Members from Docker, Kubernetes and Mesos worked on
creating a specification that could be established as a
storage standard across Container Orchestrators.
• Thus, Container Storage Interface (CSI) specification
emerged.
• CSI 1.0 spec shipped with Kubernetes 1.13 GA.
• Docker Enterprise supports Kubernetes CSI.
CSI

10. Docker Enterprise +
Kubernetes

11. Storage Jargon in k8s
● A storage claim made by a
user
● PVC consume Volume
resources.
● PVCs can request specific
sizes and access modes.
● Pods reference PVC
Persistent Volume
Claim (PVC)
Persistent Volume (PV)
● Storage resources in a cluster
● Lifecycle independent of a Pod
Pods
● A set of running containers
representing a workload

12. ● Provisioning is the creation of Persistent Volumes
● Static and Dynamic Provisioning
○ Static: pre creation of PV
○ Dynamic: on-demand creation of PV based on size,
permissions and storage backend.
● Dynamic Provisioning through Storage Classes
○ Provides a way for Admins to describe “classes” of
storage available. Example, storage classes can
represent different performance SLAs.
○ Backed by a provisioner. Provisioner is process that
provisions volumes for a specific storage backend
PV Provisioning

13. Provisioning Workflow (static)
1. Cluster Admin
pre-provisions
volumes and
registers PVs
2. Developer
claims a PV
from the pool
4. Developer
references the
claim in a Pod
3. Controller BINDS
PV to PVC
Pool of Persistent Volumes
NFS PV iSCSI PV EBS PV
Claim
`Claim
ClaimClaim
Pod
Claim
5. Controller inspects
Claim and MOUNTs
the PV into the Pod.

14. Provisioning Workflow (dynamic)
1. Cluster Admin
registers
Storage Classes
2. Developer defines a
claim by referring a
storage class
5. Developer
references the
claim in a Pod
4. Controller BINDS
PV to PVC
6. Controller
inspects Claim
and MOUNTs the
PV into the Pod.
SSD
Pod
Claim
Claim
Persistent
Volumes
Storage
Classes
Slow
Fast
3. Controller
provisions
volumes
on-demand

15. High Availability (HA)
● HA involves handling Node failures and Storage failures.
● Node failures:
○ Node drain: Maintenance
○ Node failures: Node lost/ kubelet crash
● Kubernetes provides built-in support for node failures. The
attach-detach controller on the master is in charge of
interacting with the volume plugin and moving PVs across
nodes.
● If delays/instability with attach/detach is not desired, use a
Software Defined Storage (SDS) solution.

16. Plugins & Ecosystem

17. Plugin Type Pros Cons Future
InTree
No extra installation
necessary
Release cycles tied to
k8s releases
Superseded by CSI
External
Provisioner
Code maintained
independently
Limited customization for
attach and mount ops
Superseded by CSI
FlexVolume
Highly customizable code
maintained independently
Host based exec model
reduces portability
Deprecated for Linux
Will be used for Windows
CSI
Highly customizable code
maintained independently
and based on standard
Requires installation and
configuration
The future of storage
plugins
Kubernetes Storage Plugin Ecosystem

18. In tree Plugin Architecture
Worker Worker
Master
Storage Service
Kubelet Kubelet
AttachVolume
DetachVolume
CreateVolume
DeleteVolume
Kube Core Components
MountVolume
UnmountVolume
AttachVolume
DetachVolume
CreateVolume
Delete Volume
In-tree Plugins
PV Claim
API Server

19. In-tree Plugin Scenarios
Cloud Protocol Ephemeral SDS Extensions
AWS EBS NFS local Portworx FlexVolume
GCE PD iSCSI hostpath StorageOS CSI
Azure Disk/File Fibre Channel config_map ScaleIO
Openstack Cinder secret
vSphereVolume

20. External Provisioner Architecture
Worker Worker
Master
Storage Service
Kubelet Kubelet
External Provisioner
(StatefulSet/DeploymentPod)
Kube Core Components
Externalcomponents
API Server
MountVolume
UnmountVolume
AttachVolume
DetachVolume
CreateVolume
Delete Volume
In-tree Plugins
PV Claim
PV

21. External Provisioner Scenarios
In-tree PV source Implementations
[not certified with UCP yet]
iSCSI
Netapp-Trident, Dell/EMC-Isilon, HPE-Nimble, HPE-3PAR, Nutanix,
OpenEBS-iscsi
NFS Netapp-Trident, AWS-EFS, Dell-EMC Isilon

22. CSI Plugins
Worker Worker
Storage Service
Kubelet Kubelet
External
Provisioner
(Deployment/
StatefulSet)
CreateVolume
DeleteVolume
External
Attacher
(Deployment/
StatefulSet)
CSI Controller
(Deployment/
StatefulSet)
CSI Node
(DaemonSet)
Kube Core Components Kube Sidecarcontainers CSI plugin components
ControllerPublish
ControllerUnPublish
Master
API Server
NodeStage/NodeUnstage NodePublish/NodeUnpublish
PV
PV Claim
volume
attachment
CSI In-
Tree

23. Future enterprise features in CSI
❖Backup
➢Snapshot/restore
➢Application consistent/triggered snapshots
❖Volume resizing
❖Cloning and Replication

24. Demo: PV Provisioning & HA using CSI AWS
EBS
Worker Worker
Master
AWS EBS Service

25. OS specific considerations
● Cluster may support Linux and Windows nodes
● Cluster-scoped operations can be OS agnostic
▪ Create, Delete
▪ Attach, Detach
● Node-scoped operations need to be OS aware
▪ Device enumeration
▪ Format, Mount, Dismount

26. OS specific considerations
● File system support varies across OS
▪ NTFS and SMB in Windows
▪ ext, xfs and NFS in Linux
● Support for privileged containers absent in Windows
▪ Plugins running directly on host have no problems
▪ Fully containerized plugins cannot function
▪ Host proxy to support CSI plugins being investigated

27. Varied deployments on-premise and on cloud have several storage options.
Shop around.
The plugin model is standardized through a common industry standard spec -
CSI
Summary
Try Docker and Docker Enterprise

28. Thank You!