This document discusses persistent storage options for Windows workloads in Kubernetes. It provides an overview of Kubernetes and Windows support, persistent storage concepts in Kubernetes including provisioning workflows, and storage plugin options for Windows including in-tree, FlexVolume, and CSI plugins. It also demonstrates in-tree storage using Azure File and external provisioning with iSCSI.

1. Anusha Ragunathan
Senior Software Engineer, Docker Inc.
Persistent Storage for Windows
workloads in Kubernetes
Deep Debroy
Engineering Manager, Docker Inc.

2. o Kubernetes and Windows
o Persistent Storage in Kubernetes
o Storage Plugins in Windows
Agenda

3. Kubernetes &
Windows

4. Kubernetes and Windows
o Windows support is stable in Kubernetes v1.14
o Worker nodes need to run Windows Server 2019
o Kubernetes master nodes need to be Linux
o Docker engine is the supported container runtime

5. Kubernetes and Windows
q Active Directory integration through GMSA
o Alpha in v1.14
q Support for ContainerD through CRI
o In design phase
q Support for multiple CNI plugins
o Azure CNI, Docker CNI, Flannel
q Persistent Storage

6. Kubernetes Architecture
Kubernetes Master
Scheduler
API Server
Controller
Node
Kubelet
Kube-proxy
CNI/CRI/CSI plugins
Cloud Datacenter
Network Fabric
Datacenter
Storage Fabric
Node
Kubelet
Kube-proxy
CNI/CRI/CSI plugins
Node
Kubelet
Kube-proxy
CNI/CRI/CSI plugins

7. q Cluster-wide controller loops in master nodes
o PV controller
o Attach/Detach controller
o Node controller
q OS-specific logic in worker nodes
o Kubelet
o Container runtimes
o Flexvolume and CSI node plugins
Key components of Kubernetes

8. Persistent storage in
Kubernetes

9. Kubernetes Jargon
● A storage claim made by a
user
● Just like how Pods consume
Compute resources, PVC
consume Volume resources.
● Just like how Pods can
request specific levels CPU &
memory, PVCs can request
specific sizes and access
modes.
● Pods reference PVC
Persistent Volume
Claim (PVC)
Persistent Volume (PV)
● Storage resources in a cluster
● Lifecycle independent of a Pod
Pods
● A set of running containers
representing a workload

10. Lifeof a PV
CreateVolume
AttachVolume (to a Node)
UnMountVolume (from Pod)
MountVolume (to Pod and
format if necessary)
DetachVolume (from Node)
DeleteVolume (after use)
Storage Service
PodPod PodPod
PodPod

11. ● Provisioning is the creation/allocation of Persistent Volumes
● Static and Dynamic Provisioning
○ Static: pre creation of PV
○ Dynamic: automatic creation of PV based on size,
permissions requested.
● Dynamic Provisioning through Storage Classes
○ Provides a way for Admins to describe “classes” of storage
available. Eg, different performance SLAs, value-add features
such as replication, backup, etc
○ Backed by a provisioner
PV Provisioning

12. Provisioning Workflow (static)
1. Cluster Admin
pre-provisions
volumes and
registers PVs
2. Developer
claims a PV
from the pool
4. Developer
references the
claim in a Pod
3. Controller BINDS
PV to PVC
Pool of Persistent Volumes
NFS PV iSCSI PV EBS PV
Claim
`Claim
ClaimClaim
Pod
Claim
5. Controller inspects
Claim and MOUNTs
the PV into the Pod.

13. Provisioning Workflow (dynamic)
1. Cluster Admin
registers
Storage Classes
2. Developer defines a
claim by referring a
storage class
5. Developer
references the
claim in a Pod
4. Controller BINDS
PV to PVC
6. Controller
inspects Claim
and MOUNTs the
PV into the Pod.
SSD
Pod
Claim
Claim
Persistent
Volumes
Storage
Classes
Slow
Fast
3. Controller
provisions
volumes
on-demand

14. • In-tree volume plugins
• Flexvolume plugins
• CSI plugins
• External provisioners
Persistent storage interfaces

15. Plugin Type Pros Cons Future
InTree
No extra installation
necessary
Release cycles tied to
k8s releases
Superseded by CSI
External
Provisioner
Code maintained
independently
Limited customization for
attach and mount ops
Superseded by CSI
FlexVolume
Customizable code for
attach/mount maintained
independently
Host based exec model
reduces portability.
Plugin lifecycle is non-
native to k8s.
Deprecated for Linux
Will be used for Windows
CSI
Highly customizable code
maintained independently
and based on standard
Requires installation and
configuration
The future of storage
plugins
Persistent Storage Interfaces

16. Storage Plugins in
Windows

17. Considerations for Windows
q Disk device and Volume enumeration interfaces
o Object based (rather than file based)
q File System support
o NTFS for block
o SMB for shared
q Powershell cmdlets/scripts for many actions

18. In-tree storage support for Windows
q Core support in kubelet.exe to:
o Partition disk and format volumes with NTFS
o Link volumes to container's file system
o Link SMB shares to container's file system

19. In-tree plugins for Windows
q File based cloud volumes:
o Azure File through SMB
q Block based cloud volumes with attach support:
o Azure Disk
o GCE PD
o EBS [WIP]

20. In tree Plugin Architecture
Worker Worker
Master
Storage Service
Kubelet Kubelet
AttachVolum
e
DetachVolu
me
CreateVolum
e
DeleteVolum
e
Kube Core Components
MountVolume
UnmountVolume
AttachVolume
DetachVolume
CreateVolume
Delete Volume
In-tree
Plugins
PV Claim
API Server

21. Demo 1: In-tree storage with Azure File
Worker Worker
Master
Azure File Storage

22. Flexvolume plugins for Windows
q Attach/mount pre-provisioned volumes over:
o SMB (shared file system)
o iSCSI (dedicated block)
q Dynamic provisioning with external provisioners:
o Set PV source to supported Flexvolume plugins

23. ExternalProvisionerArchitecture
Worker Worker
Master
Storage Service
Kubelet Kubelet
External Provisioner
(StatefulSet/Deployment Pod)
Kube Core Components
External components
API Server
MountVolume
UnmountVolume
AttachVolume
DetachVolume
CreateVolume
Delete Volume
In-tree Plugins
PV Claim
PV
FlexVolume Plugin

24. Demo 2: External Provisioning + iSCSI
Worker Worker
Master
ISCSI Target Server
iSCSI LUN

25. CSI plugins for Windows
q CSI support for Windows is a Work in Progress.
q No privileged support for containers in Windows yet
q Options for deploying plugin
o Host process
o Container with privileged host-based proxy

26. Summary
q In-tree and Flexvolume plugins ready to be used.
q External provisioners coming soon.
q CSI plugins for Windows node plugins in future.

27. What's next
Support for Windows workloads in Kubernetes clusters
coming in Docker Enterprise Tech Preview!!

28. Thank You
Q&A

29. Backup slides

30. CSI node plugins as host process
q Plugin process runs directly on host
q Handles plugin registration with kubelet
o Cannot use node driver registrar sidecar
o Other side cars can be scheduled on Linux
masters.
q Independent life-cycle and log management

31. CSI node plugins as container
q Plugin process packaged and deployed as containers
q Use node driver registrar for plugin registration
q Life-cycle and log management through Kubernetes
q Requires a privileged proxy in host for operations

32. CSIPlugins
Worker Worker
Storage Service
Kubelet Kubelet
External
Provisioner
(Deployment/
StatefulSet)
CreateVolume
DeleteVolume
External
Attacher
(Deployment/
StatefulSet)
CSI Controller
(Deployment/
StatefulSet)
CSI Node
(DaemonSet)
Kube Core Components Kube Sidecar containers CSI plugin components
ControllerPublish
ControllerUnPublish
Master
API Server
NodeStage/NodeUnstage
NodePublish/NodeUnpublish
PV
PV Claim
volume
attachment
CSI
InTree