Anshul Pundir - Senior Software Engineer, Docker Anusha Ragunathan - Senior Software Engineer, Docker Inc In this talk, we will discuss storage concepts related to containers on the Docker platform with the perspective of what is important throughout the lifecycle of an application., We will focus on application provisioning: creating persistent volumes and policies for stateful data and management: replication and failover scenarios, backup/restore, monitoring etc. Through this talk, we will cover the latest storage features and also some of the current and future direction of container storage. Key concepts covered about running stateful applications: - Persistent Volumes - Provisioning (Static vs Topology-aware) - Data Availability (failover with scheduler policies) - Data Protection (using Backup/Restore) - Monitoring (using Prometheus/Grafana dashboards) We will look at each of the characteristics in detail with demos.

1. Docker Inc
@AnushaRagunatha
Anusha Ragunathan
Docker Inc
@anshulpundir
Anshul Pundir

3. Anusha Ragunathan
Sr Software Engineer, Docker
Provisioning and Managing
Storage for Containers
Anshul Pundir
Sr Software Engineer, Docker

4. Disclaimer: Tech Preview

5. Agenda
Storage concepts and use-cases
Storage life-cycle: Provisioning, HA and DR
Future: CSI

6. Containers are
Ephemeral, Storage
is Not

7. Container Storage Ecosystem
is Vast
Container Storage
Specifications
Orchestrat
ion
CI/CD Database ApplicationsBig Data AI/ML Configs Archive
Enterprise Container Platform
Docker Enterprise
Docker Plugins, CSI/Flexvol/InTree/External
Storage
Block/File/Object/SDS

8. Storage Ops is Hard

9. Storage Ops is Hard
A key demand enterprises have is that developers should be able to store data
in clusters without having to worry about how persistent storage is working
under the hood.
Provision
Volumes
Deploy
Apps
HA DR
Understand
Your
use-case
Map
use-case to
tools

10. Characteristics,
protocols and use-cases
Storage Primer

11. Container Platform &
Persistent State
Kubernetes
Workload
Docker Enterprise: Enterprise Container Platform
Docker Swarm
Tasks
Storage Control
Plane
In Tree, External Provisioner,
Flex, CSI
Orchestrator +
Engine (CO)
Docker Volume
Plugin
Storage Engine
Docker Swarm +
Docker Engine +
containerd
Kubernetes +
Docker Engine +
containerd
Applications

12. Storage Types
Image credit: https://bashfulbytes.com/posts/cloudstorage.html

13. Storage use-cases and properties
Type Storage Fit Amount of data Latency Examples
Block High performance,
primary/secondary
Med Very low Structured,
transactional,
Relational DBs
File Capacity based
secondary
Low Trade-off latency for
simplicity
Unstructured, file
backup, archival
Object Highly reliable,
cloud-scale,
primary/secondary
High Low-med Unstructured, big-data
analytics

14. Mapping Applications to Storage
Application
Workloads
Data
Properties
Storage types
Traditional - CRM, CMS, Data Warehousing, Big Data,
Modern - AI/ML, IoT, Genomics, Media Processing
Latency, IOPs, Availability, Volume/Size,
Non/Transactional, Durability, Scalability,
Accessibility
File, Block, Object

15. Provisioning, HA & Backup
Docker Platform +
Kubernetes Storage

16. Kubernetes Jargon
● A storage claim made by a
user
● Just like how Pods consume
Compute resources, PVC
consume Volume resources.
● Just like how Pods can
request specific levels CPU &
memory, PVCs can request
specific sizes and access
modes.
● Pods reference PVC
Persistent Volume
Claim (PVC)
Persistent Volume (PV)
● Storage resources in a cluster
● Lifecycle independent of a Pod
Pods
● A set of running containers
representing a workload

17. ● Static and Dynamic Provisioning
● Dynamic Provisioning through Storage Classes
○ Provides a way for Admins to describe “classes” of
storage available. Eg, different performance SLAs,
value-add features such as replication, backup, etc
○ Backed by a provisioner
● Provisioner is a piece of code responsible for provisioning
storage
○ In-tree
○ External
● Kube API Objects involved: PersistentVolume,
PersistentVolumeClaim
PV Provisioning

18. Provisioning Workflow (static)
1. Cluster Admin
pre-provisions
volumes and
registers PVs
2. Developer
claims a PV
from the pool
4. Developer
references the
claim in a Pod
3. Controller BINDS
PV to PVC
Pool of Persistent Volumes
NFS PV iSCSI PV EBS PV
Claim
`Claim
ClaimClaim
Pod
Claim
5. Controller inspects
Claim and MOUNTs
the PV into the Pod.

19. Provisioning Workflow (dynamic)
1. Cluster Admin
registers
Storage Classes
2. Developer defines a
claim by referring a
storage class
5. Developer
references the
claim in a Pod
4. Controller BINDS
PV to PVC
6. Controller
inspects Claim
and MOUNTs the
PV into the Pod.
SSD
Pod
Claim
Claim
Persistent
Volumes
Storage
Classes
Slow
Fast
3. Controller
provisions
volumes
on-demand

20. Demo 1: PV Provisioning using iSCSI
Worker Worker
Master
ISCSI Target Server

21. High Availability (HA)
● HA involves handling Node failures and Storage failures.
● Node failures:
○ Node drain: Maintenance
○ Node failures: Node lost/ kubelet crash
● Kubernetes provides built-in support for node failures. The
attach-detach controller on the master is in charge of
interacting with the volume plugin and moving PVs across
nodes.
● If delays/instability with attach/detach is not desired, use a
Software Defined Storage (SDS) solution.

22. Demo 2: Storage HA
Worker Worker
Master
ISCSI Target Server
Attach-Detach controller
interacts with volume
plugins to relocate PVs

23. Snapsnot/Restore Concepts
• Storage operators need ability to backup data from snapshots and restore
data from snapshots when desired.
• Requires coordination between container orchestrator and volume
snapshot capabilities of the storage backend exposed through plugins.
• Support in Kubernetes is implemented through CSI in
external-snapshotter side-car controller that invokes plugin APIs.
− Snapshots created as out-of-tree VolumeSnapshot CRD objects
− Restore driven through PersistentVolumeClaim objects with
dataSource pointing to desired VolumeSnapshot

24. Demo 3: Snapshot/Restore of PV
Worker
Master
GCE PD volumes
Start with initial PV
backed by GCE PD
Create VolumeSnapshot object PVSnap1
PV Data1 PV Data1’
Restore by creating PV through a PVC with
DataSource set to VolumeSnapshot PVSnap1

25. Container Storage
Interface (CSI)
Future

26. CSI
Container Storage Interface is an industry standard
storage interface that enables storage vendors to
develop a plugin once and have it work across a
number of container orchestration systems.

27. Why CSI?
Container Orchestrators
Pluggable Storage Interface
Storage Providers
EBS GCE PD Ceph Portworx NetApp

28. Why is CSI a Big Deal?
Provides a
Predictable and
Functional
Interface for
Most Common
Use Cases.
Driven by User
Needs and
Community, not a
Storage Company.
Offers a simple,
community-driven
approach.

29. How will CSI change your life?
Container Orchestration Engines
Implement CSI Directly
Docker Mesos Kubernetes Cloud Foundry
Storage Drivers
Any Storage or Cloud Platform
Kernel Driver FC SAN IP SAN NAS Object Cloud
Other
Frameworks

30. How will CSI change your life?

32. Understanding the use-case,
requirements etc.
Mapping use-case to the right tools.
Understanding how to deploy
fault-tolerant, highly available storage
which can be easily used by the
developers in your organisation.
Storage is hard, and ecosystem is vast, but it can be made manageable by:
Summary

33. Docker is making storage a priority.
Lots of new things are coming, including CSI
support for Kube and iSCSI support.
What’s next?
We are here to help you: assessing your use
cases, requirements and recommending best
practices.

34. Take A Breakout Survey
Access your session and/or workshop surveys for the conference at any time by tapping the Sessions
link on the navigation menu or block on the home screen.
Find the session/workshop you attended and tap on it to view the session details. On this page, you will
find a link to the survey.

35. https://dockr.ly/Forrester
Get the Forrester Report
on Container Platforms
For more information visit:

36. https://dockr.ly/WindowsServerUpgrade
Migrate Legacy Windows
Before End of Support
For more information visit:

37. Come Join Us In San Francisco
April 29-May 2
2019

38. Thank You!

39. BackupSlides

40. Enterprise Container Platform
Software Defined Storage
Orchestration
Storage
~
Enterprise Edition
Containers/Pods
Database
Container Engine
Applications Monitoring Logging
Container Platforms & Persistent State

41. Storage Centric Applications
Databases Analytics: realtime and
offline, CI/CD
e.g. Jenkins slaves, log
analysis e.g. Splunk, big
data/ML workloads etc.

42. Use Cases Category Swarm Kubernetes Properties Cloud
___ Shared Files Native Swarm
Volume, Native
Service Volume
PV/PVC • Medium Term Storage
• Medium Access
• medium Access Speed
• Medium Cost
File Storage
e.g. NFS, SMB,
EFS, AFS
Application State
(RelationaDB, NoSQL
DB)
Monitoring (Time
Series DB)
Structured/
Partially
structured Data
• Long Term Storage
• High Access Count
• High Access Speed
• High Cost
Block Storage
e.g. iSCSI, EBS
Audit Log
Application/Service
Logging
Data Analytics
Genomics
IoT
Unstructured
Data
• Very Long Term
Storage
• Low Access Count
• Low Access Speed
• Low Cost
File Storage
e.g. NFS, SMB,
EFS, AFS
Storage Use Cases

43. Structured Data
• High IOPS
• Low Latency
(consistent)
• Highly Distributed
• High Availability
• High Data
Volume (Up to
Petabytes)
• Transactional
Storage Type
• Block Storage
• e.g. iSCSI, Fiber
• Channel, EBS,
• Google PD, Azure
• Storage Disk -
• Page Blobs or
• Azure Storage
Disk
• Premium Storage
Properties
• Long Term
Storage
• High Access
Count
• High Access
Speed
• High Cost
Applications
• Relational databases
Structured Data

44. Files
Structured Data
• Low IOPS
• Medium Latency
• High Availability
• Low Data Volume
• Non-Transactional
Storage Type
• File Storage
• E.g. NFS, EFS,
AFS etc.
Properties
• Medium Term
Storage
• Medium Access
Count
• Medium Access
Speed
• Medium Cost
Applications

45. Unstructured Data
Structured Data
• High IOPS
• Low Latency
(Consistent)
• Highly Distributed
• High Availability
• High Data Volume
(Up to Petabytes)
• Transactional
Storage Type
• Object Storage
• E.g. S3, Ceph,
Azure Storage
- Block Blob,
Google Cloud
Storage etc.
Properties
• Long Term
Storage
• High Access
Speed
• High Cost
Applications

46. Archival Data
Structured Data
• Low IOPS
• High Latency
• Low Availability
• High Data Volume
• Non-Transactional
WORM
Storage Type
• File Storage
• E.g. NFS, EFS,
AFS etc.
Properties
• Very Long Term
Storage
• Low Access
Count
• Low Access
Speed
• Low Cost
Applications

47. Provisioning Demo
1. Create a targetd account with
credentials
$export NS=default
$kubectl create secret generic
targetd-account
--from-literal=username=admin
--from-literal=password=ciao -n $NS

48. Provisioning Demo
Deploy the external provisioner.
$kubectl apply –f
iscsi-provisioner-d.yaml –n $NS

49. Provisioning Demo
Define the storage class and
persistent volume claims. Observe
persistent volumes getting created.
$ kubectl apply -f sc.yaml -n $NS
$ kubectl apply -f
iscsi-provisioner-pvc.yaml -n $NS

50. Provisioning Demo
Deploy a Pod
kubectl apply -f iscsi-test-pod2.yaml

51. Provisioning Concepts
Volume Types
• Ephemeral Volumes
- Typically used
by secrets
• Remote Volumes
- Cloud e.g.
EBS/EFS, GCE-PD,
Azure Disk.
- On-prem e.g.
vSphere, Glusterfs,
Portworx.
• Local Volumes
• Out of Tree
- e.g. CSI, Flexvol etc
Binding Mode
• Topology Aware
Binding
• Delayed Binding

52. More Provisioning: HA
Scheduler
• Node affinity:
• Topology awareness for volumes:
o Zone aware scheduling of Pods, due to PVs being available only on specific zones. Eg. AWS zones
o GCP zones have instant availability: Regional PDs like RePD in GCP allows instant availability of PVs
across zones.
Storage Availability vs Compute Availability
• SDS offers storage availability by replicating the PVs across
the cluster.
• Kube has built-in availability for PVs, by detaching and
attaching to nodes.