Presented at FOSDEM 2019 K8s as a universal control plane to deploy containerised applications • Public cloud is moving on premises (GKE, Outpost) • K8s capable of doing more then containers due to controllers (VMs)

1. Containerized Storage for Containers
@JeffryMolanus
@openEBS
https://openebs.io

2. OpenEBS; asymmetrical block
layer in user-space breaking the
million IOPS barrier

3. • Screwed up the recording — hope its all good for this year
• Touched briefly on storage history, how SAN and NAS came to be
• Mostly to set the context here
• Introduced the concept of Container Attached Storage (CAS)
Today
• Talk about progress we made, our maiden voyage with RUST, and go over
some of the concepts that we are working on
• What you see here today is only worked on by 2 persons
• Hopefully a quick demo
• If what your hear today somewhat excites you; we are (remote) hiring
OpenEBS last year (2018)

4. • Open source project started now roughly 2 years ago
• Sponsored by my employee MayaData
• Provide a cloud native storage abstraction — data plane as well as
control plane which is operated by means of declarative intent such that
it provides a platform for persistent cloud native workloads
• Build on top of Kubernetes which has demonstrated that abstraction and
intent with reconciliation allows developers to focus on the deployment of
the app rather the underlying infra structure
• What k8s does for apps we inspire to do for data
About openEBS

5. How does that look?
on premises Google packet.net
MayaOnline
Analytics
Alerting
Compliance
Policies
Declarative Data Plane
A
P
I
Advisory
Chatbot

6. Motivation
• Applications have changed and someone forgot to tell storage
• The way modern day software is developed and deployed has changed
a lot due to introduction of docker (tarball on steroids)
• Scalability and availability “batteries” are included
• Small teams of people need to deliver “fast and frequently” and
innovations tends to happen in so called shadow IT (skunkworks)
• Born in the cloud — adopts cloud native patterns
• Hardware trends enforce a change in the way we do things
• These change propagate into our software, and the languages we use
• K8s as a universal control plane to deploy containerised applications
• Public cloud is moving on premises (GKE, Outpost)
• K8s capable of doing more then containers due to controllers (VMs)

7. • Register a set of “mountable” things to k8s cluster (PV)
• Take ownership of such a mountable thing — by claiming it (PVC)
• Refer to the PVC in the application
• To avoid having to fill the up a pool of PVs — a dynamic provisioner can
be used that does that automatically
• Potential implications may vary per storage solution (max LUs)
• Storage typically the mother of all snowflakes
• To avoid a wild fire of plugins, a Container Storage Interface (CSI) has
been developed by community members
• Vendor specific implementation (or black magic) hidden from the user
• Make it a pure consumption model
PVs and PVCs in a nutshell

8. Using host paths
apiVersion: v1
kind: Pod
metadata:
name: test-pd
spec:
containers:
- image: k8s.gcr.io/test-webserver
name: test-container
volumeMounts:
- mountPath: /test-pd
name: test-volume
volumes:
- name: test-volume
hostPath:
# directory location on host
path: /data
Unless…

9. The canonical way
kind: PersistentVolume
apiVersion: v1
metadata:
name: task-pv-volume
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/data"
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: task-pv-claim
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
kind: Pod
apiVersion: v1
metadata:
name: mypod
spec:
containers:
- name: myfrontend
image: nginx
volumeMounts:
- mountPath: "/var/www/html"
name: mypd
volumes:
- name: mypd
persistentVolumeClaim:
claimName: task-pv-claim

10. Generic flow of PV/PVCs
Node Node
POD
PVC

11. • How does a developer compose its volume in terms of storage specific
features for that particular workload?
• snapshots, clones, compression, encryption — persona in control
• How do we unify storage differences between different cloud providers
and/or storage vendors?
• They are as incompatible as they can be by design
• How to provide cloud native “EBS volume” look and feel on premisses
using your existing storage infra?
• Don’t trow away existing storage solutions and or vendors
• Make storage as agile — as they applications that they serve
Problem solved?

12. • As data grows — it has the tendency to pull applications towards it
• Everything evolves around the storage systems
• Latency, throughput — IO blender
• If the sun goes super nova, all the apps around it will be gone instantly
i.e huge blast radius
• Typically you have far more PV/PVC’s then you have LUs in a virtual
environment — 1000?
• Typical solution let us replicate the sun!
• Exacerbates the problem instead of solving it?
Data gravity

14. • Data placement is expressed in YAML as part of the application
• Replication factors can be dynamically changed (patch)
• Provide a set of composable transformations layers that can be enabled
based on application specific needs
• As monolithic apps are decomposed — so are their storage needs
• Volumes typically small, allows for data agility
• Allows us to reimagine the how we manage the data
• Runs in containers for containers — prevents depending feature mismatch
between different kernel flavours across distributions and “cloud” images
• Decompose the data in to a collection of small stars
• Monolith vs Micro
OpenEBS approach

16. • The user is faced with writing an application that might run in DC1 or DC2
as the k8s cluster is spanning both.
• DC1 happens to have vendor A and DC2 has vendor B
• typically, vendor A does not work with vendor B — efficiently
• OpenEBS can be used to abstract away the differences between the to
storage systems and make the volume available in both DCs
• Almost like a ‘real’ EBS volume except — we have more control
Data availability example

17. Simple replication of small datasets
PV
CAS
TheBox 1 TheBox 2 TheBox 3
• Data routing, you specify where you want
your data to go
• It is openEBS that connects to TheBox — not
the OS
• The openEBS operator, not shown, instantiates
the needed virtual devices on the fly

18. • Facing different type of storage protocols and performance tiers
• OpenEBS cant fill the performance gap, it is storage not magic
• As time moves one, we want to get “rid” of the slow tier as a faster tier has
become available
• PVs come and go all the time, like the slow tier will be repurposed
• The alternative is to “not deploy” and wait for storage
• How-to move the data, non disruptive?
• Hydrate and then vacate, formerly known as migration aka copy =)
Data Mobility use case

19. Data hydration and mobility
PV
iSCSI iSCSI NBD
iSCSI
hydrate/mirror
• Asymmetrical backends, performance depends on replication mode and
interconnect
• async, semi-sync and sync
• Data migration and hydration — small is the new big we copying GBs not PBs!
CAS

20. • Volumes are small, rebuild in general is quick, how to know what to rebuild
• Although small — you really don’t want to rebuild unused blocks
• General approach is to segment the drive(s) into fixed blocks (e.g 16MB)
• Keep a bitmap of dirty segments as writes come in
• Where to store the bitmap?
• Remember: small (Bonwick on Spacemaps)
• As a new drive/LU is added write out the marked segments to the other
drive(s)
• But, what about thin provisioning, clones, snapshots?
• We have something that does that, but.. maybe next year
• Most of this is not new — standing on the shoulder of giants
• “The design and implementation of a Log Structured filesystem”
Rebuilding

21. Composable storage
PV
Ingress
local remote
T(x)
T(x)
T(x)
Egress
compress, encrypt, mirror

22. Defeating the optimiser

23. Mirror snippet

24. Mirror device snippet

25. (Redacted to make it fit)

26. Declarative

27. Protocols (ingress, egress)
PV CAS
? iSCSI
nvmf-tcp
nvmf-rdma
virtio-fam
NBD
iSCSI
NVMe
nvmf-rdma
virtio-fam
AIO
gluster
Custom
Custom

28. Performance

29. Hugepages

30. Lord of the rings..

31. Lord of the rings..

32. SQs and CQs

33. Rings in CAS
reactor
func(arg1, arg2)
core(n) grpc
poller: func(arg1, interval)
iscsi
Dev

34. Efficiency
SAMSUNG MZWLK1T6HCHP

35. Nvmf-tcp

36. • Some results using a Micro 9200 1.9TB NVMe SSD (840K IOPS on paper)
Protocols matter
0
250
500
750
1.000
NVMe (UIO) NBD iSCSI
840
spec

37. • Note not a very good test — it was on my laptop! but….
• ~30% increase
iscsi vs nvme-tcp
0
2.000
4.000
6.000
8.000
iSCSI nvme-tcp

38. NVMe change is substantial

39. Million IOPS barrier
Very quick demo

40. QUESTIONS?
Did I mention we are hiring?
gila@openebs.io