GlusterFS can be used with Kubernetes in several ways: 1) As a volume driver to provide persistent storage and shared access to data across containers using existing GlusterFS volumes. 2) Through local volumes which use hostPath provisioning to leverage GlusterFS mounts but are not suitable for production. 3) With Heketi which provides dynamic provisioning of GlusterFS volumes through a REST API and integration with Kubernetes. 4) Potentially through Rook which aims to integrate storage services like Ceph and GlusterFS to provide turnkey storage and currently supports Ceph.

1. GlusterFS &
Kubernetes
The State of GlusterFS Support in
Kubernetes

2. Joe Julian
● Long time community member.
● Senior Staff Engineer with Samsung SDS/CNCT.
● Certified Kubernetes Administrator (my first cert ever, hope it’s worth the
paper it’s printed on)(I haven’t printed it).
● Insufferable know-it-all with an opinion about everything.

3. What is Kubernetes
Kubernetes is ancient Greek for "Helmsman". Root
of the word "Governor", "Cybernetics".
Kubernetes is a "Container Orchestrator" or "Cluster
Manager".
● Places containers on nodes
● Recovers automatically from failure
● Basic monitoring, logging, health checking
● Enables containers to find each other.

4. How can gluster be used with Kubernetes
● Persistent state Containers can be restarted
at any time due to failed
liveness check or
deployment.

5. How can gluster be used with Kubernetes
● Persistent state
● Shared Storage
Multiple containers may
need to operate on the
same data set.

6. How can gluster be used with Kubernetes
● Persistent state
● Shared Storage
● Cats

7. How can gluster be used with Kubernetes
● Persistent state
● Shared Storage
● Cats

8. GlusterFS volume driver

9. GlusterFS Volume Driver
volumes:
- name: glusterfsvol
glusterfs:
endpoints: glusterfs-cluster
path: kube_vol
readOnly: true
kind: Endpoints
metadata:
name: glusterfs-cluster
subsets:
- addresses:
- ip: 10.0.0.1
- addresses:
- ip: 10.0.0.1

10. GlusterFS Volume Driver
containers:
- name: glusterfs
image: nginx
volumeMounts:
- mountPath: "/mnt/glusterfs"
name: glusterfsvol
Mounts the volume to /mnt/glusterfs
in the container’s chroot.

11. GlusterFS Volume Driver
Pros:
● Works with existing volumes
● Shared data
Cons:
● Not dynamic
● Must be managed externally
● Only shared data

12. Local-volume storage class

13. Local-volume storage class
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: myglustervol
namespace: kube-system
provisioner: kubernetes.io/host-path
The code says:
// This Provisioner is meant for
// development and testing only and
// WILL NOT WORK in a multi-node
// cluster.

14. Local-volume storage class
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: myglustervol
namespace: kube-system
provisioner: kubernetes.io/host-path
The code says:
// This Provisioner is meant for
// development and testing only and
// WILL NOT WORK in a multi-node
// cluster.

15. Local-volume storage class
● Can only use /tmp/hostpath_pv
● Awkward mount
● No actual allocation restriction
● Works great!
/tmp/hostpath_pv glusterfs gluster:/myvol ...
/tmp/hostpath_pv/{{ uuid }}
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: deluge-config
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 50Mi

16. Heketi
GlusterFS storage class

17. GlusterFS+Heketi
Heketi provides a REST api for the
configuration of GlusterFS which can be
(and is) used by several infrastructure tools
including Kubernetes. Support for
GlusterFS using Heketi is part of the main
kubernetes tree and has strong
downstream acceptance.
Heketi is the Taíno (a Caribbian
aboriginal) word for “One”.

18. Heketi
● GlusterFS in pods
● Strongly opinionated cluster design
● Allows metered private PVCs
● Provides an excellent api interface to GlusterFS

19. Heketi
● GlusterFS in pods (privileged)
● Strongly opinionated cluster design
○ Whole disk
○ volume-per-PVC
● Requires prior knowledge
● Doesn’t follow best practices wrt hostnames vs IP addresses

20. Rook

21. Rook
Rook is a completely cloud-native toolset to
provide turnkey storage to kubernetes. It
currently only does so with Ceph, but they
want to add GlusterFS support and are
seeking help.

22. Rook
What I like about rook:
$ helm repo add rook-alpha https://charts.rook.io/alpha
$ helm install rook-alpha/rook

23. Rook
● Established provider of ceph storage to kubernetes
● Downstream user = community contributions!
● Enhance the user experience

24. Rook
● Automated decision making
● Too easy?
● Small team, needs contributions

25. Glusterd V2?

26. Questions?

27. Joe Julian, Samsung SDS
me@joejulian.name
Think about where we’re going with respect to kubernetes.
Contribute to Kubernetes, Rook, Kraken.
Be part of the community!
● Join the sig-storage meetings
● Hang out in #sig-storage on the
kubernetes slack
● Join the rook slack
● Hang out with me in #gluster on IRC
github.com/samsung-cnct/kraken
github.com/kubernetes/kubernetes
github.com/rook/rook
Comment on the Rook + GlusterFS design
document! https://goo.gl/L62Bi2