Containers package code and runtime dependencies to offer greater portability to cloud-native applications. But containers are ephemeral by design. If containers fail, stateful applications lose all of their data, leaving your enterprise open to the risk of lost revenue and lower customer satisfaction. As you consider deploying containers in production, you'll need enterprise-calibre persistent storage that's scalable, secure, and container-aware. Red Hat is the ideal provider for versatile, multi-purpose storage for containerized applications. Red Hat offers storage for containers, letting you attach modern software-defined storage to container platforms or bridge to traditional storage. In addition, Red Hat offers storage in containers, orchestrated by Kubernetes, delivering storage services and applications out of the same containers. Container native storage reaches a new level of storage capabilities on the OpenShift Container Platform. Container-native storage can now be used for all the key infrastructure pieces of OpenShift: the registry, logging, and metrics services.

1. Container Native Storage on Openshift
Container Platform
Jaspreet Kaur
Senior TSE(Openshift Support)

2. AGENDA
● Introduction
● The Journey So Far for OpenShift Storage
● Why CNS?
● What’s shipping today
● New features in CNS 3.6
● Ref Architectures
● Q/A

3. - Break the assumption that “Containers are for stateless workloads only“.
Containers are not persistent by default. App data is lost when containers die.
- Why is container storage a pain point?
Complex. Even more complicated when container orchestration is involved and applications
need to be scaled out.Provide a solution that will run GlusterFs as containers in Openshift
pods
- Why Red Hat Container-Native Storage (CNS)?
Advanced storage capabilities, deeper integration with OpenShift, better price/performance
than traditional storage CNS is open, scalable and has consistent user experience across the
hybrid cloud (compliments the OpenShift value proposition).
Introduction

4. The Journey So Far

5. Storage Options for OpenShift: Cloud
Provider Storage
AUTOMATED CONFIGURATIONNFS
Amazon EBS
Azure Disk
GCE Disk
iSCSI
Ceph RBD
SINGLE CONTROL PANEL
GlusterFS
* very new, handle with care
CHOICE FOR PERSISTENT STORAGE
ISCSI
Amazon EBS
Azure Disk
GCE Disk

6. Storage Options for OpenShift: NFS
WELL UNDERSTOOD, EASY TO DEMO
PROVIDES RWX STORAGE
NO HA, NOT AVAILABLE IN THE CLOUD
NFS
REQUIRES PRE-PROVISIONING

7. Storage Options for OpenShift: iSCSI
WELL UNDERSTOOD IN THE DC
BLOCK STORAGE, NO RWX SUPPORT
NOT AVAILABLE IN THE CLOUD
NFS
REQUIRES PRE-PROVISIONING
iSCSI

8. Storage Options for OpenShift: Cloud
Provider Storage
AVAILABLE IN THE CLOUD
NOT AVAILABLE ACROSS AZs
PERFORMANCE DEPENDS ON SIZE
NFS
Azure Disk
GCE Disk
BLOCK STORAGE, NO RWX SUPPORT
iSCSI
Amazon EBS

9. Storage Options for OpenShift: Cloud
Provider Storage
SCALABLE, HA
CINDER PREFERRED FOR OCP on OSP
NFS
Amazon EBS
Azure Disk
GCE Disk
iSCSI
Ceph RBD
BLOCK STORAGE, NO RWX SUPPORT

10. Storage Options for OpenShift: Cloud
Provider Storage
RUNS ON TOP OF OPENSHIFT
GEO-REPLICATION, SNAPSHOTS*
NFS
Amazon EBS
Azure Disk
GCE Disk
iSCSI
Ceph RBD
SCALABLE FILE STORAGE, RWX, RWO*
SUPPORT
GlusterFS
* very new, handle with care
AVAILABLE EVERYWHERE
S3 OBJECT STORAGE FOR APPS
ISCSI
Amazon EBS
Azure Disk
GCE Disk

11. ● Storage provided by a
dedicated Red Hat Gluster
Storage Cluster over the
network outside OpenShift
● Red Hat Gluster Storage can
run bare metal or VMs
● Storage and compute can
scale independently
Deployment Option-1: Container Ready Storage (CRS)

12. Paradigm Shift
Traditional Storage Management
Converged Storage Management

13. ● Fully featured scale-out
storage platform (RHGS) runs
inside OpenShift in containers
(kubernetes pods)
● Application and storage
containers can be co-located
● Storage services are managed,
scaled and upgraded like app
containers
Deployment Option-2: Container-Native Storage (CNS)

14. Why should you care ?

15. Consistent Storage Experience

16. Storage Consolidation
Container-native
Storage
EBS gp2
EBS gp2
EBS st1
EBS io1
vs.
EBS sc1

17. Simplify Container Availability
AVAILABILITY
ZONE A
AVAILABILITY
ZONE B
AVAILABILITY
ZONE C
Node Node Node Node Node Node
GLOBAL STORAGE NAMESPACE

18. ● Storage can run as a set of micro-services inside
OpenShift - no external appliance needed
● Consistent storage experience across the hybrid cloud
● Storage Consolidation
● Multi purpose (RWO, RWX, object), flexible & versatile
Container Native Storage - Our Advantage

19. ● CNS: providing dynamic persistent storage for openshift with
GlusterFS in a hyper-converged fashion
● Heketi: the high-level service interface to manage the lifecycle of
volumes in multiple Gluster clusters.
Openshift ⇐⇒ Heketi ⇐⇒ Gluster
What is CNS … and what is Heketi?

20. ● openshift/kubernetes
○ dynamic glusterfs provisioner
○ glusterfs plugin
● Heketi
○ high-level service interface for gluster volume lifecycle management
● glusterfs:
○ one or more glusterfs clusters
○ running hyper-converged in openshift
● cns-deploy:
○ tool to deploy gluster and heketi into an existing openshift cluster
Components of CNS

21. PVC(created by user) references the glusterfs provisioner.
● glusterfs provisioner extracts details from PVC
● provisioner tells heketi to create a volume if given size and type.
> heketi looks for a gluster cluster that can satisfy this request
> if found, heketi tells the gluster instance to create the volume.
> gluster creates a volume
> Heketi hands volume back to provisioner
● provisioner creates PV and puts the gluster volume details into it.
● provisioner puts glusterfs as the mount plugin into the pv
● provisioner returns PV to the caller
PVC is bound to the PV and can later be in a pod by the user
PV creation: glusterfs dynamic provisioner

22. Workflow with Dynamic Provisioning
Persistent Volume Claim is
submitted

23. Workflow with Dynamic Provisioning
Openshift request volume to be created

24. Workflow with Dynamic Provisioning
Persistent volume is created by storage system
and registered with Openshift

25. Workflow with Dynamic Provisioning
Volume can now be used by Pod to provide
persistent storage

26. What’s Shipping Today

27. CONTAINER-NATIVE STORAGE ON OPENSHIFT
INFRA

28. CONTAINER-NATIVE STORAGE ON OPENSHIFT
POD
CONTAINERIZED
RED HAT GLUSTER STORAGE
RHEL HOST OS
RED HAT
GLUSTER POD
GLUSTER
VOLUMES
PHYSICAL DISKHOST NETWORK

29. CONTAINER-NATIVE STORAGE ON OPENSHIFT
POD
RHGS
POD
RHGS
POD
RHGS
POD
RHGS
POD
INFRA
POD
INFRA
HEKETI
RESTFUL GLUSTERFS
MANAGEMENT
RED HAT
HEKETI POD
REST API
GLUSTERFS PODS

30. ● Container Ready Storage serving out storage to OpenShift with:
○ RHGS in stand-alone bare-metal storage clusters
○ RHGS inside VMs (VMware most popular) on bare-metal hosts
○ RHGS inside VMs fronting Enterprise Storage Arrays
■ RHGS sits between OpenShift and Storage Arrays
■ LUNs are served out as RHGS bricks
○ RHGS in AWS, Azure, Google Cloud
● Container-Native Storage
○ RHGS runs containerized inside OpenShift Container Platform (always
with Heketi)
○ CNS runs anywhere OCP runs!
Container Storage Deployments

31. Container Storage Use Cases
● Local Storage for Container Images
● OpenShift Infra (registry,logging &
metrics)
● Persistent Storage for Applications

32. OpenShift Registry on CNS
OPENSHIFT INTERNAL
REGISTRY
POD
OPENSHIFT
REGISTRY
OPENSHIFT INTERNAL
REGISTRY
POD
OPENSHIFT
REGISTRY
OPENSHIFT INTERNAL
REGISTRY
POD
OPENSHIFT
REGISTRY
RED HAT GLUSTER CONTAINER-NATIVE STORAGE
Scalable Highly-Available Automated Integrated
PERSISTENT VOLUME

33. ● Brick multiplexing introduced in CNS 3.6
● Optimized storage for logging and metrics.
● Support a scaleout registry.
● S3 object storage for applications.
New features of CNS 3.6

34. Brick multiplexing in CNS 3.6
● With brick-multiplexing, only one glusterfsd
process is governing the bricks such that the
amount of memory consumption of GlusterFS
pods is drastically reduced and the scalability is
significantly improved.
● In 3.6, we are able to support over 1,000
PersistentVolumes in a single container-
native storage cluster. The amount of memory
consumed increases linearly, so that 32GB of
RAM are only needed at the high end of that.
The rule of thumb is roughly 30-35 MB RAM per
volume on each of the participating GlusterFS
pods.

35. Optimized storage for Logging and Metrics
● Container-Native Storage version 3.6 now
provides backend storage for OpenShift
Logging and OpenShift Metrics with gluster-
block. For the moment, the use of gluster-block
in production is only supported for OpenShift
Logging and Metrics services, but use of gluster-
block beyond that is under qualification, and
support is expected to be extended soon.

36. ● The OpenShift Advanced Installer now supports deploying container-native storage and the registry on
container-native storage out of the box.
● All you have to do since OpenShift Container Platform 3.6 is add a few lines to your Ansible inventory file.To deploy an
OpenShift registry backed by container-native storage, first add the following variable definition in the [OSEv3:vars]
section:
openshift_hosted_registry_storage_kind=glusterfs
● And then add a new host group defining the container-native storage nodes to the inventory, for example:
[glusterfs_registry]
infra-1.lab glusterfs_devices='[ "/dev/sdd" ]'
infra-2.lab glusterfs_devices='[ "/dev/sdd" ]'
infra-3.lab glusterfs_devices='[ "/dev/sdd" ]'
Support a scaleout registry

37. S3 object storage for applications
● Container-Native Storage 3.6 now provides an
S3 object storage interface as a TechPreview.
Application developers have a ready-to-use REST
API at hand to provide object storage to workloads
on OpenShift, just a HTTP PUT or GET request
away.
● Object storage in CNS 3.6 provides a simple yet
scalable storage layer for distributed applications
that were previously tied to specific cloud provider
S3 object storage. These application now run with
little or no modification on OpenShift.

38. AWS Reference Architecture Description
OpenShift 3.5 with Container Native Storage on Amazon Web Services Reference Architecture:
https://access.redhat.com/articles/3018151. Ansible playbooks and deplyment scripts found here:
https://github.com/openshift/openshift-ansible-contrib/tree/master/reference-architecture.

39. VMware Reference Architecture Description
OpenShift 3.6 with Container Native Storage on VMware Reference Architecture: New RA
Available July 2017. Ansible playbooks and deplyment scripts found here:
https://github.com/openshift/openshift-ansible-contrib/tree/master/reference-architecture.

40. ● https://github.com/gluster/gluster-kubernetes
● https://github.com/heketi/heketi
● https://github.com/kubernetes/kubernetes : pkg/volume/glusterfs/
● https://github.com/gluster/glusterfs
Open Source! ⇒ Upstream Bits

41. ● pod: group of one or more containers that form an entity
● persistent volume (PV): to be mounted by application pod
● provisioner: to provide PVs upon request
● plugin: mechanism to mount the PV, referenced in PV
● persistent volume claim (PVC): mechanism for a user to request a PV
● Access types for volumes:
○ RWO - read write once (single node)
○ RWX - read write many (multiple nodes)
○ ROX - read only many (multiple nodes)
● flavors of provisioning: dynamic and static
Glossary: OpenShift/Kube Storage

42. THANK YOU