In this webinar, we talk about how to set up redundant and highly available storage for your repos to make your key repositories easier to deploy, more reliable, and easier to back up or move to a different cloud. We reviewed the current practices for highly-available CI/CD and showcased how there’s a better way to do it with OpenEBS.

1. Building a Multi-cloud
Kubernetes Storage
Environment for GitLab
Brian Matheson
bmath@mayadata.io
@brian_matheson
Feb 12th 2020

2. MayaData - Data Agility
STORE AND PROTECT VALIDATE AND HARDEN UNDERSTAND AND CONTROL
CONTAINER ATTACHED STORAGE CHAOS ENGINEERING DATA MIGRATION, HA, DR

3. Agenda
● Introduction
● Demo
● Multi-cloud use cases
○ App Volume Backups on different cloud
○ Active / Active multi-cloud
○ In-service migration to a new cloud
● OpenEBS Release updates
● Q&A

4. Demo
● The Application:
○ GitLab (Gitaly in particular)
● The Clouds:
○ Packet
○ Digital Ocean
● The Topology:
○ Simple single disk Cstor Pool
○ Two storage nodes in Packet, two in DO
○ Worker and master nodes in Packet
○ Storage on SSDs (bare metal in Packet, Virtualized in DO)

5. 5
GitLab
● Free Community Edition
● Helm Chart Available
○ With some caveats
● Awesome Code Repo
○ Repo Data stored in Gitaly
○ Dynamic Persistent Storage
Recommended
○ Need backups
○ Needs to be available for
devs to work
● CI/CD and K8s management too

6. 6
Master
Master
Master
Worker
Worker
Worker
Worker
Storage
Storage
Storage
Storage
Environment
Packet Digital Ocean
Storage
Storage
● Single cluster
● Stretched between Packet and DO
● Same metro region
○ Low network latency
○ ~1.5ms rtt
● Relies on external transit

7. 7
node1
Storage
Configuration
Packet Digital Ocean
● GitLab installed on Packet
● Gitaly process (repo data)
● Talks to iSCSI Target service
● Cstor Storage Engine
○ Robust
○ Replication to three nodes
○ Fast rebuild on network
interruption
● Writes on three different hard disks
in three different nodes
node11
Storage
node3
Storage
node12
Storage
Gitaly
iSCSI
Target

8. A look at the environment

9. Further Investigations:
● Security vetting of the stretched cluster
● Ingress in DO
● Application component placement in DO
● Service mesh testing
● Adding a third cloud
● Chaos testing with Litmus

10. Introduction - Recent Storage Outages
● GCP Persistent Disk Incident Snowballed into a 23-Hour Outage
○ ~4 hours actual incident
○ Affected : us-east1-b
○ High IO wait times, failed IO operations, or IO Operations taking
over 2 seconds to complete
● How @DigitalOcean just killed our company @raisupcom. A long
thread for a very sad story.
○ AI shutdown the instances
○ 2 days with support and social media system was recovered.

11. Introduction - Mitigation Questions
● Why were you not using the multi-region deployments?
● Why were you only hosting one JUST ONE cloud provider?
● Why didn't you have backups outside of JUST ONE cloud provider?
● Why didn't you have a monthly validated Disaster Recovery plan?
● Why did you use that <Cloud Provider> in the first place?
● Why isn't everyone on bare metal?

12. Kubernetes for Stateful Applications
Kubernetes is evolving its capabilities around the stateful applications.
● Cloud Managed Services - are powered through Open source projects
○ Data Lock-In
○ Rising Cloud Costs
● Hybrid and Multi-cloud Strategy
○ Kubernetes is winning
○ Open source projects are getting Kubernetes friendly
○ Increased adoption of Container Attached Storage like OpenEBS
○ Early adopters to early majority

13. Kubernetes - Multi-cloud Strategy
Avoid single zone / data center clusters for production
● Multi-zonal clusters
○ Persistent Storage across zones
○ Asynchronous Backup on another provider
● Multi-provider clusters
○ Nodes from different providers on same cluster.
○ Persistent Storage distributed across providers
● Federated Clusters*

14. GitLab Components - Building Storage
GitLab
(Gitaly) RedisPostgreSQLMinio
Data! Artifacts and Backup User Information Cache/Session Data
Highly Resilient Highly Resilient Highly Resilient Highly performant
Storage
Replication
Non-replicated* Non-replicated* NA**
Needs Backup Needs Backup Needs Backup NA**
Read/Write High on writes High on reads Read Intensive

15. Kubernetes - native Persistent Storage
● Built into Kubernetes
● Easy to setup and manage
● Any platform, any storage
● Protect Data with multi
region/provider storage
● Easily migrate across providers
(hybrid)
● Granular controls per workloads
Completely Open source
Gain control over data
Reduce cloud spending

16. OpenEBS Release Updates
● Monthly Releases!
● 1.6 updates
○ Added support to use Local PV on nodes with taints
○ Add an option to helm chart to work in PSP enabled clusters.
○ Optimize the Jiva replica rebuild process in case of controller or replica restart.
○ Fixes an issue where cleanup of OpenEBS Local PV with hostpaths in OpenShift 4.2 environment
was failing.
○ Fixes an issue where cStor cloned volume was always defaulting to ext4 filesystem. #2809
○ Fixes an issue with openebs velero plugin when used on applications that use annotations to specify
the storage class.
● 1.7 and upcoming releases
○ MayaStor
○ ZFS Local PV
○ CSI Driver for cStor and Jiva
○ Automating ARM builds tracked under #1295

17. Questions!

18. Next Webinar: 4th March 2020
Webinar: Using Litmus Chaos Engineering
& AI for auto incident detection

20. Thank You