Link: https://youtu.be/YVXEpcSclwY https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK In a continuation of a talk given at DoK day at KubeCon EU 2022, join Dinesh Majrekar, Civo's CTO as they walk through their evaluation process of the CNCF Storage market. Civo offers managed Kubernetes clusters powered by K3s to customers around the world. We manage thousands of Virtual Machines and stateful customer data within multiple data centres across several continents. In late 2021, Civo had the opportunity to evaluate the CNCF storage landscape to move to a new technology stack. During the migration project, Civo evaluated Mayastor, Ondat, Ceph and Longhorn against the following metrics: Scalability Performance Ease of Support Attendants will see practical examples on how they could carry out their own similar evaluation and see some of the results of the Civo research project. BIO Dinesh is CTO at Civo. Having worked in the hosting industry for many years, Dinesh has a passion for creating solutions that operate at scale. This not only applies to the technology stack, but for nurturing engineers through their career.

1. Dinesh Majrekar
@DMajrekar
@CivoCloud
Evaluating Cloud Native
Storage Vendors
Dok Talks #147
1st September, 2022

2. @CivoCloud
Dinesh Majrekar
CTO @ Civo
• 15 years experience in the web hosting industry
• Developer, System Architect, DevOps Engineer
• Lead Architect for the CivoStack platform

3. @CivoCloud
Civo
• The Cloud Native Service Provider
• Focused on delivering Kubernetes Clusters
• Simplified developer experience
• Currently hosting over 300TB of end user data across 3
geographic regions

4. @CivoCloud
Why are we migrating?
● Contract was up for renewal, with a price increase
● Experiencing some outages with the current provider
● Allowed us an opportunity to review other offerings in the market
to check if renewal price was justified

5. @CivoCloud
Mayastor
● Evaluation in Summer 2021
● Decided to move to Mayastor from
OpenEBS with support from DataCore (then
Mayastor)

6. @CivoCloud
What were we looking for?
● A partnership where we can grow together
● Enterprise support options
● Open Source / Avoid vendor lock-In
● Underlying technology
● Performance
● Scalability and durability

7. @CivoCloud
Performance Testing
● DD
○ Simple to understand
○ Non representative real world results
● FIO
○ Lots of options
○ Lots of output information

8. @CivoCloud
Demo Environment

9. @CivoCloud
Performance Testing - FIO
● Time Based for demo
○ --time-based --runtime=10
● Impact of Block size
○ --bs=4k / --bs=64k
● Sequential Read vs Random read
○ --rw=read / --rw=randread
○ --rw=write / --rw=randwrite

10. @CivoCloud
Performance Testing - KBench
● Kubernetes Cluster on these nodes
● https://github.com/yasker/kbench
● Quick Testing
● Example of how replication can be tested
○ 1 Replica
○ 2 Replicas

11. @CivoCloud
Demo!!

12. Demo Commands
@CivoCloud
sudo dd if=/dev/zero of=/volume/testfile bs=10G
count=1 oflag=direct
sudo fio --name=demo --filename=/volume/testfile
--direct=1 --size=10GB --bs=64k --ioengine=libaio
--iodepth=64 --runtime=10 --time_based
--group_reporting --eta-newline=1 --numjobs=4
--readwrite=read

13. @CivoCloud
Demo Definitions - PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ms-volume-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
storageClassName: ms-xfs-1-replica-local
# storageClassName: ms-xfs-2-replicas-local

14. apiVersion: batch/v1
kind: Job
metadata:
name: kbench
spec:
template:
metadata:
labels:
kbench: fio
spec:
containers:
- name: kbench
image: yasker/kbench:latest
imagePullPolicy: Always
env:
- name: QUICK_MODE
value: "1"
- name: FILE_NAME
value: "/volume/test"
- name: SIZE
value: "10G" # must be at least 10% smaller than the PVC size
volumeMounts:
- name: ms-volume
mountPath: /volume/
restartPolicy: Never
volumes:
- name: ms-volume
persistentVolumeClaim:
claimName: ms-volume-claim
@CivoCloud
Demo Definitions - Job

15. @CivoCloud
What can influence the results?
● Hardware
● CPU Usage
● Replication
● Your usage profile, reads vs writes, block sizes

16. @CivoCloud
Civo Use Case
● Multi Tenant Public Cloud Environment
● Bare Metal Deployment Stack
● Hyper Converged Hardware already in place
● Day One Deployment of > 2000 PVC, 300TB of Raw Storage

17. @CivoCloud
What was important to us?
● Installation and integration with our existing deployment pipelines
● Comparing speed of Cluster launches
● Evaluating performance loss compared to raw disk access
● Day 2 Operations
○ Monitoring
○ Upgrades
● Commercials

18. @CivoCloud
Thoughts Are From
Summer 2021
Projects have moved on since!!!!!

19. @CivoCloud
Evaluation - Ondat
● Well understood performance profile
● Closed Source
● Support process well known
● Pricing Structure under change

20. @CivoCloud
Evaluation - Longhorn
● Single Pod per PVC
● Team Experience with Longhorn through testing
● Backed by Rancher / SUSE
● Would fit in well with Civo’s k3s usage

21. @CivoCloud
Evaluation - Rook / Ceph
● Previous Team Experience with OpenStack
● Size of Ceph Install small
● HCI not the ideal Ceph deployment pattern
● Mature Stack, great community support
● Proven at scale

22. @CivoCloud
Evaluation - OpenEBS
● Multiple Deployment Drivers
● zfs based options are well proven
● Slow performance
● Feature rich

23. @CivoCloud
Evaluation - Mayastor
● New project - in beta (now progressed to v1.x)
● Commercial Support available from DataCore (Mayadata)
● Lots of features on the roadmap
● Impressive Performance
● Designed for HCI

24. @CivoCloud
Why we choose Mayastor?
● OpenSource
● Commercial Support
● Engagement from DataCore, technical and commercial
● Potential Development Opportunities
○ NVMe over TCP / RDMA

25. @CivoCloud
Mayastor - One year on
● Smooth transition between Mayadata and Datacore
● Delays with first stable release
● Stability of the software has been excellent
● Still missing some features, eg. Snapshots

26. @CivoCloud
Summary
● Tailor your evaluation to your specific use case
● Performance is key, but performance isn’t everything
● Try and find a partner to support your growth, not just a solution
● Design for change

27. @CivoCloud
Learn More
● Learn more with our free Academy - civo.com/academy
● Meet us at KubeCon NA
● Join us in person at Civo Navigate - civo.com/navigate
● Reach out on the Civo Community Slack channel
Questions?