This document discusses enhancing data protection workflows with Kanister and Argo Workflows. It begins with discussing the need for data protection of stateful workloads on Kubernetes and challenges with current approaches. It then provides an overview of Kanister, an open source tool for application-level data protection on Kubernetes. Kanister uses custom resources and functions to abstract away complex data protection workflows. It also works with Argo Workflows to scale parallel data operations. The document concludes with a demo of using Kanister's CSI functions to create and restore snapshots and scaling snapshots with Argo Workflows.

1. Michael Cade
Senior Technologist
@MichaelCade1
Ivan Sim
Software Engineer
@ihcsim
Enhancing Data Protection
Workflows With Kanister And
Argo Workflows

2. Agenda
Stateful workloads on Kubernetes
Data Protection with Kubernetes Container Storage
Interface (CSI)
Challenges with data protection workflows
Application-Level data protection with Kanister
CSI data snapshot operations with Kanister
Scaling parallel data operations with Kanister And
Argo Workflows
Q & A

3. © 2022 Kasten by Veeam. All rights reserved. All trademarks are the property of their respective owners.
Why Stateful Workloads On Kubernetes
• Users want:
• Control over compute specification (i.e. compute size, IOPS, cost)
• Utilization of Kubernetes' neutral API (e.g., pod disruption budget, pod affinity/anti-
affinity, resource requirements and limits, workload images and registry, load balancing)
• Specific at-rest encryption scheme
• Compliance with data sovereignty regulation
• Neutral data protection strategy
• Visibility and control into upgrade and rollback mechanism

4. © 2022 Kasten by Veeam. All rights reserved. All trademarks are the property of their respective owners.
Why Stateful Workloads On Kubernetes
• What has changed
• Emergence of operator pattern to ease the automation of data service operation and
maintenance
• Continuous growth and improvement of Container Storage Interface (CSI)
• Grow experience and expertise with cloud native stateful workloads
• Spoiler: managed data services are probably running on Kubernetes anyway

5. © 2022 Kasten by Veeam. All rights reserved. All trademarks are the property of their respective owners.
Container Storage Interface (CSI)
• A standard for exposing arbitrary storage systems to
containerized workloads on orchestration systems like
Kubernetes.
• Manages volume lifecycle with out-of-tree CSI drivers –
provision, attach, mount, snapshot, delete
• Provides a set of optional sidecar containers, maintained
by CSI community, to encapsulate common storage
operations, to be bundled with CSI drivers.
• Not coupled to the Kubernetes core release cycle like the
in-tree volume plugins.

6. © 2022 Kasten by Veeam. All rights reserved. All trademarks are the property of their respective owners.
Kubernetes Container Storage Interface (CSI)
• Some useful features:
• Volume expansion via the PersistentVolumeClaim API
• Volume snapshot via the VolumeSnapshot API
• Volume cloning via the PersistentVolumeClaim data source API
• Volume data populator via the PersistentVolumeClaim data source ref API
• Integrates with secrets stores via the Secrets Stores CSI driver
• etc.

7. © 2022 Kasten by Veeam. All rights reserved. All trademarks are the property of their respective owners.
Data Protection on Kubernetes
• Data protection working group
• Data protection workflows white paper
• Proposed features:
• Volume backups
• Changed block tracking
• Quiesce and unquiesce hooks
• Volume group and group snapshot
• Backup repositories
• Application snapshots and backups
#wg-data-protection
https://github.com/kubernetes/community/blob
/master/wg-data-protection/data-protection-
workflows-white-paper.md

8. © 2022 Kasten by Veeam. All rights reserved. All trademarks are the property of their respective owners.
64.69% of respondents in the
CNCF Survey Report 2021 are
using or planning to use stateful
applications in containers in
production 📈
Cloud native infrastructure and
application architecture have
changed a lot over the years,
while data protection
architecture has fallen
behind 🛠🛠
Kubernetes enables
development and operation
teams to manage resources and
enforce policies via a set of
common APIs. Data protection
solutions should be
managed the same way 🛠
Motivation
Why bother with data protection

9. © 2022 Kasten by Veeam. All rights reserved. All trademarks are the property of their respective owners.
Challenges With Data Protection Workflows
Different snapshot and backup strategy
• Storage-centric snapshots provided by the underlying file or
block storage, limited to crash level consistency only
• Storage-centric with data service hooks to freeze/unfreeze
data service layer during snapshot process
• Data service-centric depends on database specific utilities like
mysqldump, pgdump, mongodump etc. which require direct
access
• Application-centric exercises all the above capabilities in a
coordinated manner

10. © 2022 Kasten by Veeam. All rights reserved. All trademarks are the property of their respective owners.
Challenges With Data Protection Workflows
Many Moving Parts
• An application may cross different domains
• Application Lifecycle: coordinate and
orchestrate scaling up/down workloads,
quiesce/unquiesce
• Different types of targets: object storage,
vendor targets

11. Kanister Overview

12. © 2022 Kasten by Veeam. All rights reserved. All trademarks are the property of their respective owners.
Kanister
• An extensible open source software for application-level data protection on
Kubernetes.
• Abstract away tedious details of curating data protection workflows via a set of
cohesive custom resource definition APIs.
• Implemented as a Kubernetes controller to work well with existing Kubernetes
resources.
• Extensible via custom blueprints and Kanister Functions.
What is it?

13. © 2022 Kasten by Veeam. All rights reserved. All trademarks are the property of their respective owners.
Kanister
API Specifications
Blueprint consists of a set of instructions that tell the controller how to
perform actions on a specific application.
ActionSet instructs the controller to run an action with a set of runtime
parameters, with outputs captured in the status sub resource.
Profile captures information about a location to store data operation
artifacts.

14. © 2022 Kasten by Veeam. All rights reserved. All trademarks are the property of their respective owners.
Kanister Functions
Kanister Functions (https://docs.kanister.io/functions.html)
• Go functions that encapsulate atomic data operations
• Accept input arguments, returns output values
• Structured safe logging, captured by Kanister controller
Built-In Data Operation Functionalities
CreateCSISnapshot
•args:
•name
•pvc
•namespace
•snapshotClass
•outputs:
•name
•pvc
•namespace
•restoreSize
•snapshotContent
RestoreCSISnapshot
•args:
•name
•pvc
•namespace
•restoreSize
•storageClass
•accessModes
•volumeMode
•outputs:
•name
•pvc
•Namespace
•restoreSize
CreateRDSSnapshot
•args:
•instanceID
•dbEngine
•outputs:
•snapshotID
•instanceID
•securityGroupID
•allocatedStorage
ExportRDSSnapshotToLocation
•args:
•instanceID
•namespace
•SnapshotID
•dbEngine
•username
•Password
•databases
•securityGroupID
•outputs:
•snapshotID
•instanceID
•backupID
•securityGroupID
RestoreRDSSnapshot
•args:
•instanceID
•namespace
•SnapshotID
•dbEngine
•username
•Password
•databases
•securityGroupID
•outputs
•endpoint

15. © 2022 Kasten by Veeam. All rights reserved. All trademarks are the property of their respective owners.
Architecture
Execution Walkthrough
Controller
Blueprint

16. © 2022 Kasten by Veeam. All rights reserved. All trademarks are the property of their respective owners.
Architecture
Execution Walkthrough
Controller
Blueprint
ActionSet

17. © 2022 Kasten by Veeam. All rights reserved. All trademarks are the property of their respective owners.
Architecture
Execution Walkthrough
Controller
Blueprint
ActionSet
orchestrate
steps
watches and
read input
parameters

18. © 2022 Kasten by Veeam. All rights reserved. All trademarks are the property of their respective owners.
Architecture
Execution Walkthrough
Controller
Blueprint
ActionSet
Database
Workload
Kanister
Function
orchestrate
steps exec function
watches and
read input
parameters

19. © 2022 Kasten by Veeam. All rights reserved. All trademarks are the property of their respective owners.
Architecture
Execution Walkthrough
Controller
Blueprint
ActionSet
Database
Workload
Kanister
Function
Remote
Storage
orchestrate
steps exec function
watches and
read input
parameters
export artifacts

20. © 2022 Kasten by Veeam. All rights reserved. All trademarks are the property of their respective owners.
Architecture
Execution Walkthrough
Controller
Blueprint
ActionSet
Database
Workload
Remote
Storage
update
progress
and status

21. Demo

22. © 2022 Kasten by Veeam. All rights reserved. All trademarks are the property of their respective owners.
Demo
Objectives
• Create and restore crash-consistency
EBS snapshots using Kanister's CSI
functions
CSI Snapshot & Restore

23. © 2022 Kasten by Veeam. All rights reserved. All trademarks are the property of their respective owners.
Demo
Objectives
• Perform multiple parallel snapshot
operations
Scaling Parallel Data Operations

24. © 2022 Kasten by Veeam. All rights reserved. All trademarks are the property of their respective owners.
Opportunities
Try out Kanister today:
Join our community meeting:
Follow for upcoming exciting roadmap.
Feedback and contributions welcome.
Get Involved!
kanisterio.slack.com
$ helm repo add kanister http://charts.kanister.io
"kanister" has been added to your repositories
$ helm install kanister -n kanister --create-
namespace kanister/kanister-operator
Every other Thursdays, 16:00 UTC
github.com/kanisterio/kanister

25. Questions?

26. Thank You