This document provides an overview of Redis Enterprise and its journey towards being deployed and managed as a Kubernetes operator. Some key points: - Redis Enterprise allows for multi-tenancy by supporting over 200 applications on a single cloud instance through its shared-nothing cluster architecture. - Running stateful applications on Kubernetes presents challenges around ordering operations, persistent storage, and supporting multiple Kubernetes platforms/distributions. - The operator approach was taken to gain full control over the application lifecycle through Go code running as a Kubernetes deployment, including upgrades, validation of configuration changes, and tying the deployment to custom resources. - This provides a simpler way to package, deploy and manage Redis Enterprise on Kubernetes compared to static YAML files or

1. PRESENTED BY
Kubernetes Operators and
The Redis Enterprise Journey
Michal Rabinowitch
Redis Labs, Software Engineer

2. PRESENTED BY

3. PRESENTED BY
Redis Labs
Open source. The leading in-memory database platform,
supporting any high performance operational, analytics or
hybrid use case.
The open source home and commercial provider of
Redis Enterprise (Redise) technology, platform, products
& services.

4. PRESENTED BY
Redis Enterprise Technology
Redis Enterprise Node Redis Enterprise Cluster
• Shared nothing cluster architecture
• Fully compatible with open source
commands & data structures

5. PRESENTED BY
Redis Enterprise : Multi-Tenancy Maximizes Resource Utilization
200+ applications or shards on a single 4vcore cloud instance
Application A
Application B
Application N

6. PRESENTED BY
Multiple Delivery Models
Cloud
Essentials
Cloud
Pro
Downloadable Native Kubernetes
Essentials
PCF over BOSH or PKS
Essentials
OpenShift over
Kubernetes
DATABASE-AS-A-SERVICE SOFTWARE
Fully-managed,
server-less and
hosted
Fully-managed,
hosted or in
customer’s VPC

7. PRESENTED BY
Redis Enterprise Software
and K8s

8. PRESENTED BY
Who Does What
• Node auto-healing
• Node scaling
• Failover & scaling
• Configuration & monitoring
• Service discovery
• Upgrade
+

9. PRESENTED BY
Challenge 1: Running a Stateful App
• Ordered Operations
• Persistent Storage

10. PRESENTED BY
Challenge 1: Running a Stateful App
• Ordered Operations
• Persistent Storage Stateful Set
PV-0 PV-1 PV-2
pod-1pod-0 pod-2pod-1

11. PRESENTED BY
• Supporting multiple K8s platforms and distributions
• storage
• security
• ingress
• Packaging
Challenge 2: Vendors
Operator

12. PRESENTED BY
Building Our Deployment Puzzle
Persistence: Stateful Set with PVs
PV-0 PV-1 PV-2
pod-0 pod-1 pod-2
StatefulSet

13. PRESENTED BY
Building Our Deployment Puzzle
Persistence: Stateful Set with PVs DB Connectivity:
Designated Deployment
Deployment
Services
Rigger
db-1.ns.svc
db-2.ns.svc
PV-0 PV-1 PV-2
pod-0 pod-1 pod-2
StatefulSet

14. PRESENTED BY
Building Our Deployment Puzzle
Persistence: Stateful Set with PVs DB Connectivity:
Designated Deployment
Deployment
Services
Rigger
db-1.ns.svc
db-2.ns.svc
UI service
API service
UI, API: Services
PV-0 PV-1 PV-2
pod-0 pod-1 pod-2
StatefulSet

15. PRESENTED BY
Building Our Deployment Puzzle
Persistence: Stateful Set with PVs DB Connectivity:
Designated Deployment
High Availability: Anti Affinity
Deployment
Services
Rigger
db-1.ns.svc
db-2.ns.svc
UI service
API service
UI, API: Services
K8s Node1 K8s Node2 K8s Node3
PV-0 PV-1 PV-2
pod-0 pod-1 pod-2
StatefulSet
more
resources
...

16. PRESENTED BY
Our Journey Towards an Operator
Started out with 9 static yaml files
– Hard to maintain, no parameterization
– Hard to distribute - poor UX
– No control over the deployment life-cycle
.yaml

17. PRESENTED BY
Our Journey Towards an Operator
Helm - A package manager
– Customized deployment - templating
– Requires installation inside the cluster
– No control over the deployment life-cycle

18. PRESENTED BY
Our Journey Towards an Operator
Operator
– K8s native - no plugin needed
– Simple deployment
– Simple packaging
– Operator SDK to minimize boilerplate
– Go code that runs as a K8s deployment

19. PRESENTED BY
Our Journey Towards an Operator
Operator
– Add domain specific knowledge and
full control over life-cycle
• Upgrade
• Validate configuration changes e.g. downgrade
• Reconfiguration of deployments
• The spec is saved to k8s and not some config file

20. PRESENTED BY
Enter Operator
Operator
manage
Custom Resource
Redis Enterprise Cluster

21. PRESENTED BY
Enter Operator
Deployment
Services
Rigger
db-1.ns.svc
db-2.ns.svc
UI service
API service
K8s Node1 K8s Node2 K8s Node3
PV-0 PV-1 PV-2
pod-0 pod-1 pod-2
StatefulSet
more
resources
...
Operator
Custom Resource: Redis Enterprise Cluster
manage

22. PRESENTED BY
22
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
labelSelector:
matchExpressions:
key: app
operator: In
values:
{{ template "redisenterprise.name" . }}
key: release
operator: In
values:
{{ .Release.Name }}
key: redis.io/role
operator: In
values:
node
topologyKey: kubernetes.io/hostname
terminationGracePeriodSeconds: 31536000
serviceAccountName: {{ template "redisenterprise.serviceAccountName" . }}
{{ with .Values.imagePullSecrets }}
imagePullSecrets:
{{ toYaml . | indent 8 }}
{{ end }}
containers:
name: redis
image: {{ .Values.redisImage.repository }}:{{ .Values.redisImage.tag }}
imagePullPolicy: {{ .Values.redisImage.pullPolicy }}
readinessProbe:
exec:
command:
# check that the node is bootstrapped and that its connected and synced.
bash
c
curl silent localhost:8080/v1/bootstrap && /opt/redislabs/bin/rladmin status
nodes | grep node:$(cat /etc/opt/redislabs/node.id) | grep OK
initialDelaySeconds: 20
timeoutSeconds: 5
lifecycle:
preStop:
exec:
command:
# enslave the node, if this current node is master, change the master to
the first slave node.
bash
c
/opt/redislabs/bin/rladmin node $(cat /etc/opt/redislabs/node.id) enslave
&& ((/opt/redislabs/bin/rladmin status nodes | grep node:$(cat
/etc/opt/redislabs/node.id) | grep q master) && /opt/redislabs/bin/rlutil
change_master master=$(/opt/redislabs/bin/rladmin status nodes | grep slave |
head 1 | cut d " " f 1| cut d ":" f2) && sleep 10) || /bin/true
resources:
{{ toYaml .Values.redisResources | indent 10 }}
ports:
containerPort: 8001
containerPort: 8443
containerPort: 9443
securityContext:
capabilities:
add:
SYS_RESOURCE
{{ if .Values.persistentVolume.enabled }}
volumeMounts:
mountPath: "/opt/persistent"
name: redisstorage
{{ end }}
env:
name: K8S_ORCHASTRATED_DEPLOYMENT
value: "yes"
name: JOIN_HOSTNAME
value: {{ template "redisenterprise.fullname" . }}
{{ if .Values.persistentVolume.enabled }}
name: PERSISTANCE_PATH
value: /opt/persistent
{{ end }}
name: K8S_SERVICE_NAME
value: {{ template "redisenterprise.fullname" . }}
name: BOOTSTRAP_HANDLE_REDIRECTS
value: "enabled"
name: BOOTSTRAP_CLUSTER_FQDN
value: {{ template "redisenterprise.clusterDNS" . }}
name: BOOTSTRAP_DMC_THREADS
value: "10"
name: BOOTSTRAP_USERNAME
valueFrom:
secretKeyRef:
name: {{ template "redisenterprise.fullname" . }}
key: username
name: BOOTSTRAP_PASSWORD
valueFrom:
secretKeyRef:
name: {{ template "redisenterprise.fullname" . }}
key: password
name: BOOTSTRAP_LICENSE
valueFrom:
secretKeyRef:
name: {{ template "redisenterprise.fullname" . }}
key: license
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: {{ template "redisenterprise.statefulsetName" . }}
labels:
app: {{ template "redisenterprise.name" . }}
chart: {{ template "redisenterprise.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
{{ if .Values.persistentVolume.enabled }}
volumeClaimTemplates:
metadata:
name: redisstorage
labels:
app: {{ template "redisenterprise.name" . }}
chart: {{ template "redisenterprise.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: {{ .Values.persistentVolume.size | quote }}
{{ if .Values.persistentVolume.storageClass }}
{{ if (eq "" .Values.persistentVolume.storageClass) }}
storageClassName: ""
{{ else }}
storageClassName: "{{ .Values.persistentVolume.storageClass }}"
{{ end }}
{{ end }}
{{ end }}
serviceName: {{ template "redisenterprise.fullname" . }}
replicas: {{ .Values.replicas }}
updateStrategy:
type: "RollingUpdate"
template:
metadata:
labels:
redis.io/role: node
app: {{ template "redisenterprise.name" . }}
chart: {{ template "redisenterprise.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
{{ with .Values.nodeSelector }}
nodeSelector:
{{ toYaml . | indent 8 }}
{{ end }}

23. PRESENTED BY
23
custom_resource.yaml

24. PRESENTED BY
24
Example: Upgrade Flow
Update Spec with new version1
Running

25. PRESENTED BY
25
Example: Upgrade Flow
Update Spec with new version
Operator Validates
- An upgrade is not taking place at the moment
- This is not a downgrade
- Redis Enterprise Cluster is ready
1
2
Running
Invalid
Upgrade

26. PRESENTED BY
26
Example: Upgrade Flow
User edits RedisEnterpriseCluster Spec with new image
Operator Validates
1
2
3 Upgrade
- Edit Stateful Set spec
- Switch Operator state to “Upgrade”
Running
Upgrade

27. PRESENTED BY
Demo

28. PRESENTED BY
Geo distributed Active Active database
running on multiple clouds environments
deployed using an Operator

29. PRESENTED BY
Active-Active Geo Distribution Database (CRDT-Based)
App
App
App

30. PRESENTED BY
Multi Cloud, Different Deployment Methods
Google
Kubernetes
Engine
US EASTUS WEST
EU WEST
Google
Kubernetes
Engine
Demo

31. PRESENTED BY
Getting started with Redis Enterprise Operator
https://docs.redislabs.com/latest/rs/getting-started/k8s-openshift/

32. Thank you!