A short introduction into how the GitOps toolkit can be used to deploy Confluent for Kubernetes. The demo covers: 1. Building a clear Kafka vision 2. Declarative cluster management (including Connectors) 3. Automating Confluent Cloud 4. Demo’ing GitOps with Terraform provision of Confluent Cloud All code for this Demo can be found here: https://github.com/osodevops/confluent-gitops-demo

1. Cloud Native Kafka
Using Kubernetes Operators
May 2022

2. 02
The Design
03
The Solutions
01
The Vision
04
Demo
Your trusted, strategic technology partner
Helping companies adopt emerging technologies to keep competitive and drive business growth
The art of the possible
with kafka
How can we structure the
deliverables
What tooling should we
use where
All singing and dancing
kafka
2

3. 01
The Vision

4. Siloed Kafka
Development teams are
responsible for building and
running their own Kafka in siloes
Standardised way of adopting
Kafka throughout your
organisation
Confluent Cloud
Pay-as-you consume, without
infrastructure and operational
complexities
Where are you today?
4
Centralised Kafka
01. Vision
Kafka as a Service (KaaS)

5. Investment & Time
Value
3
4
5
1
2
Product, Support, Training, Duel Delivery, Technical Account Management
Experimentation /
early interest
Central Nervous
System
Mission critical, in
production, but
disparate lines of
business (LOBs)
Identify a project
Mission-critical,
connected LOBs
Individual Use Case
Individual Team Project
Multi-Use Case Multi-Team
Platform
5
01. Vision
The journey of event streaming

6. 6
Goals
+ Lower the barrier to entry
+ Build confidence in the
value of event driven use
cases
+ Share best practice across
teams
+ Increase project success
■ Efficient, focused delivery squads working to achieve clear goals
■ Build confidence in the technology and delivery model
■ Low risk for new users of Kafka
■ Well defined operating model
■ Clear pathway to production
Reusable enablers
Kafka
adoption
→
Siloed Kafka
01. Vision
Increasing your Kafka adoption curve

7. 7
01. Vision
Adopting a central nervous system
...
Device
Logs ... ...
...
Data Stores Logs 3rd Party Apps Custom Apps / Microservices
Real-time
Portfolio
Real-time Fraud
Detection
Real-time
Customer 360
Machine
Learning
Models
Real-time Data
Transformation ...
Data in Motion Applications
Universal Event Pipeline
SaaS
apps

8. 02
The Design

9. 02. Design
Smart Analytics
(AI/ML)
Anomaly
Detection
Application
Modernization
Data
Exchange
IT Observability &
SIEM Optimization
Compliance &
Regulations
Microservice /
Event Sourcing
Streaming
ETL
Log
Aggregation
IoT / Edge
Analytics
Cyber
Security
DATA INFRASTRUCTURE USE CASES BUSINESS APPLICATION USE CASES
Data
Pipelines
Hybrid & Multicloud
Integration
Customer
360
Mainframe
Augmentation
Data Warehouse
Modernization
Messaging
Modernization
Database
Modernization
Real-time Data beats Slow Data

10. Introducing
Confluent for
Kubernetes
Complete, declarative API
to deploy and operate
Confluent as a
cloud-native system on
Kubernetes
02. Design
Confluent Platform
Confluent for Kubernetes (CFK)
Kubernetes
Red Hat Local Raspberry Pi
AWS Azure
Docker Images
GCP
Standardised way of deploying components no matter the environment or
underlying infrastructure.
10

11. 11
Key benefits of CFK
Operability
> Everything is declared in YAML which removes ambiguity around Terraform
controlling configuration
Reliability
> Operator pattern closes the gap between SRE and Kafka SME. Lower the
barrier to entry with isolated deployments
Maintainability
> Operator pattern is designed to do the heavy lifting. Broker rebalancing, disk
attachment and certificates management are built into CFK
02. Design

12. Local
Creating and validating Kafka
configuration
Sandbox
Testing your automation and GitOps
process
Production
Delivering KaaS in a controlled
environment
How to deliver configuration as code
12
Rapidly prototype solutions
High developer velocity
Build trust in the technology
Automate delivery pipeline
Monitor operational excellence
Consistent delivery
02. Design
GitOps
Process
GitOps
Process

13. 13
+ Multi tenancy allowed many
teams to co-exist in the same
cluster
+ Tenants have segregated
namespaces with RBAC
+ Separate Git repos per tenant
+ Easily automated for onboarding
Multi tenancy using namespaced deployments
02. Design
Tenant A
Kafka
Tenant B
Kafka
Tenant C
Kafka
Flux System
Flux

14. 03
The Solutions

15. Kubernetes Native CRDs for Confluent services
15
03. Solutions
{ } { }
zookeeper.yaml
{ }
schemaregistry
.yaml
{ }
controlcenter
.yaml
K8s Resource
{ }
connect.yaml
● Affinity
● Annotations
● Labels
● Environment Variables
● Tolerations
{ }
Enables
Integrates with Kubernetes
ecosystem
Configuration Overrides
● Server properties
● JVM
● Log4j
Enables
Parity with all Confluent
Platform features
ksqldb.yaml
kafka.yaml

16. 16
How CFK operator works
03. Solutions
Building a reactive platform to automatically
respond to tenant demands.
+ Deploy Confluent operator to handle
Kafka operations
+ Provides the ability to programmatically
deploy clusters
+ Well defined division of responsibilities
{ }
Tenant A
YAML
GitOps Pipeline
Confluent
Operator
Confluent CRDs
Watches
Create/update cluster
Tenant A Namespace
Zookeeper Cluster
pod pod pod
Kafka Cluster
pod pod pod pod
Create/Deploy Scale

17. 17
Default configuration defined in base YAML
with variations defined per tenant.
+ Supports reuse of configuration via the
concept of layering
+ Centralled controlled by Kustomize
+ One-to-one mapping of each environment
Configuration management using kustomize.io
03. Solutions
{ } { }
{ }
{ }
Base YAML Tenant B
Tenant A
Tenant C
Tenant B
Namespace
Tenant A
Namespace
Tenant C
Namespace
Kustomize
Git

18. Tenant A
18
Manage Kafka infrastructure and deployments using GitOps
03. Solutions
The Kustomize and Source controller apply
configuration in a standardised way.
+ Git is the single source of truth
+ Automatic cluster reconciliation. e.g. Flux
+ Multiple clusters from a central repository
+ Operations are committed by pull requests
Core Platform
Kustomization
Security
Policy
Source
Controller
Kubernetes
API
Tenant A
Namespace
(Tenant A)
Confluent
Operator
Confluent
CRDs
Kafka
Cluster
Tenant B
Kustomize
Controller
Kafka Config
Change
Poll
Reconcile
Git

19. 19
TF controller features
Multi-Tenancy
> Supports multi-tenancy by running Terraform plan and apply inside Runner Pods
GitOps Automation for Terraform
> Terraform object to perform plan, apply its associated Terraform resources. It then
stores the TFSTATE of the applied resources as a Secret
Drift detection
> The controller detects and fixes drift for your infrastructures, based on the
Terraform resources and their TFSTATE
Plan and Manual Approve
> Separate the plan, out of the apply step, just like the Terraform workflow
03. Solutions

20. 04
Demo

21. Overview
21
04. Demo
1. Minikube overview
2. Deploy GitOps toolkit
3. Install the Confluent Operator (CFK)
4. Connect Source Controller to GitHub repo
5. Deploy production cluster
6. Deploy sandbox cluster
7. Deploy TF Controller
8. Run Terraform to provision Confluent Cloud

22. Thank You!
Contact me
sion@oso.sh
https://www.linkedin.com/in/sionsmith/