Every team in your organization wants to leverage the latest and greatest technologies, Kafka is of course at the top of the list but should you run one shared cluster or one per team? Should you build and deploy it yourself or leverage a managed service? This talk will dive into the journey you must take in order to reach your ultimate goal making Kafka the commodity all your development teams run on. Building a shared platform service is not like building and shipping an application in your favorite shiny modern programming language. Kafka is stateful and evolves with the use cases built upon it. Building a cloud native Kafka-as-a-Service offering in house is hard, it requires a clear vision, sound automation and committed engineering team with a lot of patience. Confluent Kafka for Kubernetes is a cloud native way of deploying Kafka, at scale, in a self managed way. But how does this help? Combining this technology with GitOps enables developers to precisely describe the desired state of their clusters in each environment. I will be showcasing the key tools and techniques required to make Kafka a commodity and automate operations through Git in a repeatable, secure and controlled manner.

1. The Age of the Clusters
Offering Kafka as a Service in Your Organisation
Sion Smith CTO @ OSO

2. Why choose Kafka as
a Service
Design and Build
the right capability
Introduction
2
Scale your Kafka
+ Your Kafka options
+ Increasing your adoption
curve
+ Explaining KaaS
+ Overview of design
principles
+ Confluent for Kubernetes
+ GitOps
+ 5 design principles
+ Establishing a Centre of
Excellence
+ Team structure
+ More than technology

3. Why choose Kafka
as a Service?

4. Siloed Kafka
Development teams are
responsible for building and
running their own Kafka in siloes
Standardised way of adopting
Kafka throughout your
organisation
Confluent Cloud
Pay-as-you consume, without
infrastructure and operational
complexities
What are your options?
4
Centralised Kafka
Why
Kafka as a Service (KaaS)

5. 5
Goals
+ Lower the barrier to entry
+ Build confidence in the
value of event driven use
cases
+ Share best practice across
teams
+ Increase project success
■ Efficient, focused delivery squads working to achieve clear goals
■ Build confidence in the technology and delivery model
■ Low risk for new users of Kafka
■ Well defined operating model
■ Clear pathway to production
Reusable enablers
Kafka
adoption
→
Siloed Kafka
Why
Increasing your Kafka adoption curve

6. Why build Kafka as a
service capability
+ Focus on your data
+ Reduce operational complexity
+ Maintain data sovereignty
+ Leverage organisation
governance
+ Backbone of your organisation
Why
6
DB
DB
APP
APP
Connector
Connector
DB APP
Connector
Stream
processing
■ The set up needs careful consideration, from build
to managing and scaling
■ It is not like building and shipping an application in
your favorite language

7. Investment & Time
Operational
Capacity
7
High level roadmap
Define success factors at each stage
Why
Design & build
Design and build
reusable foundations
Scale
Iterative development

8. Design and build

9. Best practice Kafka by default
Configurable
01
Visualise, monitor and react to
important changes
Observable
05
Ability to manage and deploy
multiple clusters
Automated
03
Guardrails which foster innovation
in a controlled environment
Secure
04
KaaS design principles
9
Design and build
Programmatically available
on-demand
Elastic
02

10. 10
GitOps Process
Confluent Platform
Docker Images
Confluent for Kubernetes (CFK)
Kubernetes
Design and build
Introducing Confluent
for Kubernetes (CFK)
Complete, declarative API to deploy and
operate Confluent as a cloud-native
system on Kubernetes

11. 11
GitOps Process
Confluent Platform
Kubernetes
Configuration as
code
Git Source controller
Kustomize controller
Flux CD
2.
Desired system state is
versioned
3.
System continuously
polls Git for changes
4.
Approved changes to the
desired state are
automatically applied
5.
Software agents ensure
correctness and alert on
divergence
+ A self-service developer friend
experience to deploying Kafka.
+ Manage Kafka through Git and treat
your brokers as your source code.
+ Provide every product team the ability
to deploy Kafka in a simple, compliant
and repeatable manner.
1.
System is described
declaratively
Design and build
What does the GitOps
process look like?

12. Local
Creating and validating Kafka
configuration
Sandbox
Testing your automation and GitOps
process
Production
Delivering KaaS in a controlled
environment
How to deliver configuration as code
12
Rapidly prototype solutions
High developer velocity
Build trust in the technology
Automate delivery pipeline
Monitor operational excellence
Consistent delivery
Design and build
GitOps
Process
GitOps
Process

13. 13
Design and build
Responsibilities for
design and build
+ Make risk-aware decisions
+ Enable built-in compliance
+ Implement progressive
delivery Producers Consumers
CFK Operator Zookeeper Storage
Brokers
Topics &
partitions
Schemas
ACLs/RBAC
Monitoring, performance, operational
tasks, capacity planning
GitOps process
Kubernetes
Design and build
responsibilities
Tenant responsibilities

14. 14
Default configuration defined in base YAML
with variations defined per tenant.
+ Supports reuse of configuration via the
concept of layering
+ Centralled controlled by Kustomize
+ One-to-one mapping of each environment
Configurable Elastic Automated Secure Observable
Configuration management using kustomize.io
Design and build
{ } { }
{ }
{ }
Base YAML Tenant B
Tenant A
Tenant C
Tenant B
Namespace
Tenant A
Namespace
Tenant C
Namespace
Kustomize
Git

15. 15
Configurable Elastic Automated Secure Observable
Dynamic Kafka provisioning using Kubernetes operator pattern
Design and build
Building a reactive platform to automatically
respond to tenant demands.
+ Deploy Confluent operator to handle
Kafka operations
+ Provides the ability to programmatically
deploy clusters
+ Well defined division of responsibilities
{ }
Tenant A
YAML
GitOps Pipeline
Confluent
Operator
Confluent CRDs
Watches
Create/update cluster
Tenant A Namespace
Zookeeper Cluster
pod pod pod
Kafka Cluster
pod pod pod pod
Create/Deploy Scale

16. Tenant A
16
Configurable Elastic Automated Secure Observable
Manage Kafka infrastructure and deployments using GitOps
Design and build
The Kustomize and Source controller apply
configuration in a standardised way.
+ Git is the single source of truth
+ Automatic cluster reconciliation. e.g. Flux
+ Multiple clusters from a central repository
+ Operations are committed by pull requests
Core Platform
Kustomization
Security
Policy
Source
Controller
Kubernetes
API
Tenant A
Namespace
(Tenant A)
Confluent
Operator
Confluent
CRDs
Kafka
Cluster
Tenant B
Kustomize
Controller
Kafka Config
Change
Poll
Reconcile
Git

17. 17
Configurable Elastic Automated Secure Observable
Assess, audit and govern your Kafka clusters using Open Policy Agent
Design and build
Using policy as code to establish Kafka
guardrails, enforcing built-in compliance.
+ Validate tenant configuration before
its applied to Kafka clusters
+ Validation rules written in Rego
+ Configurable failure notification
{ }
Tenant A
developer
Source
Controller
Kustomize
Controller
Notification
Controller
Kubernetes
API
etcd
OPA
Gatekeeper
Push Pull
Apply
Validate
Passed
Failures

18. 18
Configurable Elastic Automated Secure Observable
Auditable single source of truth through events and API calls
Design and build
Standardised feedback loop providing tenants
self-service developer experience.
+ Flexible notifications on important changes
of the Kafka state
+ Plug into external systems
(e.g. ServiceNow / Jira / Jenkins)
+ Ability to track changes through time
Notification Controller
Core Platform Repository
Kustomize
Controller
Source
Controller
Confluent
Operator
Git Tenant A
Git Tenant B
Schedule
Kubernetes
Resource
Kafka Stream
Slack
Service Now
Platform upgrade event
Git push event
Source
changes
Cluster
changes
SRE
alerts

19. Scale your Kafka

20. 20
Establish a Centre of Excellence
A standardised way of bringing Kafka adoption, governance
and operational best practices to your organisation.
Sharing experiences ensure the benefits are realised beyond
the initial build stage.
Scale
Centre of
excellence
Scale
Design
& build
Transition from
design and build to
Scale
Benefits
+ Agile delivery of new features from a backlog
+ Consistent adoption of Kafka
+ Priorities based on tenant requirements
+ Operational efficiency scaling of your Kafka

21. Your Centre of Excellence team
21
Product Owner
Manage backlog and feature
requests from tenants
Devops Engineer
Automate operations through
repeatable processes
Kafka Developer
Experienced in building best
practice event driven applications
Security Engineer
Identify security risks that arise
from event driven architectures
Support
Provide 1st and 2nd line Kafka
support through service desk
Tester
Validate platform functionality
against business requirements
Outer circle: Tasks performed
Inner circle: Relationships between roles
Scale

22. 22
Futureproof Kafka Operating Model
Scale
Backlog
Feature
requests
Product
Owner
Deploy to
cluster
Create Topics/
Schemas/ ACLs
Producer /
Consumer
Operations
Tenants
KaaS
Boundary
Create cluster
config
Deploy cluster
Centre of
Excellence team
Management
(operations)

23. Thank You!
Contact me
sion@oso.sh
https://www.linkedin.com/in/sionsmith/