1.
Data-Streaming at DKV
Tobias Gockel, Alexander Kropp
Frankfurt, 20.10.2022
GitOps-Integration for the management of Kafka resources

2.
Management Summary
Tobias Gockel
Team manager Platform
Customer Product Services
@ DKV Mobility
Alexander Kropp
IT Consultant
Starting with the context of how our
organization is built around cross functional
product teams and how we are developing
software at DKV we want to describe two
solutions for working with Kafka:
Management for Kafka is a tool that applies
the GitOps approach1 to Kafka cluster
orchestration.
The CLI pod for Kafka allows developers to
comfortably debug following the principle of
least privilege2.
1 GitOps is an operational framework that takes DevOps best practices used for
application development such as version control, collaboration, compliance, and CI/CD
tooling, and applies them to infrastructure automation.
2 A subject should be given only those privileges needed for it to complete its task.
2

3.
Data-Streaming: Organizational context
Customer Product Services:
CPS develops digital products in the cloud for our customers.
Platform-Team
Cockpit
Framework
Pricing
and
Maps
Produkt-Team
1
Produkt-Team
2
Produkt-Team
3
Produkt-Team
4
Produkt-Team
5
Produkt-Team
6
Platform-Team
Cloud Infrastructure
CI/CD Toolchain
Kafka Cluster
IAM
Integrated Portal Smartphone App
Framework Development Product Development
Customer Product Services
App
Framework
3
Self-enablement Scalability Security Automation
Key principles:

4.
Data-Streaming: Motivation
Kafka orchestration at DKV Mobility
Agile Software Development at DKV Mobility
Fairly mature
and modern
stack and
process
GUI via confluent Control Center
No access for developers
Cumbersome workflow for
creating Kafka resources
- Topics
- Service Accounts
- ACLs1
- API-Keys
No agile
development and
not a mature and
efficient process
1 Access Control Lists
Local development Remote development Dev/Test stage
Test/PreProd stage Prod stage Operations
4

5.
Management for Kafka: Technical context
VNET peering
Confluent Cloud DKV Azure Cloud
Dev
Test
Pre
Prod
Kubernetes
Cluster
Kafka Connect
Cluster inside
Kubernetes
Azure
Resources
Self-hosted
agents
Self-enablement Scalability Security Automation
Key principles:
Dev/Test
Pre
Prod
Dedicated
Cluster
VNET
peered
Multi
Availability
Zones
5

6.
Management for Kafka - Architecture
6
Management for Kafka
Entity Management
Topics
Service
Accounts
ACLs
Validation
Data
Schema
Policies
Consistency
Checks
Planning
Desired
State
Current
State
Changes
Deployment
Creation Deletion
Updating
Secret
Handling

7.
Management for Kafka – Entity Management
7
Management for Kafka
Entity Management
Topics
Service
Accounts
ACLs
Validation
Data
Schema
Policies
Consistency
Checks
Planning
Desired
State
Current
State
Changes
Deployment
Creation Deletion
Updating
Secret
Handling
• Structured storing of Kafka entities
• Parameterized scripts to create, update or delete
Kafka entities
• Uses the validation component to check if changes
are allowed
• Parameterized pipelines to simplify the Kafka entity
management
• Avoid wrong inputs
• Simplifies to be compliant to policies
• Automated pull requests
• Create multiple resources at once
• Low complexity
• Alternatively manual pull requests
Characteristics

8.
Management for Kafka - Validation
8
Management for Kafka
Entity Management
Topics
Service
Accounts
ACLs
Validation
Data
Schema
Policies
Consistency
Checks
Planning
Desired
State
Current
State
Changes
Deployment
Creation Deletion
Updating
Secret
Handling
• Validates data schemes of Kafka entities
• Topic config in the correct format?
• Config contains only possible values and no
nonsense?
• Customizable policies of Kafka entities
• Naming conventions
• Required metadata
• Config restrictions for topics or service
accounts
• Consistency checks
• Is there a specific stage order? (e. g. topics
should only exist on higher stages if they are
already on lower stages)
• Should a Kafka entity exist and not exist at
the same time?
Characteristics

9.
Management for Kafka - Planning
9
Management for Kafka
Entity Management
Topics
Service
Accounts
ACLs
Validation
Data
Schema
Policies
Consistency
Checks
Planning
Desired
State
Current
State
Changes
Deployment
Creation Deletion
Updating
Secret
Handling
• Combines all stored Kafka entities to create a
desired state
• Validation component to check if Kafka entities
are valid
• Topics and service accounts which should or
should not exist
• Topic configurations
• API-Keys which should exist (supports Kafka
cluster and Confluent schema registry)
• ACLs which should exist
• Uses API to get the current state
• Compares current state with desired states
• Creates a plan with changes
• Does not change resources which are not part
of Kafka management
• Wrapped inside a parameterizable pipeline
Characteristics

10.
Management for Kafka - Deployment
10
Management for Kafka
Entity Management
Topics
Service
Accounts
ACLs
Validation
Data
Schema
Policies
Consistency
Checks
Planning
Desired
State
Current
State
Changes
Deployment
Creation Deletion
Updating
Secret
Handling
• Deploys changes according to the plan
• Outputs results
• Actual created, updated or deleted Kafka
entities
• Stores API-Keys
• Wrapped inside a parameterizable pipeline
• Approvals
• Stores API-Keys automatically to dedicated
Azure Key Vaults
Characteristics

11.
Management for Kafka - Summary
11
Management for Kafka
Entity Management
Topics
Service
Accounts
ACLs
Validation
Data
Schema
Policies
Consistency
Checks
Planning
Desired
State
Current
State
Changes
Deployment
Creation Deletion
Updating
Secret
Handling
Self-enablement
Scalability
Security
Automation
Key principles:
Easy to use

12.
Debugging Kafka - Motivation
12
Services access the Kafka cluster with dedicated service
accounts with a limited set of ACLs
Initial situation
principle of
least privilege
Why should developers do that differently?
(e. g. personal accounts with access on (almost) everything)
Let us create a self-service to enable the developers to
securely debug their specific Kafka resources!
Our vision ☺

13.
Debugging Kafka - Solution
13
1
2
4
3
5
Request Pod via Azure DevOps
Get credentials for Service Account
Deploy
Debug service
Clean up automatically
Step description
1
2
3
4
5
Workflow

14.
Debugging Kafka - Solution
14
• Self-service to create Kubernetes deployment to debug
Kafka
• Same approval rules as other deployments
• Contains Kafka-CLI scripts and custom scripts
• Access rights of a specific service account
• Enforces that teams can only use service accounts
which belong to them
• Gets cleaned up automatically
• Convenient and secure way to debug Kafka applications
Characteristics
Self-enablement Scalability Security Automation
Key principles:
1
2
3
4
5
Workflow
Request Pod
Get credentials
Deploy
Debug
Clean up

15.
Data Streaming @ DKV – Wrap Up
Kafka orchestration at DKV Mobility
Agile Software Development at DKV Mobility
Fairly mature
and modern
stack and
process
Local development Remote development Dev/Test stage
Test/PreProd stage Prod stage Operations
Fairly mature
and modern
stack and
process
Key principles
Self-enablement
Scalability
Security
Automation
15

16.
16