SlideShare a Scribd company logo
Where Apache Kafka
meets OpenShift
oc whoami
$ oc whoami
rmarting, jromanmartin
$ oc describe user rmarting jromanmartin
Name: Jose Roman Martin Gil
Created: 43 years ago
Labels: father, husband, friend, runner, curious, red hatter,
developer (in any order)
Annotations: Principal Middleware Architect @ Red Hat
Identities:
GitHub: https://github.com/rmarting
LinkedIn: https://www.linkedin.com/in/jromanmartin/ @jromanmartin
Spoiler!!
It is a true love story!
What is Apache Kafka?
Apache Kafka is a distributed system
designed for streams. It is built to be an
horizontally-scalable, fault-tolerant,
commit log, and allows distributed data
streams and stream processing
applications.
What is Apache Kafka?
● Developed at LinkedIn back in 2010, open sourced in 2011
● Designed to be fast, scalable, durable and available
● Distributed by nature
● Data partitioning (sharding)
● High throughput / low latency
● Ability to handle huge number of consumers
Main Use Cases
Rebuild user activity tracking
pipeline as a set of real-time
publish-subscribe feeds. Activity is
published to central topics with
one topic per activity type.
Replacement of traditional
message broker, has better
throughput, built-in
partitioning, replication, and
fault-tolerance. Provides strong
durability.
Messaging Website Activity Tracker
Aggregation of statistics
from distributed
applications to produce
centralized feeds of
operational data.
Metrics
Main Use Cases
Stream Processing Data Integration
Enables continuous, real-time
applications built to react to,
process, or transform streams.
Captures streams of events or
data changes and feeds these
to other data systems.
Abstracts details of files an gives
event data as stream of
messages. Offers good
performance, stronger durability
guarantees due to replication.
Log Aggregation
Apache Kafka Core Components
Zookeeper
Kafka
Applications
Admin tools
Topics and Partitions
● Messages/records are sent to/received from topic
○ Topics are split into one or more partitions
○ All actual work is done on partition level, topic is just a virtual object
● Each message is writing only into one selected partition
○ Partitioning is usually done based on the message key
○ Messaging ordering within the partition is fixed
● Retention policies
○ Based on size/message age
○ Compacted based on message key
● Replication
○ Each partition can exist in one or more backup copies to achieve HA
Topics and Partitions - Producers
old new
0 1 2 3 4 5 6 7 8 9
1
0
1
1
0 1 2 3 4 5 6
0 1 2 3 4 5 6 7 8 9
1
0
Producer
Partition 0
Partition 1
Partition 2
Topics and Partitions - Consumers
old new
0 1 2 3 4 5 6 7 8 9
1
0
1
1
0 1 2 3 4 5 6
0 1 2 3 4 5 6 7 8 9
1
0
Consumer
Partition 0
Partition 1
Partition 2
High Availability
Broker 1
T1 - P1
T1 - P2
T2 - P1
T2 - P2
Broker 2
T1 - P1
T1 - P2
T2 - P1
T2 - P2
Broker 3
T1 - P1
T1 - P2
T2 - P1
T2 - P2
Replication from leaders to followers
High Availability
Broker 1
T1 - P1
T1 - P2
T2 - P1
T2 - P2
Broker 2
T1 - P1
T1 - P2
T2 - P1
T2 - P2
Broker 3
T1 - P1
T1 - P2
T2 - P1
T2 - P2
New leaders on different brokers
High Availability
Broker 1
T1 - P1
T1 - P2
T2 - P1
T2 - P2
Broker 2
T1 - P1
T1 - P2
T2 - P1
T2 - P2
Broker 3
T1 - P1
T1 - P2
T2 - P1
T2 - P2
Producer Consumer
Consumer
Producer
Consumer
Reading and writing to leaders
Partitions Assignment
Topic
Consumer Groups
Partition 0
Partition 1
Partition 2
Partition 3
Group 1
Consumer
Consumer
Group 2
Consumer
Consumer
Consumer
Consumer Groups
Rebalancing
Topic
Partition 0
Partition 1
Partition 2
Partition 3
Group 1
Consumer
Consumer
Group 2
Consumer
Consumer
Consumer
Max parallelism & Idle Consumer
Consumer Groups
Topic
Partition 0
Partition 1
Partition 2
Partition 3
Group 1
Consumer
Consumer
Consumer
Consumer
Consumer
Applications
● They are really “smart” (unlike “traditional” messaging)
● Configured with a “bootstrap servers” list for fetching first metadata
○ Where are interested topics? Connect to broker which holds partition leaders
○ Producer specifies destination partition
○ Consumer handles messages offsets to read
○ If error happens, refresh metadata (something is changed in the cluster)
● Batching on producing/consuming
What is OpenShift?
A distribution of Kubernetes optimized
for continuous application development
and multi-tenant deployment.
SPRING & JAVA EE MICROSERVICES FUNCTIONS
LANGUAGES DATABASES APPLICATION SERVICES
CODE
BUILD TEST DEPLOY
MONITORREVIEW
Self-service
Infrastructure
Automated
build & deploy
CI/CD
pipelines
Consistent
environments
Configuration
management
App logs &
metrics
OpenShift enables Developer Productivity
But ...
● Apache Kafka is stateful which means we require:
○ A stable broker identity
○ A way for the brokers to discover each other on the network
○ Durable broker state
○ The ability to recover broker state after a failure
● All the above are true for Zookeeper as well and …
○ Each node has the configuration of the others
○ To have notes able to communicate each others
● Accessing Kafka is not so simple
● OpenShift provides several services …
Who is going to help me?
Strimzi is a set of enabling services that
allow Apache Kafka to work in OpenShift
as a first class citizen, be installed easily
and configured and managed simply.
Strimzi is our hero!
What is Strimzi?
● Simplifies Apache Kafka deployment on OpenShift
● Uses the OpenShift native mechanisms for:
○ Provisioning Kafka clusters
○ Managing topics and users
● Provides:
○ Linux Container images for running Apache Kafka and Zookeeper
○ Tooling for managing and configuring Apache Kafka clusters, topics and users
● Follows the Kubernetes Operator Model
● Available from OperatorHub.io
OperatorHub.io
● Home for the Kubernetes community to share Operators
● Many different Operators available …
○ Big Data, Cloud Provider, Database, Logging & Tracing, Monitoring, ...
● … And growing up !!!
Kubernetes Operators
● Application-specific controller is used to create, configure and manage
other complex application:
○ The controller contains specific domain/application knowledge
○ Usually used for stateful applications (databases, …) which are non-trivial to
operate on Kubernetes/OpenShift
● Controller operates based on input from Config Maps or Custom
Resource Definitions
○ User describes the desired state
○ Controller applies this state to the application
Operator Pattern
● Monitor the current state of the application
● Compare the actual state to the desire state
● Resolve any differences between actual and desired state
Observe
Analyze
Act
Strimzi Operators
Kafka Operator Topic Operator
User Operator
Strimzi Operators in a glance
Strimzi
OpenShift
Custom Resources
KafkaTopic
Kafka
KafkaUser
Operators
User
Operator
Topic
Operator
Kafka
Operator
Kafka Environment
Deploys
&
Manage
Manage
Users
Manage
Topics
Kafka Cluster
Kafka Broker
Kafka Broker
Kafka Broker
Zookeeper Cluster
Zookeeper Node
Zookeeper Node
Zookeeper Node
Kafka Operator
● Creating and managing Apache Kafka clusters
○ Number of Zookeeper, Kafka Brokers and Kafka Connect nodes
○ Configuration of Kafka and Kafka Connect
● Responsible for
○ Deployment
○ Storage
○ Scale up / Scale Down
○ Metrics
○ Healthchecks
● One Cluster Operator can manage several clusters in parallel
○ Can cover one or more projects
Topic Operator
● Creating and managing Kafka topics
● Some Kafka components (Streams, Connect) often create their own topics
○ Bi-directional synchronization
○ Changes done directly in Kafka/Zookeeper are applied to Custom Resource
Definitions
○ Changes done in Custom Resource Definitions are applied to Kafka topics
● Topic Operator solves this by using 3-way diff
○ Our own Zookeeper-based store
○ Apache Kafka / Zookeeper
○ Custom Resource Definitions
User Operator
● Creating and managing Kafka users
● Unlike Topic Operator, does not sync any changes from the Kafka cluster
with OCP
● It is not expected that the users will be managed directly in Kafka cluster
in parallel with the User operator
● User Credentials managed in a Secret
● Manages authorization rules
Strimzi Custom Resource Definitions
● Extensions of the Kubernetes API to store a collection of API objects
● Allows to introduce an own API into a cluster
● Strimzi CRD
○ Kafka: Definition of a Kafka Cluster
○ KafkaTopic: Definition of a Kafka Topic
○ KafkaUser: Definition of a User
Kafka CRD
apiVersion: kafka.strimzi.io/v1alpha1
kind: Kafka
metadata:
name: meetup-cluster
spec:
kafka:
replicas: 2
# listeners, config, metrics, jvmOptions, resources
storage:
type: persistent-claim
size: 12Gi
zookeeper:
replicas: 3
# jvmOptions, resources
storage:
type: persistent-claim
size: 5Gi
entityOperator:
# topicOperator, userOperator, tlsSidecar
KafkaTopic & KafkaUser CRD
apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaTopic
metadata:
name: meetings
labels:
strimzi.io/cluster: meetup-cluster
spec:
partitions: 10
replicas: 2
apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaUser
metadata:
name: roman
labels:
strimzi.io/cluster: meetup-cluster
spec:
authentication:
type: tls
authorization:
type: simple
acls:
- resource:
type: topic
name: meetings
patternType: literal
operation: Read
Strimzi is much more ...
Demo Time !!!
Get Started !!!
● Strimzi:
○ https://strimzi.io
○ https://github.com/strimzi
● OpenShift:
○ https://www.openshift.com
○ http://learn.openshift.com
● Apache Kafka:
○ https://kafka.apache.org
● OperatorHub.io:
○ https://www.operatorhub.io
Questions?
Thank you!

More Related Content

What's hot

Event Sourcing, Stream Processing and Serverless (Benjamin Stopford, Confluen...
Event Sourcing, Stream Processing and Serverless (Benjamin Stopford, Confluen...Event Sourcing, Stream Processing and Serverless (Benjamin Stopford, Confluen...
Event Sourcing, Stream Processing and Serverless (Benjamin Stopford, Confluen...confluent
 
Introducing Kafka's Streams API
Introducing Kafka's Streams APIIntroducing Kafka's Streams API
Introducing Kafka's Streams APIconfluent
 
Westpac Bank Tech Talk 1: Dive into Apache Kafka
Westpac Bank Tech Talk 1: Dive into Apache KafkaWestpac Bank Tech Talk 1: Dive into Apache Kafka
Westpac Bank Tech Talk 1: Dive into Apache Kafkaconfluent
 
Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...
Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...
Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...confluent
 
Creating Connector to Bridge the Worlds of Kafka and gRPC at Wework (Anoop Di...
Creating Connector to Bridge the Worlds of Kafka and gRPC at Wework (Anoop Di...Creating Connector to Bridge the Worlds of Kafka and gRPC at Wework (Anoop Di...
Creating Connector to Bridge the Worlds of Kafka and gRPC at Wework (Anoop Di...confluent
 
From Zero to Hero with Kafka Connect
From Zero to Hero with Kafka ConnectFrom Zero to Hero with Kafka Connect
From Zero to Hero with Kafka Connectconfluent
 
How to over-engineer things and have fun? | Oto Brglez, OPALAB
How to over-engineer things and have fun? | Oto Brglez, OPALABHow to over-engineer things and have fun? | Oto Brglez, OPALAB
How to over-engineer things and have fun? | Oto Brglez, OPALABHostedbyConfluent
 
Kafka Needs No Keeper
Kafka Needs No KeeperKafka Needs No Keeper
Kafka Needs No KeeperC4Media
 
Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservi...
Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservi...Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservi...
Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservi...HostedbyConfluent
 
Synchronous Commands over Apache Kafka (Neil Buesing, Object Partners, Inc) K...
Synchronous Commands over Apache Kafka (Neil Buesing, Object Partners, Inc) K...Synchronous Commands over Apache Kafka (Neil Buesing, Object Partners, Inc) K...
Synchronous Commands over Apache Kafka (Neil Buesing, Object Partners, Inc) K...confluent
 
Follow the (Kafka) Streams
Follow the (Kafka) StreamsFollow the (Kafka) Streams
Follow the (Kafka) Streamsconfluent
 
Writing Blazing Fast, and Production-Ready Kafka Streams apps in less than 30...
Writing Blazing Fast, and Production-Ready Kafka Streams apps in less than 30...Writing Blazing Fast, and Production-Ready Kafka Streams apps in less than 30...
Writing Blazing Fast, and Production-Ready Kafka Streams apps in less than 30...HostedbyConfluent
 
Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...
Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...
Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...HostedbyConfluent
 
A Modern C++ Kafka API | Kenneth Jia, Morgan Stanley
A Modern C++ Kafka API | Kenneth Jia, Morgan StanleyA Modern C++ Kafka API | Kenneth Jia, Morgan Stanley
A Modern C++ Kafka API | Kenneth Jia, Morgan StanleyHostedbyConfluent
 
Apache Kafka - Scalable Message Processing and more!
Apache Kafka - Scalable Message Processing and more!Apache Kafka - Scalable Message Processing and more!
Apache Kafka - Scalable Message Processing and more!Guido Schmutz
 
Everything you ever needed to know about Kafka on Kubernetes but were afraid ...
Everything you ever needed to know about Kafka on Kubernetes but were afraid ...Everything you ever needed to know about Kafka on Kubernetes but were afraid ...
Everything you ever needed to know about Kafka on Kubernetes but were afraid ...HostedbyConfluent
 
Cross the streams thanks to Kafka and Flink (Christophe Philemotte, Digazu) K...
Cross the streams thanks to Kafka and Flink (Christophe Philemotte, Digazu) K...Cross the streams thanks to Kafka and Flink (Christophe Philemotte, Digazu) K...
Cross the streams thanks to Kafka and Flink (Christophe Philemotte, Digazu) K...confluent
 
Understanding Apache Kafka® Latency at Scale
Understanding Apache Kafka® Latency at ScaleUnderstanding Apache Kafka® Latency at Scale
Understanding Apache Kafka® Latency at Scaleconfluent
 
Improving Logging Ingestion Quality At Pinterest: Fighting Data Corruption An...
Improving Logging Ingestion Quality At Pinterest: Fighting Data Corruption An...Improving Logging Ingestion Quality At Pinterest: Fighting Data Corruption An...
Improving Logging Ingestion Quality At Pinterest: Fighting Data Corruption An...HostedbyConfluent
 
Failing to Cross the Streams – Lessons Learned the Hard Way | Philip Schmitt,...
Failing to Cross the Streams – Lessons Learned the Hard Way | Philip Schmitt,...Failing to Cross the Streams – Lessons Learned the Hard Way | Philip Schmitt,...
Failing to Cross the Streams – Lessons Learned the Hard Way | Philip Schmitt,...HostedbyConfluent
 

What's hot (20)

Event Sourcing, Stream Processing and Serverless (Benjamin Stopford, Confluen...
Event Sourcing, Stream Processing and Serverless (Benjamin Stopford, Confluen...Event Sourcing, Stream Processing and Serverless (Benjamin Stopford, Confluen...
Event Sourcing, Stream Processing and Serverless (Benjamin Stopford, Confluen...
 
Introducing Kafka's Streams API
Introducing Kafka's Streams APIIntroducing Kafka's Streams API
Introducing Kafka's Streams API
 
Westpac Bank Tech Talk 1: Dive into Apache Kafka
Westpac Bank Tech Talk 1: Dive into Apache KafkaWestpac Bank Tech Talk 1: Dive into Apache Kafka
Westpac Bank Tech Talk 1: Dive into Apache Kafka
 
Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...
Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...
Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...
 
Creating Connector to Bridge the Worlds of Kafka and gRPC at Wework (Anoop Di...
Creating Connector to Bridge the Worlds of Kafka and gRPC at Wework (Anoop Di...Creating Connector to Bridge the Worlds of Kafka and gRPC at Wework (Anoop Di...
Creating Connector to Bridge the Worlds of Kafka and gRPC at Wework (Anoop Di...
 
From Zero to Hero with Kafka Connect
From Zero to Hero with Kafka ConnectFrom Zero to Hero with Kafka Connect
From Zero to Hero with Kafka Connect
 
How to over-engineer things and have fun? | Oto Brglez, OPALAB
How to over-engineer things and have fun? | Oto Brglez, OPALABHow to over-engineer things and have fun? | Oto Brglez, OPALAB
How to over-engineer things and have fun? | Oto Brglez, OPALAB
 
Kafka Needs No Keeper
Kafka Needs No KeeperKafka Needs No Keeper
Kafka Needs No Keeper
 
Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservi...
Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservi...Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservi...
Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservi...
 
Synchronous Commands over Apache Kafka (Neil Buesing, Object Partners, Inc) K...
Synchronous Commands over Apache Kafka (Neil Buesing, Object Partners, Inc) K...Synchronous Commands over Apache Kafka (Neil Buesing, Object Partners, Inc) K...
Synchronous Commands over Apache Kafka (Neil Buesing, Object Partners, Inc) K...
 
Follow the (Kafka) Streams
Follow the (Kafka) StreamsFollow the (Kafka) Streams
Follow the (Kafka) Streams
 
Writing Blazing Fast, and Production-Ready Kafka Streams apps in less than 30...
Writing Blazing Fast, and Production-Ready Kafka Streams apps in less than 30...Writing Blazing Fast, and Production-Ready Kafka Streams apps in less than 30...
Writing Blazing Fast, and Production-Ready Kafka Streams apps in less than 30...
 
Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...
Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...
Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...
 
A Modern C++ Kafka API | Kenneth Jia, Morgan Stanley
A Modern C++ Kafka API | Kenneth Jia, Morgan StanleyA Modern C++ Kafka API | Kenneth Jia, Morgan Stanley
A Modern C++ Kafka API | Kenneth Jia, Morgan Stanley
 
Apache Kafka - Scalable Message Processing and more!
Apache Kafka - Scalable Message Processing and more!Apache Kafka - Scalable Message Processing and more!
Apache Kafka - Scalable Message Processing and more!
 
Everything you ever needed to know about Kafka on Kubernetes but were afraid ...
Everything you ever needed to know about Kafka on Kubernetes but were afraid ...Everything you ever needed to know about Kafka on Kubernetes but were afraid ...
Everything you ever needed to know about Kafka on Kubernetes but were afraid ...
 
Cross the streams thanks to Kafka and Flink (Christophe Philemotte, Digazu) K...
Cross the streams thanks to Kafka and Flink (Christophe Philemotte, Digazu) K...Cross the streams thanks to Kafka and Flink (Christophe Philemotte, Digazu) K...
Cross the streams thanks to Kafka and Flink (Christophe Philemotte, Digazu) K...
 
Understanding Apache Kafka® Latency at Scale
Understanding Apache Kafka® Latency at ScaleUnderstanding Apache Kafka® Latency at Scale
Understanding Apache Kafka® Latency at Scale
 
Improving Logging Ingestion Quality At Pinterest: Fighting Data Corruption An...
Improving Logging Ingestion Quality At Pinterest: Fighting Data Corruption An...Improving Logging Ingestion Quality At Pinterest: Fighting Data Corruption An...
Improving Logging Ingestion Quality At Pinterest: Fighting Data Corruption An...
 
Failing to Cross the Streams – Lessons Learned the Hard Way | Philip Schmitt,...
Failing to Cross the Streams – Lessons Learned the Hard Way | Philip Schmitt,...Failing to Cross the Streams – Lessons Learned the Hard Way | Philip Schmitt,...
Failing to Cross the Streams – Lessons Learned the Hard Way | Philip Schmitt,...
 

Similar to Strimzi - Where Apache Kafka meets OpenShift - OpenShift Spain MeetUp

Copy of Kafka-Camus
Copy of Kafka-CamusCopy of Kafka-Camus
Copy of Kafka-CamusDeep Shah
 
Apache Kafka Architecture & Fundamentals Explained
Apache Kafka Architecture & Fundamentals ExplainedApache Kafka Architecture & Fundamentals Explained
Apache Kafka Architecture & Fundamentals Explainedconfluent
 
Apache Kafka's Common Pitfalls & Intricacies: A Customer Support Perspective
Apache Kafka's Common Pitfalls & Intricacies: A Customer Support PerspectiveApache Kafka's Common Pitfalls & Intricacies: A Customer Support Perspective
Apache Kafka's Common Pitfalls & Intricacies: A Customer Support PerspectiveHostedbyConfluent
 
14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...
14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...
14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...Athens Big Data
 
Netflix Data Pipeline With Kafka
Netflix Data Pipeline With KafkaNetflix Data Pipeline With Kafka
Netflix Data Pipeline With KafkaSteven Wu
 
PortoTechHub - Hail Hydrate! From Stream to Lake with Apache Pulsar and Friends
PortoTechHub  - Hail Hydrate! From Stream to Lake with Apache Pulsar and FriendsPortoTechHub  - Hail Hydrate! From Stream to Lake with Apache Pulsar and Friends
PortoTechHub - Hail Hydrate! From Stream to Lake with Apache Pulsar and FriendsTimothy Spann
 
Insta clustr seattle kafka meetup presentation bb
Insta clustr seattle kafka meetup presentation   bbInsta clustr seattle kafka meetup presentation   bb
Insta clustr seattle kafka meetup presentation bbNitin Kumar
 
An Introduction to Apache Kafka
An Introduction to Apache KafkaAn Introduction to Apache Kafka
An Introduction to Apache KafkaAmir Sedighi
 
Princeton Dec 2022 Meetup_ NiFi + Flink + Pulsar
Princeton Dec 2022 Meetup_ NiFi + Flink + PulsarPrinceton Dec 2022 Meetup_ NiFi + Flink + Pulsar
Princeton Dec 2022 Meetup_ NiFi + Flink + PulsarTimothy Spann
 
Apache Kafka 101 by Confluent Developer Friendly
Apache Kafka 101 by Confluent Developer FriendlyApache Kafka 101 by Confluent Developer Friendly
Apache Kafka 101 by Confluent Developer Friendlyitplanningandarchite
 
Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache KafkaRicardo Bravo
 
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...confluent
 
Devfest uk & ireland using apache nifi with apache pulsar for fast data on-r...
Devfest uk & ireland  using apache nifi with apache pulsar for fast data on-r...Devfest uk & ireland  using apache nifi with apache pulsar for fast data on-r...
Devfest uk & ireland using apache nifi with apache pulsar for fast data on-r...Timothy Spann
 
Timothy Spann: Apache Pulsar for ML
Timothy Spann: Apache Pulsar for MLTimothy Spann: Apache Pulsar for ML
Timothy Spann: Apache Pulsar for MLEdunomica
 
Streaming Processing with a Distributed Commit Log
Streaming Processing with a Distributed Commit LogStreaming Processing with a Distributed Commit Log
Streaming Processing with a Distributed Commit LogJoe Stein
 
Introduction to apache kafka
Introduction to apache kafkaIntroduction to apache kafka
Introduction to apache kafkaSamuel Kerrien
 
bigdata 2022_ FLiP Into Pulsar Apps
bigdata 2022_ FLiP Into Pulsar Appsbigdata 2022_ FLiP Into Pulsar Apps
bigdata 2022_ FLiP Into Pulsar AppsTimothy Spann
 
Apache Kafka - Event Sourcing, Monitoring, Librdkafka, Scaling & Partitioning
Apache Kafka - Event Sourcing, Monitoring, Librdkafka, Scaling & PartitioningApache Kafka - Event Sourcing, Monitoring, Librdkafka, Scaling & Partitioning
Apache Kafka - Event Sourcing, Monitoring, Librdkafka, Scaling & PartitioningGuido Schmutz
 

Similar to Strimzi - Where Apache Kafka meets OpenShift - OpenShift Spain MeetUp (20)

Copy of Kafka-Camus
Copy of Kafka-CamusCopy of Kafka-Camus
Copy of Kafka-Camus
 
Apache KAfka
Apache KAfkaApache KAfka
Apache KAfka
 
Apache Kafka Architecture & Fundamentals Explained
Apache Kafka Architecture & Fundamentals ExplainedApache Kafka Architecture & Fundamentals Explained
Apache Kafka Architecture & Fundamentals Explained
 
Apache Kafka's Common Pitfalls & Intricacies: A Customer Support Perspective
Apache Kafka's Common Pitfalls & Intricacies: A Customer Support PerspectiveApache Kafka's Common Pitfalls & Intricacies: A Customer Support Perspective
Apache Kafka's Common Pitfalls & Intricacies: A Customer Support Perspective
 
14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...
14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...
14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...
 
Netflix Data Pipeline With Kafka
Netflix Data Pipeline With KafkaNetflix Data Pipeline With Kafka
Netflix Data Pipeline With Kafka
 
Netflix Data Pipeline With Kafka
Netflix Data Pipeline With KafkaNetflix Data Pipeline With Kafka
Netflix Data Pipeline With Kafka
 
PortoTechHub - Hail Hydrate! From Stream to Lake with Apache Pulsar and Friends
PortoTechHub  - Hail Hydrate! From Stream to Lake with Apache Pulsar and FriendsPortoTechHub  - Hail Hydrate! From Stream to Lake with Apache Pulsar and Friends
PortoTechHub - Hail Hydrate! From Stream to Lake with Apache Pulsar and Friends
 
Insta clustr seattle kafka meetup presentation bb
Insta clustr seattle kafka meetup presentation   bbInsta clustr seattle kafka meetup presentation   bb
Insta clustr seattle kafka meetup presentation bb
 
An Introduction to Apache Kafka
An Introduction to Apache KafkaAn Introduction to Apache Kafka
An Introduction to Apache Kafka
 
Princeton Dec 2022 Meetup_ NiFi + Flink + Pulsar
Princeton Dec 2022 Meetup_ NiFi + Flink + PulsarPrinceton Dec 2022 Meetup_ NiFi + Flink + Pulsar
Princeton Dec 2022 Meetup_ NiFi + Flink + Pulsar
 
Apache Kafka 101 by Confluent Developer Friendly
Apache Kafka 101 by Confluent Developer FriendlyApache Kafka 101 by Confluent Developer Friendly
Apache Kafka 101 by Confluent Developer Friendly
 
Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache Kafka
 
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...
 
Devfest uk & ireland using apache nifi with apache pulsar for fast data on-r...
Devfest uk & ireland  using apache nifi with apache pulsar for fast data on-r...Devfest uk & ireland  using apache nifi with apache pulsar for fast data on-r...
Devfest uk & ireland using apache nifi with apache pulsar for fast data on-r...
 
Timothy Spann: Apache Pulsar for ML
Timothy Spann: Apache Pulsar for MLTimothy Spann: Apache Pulsar for ML
Timothy Spann: Apache Pulsar for ML
 
Streaming Processing with a Distributed Commit Log
Streaming Processing with a Distributed Commit LogStreaming Processing with a Distributed Commit Log
Streaming Processing with a Distributed Commit Log
 
Introduction to apache kafka
Introduction to apache kafkaIntroduction to apache kafka
Introduction to apache kafka
 
bigdata 2022_ FLiP Into Pulsar Apps
bigdata 2022_ FLiP Into Pulsar Appsbigdata 2022_ FLiP Into Pulsar Apps
bigdata 2022_ FLiP Into Pulsar Apps
 
Apache Kafka - Event Sourcing, Monitoring, Librdkafka, Scaling & Partitioning
Apache Kafka - Event Sourcing, Monitoring, Librdkafka, Scaling & PartitioningApache Kafka - Event Sourcing, Monitoring, Librdkafka, Scaling & Partitioning
Apache Kafka - Event Sourcing, Monitoring, Librdkafka, Scaling & Partitioning
 

Recently uploaded

A Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdfA Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdfkalichargn70th171
 
JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)Max Lee
 
Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdfMastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdfmbmh111980
 
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...rajkumar669520
 
Studiovity film pre-production and screenwriting software
Studiovity film pre-production and screenwriting softwareStudiovity film pre-production and screenwriting software
Studiovity film pre-production and screenwriting softwareinfo611746
 
top nidhi software solution freedownload
top nidhi software solution freedownloadtop nidhi software solution freedownload
top nidhi software solution freedownloadvrstrong314
 
OpenChain @ LF Japan Executive Briefing - May 2024
OpenChain @ LF Japan Executive Briefing - May 2024OpenChain @ LF Japan Executive Briefing - May 2024
OpenChain @ LF Japan Executive Briefing - May 2024Shane Coughlan
 
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...Alluxio, Inc.
 
How to install and activate eGrabber JobGrabber
How to install and activate eGrabber JobGrabberHow to install and activate eGrabber JobGrabber
How to install and activate eGrabber JobGrabbereGrabber
 
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with StrimziStrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzisteffenkarlsson2
 
AI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning FrameworkAI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning FrameworkAlluxio, Inc.
 
A Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data MigrationA Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data MigrationHelp Desk Migration
 
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdfImplementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdfVictor Lopez
 
GraphAware - Transforming policing with graph-based intelligence analysis
GraphAware - Transforming policing with graph-based intelligence analysisGraphAware - Transforming policing with graph-based intelligence analysis
GraphAware - Transforming policing with graph-based intelligence analysisNeo4j
 
Advanced Flow Concepts Every Developer Should Know
Advanced Flow Concepts Every Developer Should KnowAdvanced Flow Concepts Every Developer Should Know
Advanced Flow Concepts Every Developer Should KnowPeter Caitens
 
Agnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in KrakówAgnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in Krakówbim.edu.pl
 
De mooiste recreatieve routes ontdekken met RouteYou en FME
De mooiste recreatieve routes ontdekken met RouteYou en FMEDe mooiste recreatieve routes ontdekken met RouteYou en FME
De mooiste recreatieve routes ontdekken met RouteYou en FMEJelle | Nordend
 
Breaking the Code : A Guide to WhatsApp Business API.pdf
Breaking the Code : A Guide to WhatsApp Business API.pdfBreaking the Code : A Guide to WhatsApp Business API.pdf
Breaking the Code : A Guide to WhatsApp Business API.pdfMeon Technology
 
A Guideline to Gorgias to to Re:amaze Data Migration
A Guideline to Gorgias to to Re:amaze Data MigrationA Guideline to Gorgias to to Re:amaze Data Migration
A Guideline to Gorgias to to Re:amaze Data MigrationHelp Desk Migration
 
iGaming Platform & Lottery Solutions by Skilrock
iGaming Platform & Lottery Solutions by SkilrockiGaming Platform & Lottery Solutions by Skilrock
iGaming Platform & Lottery Solutions by SkilrockSkilrock Technologies
 

Recently uploaded (20)

A Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdfA Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
 
JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)
 
Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdfMastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
 
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
 
Studiovity film pre-production and screenwriting software
Studiovity film pre-production and screenwriting softwareStudiovity film pre-production and screenwriting software
Studiovity film pre-production and screenwriting software
 
top nidhi software solution freedownload
top nidhi software solution freedownloadtop nidhi software solution freedownload
top nidhi software solution freedownload
 
OpenChain @ LF Japan Executive Briefing - May 2024
OpenChain @ LF Japan Executive Briefing - May 2024OpenChain @ LF Japan Executive Briefing - May 2024
OpenChain @ LF Japan Executive Briefing - May 2024
 
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
 
How to install and activate eGrabber JobGrabber
How to install and activate eGrabber JobGrabberHow to install and activate eGrabber JobGrabber
How to install and activate eGrabber JobGrabber
 
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with StrimziStrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi
 
AI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning FrameworkAI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning Framework
 
A Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data MigrationA Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data Migration
 
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdfImplementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
 
GraphAware - Transforming policing with graph-based intelligence analysis
GraphAware - Transforming policing with graph-based intelligence analysisGraphAware - Transforming policing with graph-based intelligence analysis
GraphAware - Transforming policing with graph-based intelligence analysis
 
Advanced Flow Concepts Every Developer Should Know
Advanced Flow Concepts Every Developer Should KnowAdvanced Flow Concepts Every Developer Should Know
Advanced Flow Concepts Every Developer Should Know
 
Agnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in KrakówAgnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in Kraków
 
De mooiste recreatieve routes ontdekken met RouteYou en FME
De mooiste recreatieve routes ontdekken met RouteYou en FMEDe mooiste recreatieve routes ontdekken met RouteYou en FME
De mooiste recreatieve routes ontdekken met RouteYou en FME
 
Breaking the Code : A Guide to WhatsApp Business API.pdf
Breaking the Code : A Guide to WhatsApp Business API.pdfBreaking the Code : A Guide to WhatsApp Business API.pdf
Breaking the Code : A Guide to WhatsApp Business API.pdf
 
A Guideline to Gorgias to to Re:amaze Data Migration
A Guideline to Gorgias to to Re:amaze Data MigrationA Guideline to Gorgias to to Re:amaze Data Migration
A Guideline to Gorgias to to Re:amaze Data Migration
 
iGaming Platform & Lottery Solutions by Skilrock
iGaming Platform & Lottery Solutions by SkilrockiGaming Platform & Lottery Solutions by Skilrock
iGaming Platform & Lottery Solutions by Skilrock
 

Strimzi - Where Apache Kafka meets OpenShift - OpenShift Spain MeetUp

  • 2. oc whoami $ oc whoami rmarting, jromanmartin $ oc describe user rmarting jromanmartin Name: Jose Roman Martin Gil Created: 43 years ago Labels: father, husband, friend, runner, curious, red hatter, developer (in any order) Annotations: Principal Middleware Architect @ Red Hat Identities: GitHub: https://github.com/rmarting LinkedIn: https://www.linkedin.com/in/jromanmartin/ @jromanmartin
  • 3. Spoiler!! It is a true love story!
  • 4. What is Apache Kafka? Apache Kafka is a distributed system designed for streams. It is built to be an horizontally-scalable, fault-tolerant, commit log, and allows distributed data streams and stream processing applications.
  • 5. What is Apache Kafka? ● Developed at LinkedIn back in 2010, open sourced in 2011 ● Designed to be fast, scalable, durable and available ● Distributed by nature ● Data partitioning (sharding) ● High throughput / low latency ● Ability to handle huge number of consumers
  • 6. Main Use Cases Rebuild user activity tracking pipeline as a set of real-time publish-subscribe feeds. Activity is published to central topics with one topic per activity type. Replacement of traditional message broker, has better throughput, built-in partitioning, replication, and fault-tolerance. Provides strong durability. Messaging Website Activity Tracker Aggregation of statistics from distributed applications to produce centralized feeds of operational data. Metrics
  • 7. Main Use Cases Stream Processing Data Integration Enables continuous, real-time applications built to react to, process, or transform streams. Captures streams of events or data changes and feeds these to other data systems. Abstracts details of files an gives event data as stream of messages. Offers good performance, stronger durability guarantees due to replication. Log Aggregation
  • 8. Apache Kafka Core Components Zookeeper Kafka Applications Admin tools
  • 9. Topics and Partitions ● Messages/records are sent to/received from topic ○ Topics are split into one or more partitions ○ All actual work is done on partition level, topic is just a virtual object ● Each message is writing only into one selected partition ○ Partitioning is usually done based on the message key ○ Messaging ordering within the partition is fixed ● Retention policies ○ Based on size/message age ○ Compacted based on message key ● Replication ○ Each partition can exist in one or more backup copies to achieve HA
  • 10. Topics and Partitions - Producers old new 0 1 2 3 4 5 6 7 8 9 1 0 1 1 0 1 2 3 4 5 6 0 1 2 3 4 5 6 7 8 9 1 0 Producer Partition 0 Partition 1 Partition 2
  • 11. Topics and Partitions - Consumers old new 0 1 2 3 4 5 6 7 8 9 1 0 1 1 0 1 2 3 4 5 6 0 1 2 3 4 5 6 7 8 9 1 0 Consumer Partition 0 Partition 1 Partition 2
  • 12. High Availability Broker 1 T1 - P1 T1 - P2 T2 - P1 T2 - P2 Broker 2 T1 - P1 T1 - P2 T2 - P1 T2 - P2 Broker 3 T1 - P1 T1 - P2 T2 - P1 T2 - P2 Replication from leaders to followers
  • 13. High Availability Broker 1 T1 - P1 T1 - P2 T2 - P1 T2 - P2 Broker 2 T1 - P1 T1 - P2 T2 - P1 T2 - P2 Broker 3 T1 - P1 T1 - P2 T2 - P1 T2 - P2 New leaders on different brokers
  • 14. High Availability Broker 1 T1 - P1 T1 - P2 T2 - P1 T2 - P2 Broker 2 T1 - P1 T1 - P2 T2 - P1 T2 - P2 Broker 3 T1 - P1 T1 - P2 T2 - P1 T2 - P2 Producer Consumer Consumer Producer Consumer Reading and writing to leaders
  • 15. Partitions Assignment Topic Consumer Groups Partition 0 Partition 1 Partition 2 Partition 3 Group 1 Consumer Consumer Group 2 Consumer Consumer Consumer
  • 16. Consumer Groups Rebalancing Topic Partition 0 Partition 1 Partition 2 Partition 3 Group 1 Consumer Consumer Group 2 Consumer Consumer Consumer
  • 17. Max parallelism & Idle Consumer Consumer Groups Topic Partition 0 Partition 1 Partition 2 Partition 3 Group 1 Consumer Consumer Consumer Consumer Consumer
  • 18. Applications ● They are really “smart” (unlike “traditional” messaging) ● Configured with a “bootstrap servers” list for fetching first metadata ○ Where are interested topics? Connect to broker which holds partition leaders ○ Producer specifies destination partition ○ Consumer handles messages offsets to read ○ If error happens, refresh metadata (something is changed in the cluster) ● Batching on producing/consuming
  • 19. What is OpenShift? A distribution of Kubernetes optimized for continuous application development and multi-tenant deployment.
  • 20. SPRING & JAVA EE MICROSERVICES FUNCTIONS LANGUAGES DATABASES APPLICATION SERVICES CODE BUILD TEST DEPLOY MONITORREVIEW Self-service Infrastructure Automated build & deploy CI/CD pipelines Consistent environments Configuration management App logs & metrics OpenShift enables Developer Productivity
  • 21. But ... ● Apache Kafka is stateful which means we require: ○ A stable broker identity ○ A way for the brokers to discover each other on the network ○ Durable broker state ○ The ability to recover broker state after a failure ● All the above are true for Zookeeper as well and … ○ Each node has the configuration of the others ○ To have notes able to communicate each others ● Accessing Kafka is not so simple ● OpenShift provides several services …
  • 22. Who is going to help me?
  • 23. Strimzi is a set of enabling services that allow Apache Kafka to work in OpenShift as a first class citizen, be installed easily and configured and managed simply. Strimzi is our hero!
  • 24. What is Strimzi? ● Simplifies Apache Kafka deployment on OpenShift ● Uses the OpenShift native mechanisms for: ○ Provisioning Kafka clusters ○ Managing topics and users ● Provides: ○ Linux Container images for running Apache Kafka and Zookeeper ○ Tooling for managing and configuring Apache Kafka clusters, topics and users ● Follows the Kubernetes Operator Model ● Available from OperatorHub.io
  • 25. OperatorHub.io ● Home for the Kubernetes community to share Operators ● Many different Operators available … ○ Big Data, Cloud Provider, Database, Logging & Tracing, Monitoring, ... ● … And growing up !!!
  • 26. Kubernetes Operators ● Application-specific controller is used to create, configure and manage other complex application: ○ The controller contains specific domain/application knowledge ○ Usually used for stateful applications (databases, …) which are non-trivial to operate on Kubernetes/OpenShift ● Controller operates based on input from Config Maps or Custom Resource Definitions ○ User describes the desired state ○ Controller applies this state to the application
  • 27. Operator Pattern ● Monitor the current state of the application ● Compare the actual state to the desire state ● Resolve any differences between actual and desired state Observe Analyze Act
  • 28. Strimzi Operators Kafka Operator Topic Operator User Operator
  • 29. Strimzi Operators in a glance Strimzi OpenShift Custom Resources KafkaTopic Kafka KafkaUser Operators User Operator Topic Operator Kafka Operator Kafka Environment Deploys & Manage Manage Users Manage Topics Kafka Cluster Kafka Broker Kafka Broker Kafka Broker Zookeeper Cluster Zookeeper Node Zookeeper Node Zookeeper Node
  • 30. Kafka Operator ● Creating and managing Apache Kafka clusters ○ Number of Zookeeper, Kafka Brokers and Kafka Connect nodes ○ Configuration of Kafka and Kafka Connect ● Responsible for ○ Deployment ○ Storage ○ Scale up / Scale Down ○ Metrics ○ Healthchecks ● One Cluster Operator can manage several clusters in parallel ○ Can cover one or more projects
  • 31. Topic Operator ● Creating and managing Kafka topics ● Some Kafka components (Streams, Connect) often create their own topics ○ Bi-directional synchronization ○ Changes done directly in Kafka/Zookeeper are applied to Custom Resource Definitions ○ Changes done in Custom Resource Definitions are applied to Kafka topics ● Topic Operator solves this by using 3-way diff ○ Our own Zookeeper-based store ○ Apache Kafka / Zookeeper ○ Custom Resource Definitions
  • 32. User Operator ● Creating and managing Kafka users ● Unlike Topic Operator, does not sync any changes from the Kafka cluster with OCP ● It is not expected that the users will be managed directly in Kafka cluster in parallel with the User operator ● User Credentials managed in a Secret ● Manages authorization rules
  • 33. Strimzi Custom Resource Definitions ● Extensions of the Kubernetes API to store a collection of API objects ● Allows to introduce an own API into a cluster ● Strimzi CRD ○ Kafka: Definition of a Kafka Cluster ○ KafkaTopic: Definition of a Kafka Topic ○ KafkaUser: Definition of a User
  • 34. Kafka CRD apiVersion: kafka.strimzi.io/v1alpha1 kind: Kafka metadata: name: meetup-cluster spec: kafka: replicas: 2 # listeners, config, metrics, jvmOptions, resources storage: type: persistent-claim size: 12Gi zookeeper: replicas: 3 # jvmOptions, resources storage: type: persistent-claim size: 5Gi entityOperator: # topicOperator, userOperator, tlsSidecar
  • 35. KafkaTopic & KafkaUser CRD apiVersion: kafka.strimzi.io/v1alpha1 kind: KafkaTopic metadata: name: meetings labels: strimzi.io/cluster: meetup-cluster spec: partitions: 10 replicas: 2 apiVersion: kafka.strimzi.io/v1alpha1 kind: KafkaUser metadata: name: roman labels: strimzi.io/cluster: meetup-cluster spec: authentication: type: tls authorization: type: simple acls: - resource: type: topic name: meetings patternType: literal operation: Read
  • 36. Strimzi is much more ...
  • 38. Get Started !!! ● Strimzi: ○ https://strimzi.io ○ https://github.com/strimzi ● OpenShift: ○ https://www.openshift.com ○ http://learn.openshift.com ● Apache Kafka: ○ https://kafka.apache.org ● OperatorHub.io: ○ https://www.operatorhub.io