Kafka with IBM Event Streams - Technical Presentation

© 2019 IBM CorporationIBM Event Streams
IBM Event Streams
Apache Kafka® for the Enterprise
A core part of Cloud Pak for Integration
Technical Overview | November 2019

IBM Event Streams is fully
supported Apache Kafka®
with value-add capabilities
2
Powerful Ops Tooling
Award-Winning User Experience
Connector Catalog
Support you can Trust
Unrivalled MQ connectivity
Schema Registry
Geo-replication for DR
IBMEventStreams

Deployment Options to Suit Different Needs
3
Event Streams
for IBM Cloud
Isolated
aaS
Multi-tenant
aaS
IBM Event Streams
Red Hat OpenShift
&
ICP
Container–native Software
X86_64 zLinux
Cloud Pak for Integration
Hosted service on IBM public cloud
Event Streams
API Connect
App Connect
Gateways
Aspera
MQ
Event Streams
Community Edition
Free, unrestricted, unsupported
Get Started at: ibm.com/cloud/event-streams/get-started
Deploy on OpenShift on-premise or on public cloud

IBM Event Streams Builds on Open Container Orchestration Technology
4
Automate
deployment,
scaling, and
management of
containerized
applications
Define, install, and
upgrade Kubernetes
applications
Executable package
of software that
includes everything
needed to run it
Text or image Text or image Text or image
Containers Orchestration Management
Infrastructure as
code to provision
public cloud and on-
premises
environments
Text or image
Provisioning

Apache Kafka orchestrated with Kubernetes
▪ Highly available by design
–Brokers are spread across worker nodes using anti-affinity policies
–Minimizes the risk of down-time in the event of a node outage
▪ Scale the Kafka cluster up with one command
–Safely grows the stateful set, reconfigures the network interfaces and gives you more capacity
▪ Roll out Kafka cluster configuration changes easily
–Make a single configuration change and Event Streams rolls it out across the brokers in the cluster
–Broker availability is managed using health checks to ensure that availability is maintained
5

Safe, Planned Upgrade of Apache Kafka
Upgrade Kafka versions safely and without hassle
▪ First, upgrade the release to a newer version of IBM Event Streams
–Rolling update of the Kafka brokers minimizes disruption
▪ As a separate step, upgrade the broker data and protocol version to complete the upgrade
–Until this point, you can roll back
6

© 2019 IBM CorporationIBM Event Streams 7
Making Apache Kafka Intuitive and Easy
Simple to deploy
• Kafka has many distinct components to deploy,
configure and coordinate for secure connectivity
• Container placement critical to ensure production-
level availability
• Secured network traffic ingress
• Ensuring consistent and repeatable deployment

Making Apache Kafka Intuitive and Easy
8
Monitor status at a glance
Integrated feedback and support

IBM Event Streams
9
External monitoring tools
Datadog, Splunk, etc
Integrated with Key Monitoring Tools

Security – Authentication and Access Control
▪ User and group information controlled centrally
–Integrate with your corporate LDAP through IBM Cloud Private
▪ Control access to Event Streams resources using role-based
access control policies
–Assign roles to users: Viewer, Editor, Operator, Administrator
–Optionally, specify access for individual resources, such as topic T
▪ Application access control using service IDs
–Same concepts as controlling user access
–Can restrict application access to exactly the resources they need
–Prevent accidental or intentional interference between applications
10
Example policy
Permit Bob to write to topic T
User bob
Role Editor
Service Event Streams instance R
Resource T
Type topic
Service
action
Roles Permissions
topic.read Viewer and
above
Read
messages or
config
topic.write Editor and
above
Write
messages
topic.manage Operator and
Administrator
Delete or
change config

Geo-Replication Makes Disaster Recovery Simple
11
Origin
cluster
Destination
cluster
Target is take-over of workload on the destination cluster by business applications within 15 minutes
Easy configuration using the Event Streams UI from the origin cluster sets up the replicator and security credentials
At-least-once reliability so messages are not lost
Destination
topic
Destination
topic
Origin topic
Origin topic

Enhanced resilience with clusters across multiple zones
12
AVAILABILITY ZONE 1
DATACENTER
AVAILABILITY ZONE 3
DATACENTER
AVAILABILITY ZONE 2
DATACENTER
Multi-Availability Zone
•Must have at least 3 zones
•Kafka brokers and ZooKeeper
servers span across zones
•Can tolerate failure of a zone
with no service degradation
•High-speed network with low
latency between zones
required (< 20ms)

Use Existing Data in New Ways that Yield Competitive Advantage
Unmatched Connectivity to Core Systems
Connector Catalog REST API
IBM MQ
IBM Cloud Object
Storage
IBM Watson IoT Platform
Databases
IBM Streams Cloud Pak for
Integration
IBM App Connect
Enterprise
Direct integrations
Kafka Connect
framework
IBM DataPower
IBM Z
Applications
IBM Data Replication Change Data
Capture
13
ENHANCED FOR CICS

14

It’s Easy to Connect IBM MQ to Apache Kafka
IBM has created a pair of connectors, available as source code or as part of IBM Event Streams
Source Connector
From MQ queue to Kafka topic
https://github.com/ibm-messaging/kafka-connect-mq-source
Sink Connector
From Kafka topic to MQ queue
https://github.com/ibm-messaging/kafka-connect-mq-sink
▪ Copies messages from MQ queues to Event Streams topics and vice versa
▪ Supports all current MQ versions (MQ v8 or later, all platforms)
▪ Extend the connector to support any business-specific message format
▪ Fully supported by IBM for customers with support entitlement for IBM Event Streams

Running the Connectors for IBM MQ
16
IBM MQ
FROM.MQ.TOPICTO.KAFKA.Q
IBM Event StreamsKafka Connect worker
FROM.KAFKA.Q
MQ Source
Connector
TO.MQ.TOPIC
Kafka Connect worker
MQ Sink
Connector
The connectors are deployed into a component of Apache Kafka called a Kafka Connect worker
This runs between IBM MQ and IBM Event Streams (or open-source Apache Kafka)
CLIENT
CLIENT

Unix System Services
Running the Connectors for IBM MQ on z/OS
17
IBM MQ
FROM.MQ.TOPICTO.KAFKA.Q
IBM Event StreamsKafka Connect worker
FROM.KAFKA.Q
MQ Source
Connector
TO.MQ.TOPIC
Kafka Connect worker
MQ Sink
Connector
The Kafka Connect workers can be deployed onto z/OS Unix System Services
Then, the connection to MQ can be a bindings connection
BINDINGS
BINDINGS

Publish Events from Anywhere with the REST Producer API
IBM has created a new easy-to-use REST Producer API
POST /topics/{topic_name}/records
Content-Type: text/plain
Authorization: Bearer {bearer_token}
Hello Event Streams
▪ Use it wherever it’s difficult to use a real Kafka client e.g. DataPower, z/OS
▪ Straightforward design makes it easy to use from the command line and developer tools
▪ Supports partitioning keys and headers
▪ So easy you can use it from the command line with cURL
18

{
"type": "record",
"name": "Aircraft_schema",
"namespace": "com.mycompany.schemas.aircraft",
"fields": [
{
"name": "Aircraft_number",
"type": "string"
},
{
"name": "GPS_coordinates",
"type": "string"
},
{
"name": "Departure_time",
"type": "string"
},
Avro Schema
Develop Apps Efficiently with a Flexible Schema Registry
App
Schema Registry
App
2. Validate & Serialize 6. Deserialize
3. Send 4. receive

Start Planning your Journey with the IBM Cloud Garage
20
http://ibm.biz/EventDriven
Event-Driven Architecture Center

Ready for Mission-Critical Workloads
21
All with IBM 24x7 worldwide support
IBM has years of experience running Apache Kafka across the globe

Thank You

23
Back Up

Find Out More
Explore IBM Event Streams at
https://ibm.github.io/event-streams/
and try Community Edition for free!
24
Contact us: eventstreams@uk.ibm.com

Understanding Event-driven Terminology
25
▪ An Event is a statement of a fact. Facts are immutable – they can be superseded by new
information, but they don’t change
▪ Additional characteristics of an event-driven application
– Reactive
– Responsive
– Intelligent
▪ Platform capabilities required to deliver Event-driven applications
1. Being able to communicate and persist events
2. Being able to take direct action on events
3. Processing event streams to derive real time insight/intelligence
Event A record that something happened at a particular time
Event Stream A continuous, ordered, unbounded sequence of events
Stream Processing Stateful computation optimized for continuous processing

Components of an Event Streaming Application
26
App
Event Backbone
1
2 App
3
4
Building Blocks
1 :: Event Sources
2 :: Stream Processing
3 :: Event Archive
4 :: Notifications

Message Queuing & Event Streaming focus on different aspects of Messaging
Operations Informational
Messages that represent
some current or future
processing. For instance
request and response
messages.
Messages that represent
the state of the system.
For instance logging,
measurements and
notifications.
Event StreamingMessage Queueing
27
Isn’t This “Just Messaging”?

Request / Reply
Event Streaming & Message Queuing Need Different Capabilities
28
Stream History
Transient Data
Persistence
Immutable data
Scalable
consumption
Targeted
Reliable Delivery
Event
Streaming
Message
Queueing ✓

Technologies and Standards
Messaging Queuing Event Streaming
Technologies IBM MQ
ActiveMQ
RabbitMQ
Amazon SQS
IBM Event Streams
Apache Kafka
Apache Pulsar
Amazon Kinesis
Azure Event Hubs
Protocol
Standards
AMQP (0.91 & 1.0)
JMS
Kafka (de-facto)
29

Message Queuing & Event Streaming Fulfil Different Requirements
Messaging Queuing Event Streaming
Event
Communication
Queues & Pub Sub
Each consumer has messages
delivered to its own queue
Pub Sub
Each consumer has its own offset
into the log
Immutable log No
Would need to be implemented
using separate data store
Yes
Event Sourcing
& Replay
Yes & No
Can read all messages published on
a subscription
Replay requires separate data store
Yes & Yes
30

Intro to Apache Kafka
31

Apache Kafka is an open source,
distributed streaming platform
© 2019

Apache Kafka is an open source,
distributed streaming platform
© 2019
Publish and subscribe to streams of events
Store events in durable way
Process streams of events as they occur

Brokers
© 2019

Partitions
© 2019

Replication
© 2019

Replication

Producers

Topics
© 2019
0 1 2

Topics
© 2019
0 1 2 3

Topics
© 2019
0 1 2 3 4

Topics
© 2019
0 1 2 3 4 5
Offset

Topics and Keys
0 1 2 3
TOPIC
0 1 2 3
0 1 2 3
PARTITION 0
PARTITION 1
PARTITION 2

Retention Period and Compaction
▪ Retention set in time or size
log.retention.minutes = 3000
log.retention.bytes = 1048

▪ Compacted topics are evolving data
stores
0
key:a
val:A1
1
key:b
val:B1
2
key:a
val:A2
3
key:c
val:C1
4
key:c
val:C2
5
key:b
val:B2
PARTITION 0

▪ Compacted topics are evolving data
stores
0
key:a
val:A1
1
key:b
val:B1
2
key:a
val:A2
3
key:c
val:C1
4
key:c
val:C2
5
key:b
val:B2
PARTITION 0
2
key:a
val:A2
4
key:c
val:C2
5
key:b
val:B2
PARTITION 0
(REWRITTEN)

Producers and Keys
0 1 2 3 4
TOPIC
0 1 2 3
0 1 2 3 4
PARTITION 0
PARTITION 1
PARTITION 2
4

Producers and Keys
0 1 2 3 4 5
TOPIC
0 1 2 3
0 1 2 3 4 5
PARTITION 0
PARTITION 1
PARTITION 2
4 5

Producers and Keys
0 1 2 3 4 5
TOPIC
0 1 2 3
0 1 2 3 4 5
PARTITION 0
PARTITION 1
PARTITION 2
6
4 5

Producers and Keys
0 1 2 3 4 5
TOPIC
0 1 2 3
0 1 2 3 4 5
PARTITION 0
PARTITION 1
PARTITION 2
6 7
4 5

Producers
0 1 2 3 4 5
TOPIC
0 1 2 3
0 1 2 3 4 5
PARTITION 0
PARTITION 1
PARTITION 2
6 7
4 5

Producers Producer can choose acknowledgement level:
0 Fire-and-forget
Fast, but risky
1 Waits for 1 broker to acknowledge
ALL Waits for all replica brokers to acknowledge
0 1 2 3 4 5
TOPIC
0 1 2 3
0 1 2 3 4 5
PARTITION 0
PARTITION 1
PARTITION 2
6 7
4 5

Producers Producer can choose acknowledgement level:
0 Fire-and-forget
Fast, but risky
1 Waits for 1 broker to acknowledge
ALL Waits for all replica brokers to acknowledge
Producer can choose whether to retry:
0
Do not retry
Loses messages on error
>0
Retry
Retry, might result in duplicates on error
Producer can also choose idempotence
Can retry without risking duplicates
0 1 2 3 4 5
TOPIC
0 1 2 3
0 1 2 3 4 5
PARTITION 0
PARTITION 1
PARTITION 2
6 7
4 5

Consumers

Consumers
0 1 2 3 4 5 6
CONSUMER B
Offset 5
CONSUMER A
Offset 2

Consumers
0 1 2 3 4 5 6
CONSUMER B
Offset 5
CONSUMER A
Offset 2
Consumer can choose how to commit offsets:
Commits might go faster than processing
Manual,
asynchronous Fairly safe, but could re-process messages
Safe, but slows down processing
Automatic
Manual, synchronous
A common pattern is to commit offsets on a timer

Consumers
0 1 2 3 4 5 6
CONSUMER B
Offset 5
CONSUMER A
Offset 2
Consumer can choose how to commit offsets:
Commits might go faster than processing
Manual,
asynchronous Fairly safe, but could re-process messages
Safe, but slows down processing
Can group sending messages and committing offsets into transactions
Automatic
Manual, synchronous
Exactly once semantics
Primarily aimed at stream processing applications
A common pattern is to commit offsets on a timer

Consumer Groups

Consumer Groups
CONSUMER GROUP A
p0, offset 7
p1, offset 3
p2, offset 5
TOPIC
0 1 2 3
0 1 2 3 4 5
PARTITION 0
PARTITION 1
PARTITION 2
0 1 2 3 4 5 6 7
CONSUMER
CONSUMER
CONSUMER
CONSUMER GROUP B
p0, offset 7
p1, offset 3
p2, offset 5
CONSUMER
CONSUMER

Consumer Groups
CONSUMER GROUP A
p0, offset 7
p1, offset 3
p2, offset 5
TOPIC
0 1 2 3
0 1 2 3 4 5
PARTITION 0
PARTITION 1
PARTITION 2
0 1 2 3 4 5 6 7
CONSUMER
CONSUMER
CONSUMER
CONSUMER
CONSUMER GROUP B
p0, offset 7
p1, offset 3
p2, offset 5
CONSUMER
CONSUMER

Kafka Streams
▪ Client library for processing and analyzing data stored in
Kafka
▪ Processing happens in the app
▪ Supports per-record processing – no batching
my-input
FILTER
my-output
MAP

Kafka Streams
foo
red
bar
orange
foo
yellow
bingo
green
bingo
blue
bar
purple
INPUT TOPIC

Kafka Streams
foo
red
bar
orange
foo
yellow
bingo
green
bingo
blue
bar
purple
INPUT TOPIC
KStream<String, String> source = builder.stream("my-input")
.filter((key,value) -> key.equals("bingo"))
.map((key,value) -> KeyValue.pair(key, value.toUpperCase()))
.to("my-output");

Kafka Streams
foo
red
bar
orange
foo
yellow
bingo
green
bingo
blue
bar
purple
INPUT TOPIC
KStream<String, String> source = builder.stream("my-input")
.filter((key,value) -> key.equals("bingo"))
.map((key,value) -> KeyValue.pair(key, value.toUpperCase()))
.to("my-output");
bingo
GREEN
bingo
BLUE
OUTPUT TOPIC

Kafka Connect – bridge to
cloud-native apps
Event Backbone

Kafka Connect
Over 80 connectors
HDFS
Elasticsearch
MySQL
JDBC
IBM MQ
MQTT
CoAP
+ many others

Kafka with IBM Event Streams - Technical Presentation

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Kafka with IBM Event Streams - Technical Presentation

Similar to Kafka with IBM Event Streams - Technical Presentation (20)

More from Winton Winton

More from Winton Winton (9)

Recently uploaded

Recently uploaded (20)

Kafka with IBM Event Streams - Technical Presentation