From Prague Kafka Meetup in November 2018. This session introduces Apache Kafka as event-driven open source streaming platform. Apache Kafka goes far beyond scalable, high volume messaging. In addition, you can leverage Kafka Connect for integration and the Kafka Streams API for building lightweight stream processing microservices in autonomous teams. The open source Confluent Platform adds further components such as a KSQL, Schema Registry, REST Proxy, Clients for different programming languages and Connectors for different technologies and databases.

1. 1Confidential
Introduction to Apache Kafka as
Event-Driven Open Source Streaming Platform
Kai Waehner
Technology Evangelist
kontakt@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de
Apache Kafka, Kafka Streams, KSQL, Schema Registry, and more…

6. Events
What is an event?

7. Events

8. Events
A Sale An Invoice A Trade A Customer
Experience

9. Events and State
User_data = ‘old_value’
User_data = ‘new_value’

10. Table-Stream Duality

11. All Your Data is Streams of Events

12. Events have two functions
1. Triggers
2. Data

13. Where are they?
Events haven’t
had a proper home
in infrastructure
or in code. They
are implicit.
Here!

14. Events are everywhere
A business is a series of events and the reactions to those events.

15. Event-driven Retail

16. What do we have today?
Data Stores

17. You need both these things

18. Events and Business Services
Business Service Business Service
Sales Events New Orders
to place
Processing
and Inventory

19. Events and Data Pipelines
Databases Databases
Customer Data
Updates
Unified 360
Merged Customer
Profiles

20. These are equivalent
Both are Event Stream Processing

21. The Streaming Platform

22. Haven’t we seen all
this before?

23. What’s different this time around?
Published in 2009 Published in 2004

24. What’s different this time around?

25. Microservices Mobile Machine
Learning
Internet of
Things
The World has Changed

26. Confluents Business Value per Use Case
Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save
money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate to
Cloud
Mitigate Risk
(protect money)
Key Drivers
Strategic Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital
replatforming/
Mainframe Offload
Connected Car: Navigation & improved
in-car experience: Audi
Customer 360
Simplifying Omni-channel Retail at
Scale: Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log aggregation,
Splunk replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous System)
The [Silicon Valley] Digital Natives;
LinkedIn, Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital One
Developer Velocity - Building Stateful
Financial Applications with Kafka
Streams: Funding Circle
Detect Fraud & Prevent Fraud in Real
Time: PayPal
Kafka as a Service - A Tale of Security
and Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$
Example Case Studies
(of many)

27. How do we get there?
Why Apache Kafka and Confluent?

33. à It is NOT a Queue! J

46. • Robustness and scalability
• Slow consumers
• Reprocessing

48. • Strict Ordering
• Persistence
• Exactly-Once Processing

58. Kafka Streams Execution Model

66. Apache Kafka + Kafka Connect + Kafka Streams
+ many additional components

68. 6969
Kafka Clients
Apache Kafka Native Clients
Confluent Native Clients
Community Supported Clients
Proxy http/REST
stdin/stdout

69. 7070
Confluent REST Proxy
REST Proxy
Non-Java Applications
Native Kafka Java
Applications
Schema Registry
REST / HTTP(S)
Talking to
Non-native Kafka Apps
and
Outside the Firewall

70. 7171
Confluent Schema Registry
Elastic
HDFS
Example Consumers
Serializer
App 1
Serializer
App 2
!
Kafka Topic!
Schema Registry
Deserializer
Deserializer

71. KSQL – A Streaming SQL Engine for Apache Kafka

72. KSQL – A Streaming SQL Engine for Apache Kafka
CREATE STREAM vip_actions AS
SELECT userid, page, action FROM clickstream c
LEFT JOIN users u ON c.userid = u.user_id
WHERE u.level = 'Platinum';
CREATE TABLE possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTES)
GROUP BY card_number
HAVING count(*) > 3;

74. Apache Kafka: The Rise of a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine
Add-ons
Schema Registry
REST Proxy
KSQL
Etc.

75. Event-Streaming Maturity Model / Adoption Journey
Pre-Streaming
Streaming
Awareness
and Pilot
Early Production
Streaming
Mission
Critical,
Integrated
Streaming
Global
Streaming
Central
Nervous System

76. Free material for Managers, Project lead, Architects, Developers

77. 85
Kai Waehner
Technology Evangelist
kontakt@kai-waehner.de
@KaiWaehner
www.confluent.io
www.kai-waehner.de
LinkedIn
Questions? Feedback?
Please contact me!
Download Apache Kafka
& Confluent Platform
confluent.io/download