After a quick overview and introduction of Apache Kafka, this session cover two components which extend the core of Apache Kafka: Kafka Connect and Kafka Streams/KSQL. Kafka Connects role is to access data from the out-side-world and make it available inside Kafka by publishing it into a Kafka topic. On the other hand, Kafka Connect is also responsible to transport information from inside Kafka to the outside world, which could be a database or a file system. There are many existing connectors for different source and target systems available out-of-the-box, either provided by the community or by Confluent or other vendors. You simply configure these connectors and off you go. Kafka Streams is a light-weight component which extends Kafka with stream processing functionality. By that, Kafka can now not only reliably and scalable transport events and messages through the Kafka broker but also analyse and process these event in real-time. Interestingly Kafka Streams does not provide its own cluster infrastructure and it is also not meant to run on a Kafka cluster. The idea is to run Kafka Streams where it makes sense, which can be inside a “normal” Java application, inside a Web container or on a more modern containerized (cloud) infrastructure, such as Mesos, Kubernetes or Docker. Kafka Streams has a lot of interesting features, such as reliable state handling, queryable state and much more. KSQL is a streaming engine for Apache Kafka, providing a simple and completely interactive SQL interface for processing data in Kafka.

1. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF
HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH
Kafka Connect & Kafka Streams/KSQL
Powerful Ecosystem Around Kafka Core
Guido Schmutz – 5.12.2017
@gschmutz guidoschmutz.wordpress.com

2. Guido Schmutz
Working at Trivadis for more than 20 years
Oracle ACE Director for Fusion Middleware and SOA
Consultant, Trainer Software Architect for Java, Oracle, SOA and
Big Data / Fast Data
Head of Trivadis Architecture Board
Technology Manager @ Trivadis
More than 30 years of software development experience
Contact: guido.schmutz@trivadis.com
Blog: http://guidoschmutz.wordpress.com
Slideshare: http://www.slideshare.net/gschmutz
Twitter: gschmutz
Kafka Connect & Kafka Streams/KSQL

3. Agenda
1. What is Apache Kafka?
2. Kafka Connect
3. Kafka Streams
4. KSQL
5. Kafka and "Big Data" / "Fast Data" Ecosystem
6. Kafka in Software Architecture
Kafka Connect & Kafka Streams/KSQL

4. Demo Example
Truck-2
truck/nn/
position
Truck-1
Truck-3
mqtt-
source
truck_
position
detect_danger
ous_driving
dangerous_
driving
Truck
Driver
jdbc-source
trucking_
driver
join_dangerous
_driving_driver
dangerous_dri
ving_driver
console
consumer
2016-06-02 14:39:56.605|98|27|803014426|
Wichita to Little Rock Route2|
Normal|38.65|90.21|5187297736652502631
Kafka Connect & Kafka Streams/KSQL
27, Walter, Ward, Y, 24-JUL-85, 2017-10-02 15:19:00 {"id":27,"firstName":"Walter",
"lastName":"Ward","available
":"Y","birthdate":"24-JUL-
85","last_update":150692305
2012}

5. What is Apache Kafka?
Kafka Connect & Kafka Streams/KSQL

6. Apache Kafka History
2012 2013 2014 2015 2016 2017
Cluster mirroring
data compression
Intra-cluster
replication
0.7
0.8
0.9
Data Processing
(Streams API)
0.10
Data Integration
(Connect API)
0.11
2018
Exactly Once
Semantics
Performance
Improvements
KSQL Developer
Preview
Kafka Connect & Kafka Streams/KSQL
1.0 JBOD Support
Support Java 9

7. Apache Kafka - Unix Analogy
$ cat < in.txt | grep "kafka" | tr a-z A-Z > out.txt
Kafka Connect API Kafka Connect APIKafka Streams API
Kafka Core (Cluster)
Adapted from: Confluent
KSQL
Kafka Connect & Kafka Streams/KSQL

8. Apache Kafka – A Streaming Platform
Kafka Connect & Kafka Streams/KSQL
High-Level Architecture
Distributed Log at the Core
Scale-Out Architecture
Logs do not (necessarilys) forget

9. How to get a Kafka environent
Kafka Connect & Kafka Streams/KSQL
On Premises
• Bare Metal Installation
• Docker
• Mesos / Kubernetes
• Hadoop Distributions
Cloud
• Oracle Event Hub Cloud Service
• Azure HDInsight Kafka
• Confluent Cloud
• …

10. Demo (I)
Truck-2
truck
position
Truck-1
Truck-3
console
consumer
2016-06-02 14:39:56.605|98|27|803014426|
Wichita to Little Rock Route2|
Normal|38.65|90.21|5187297736652502631
Testdata-Generator by Hortonworks
Kafka Connect & Kafka Streams/KSQL

11. Demo (I) – Create Kafka Topic
$ kafka-topics --zookeeper zookeeper:2181 --create
--topic truck_position --partitions 8 --replication-factor 1
$ kafka-topics --zookeeper zookeeper:2181 –list
__consumer_offsets
_confluent-metrics
_schemas
docker-connect-configs
docker-connect-offsets
docker-connect-status
truck_position
Kafka Connect & Kafka Streams/KSQL

12. Demo (I) – Run Producer and Kafka-Console-Consumer
Kafka Connect & Kafka Streams/KSQL

13. Demo (I) – Java Producer to "truck_position"
Constructing a Kafka Producer
private Properties kafkaProps = new Properties();
kafkaProps.put("bootstrap.servers","broker-1:9092);
kafkaProps.put("key.serializer", "...StringSerializer");
kafkaProps.put("value.serializer", "...StringSerializer");
producer = new KafkaProducer<String, String>(kafkaProps);
ProducerRecord<String, String> record =
new ProducerRecord<>("truck_position", driverId, eventData);
try {
metadata = producer.send(record).get();
} catch (Exception e) {}
Kafka Connect & Kafka Streams/KSQL

14. Demo (II) – devices send to MQTT instead of Kafka
Truck-2
truck/nn/
position
Truck-1
Truck-3
2016-06-02 14:39:56.605|98|27|803014426|
Wichita to Little Rock Route2|
Normal|38.65|90.21|5187297736652502631
Kafka Connect & Kafka Streams/KSQL

15. Demo (II) – devices send to MQTT instead of Kafka
Kafka Connect & Kafka Streams/KSQL

16. Demo (II) - devices send to MQTT instead of Kafka –
how to get the data into Kafka?
Truck-2
truck/nn/
position
Truck-1
Truck-3
truck
position raw
?
2016-06-02 14:39:56.605|98|27|803014426|
Wichita to Little Rock Route2|
Normal|38.65|90.21|5187297736652502631
Kafka Connect & Kafka Streams/KSQL

17. Kafka Connect
Kafka Connect & Kafka Streams/KSQL

18. Kafka Connect - Overview
Source
Connector
Sink
Connector
Kafka Connect & Kafka Streams/KSQL

19. Kafka Connect – Single Message Transforms (SMT)
Simple Transformations for a single message
Defined as part of Kafka Connect
• some useful transforms provided out-of-the-box
• Easily implement your own
Optionally deploy 1+ transforms with each
connector
• Modify messages produced by source
connector
• Modify messages sent to sink connectors
Makes it much easier to mix and match connectors
Some of currently available
transforms:
• InsertField
• ReplaceField
• MaskField
• ValueToKey
• ExtractField
• TimestampRouter
• RegexRouter
• SetSchemaMetaData
• Flatten
• TimestampConverter
Kafka Connect & Kafka Streams/KSQL

20. Kafka Connect – Many Connectors
60+ since first release (0.9+)
20+ from Confluent and Partners
Source: http://www.confluent.io/product/connectors
Confluent supported Connectors
Certified Connectors Community Connectors
Kafka Connect & Kafka Streams/KSQL

21. Demo (III)
Truck-2
truck/nn/
position
Truck-1
Truck-3
mqtt to
kafka
truck_
position
2016-06-02 14:39:56.605|98|27|803014426|
Wichita to Little Rock Route2|
Normal|38.65|90.21|5187297736652502631
console
consumer
Kafka Connect & Kafka Streams/KSQL

22. Demo (III) – Create MQTT Connect through REST API
#!/bin/bash
curl -X "POST" "http://192.168.69.138:8083/connectors"
-H "Content-Type: application/json"
-d $'{
"name": "mqtt-source",
"config": {
"connector.class":
"com.datamountaineer.streamreactor.connect.mqtt.source.MqttSourceConnector",
"connect.mqtt.connection.timeout": "1000",
"tasks.max": "1",
"connect.mqtt.kcql":
"INSERT INTO truck_position SELECT * FROM truck/+/position",
"name": "MqttSourceConnector",
"connect.mqtt.service.quality": "0",
"connect.mqtt.client.id": "tm-mqtt-connect-01",
"connect.mqtt.converter.throw.on.error": "true",
"connect.mqtt.hosts": "tcp://mosquitto:1883"
}
}'
Kafka Connect & Kafka Streams/KSQL

23. Demo (III) – Call REST API and Kafka Console
Consumer
Kafka Connect & Kafka Streams/KSQL

24. Demo (III)
Truck-2
truck/nn/
position
Truck-1
Truck-3
mqtt to
kafka
truck_
position
2016-06-02 14:39:56.605|98|27|803014426|
Wichita to Little Rock Route2|
Normal|38.65|90.21|5187297736652502631
console
consumer
what about some
analytics ?
Kafka Connect & Kafka Streams/KSQL

25. Kafka Streams
Kafka Connect & Kafka Streams/KSQL

26. Kafka Streams - Overview
• Designed as a simple and lightweight library in Apache
Kafka
• no external dependencies on systems other than Apache
Kafka
• Part of open source Apache Kafka, introduced in 0.10+
• Leverages Kafka as its internal messaging layer
• Supports fault-tolerant local state
• Event-at-a-time processing (not microbatch) with millisecond
latency
• Windowing with out-of-order data using a Google DataFlow-like
model
Kafka Connect & Kafka Streams/KSQL

27. Kafka Stream DSL and Processor Topology
KStream<Integer, String> stream1 =
builder.stream("in-1");
KStream<Integer, String> stream2=
builder.stream("in-2");
KStream<Integer, String> joined =
stream1.leftJoin(stream2, …);
KTable<> aggregated =
joined.groupBy(…).count("store");
aggregated.to("out-1");
1 2
lj
a
t
State
Kafka Connect & Kafka Streams/KSQL

28. Kafka Stream DSL and Processor Topology
KStream<Integer, String> stream1 =
builder.stream("in-1");
KStream<Integer, String> stream2=
builder.stream("in-2");
KStream<Integer, String> joined =
stream1.leftJoin(stream2, …);
KTable<> aggregated =
joined.groupBy(…).count("store");
aggregated.to("out-1");
1 2
lj
a
t
State
Kafka Connect & Kafka Streams/KSQL

29. Kafka Streams Cluster
Processor Topology
Kafka Cluster
input-1
input-2
store (changelog)
output
1 2
lj
a
t
State
Kafka Connect & Kafka Streams/KSQL

30. Kafka Cluster
Processor Topology
input-1
Partition 0
Partition 1
Partition 2
Partition 3
input-2
Partition 0
Partition 1
Partition 2
Partition 3
Kafka Streams 1
Kafka Streams 2
Kafka Connect & Kafka Streams/KSQL

31. Kafka Cluster
Processor Topology
input-1
Partition 0
Partition 1
Partition 2
Partition 3
input-2
Partition 0
Partition 1
Partition 2
Partition 3
Kafka Streams 1 Kafka Streams 2
Kafka Streams 3 Kafka Streams 4
Kafka Connect & Kafka Streams/KSQL

32. Stream vs. Table
Event Stream State Stream (Change Log Stream)
2017-10-02T20:18:46 11,Normal,41.87,-87.67
2017-10-02T20:18:55 11,Normal,40.38,-89.17
2017-10-02T20:18:59 21,Normal,42.23,-91.78
2017-10-02T20:19:01 21,Normal,41.71,-91.32
2017-10-02T20:19:02 11,Normal,38.65,-90.2
2017-10-02T20:19:23 21,Normal41.71,-91.32
11 2017-10-02T20:18:46,11,Normal,41.87,-87.67
11 2017-10-02T20:18:55,11,Normal,40.38,-89.17
21 2017-10-02T20:18:59, 21,Normal,42.23,-91.78
21 2017-10-02T20:19:01,21,Normal,41.71,-91.32
11 2017-10-02T20:19:02,11,Normal,38.65,-90.2
21 2017-10-02T20:19:23,21,Normal41.71,-91.32
Kafka Connect & Kafka Streams/KSQL
KStream KTable

33. Kafka Streams: Key Features
Kafka Connect & Kafka Streams/KSQL
• Native, 100%-compatible Kafka integration
• Secure stream processing using Kafka's security features
• Elastic and highly scalable
• Fault-tolerant
• Stateful and stateless computations
• Interactive queries
• Time model
• Windowing
• Supports late-arriving and out-of-order data
• Millisecond processing latency, no micro-batching
• At-least-once and exactly-once processing guarantees

34. Demo (IV)
Truck-2
truck/nn/
position
Truck-1
Truck-3
mqtt to
kafka
truck_
position_s
detect_danger
ous_driving
dangerous_
driving
console
consumer
2016-06-02 14:39:56.605|98|27|803014426|
Wichita to Little Rock Route2|
Normal|38.65|90.21|5187297736652502631
Kafka Connect & Kafka Streams/KSQL

35. Demo (IV) - Create Stream
final KStreamBuilder builder = new KStreamBuilder();
KStream<String, String> source =
builder.stream(stringSerde, stringSerde, "truck_position");
KStream<String, TruckPosition> positions =
source.map((key,value) ->
new KeyValue<>(key, TruckPosition.create(value)));
KStream<String, TruckPosition> filtered =
positions.filter(TruckPosition::filterNonNORMAL);
filtered.map((key,value) -> new
KeyValue<>(key,value._originalRecord))
.to("dangerous_driving");
Kafka Connect & Kafka Streams/KSQL

36. KSQL
Kafka Connect & Kafka Streams/KSQL

37. KSQL: a Streaming SQL Engine for Apache Kafka
• Enables stream processing with zero coding required
• The simples way to process streams of data in real-time
• Powered by Kafka and Kafka Streams: scalable, distributed, mature
• All you need is Kafka – no complex deployments
• available as Developer preview!
• STREAM and TABLE as first-class citizens
• STREAM = data in motion
• TABLE = collected state of a stream
• join STREAM and TABLE
Kafka Connect & Kafka Streams/KSQL

38. KSQL Deployment Models
Standalone Mode Cluster Mode
Source: Confluent
Kafka Connect & Kafka Streams/KSQL

39. Demo (IV)
Truck-2
truck/nn/
position
Truck-1
Truck-3
mqtt to
kafka
truck_
position_s
detect_danger
ous_driving
dangerous_
driving
console
consumer
2016-06-02 14:39:56.605|98|27|803014426|
Wichita to Little Rock Route2|
Normal|38.65|90.21|5187297736652502631
Kafka Connect & Kafka Streams/KSQL

40. Demo (V) - Start Kafka KSQL
$ docker-compose exec ksql-cli ksql-cli local --bootstrap-server broker-1:9092
======================================
= _ __ _____ ____ _ =
= | |/ // ____|/ __ | | =
= | ' /| (___ | | | | | =
= | < ___ | | | | | =
= | . ____) | |__| | |____ =
= |_|______/ __________| =
= =
= Streaming SQL Engine for Kafka =
Copyright 2017 Confluent Inc.
CLI v0.1, Server v0.1 located at http://localhost:9098
Having trouble? Type 'help' (case-insensitive) for a rundown of how things work!
ksql>
Kafka Connect & Kafka Streams/KSQL

41. Demo (IV) - Create Stream
ksql> CREATE STREAM truck_position_s
(ts VARCHAR,
truckid VARCHAR,
driverid BIGINT,
routeid BIGINT,
routename VARCHAR,
eventtype VARCHAR,
latitude DOUBLE,
longitude DOUBLE,
correlationid VARCHAR)
WITH (kafka_topic='truck_position',
value_format='DELIMITED');
Message
----------------
Stream created
Kafka Connect & Kafka Streams/KSQL

42. Demo (IV) - Create Stream
ksql> describe truck_position_s;
Field | Type
---------------------------------
ROWTIME | BIGINT
ROWKEY | VARCHAR(STRING)
TS | VARCHAR(STRING)
TRUCKID | VARCHAR(STRING)
DRIVERID | BIGINT
ROUTEID | BIGINT
ROUTENAME | VARCHAR(STRING)
EVENTTYPE | VARCHAR(STRING)
LATITUDE | DOUBLE
LONGITUDE | DOUBLE
CORRELATIONID | VARCHAR(STRING)
Kafka Connect & Kafka Streams/KSQL

43. Demo (IV) - Create Stream
ksql> SELECT * FROM truck_position_s;
1506922133306 | "truck/13/position0 | �2017-10-02T07:28:53 | 31 | 13 | 371182829
| Memphis to Little Rock | Normal | 41.76 | -89.6 | -2084263951914664106
1506922133396 | "truck/16/position0 | �2017-10-02T07:28:53 | 19 | 16 | 160405074
| Joplin to Kansas City Route 2 | Normal | 41.48 | -88.07 | -2084263951914664106
1506922133457 | "truck/30/position0 | �2017-10-02T07:28:53 | 26 | 30 | 160779139
| Des Moines to Chicago Route 2 | Normal | 41.85 | -89.29 | -2084263951914664106
1506922133485 | "truck/23/position0 | �2017-10-02T07:28:53 | 32 | 23 |
1090292248 | Peoria to Ceder Rapids Route 2 | Normal | 41.48 | -88.07 | -
2084263951914664106
1506922133497 | "truck/12/position0 | �2017-10-02T07:28:53 | 80 | 12 |
1961634315 | Saint Louis to Memphis | Normal | 41.74 | -91.47 | -
2084263951914664106
1506922133547 | "truck/14/position0 | �2017-10-02T07:28:53 | 73 | 14 |
1927624662 | Springfield to KC Via Columbia | Normal | 35.12 | -90.68 | -
2084263951914664106
Kafka Connect & Kafka Streams/KSQL

44. Demo (IV) - Create Stream
ksql> SELECT * FROM truck_position_s WHERE eventtype != 'Normal';
1506922264016 | "truck/11/position0 | �2017-10-02T07:31:04 | 27 | 11 |
1325712174 | Saint Louis to Tulsa Route2 | Lane Departure | 38.5 | -90.69 | -
2084263951914664106
1506922281156 | "truck/11/position0 | �2017-10-02T07:31:21 | 27 | 11 |
1325712174 | Saint Louis to Tulsa Route2 | Unsafe tail distance | 37.81 | -92.31
| -2084263951914664106
1506922284436 | "truck/10/position0 | �2017-10-02T07:31:24 | 93 | 10 |
1384345811 | Joplin to Kansas City | Unsafe following distance | 37.02 | -94.54 |
-2084263951914664106
1506922297887 | "truck/11/position0 | �2017-10-02T07:31:37 | 27 | 11 |
1325712174 | Saint Louis to Tulsa Route2 | Unsafe following distance | 37.09 | -
94.23 | -2084263951914664106
Kafka Connect & Kafka Streams/KSQL

45. Demo (IV) - Create Stream
ksql> CREATE STREAM dangerous_driving_s
WITH (kafka_topic= dangerous_driving',
value_format='DELIMITED')
AS SELECT * FROM truck_position_s
WHERE eventtype != 'Normal';
Message
----------------------------
Stream created and running
ksql> select * from dangerous_driving_s;
1506922849375 | "truck/11/position0 | �2017-10-02T07:40:49 | 90 | 11 | 160779139
| Des Moines to Chicago Route 2 | Overspeed | 41.48 | -88.07 |
3569183071347898366
1506922866488 | "truck/11/position0 | �2017-10-02T07:41:06 | 90 | 11 | 160779139
| Des Moines to Chicago Route 2 | Overspeed | 40.38 | -89.17 |
3569183071347898366
Kafka Connect & Kafka Streams/KSQL

46. Demo (IV) - Create Stream
ksql> describe dangerous_driving_s;
Field | Type
---------------------------------
ROWTIME | BIGINT
ROWKEY | VARCHAR(STRING)
TS | VARCHAR(STRING)
TRUCKID | VARCHAR(STRING)
DRIVERID | BIGINT
ROUTEID | BIGINT
ROUTENAME | VARCHAR(STRING)
EVENTTYPE | VARCHAR(STRING)
LATITUDE | DOUBLE
LONGITUDE | DOUBLE
CORRELATIONID | VARCHAR(STRING)
Kafka Connect & Kafka Streams/KSQL

47. Demo (V)
Truck-2
truck/nn/
position
Truck-1
Truck-3
mqtt-
source
truck_
position
detect_danger
ous_driving
dangerous_
driving
Truck
Driver
jdbc-source
trucking_
driver
join_dangerous
_driving_driver
dangerous_dri
ving_driver
27, Walter, Ward, Y, 24-JUL-85, 2017-10-02 15:19:00
console
consumer
2016-06-02 14:39:56.605|98|27|803014426|
Wichita to Little Rock Route2|
Normal|38.65|90.21|5187297736652502631
{"id":27,"firstName":"Walter",
"lastName":"Ward","available
":"Y","birthdate":"24-JUL-
85","last_update":150692305
2012}
Kafka Connect & Kafka Streams/KSQL

48. Demo (V) – Create JDBC Connect through REST API
#!/bin/bash
curl -X "POST" "http://192.168.69.138:8083/connectors"
-H "Content-Type: application/json"
-d $'{
"name": "jdbc-driver-source",
"config": {
"connector.class": "JdbcSourceConnector",
"connection.url":"jdbc:postgresql://db/sample?user=sample&password=sample",
"mode": "timestamp",
"timestamp.column.name":"last_update",
"table.whitelist":"driver",
"validate.non.null":"false",
"topic.prefix":"trucking_",
"key.converter":"org.apache.kafka.connect.json.JsonConverter",
"key.converter.schemas.enable": "false",
"value.converter":"org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable": "false",
"name": "jdbc-driver-source",
"transforms":"createKey,extractInt",
"transforms.createKey.type":"org.apache.kafka.connect.transforms.ValueToKey",
"transforms.createKey.fields":"id",
"transforms.extractInt.type":"org.apache.kafka.connect.transforms.ExtractField$Key",
"transforms.extractInt.field":"id"
}
}'
Kafka Connect & Kafka Streams/KSQL

49. Demo (V) – Create JDBC Connect through REST API
Kafka Connect & Kafka Streams/KSQL

50. Demo (V) - Create Table with Driver State
ksql> CREATE TABLE driver_t
(id BIGINT,
first_name VARCHAR,
last_name VARCHAR,
available VARCHAR)
WITH (kafka_topic='trucking_driver',
value_format='JSON');
Message
----------------
Table created
Kafka Connect & Kafka Streams/KSQL

51. Demo (V) - Create Table with Driver State
ksql> CREATE STREAM dangerous_driving_and_driver_s
WITH (kafka_topic='dangerous_driving_and_driver_s',
value_format='JSON')
AS SELECT driverid, first_name, last_name, truckid, routeid,routename,
eventtype
FROM truck_position_s
LEFT JOIN driver_t
ON dangerous_driving_and_driver_s.driverid = driver_t.id;
Message
----------------------------
Stream created and running
ksql> select * from dangerous_driving_and_driver_s;
1511173352906 | 21 | 21 | Lila | Page | 58 | 1594289134 | Memphis to Little Rock
Route 2 | Unsafe tail distance
1511173353669 | 12 | 12 | Laurence | Lindsey | 93 | 1384345811 | Joplin to Kansas
City | Lane Departure
1511173435385 | 11 | 11 | Micky | Isaacson | 22 | 1198242881 | Saint Louis to
Chicago Route2 | Unsafe tail distance
Kafka Connect & Kafka Streams/KSQL

52. Kafka and "Big Data" / "Fast Data"
Ecosystem
Kafka Connect & Kafka Streams/KSQL

53. Kafka and the Big Data / Fast Data ecosystem
Kafka integrates with many popular products / frameworks
• Apache Spark Streaming
• Apache Flink
• Apache Storm
• Apache Apex
• Apache NiFi
• StreamSets
• Oracle Stream Analytics
• Oracle Service Bus
• Oracle GoldenGate
• Oracle Event Hub Cloud Service
• Debezium CDC
• …
Additional Info: https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
Kafka Connect & Kafka Streams/KSQL

54. Kafka in Software Architecture
Kafka Connect & Kafka Streams/KSQL

55. Hadoop Clusterd
Hadoop Cluster
Big Data Cluster
Traditional Big Data Architecture
BI Tools
Enterprise Data
Warehouse
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
File Import / SQL Import
SQL
Search / Explore
Online & Mobile
Apps
Search
NoSQL
Parallel Batch
Processing
Distributed
Filesystem
• Machine Learning
• Graph Algorithms
• Natural Language Processing
Kafka Connect & Kafka Streams/KSQL

56. Event
Hub
Event
Hub
Hadoop Clusterd
Hadoop Cluster
Big Data Cluster
Event Hub – handle event stream data
BI Tools
Enterprise Data
Warehouse
Location
Social
Click
stream
Sensor
Data
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
Event
Hub
Call
Center
Weather
Data
Mobile
Apps
SQL
Search / Explore
Online & Mobile
Apps
Search
Data Flow
NoSQL
Parallel Batch
Processing
Distributed
Filesystem
• Machine Learning
• Graph Algorithms
• Natural Language Processing
Kafka Connect & Kafka Streams/KSQL

57. Hadoop Clusterd
Hadoop Cluster
Big Data Cluster
Event Hub – taking Velocity into account
Location
Social
Click
stream
Sensor
Data
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
Call
Center
Mobile
Apps
Batch Analytics
Streaming Analytics
Results
Parallel Batch
Processing
Distributed
Filesystem
Stream Analytics
NoSQL
Reference /
Models
SQL
Search
Dashboard
BI Tools
Enterprise Data
Warehouse
Search / Explore
Online & Mobile
Apps
File Import / SQL Import
Weather
Data
Event
Hub
Event
Hub
Event
Hub
Kafka Connect & Kafka Streams/KSQL

58. Container
Hadoop Clusterd
Hadoop Cluster
Big Data Cluster
Event Hub – Asynchronous Microservice Architecture
Location
Social
Click
stream
Sensor
Data
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
Call
Center
Mobile
Apps
Parallel
Batch
ProcessingDistributed
Filesystem
Microservice
NoSQLRDBMS
SQL
Search
BI Tools
Enterprise Data
Warehouse
Search / Explore
Online & Mobile
Apps
File Import / SQL Import
Weather
Data
{ }
API
Event
Hub
Event
Hub
Event
Hub
Kafka Connect & Kafka Streams/KSQL

59. Kafka Connect & Kafka Streams/KSQL
Technology on its own won't help you.
You need to know how to use it properly.