Guido Schmutz, profile picture
Uploaded byGuido Schmutz
PDF, PPTX779 views

Kafka Connect & Kafka Streams/KSQL - powerful ecosystem around Kafka core

The document provides an overview of Apache Kafka, highlighting its components such as Kafka Connect, Kafka Streams, and KSQL, which enable efficient streaming data processing. It covers various aspects of Kafka's architecture, data connectors, and transformation capabilities, along with practical demo examples related to vehicle tracking systems. The document emphasizes Kafka's functionality in handling big data and real-time applications through its robust ecosystem.

Embed presentation

Download as PDF, PPTX
BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH Kafka Connect & Kafka Streams/KSQL Powerful Ecosystem Around Kafka Core Guido Schmutz – 5.12.2017 @gschmutz guidoschmutz.wordpress.com
Guido Schmutz Working at Trivadis for more than 20 years Oracle ACE Director for Fusion Middleware and SOA Consultant, Trainer Software Architect for Java, Oracle, SOA and Big Data / Fast Data Head of Trivadis Architecture Board Technology Manager @ Trivadis More than 30 years of software development experience Contact: guido.schmutz@trivadis.com Blog: http://guidoschmutz.wordpress.com Slideshare: http://www.slideshare.net/gschmutz Twitter: gschmutz Kafka Connect & Kafka Streams/KSQL
Agenda 1. What is Apache Kafka? 2. Kafka Connect 3. Kafka Streams 4. KSQL 5. Kafka and "Big Data" / "Fast Data" Ecosystem 6. Kafka in Software Architecture Kafka Connect & Kafka Streams/KSQL
Demo Example Truck-2 truck/nn/ position Truck-1 Truck-3 mqtt- source truck_ position detect_danger ous_driving dangerous_ driving Truck Driver jdbc-source trucking_ driver join_dangerous _driving_driver dangerous_dri ving_driver console consumer 2016-06-02 14:39:56.605|98|27|803014426| Wichita to Little Rock Route2| Normal|38.65|90.21|5187297736652502631 Kafka Connect & Kafka Streams/KSQL 27, Walter, Ward, Y, 24-JUL-85, 2017-10-02 15:19:00 {"id":27,"firstName":"Walter", "lastName":"Ward","available ":"Y","birthdate":"24-JUL- 85","last_update":150692305 2012}
What is Apache Kafka? Kafka Connect & Kafka Streams/KSQL
Apache Kafka History 2012 2013 2014 2015 2016 2017 Cluster mirroring data compression Intra-cluster replication 0.7 0.8 0.9 Data Processing (Streams API) 0.10 Data Integration (Connect API) 0.11 2018 Exactly Once Semantics Performance Improvements KSQL Developer Preview Kafka Connect & Kafka Streams/KSQL 1.0 JBOD Support Support Java 9
Apache Kafka - Unix Analogy $ cat < in.txt | grep "kafka" | tr a-z A-Z > out.txt Kafka Connect API Kafka Connect APIKafka Streams API Kafka Core (Cluster) Adapted from: Confluent KSQL Kafka Connect & Kafka Streams/KSQL
Apache Kafka – A Streaming Platform Kafka Connect & Kafka Streams/KSQL High-Level Architecture Distributed Log at the Core Scale-Out Architecture Logs do not (necessarilys) forget
How to get a Kafka environent Kafka Connect & Kafka Streams/KSQL On Premises • Bare Metal Installation • Docker • Mesos / Kubernetes • Hadoop Distributions Cloud • Oracle Event Hub Cloud Service • Azure HDInsight Kafka • Confluent Cloud • …
Demo (I) Truck-2 truck position Truck-1 Truck-3 console consumer 2016-06-02 14:39:56.605|98|27|803014426| Wichita to Little Rock Route2| Normal|38.65|90.21|5187297736652502631 Testdata-Generator by Hortonworks Kafka Connect & Kafka Streams/KSQL
Demo (I) – Create Kafka Topic $ kafka-topics --zookeeper zookeeper:2181 --create --topic truck_position --partitions 8 --replication-factor 1 $ kafka-topics --zookeeper zookeeper:2181 –list __consumer_offsets _confluent-metrics _schemas docker-connect-configs docker-connect-offsets docker-connect-status truck_position Kafka Connect & Kafka Streams/KSQL
Demo (I) – Run Producer and Kafka-Console-Consumer Kafka Connect & Kafka Streams/KSQL
Demo (I) – Java Producer to "truck_position" Constructing a Kafka Producer private Properties kafkaProps = new Properties(); kafkaProps.put("bootstrap.servers","broker-1:9092); kafkaProps.put("key.serializer", "...StringSerializer"); kafkaProps.put("value.serializer", "...StringSerializer"); producer = new KafkaProducer<String, String>(kafkaProps); ProducerRecord<String, String> record = new ProducerRecord<>("truck_position", driverId, eventData); try { metadata = producer.send(record).get(); } catch (Exception e) {} Kafka Connect & Kafka Streams/KSQL
Demo (II) – devices send to MQTT instead of Kafka Truck-2 truck/nn/ position Truck-1 Truck-3 2016-06-02 14:39:56.605|98|27|803014426| Wichita to Little Rock Route2| Normal|38.65|90.21|5187297736652502631 Kafka Connect & Kafka Streams/KSQL
Demo (II) – devices send to MQTT instead of Kafka Kafka Connect & Kafka Streams/KSQL
Demo (II) - devices send to MQTT instead of Kafka – how to get the data into Kafka? Truck-2 truck/nn/ position Truck-1 Truck-3 truck position raw ? 2016-06-02 14:39:56.605|98|27|803014426| Wichita to Little Rock Route2| Normal|38.65|90.21|5187297736652502631 Kafka Connect & Kafka Streams/KSQL
Kafka Connect Kafka Connect & Kafka Streams/KSQL
Kafka Connect - Overview Source Connector Sink Connector Kafka Connect & Kafka Streams/KSQL
Kafka Connect – Single Message Transforms (SMT) Simple Transformations for a single message Defined as part of Kafka Connect • some useful transforms provided out-of-the-box • Easily implement your own Optionally deploy 1+ transforms with each connector • Modify messages produced by source connector • Modify messages sent to sink connectors Makes it much easier to mix and match connectors Some of currently available transforms: • InsertField • ReplaceField • MaskField • ValueToKey • ExtractField • TimestampRouter • RegexRouter • SetSchemaMetaData • Flatten • TimestampConverter Kafka Connect & Kafka Streams/KSQL
Kafka Connect – Many Connectors 60+ since first release (0.9+) 20+ from Confluent and Partners Source: http://www.confluent.io/product/connectors Confluent supported Connectors Certified Connectors Community Connectors Kafka Connect & Kafka Streams/KSQL
Demo (III) Truck-2 truck/nn/ position Truck-1 Truck-3 mqtt to kafka truck_ position 2016-06-02 14:39:56.605|98|27|803014426| Wichita to Little Rock Route2| Normal|38.65|90.21|5187297736652502631 console consumer Kafka Connect & Kafka Streams/KSQL
Demo (III) – Create MQTT Connect through REST API #!/bin/bash curl -X "POST" "http://192.168.69.138:8083/connectors" -H "Content-Type: application/json" -d $'{ "name": "mqtt-source", "config": { "connector.class": "com.datamountaineer.streamreactor.connect.mqtt.source.MqttSourceConnector", "connect.mqtt.connection.timeout": "1000", "tasks.max": "1", "connect.mqtt.kcql": "INSERT INTO truck_position SELECT * FROM truck/+/position", "name": "MqttSourceConnector", "connect.mqtt.service.quality": "0", "connect.mqtt.client.id": "tm-mqtt-connect-01", "connect.mqtt.converter.throw.on.error": "true", "connect.mqtt.hosts": "tcp://mosquitto:1883" } }' Kafka Connect & Kafka Streams/KSQL
Demo (III) – Call REST API and Kafka Console Consumer Kafka Connect & Kafka Streams/KSQL
Demo (III) Truck-2 truck/nn/ position Truck-1 Truck-3 mqtt to kafka truck_ position 2016-06-02 14:39:56.605|98|27|803014426| Wichita to Little Rock Route2| Normal|38.65|90.21|5187297736652502631 console consumer what about some analytics ? Kafka Connect & Kafka Streams/KSQL
Kafka Streams Kafka Connect & Kafka Streams/KSQL
Kafka Streams - Overview • Designed as a simple and lightweight library in Apache Kafka • no external dependencies on systems other than Apache Kafka • Part of open source Apache Kafka, introduced in 0.10+ • Leverages Kafka as its internal messaging layer • Supports fault-tolerant local state • Event-at-a-time processing (not microbatch) with millisecond latency • Windowing with out-of-order data using a Google DataFlow-like model Kafka Connect & Kafka Streams/KSQL
Kafka Stream DSL and Processor Topology KStream<Integer, String> stream1 = builder.stream("in-1"); KStream<Integer, String> stream2= builder.stream("in-2"); KStream<Integer, String> joined = stream1.leftJoin(stream2, …); KTable<> aggregated = joined.groupBy(…).count("store"); aggregated.to("out-1"); 1 2 lj a t State Kafka Connect & Kafka Streams/KSQL
Kafka Stream DSL and Processor Topology KStream<Integer, String> stream1 = builder.stream("in-1"); KStream<Integer, String> stream2= builder.stream("in-2"); KStream<Integer, String> joined = stream1.leftJoin(stream2, …); KTable<> aggregated = joined.groupBy(…).count("store"); aggregated.to("out-1"); 1 2 lj a t State Kafka Connect & Kafka Streams/KSQL
Kafka Streams Cluster Processor Topology Kafka Cluster input-1 input-2 store (changelog) output 1 2 lj a t State Kafka Connect & Kafka Streams/KSQL
Kafka Cluster Processor Topology input-1 Partition 0 Partition 1 Partition 2 Partition 3 input-2 Partition 0 Partition 1 Partition 2 Partition 3 Kafka Streams 1 Kafka Streams 2 Kafka Connect & Kafka Streams/KSQL
Kafka Cluster Processor Topology input-1 Partition 0 Partition 1 Partition 2 Partition 3 input-2 Partition 0 Partition 1 Partition 2 Partition 3 Kafka Streams 1 Kafka Streams 2 Kafka Streams 3 Kafka Streams 4 Kafka Connect & Kafka Streams/KSQL
Stream vs. Table Event Stream State Stream (Change Log Stream) 2017-10-02T20:18:46 11,Normal,41.87,-87.67 2017-10-02T20:18:55 11,Normal,40.38,-89.17 2017-10-02T20:18:59 21,Normal,42.23,-91.78 2017-10-02T20:19:01 21,Normal,41.71,-91.32 2017-10-02T20:19:02 11,Normal,38.65,-90.2 2017-10-02T20:19:23 21,Normal41.71,-91.32 11 2017-10-02T20:18:46,11,Normal,41.87,-87.67 11 2017-10-02T20:18:55,11,Normal,40.38,-89.17 21 2017-10-02T20:18:59, 21,Normal,42.23,-91.78 21 2017-10-02T20:19:01,21,Normal,41.71,-91.32 11 2017-10-02T20:19:02,11,Normal,38.65,-90.2 21 2017-10-02T20:19:23,21,Normal41.71,-91.32 Kafka Connect & Kafka Streams/KSQL KStream KTable
Kafka Streams: Key Features Kafka Connect & Kafka Streams/KSQL • Native, 100%-compatible Kafka integration • Secure stream processing using Kafka's security features • Elastic and highly scalable • Fault-tolerant • Stateful and stateless computations • Interactive queries • Time model • Windowing • Supports late-arriving and out-of-order data • Millisecond processing latency, no micro-batching • At-least-once and exactly-once processing guarantees
Demo (IV) Truck-2 truck/nn/ position Truck-1 Truck-3 mqtt to kafka truck_ position_s detect_danger ous_driving dangerous_ driving console consumer 2016-06-02 14:39:56.605|98|27|803014426| Wichita to Little Rock Route2| Normal|38.65|90.21|5187297736652502631 Kafka Connect & Kafka Streams/KSQL
Demo (IV) - Create Stream final KStreamBuilder builder = new KStreamBuilder(); KStream<String, String> source = builder.stream(stringSerde, stringSerde, "truck_position"); KStream<String, TruckPosition> positions = source.map((key,value) -> new KeyValue<>(key, TruckPosition.create(value))); KStream<String, TruckPosition> filtered = positions.filter(TruckPosition::filterNonNORMAL); filtered.map((key,value) -> new KeyValue<>(key,value._originalRecord)) .to("dangerous_driving"); Kafka Connect & Kafka Streams/KSQL
KSQL Kafka Connect & Kafka Streams/KSQL
KSQL: a Streaming SQL Engine for Apache Kafka • Enables stream processing with zero coding required • The simples way to process streams of data in real-time • Powered by Kafka and Kafka Streams: scalable, distributed, mature • All you need is Kafka – no complex deployments • available as Developer preview! • STREAM and TABLE as first-class citizens • STREAM = data in motion • TABLE = collected state of a stream • join STREAM and TABLE Kafka Connect & Kafka Streams/KSQL
KSQL Deployment Models Standalone Mode Cluster Mode Source: Confluent Kafka Connect & Kafka Streams/KSQL
Demo (IV) Truck-2 truck/nn/ position Truck-1 Truck-3 mqtt to kafka truck_ position_s detect_danger ous_driving dangerous_ driving console consumer 2016-06-02 14:39:56.605|98|27|803014426| Wichita to Little Rock Route2| Normal|38.65|90.21|5187297736652502631 Kafka Connect & Kafka Streams/KSQL
Demo (V) - Start Kafka KSQL $ docker-compose exec ksql-cli ksql-cli local --bootstrap-server broker-1:9092 ====================================== = _ __ _____ ____ _ = = | |/ // ____|/ __ | | = = | ' /| (___ | | | | | = = | < ___ | | | | | = = | . ____) | |__| | |____ = = |_|______/ __________| = = = = Streaming SQL Engine for Kafka = Copyright 2017 Confluent Inc. CLI v0.1, Server v0.1 located at http://localhost:9098 Having trouble? Type 'help' (case-insensitive) for a rundown of how things work! ksql> Kafka Connect & Kafka Streams/KSQL
Demo (IV) - Create Stream ksql> CREATE STREAM truck_position_s (ts VARCHAR, truckid VARCHAR, driverid BIGINT, routeid BIGINT, routename VARCHAR, eventtype VARCHAR, latitude DOUBLE, longitude DOUBLE, correlationid VARCHAR) WITH (kafka_topic='truck_position', value_format='DELIMITED'); Message ---------------- Stream created Kafka Connect & Kafka Streams/KSQL
Demo (IV) - Create Stream ksql> describe truck_position_s; Field | Type --------------------------------- ROWTIME | BIGINT ROWKEY | VARCHAR(STRING) TS | VARCHAR(STRING) TRUCKID | VARCHAR(STRING) DRIVERID | BIGINT ROUTEID | BIGINT ROUTENAME | VARCHAR(STRING) EVENTTYPE | VARCHAR(STRING) LATITUDE | DOUBLE LONGITUDE | DOUBLE CORRELATIONID | VARCHAR(STRING) Kafka Connect & Kafka Streams/KSQL
Demo (IV) - Create Stream ksql> SELECT * FROM truck_position_s; 1506922133306 | "truck/13/position0 | �2017-10-02T07:28:53 | 31 | 13 | 371182829 | Memphis to Little Rock | Normal | 41.76 | -89.6 | -2084263951914664106 1506922133396 | "truck/16/position0 | �2017-10-02T07:28:53 | 19 | 16 | 160405074 | Joplin to Kansas City Route 2 | Normal | 41.48 | -88.07 | -2084263951914664106 1506922133457 | "truck/30/position0 | �2017-10-02T07:28:53 | 26 | 30 | 160779139 | Des Moines to Chicago Route 2 | Normal | 41.85 | -89.29 | -2084263951914664106 1506922133485 | "truck/23/position0 | �2017-10-02T07:28:53 | 32 | 23 | 1090292248 | Peoria to Ceder Rapids Route 2 | Normal | 41.48 | -88.07 | - 2084263951914664106 1506922133497 | "truck/12/position0 | �2017-10-02T07:28:53 | 80 | 12 | 1961634315 | Saint Louis to Memphis | Normal | 41.74 | -91.47 | - 2084263951914664106 1506922133547 | "truck/14/position0 | �2017-10-02T07:28:53 | 73 | 14 | 1927624662 | Springfield to KC Via Columbia | Normal | 35.12 | -90.68 | - 2084263951914664106 Kafka Connect & Kafka Streams/KSQL
Demo (IV) - Create Stream ksql> SELECT * FROM truck_position_s WHERE eventtype != 'Normal'; 1506922264016 | "truck/11/position0 | �2017-10-02T07:31:04 | 27 | 11 | 1325712174 | Saint Louis to Tulsa Route2 | Lane Departure | 38.5 | -90.69 | - 2084263951914664106 1506922281156 | "truck/11/position0 | �2017-10-02T07:31:21 | 27 | 11 | 1325712174 | Saint Louis to Tulsa Route2 | Unsafe tail distance | 37.81 | -92.31 | -2084263951914664106 1506922284436 | "truck/10/position0 | �2017-10-02T07:31:24 | 93 | 10 | 1384345811 | Joplin to Kansas City | Unsafe following distance | 37.02 | -94.54 | -2084263951914664106 1506922297887 | "truck/11/position0 | �2017-10-02T07:31:37 | 27 | 11 | 1325712174 | Saint Louis to Tulsa Route2 | Unsafe following distance | 37.09 | - 94.23 | -2084263951914664106 Kafka Connect & Kafka Streams/KSQL
Demo (IV) - Create Stream ksql> CREATE STREAM dangerous_driving_s WITH (kafka_topic= dangerous_driving', value_format='DELIMITED') AS SELECT * FROM truck_position_s WHERE eventtype != 'Normal'; Message ---------------------------- Stream created and running ksql> select * from dangerous_driving_s; 1506922849375 | "truck/11/position0 | �2017-10-02T07:40:49 | 90 | 11 | 160779139 | Des Moines to Chicago Route 2 | Overspeed | 41.48 | -88.07 | 3569183071347898366 1506922866488 | "truck/11/position0 | �2017-10-02T07:41:06 | 90 | 11 | 160779139 | Des Moines to Chicago Route 2 | Overspeed | 40.38 | -89.17 | 3569183071347898366 Kafka Connect & Kafka Streams/KSQL
Demo (IV) - Create Stream ksql> describe dangerous_driving_s; Field | Type --------------------------------- ROWTIME | BIGINT ROWKEY | VARCHAR(STRING) TS | VARCHAR(STRING) TRUCKID | VARCHAR(STRING) DRIVERID | BIGINT ROUTEID | BIGINT ROUTENAME | VARCHAR(STRING) EVENTTYPE | VARCHAR(STRING) LATITUDE | DOUBLE LONGITUDE | DOUBLE CORRELATIONID | VARCHAR(STRING) Kafka Connect & Kafka Streams/KSQL
Demo (V) Truck-2 truck/nn/ position Truck-1 Truck-3 mqtt- source truck_ position detect_danger ous_driving dangerous_ driving Truck Driver jdbc-source trucking_ driver join_dangerous _driving_driver dangerous_dri ving_driver 27, Walter, Ward, Y, 24-JUL-85, 2017-10-02 15:19:00 console consumer 2016-06-02 14:39:56.605|98|27|803014426| Wichita to Little Rock Route2| Normal|38.65|90.21|5187297736652502631 {"id":27,"firstName":"Walter", "lastName":"Ward","available ":"Y","birthdate":"24-JUL- 85","last_update":150692305 2012} Kafka Connect & Kafka Streams/KSQL
Demo (V) – Create JDBC Connect through REST API #!/bin/bash curl -X "POST" "http://192.168.69.138:8083/connectors" -H "Content-Type: application/json" -d $'{ "name": "jdbc-driver-source", "config": { "connector.class": "JdbcSourceConnector", "connection.url":"jdbc:postgresql://db/sample?user=sample&password=sample", "mode": "timestamp", "timestamp.column.name":"last_update", "table.whitelist":"driver", "validate.non.null":"false", "topic.prefix":"trucking_", "key.converter":"org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enable": "false", "value.converter":"org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": "false", "name": "jdbc-driver-source", "transforms":"createKey,extractInt", "transforms.createKey.type":"org.apache.kafka.connect.transforms.ValueToKey", "transforms.createKey.fields":"id", "transforms.extractInt.type":"org.apache.kafka.connect.transforms.ExtractField$Key", "transforms.extractInt.field":"id" } }' Kafka Connect & Kafka Streams/KSQL
Demo (V) – Create JDBC Connect through REST API Kafka Connect & Kafka Streams/KSQL
Demo (V) - Create Table with Driver State ksql> CREATE TABLE driver_t (id BIGINT, first_name VARCHAR, last_name VARCHAR, available VARCHAR) WITH (kafka_topic='trucking_driver', value_format='JSON'); Message ---------------- Table created Kafka Connect & Kafka Streams/KSQL
Demo (V) - Create Table with Driver State ksql> CREATE STREAM dangerous_driving_and_driver_s WITH (kafka_topic='dangerous_driving_and_driver_s', value_format='JSON') AS SELECT driverid, first_name, last_name, truckid, routeid,routename, eventtype FROM truck_position_s LEFT JOIN driver_t ON dangerous_driving_and_driver_s.driverid = driver_t.id; Message ---------------------------- Stream created and running ksql> select * from dangerous_driving_and_driver_s; 1511173352906 | 21 | 21 | Lila | Page | 58 | 1594289134 | Memphis to Little Rock Route 2 | Unsafe tail distance 1511173353669 | 12 | 12 | Laurence | Lindsey | 93 | 1384345811 | Joplin to Kansas City | Lane Departure 1511173435385 | 11 | 11 | Micky | Isaacson | 22 | 1198242881 | Saint Louis to Chicago Route2 | Unsafe tail distance Kafka Connect & Kafka Streams/KSQL
Kafka and "Big Data" / "Fast Data" Ecosystem Kafka Connect & Kafka Streams/KSQL
Kafka and the Big Data / Fast Data ecosystem Kafka integrates with many popular products / frameworks • Apache Spark Streaming • Apache Flink • Apache Storm • Apache Apex • Apache NiFi • StreamSets • Oracle Stream Analytics • Oracle Service Bus • Oracle GoldenGate • Oracle Event Hub Cloud Service • Debezium CDC • … Additional Info: https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem Kafka Connect & Kafka Streams/KSQL
Kafka in Software Architecture Kafka Connect & Kafka Streams/KSQL
Hadoop Clusterd Hadoop Cluster Big Data Cluster Traditional Big Data Architecture BI Tools Enterprise Data Warehouse Billing & Ordering CRM / Profile Marketing Campaigns File Import / SQL Import SQL Search / Explore Online & Mobile Apps Search NoSQL Parallel Batch Processing Distributed Filesystem • Machine Learning • Graph Algorithms • Natural Language Processing Kafka Connect & Kafka Streams/KSQL
Event Hub Event Hub Hadoop Clusterd Hadoop Cluster Big Data Cluster Event Hub – handle event stream data BI Tools Enterprise Data Warehouse Location Social Click stream Sensor Data Billing & Ordering CRM / Profile Marketing Campaigns Event Hub Call Center Weather Data Mobile Apps SQL Search / Explore Online & Mobile Apps Search Data Flow NoSQL Parallel Batch Processing Distributed Filesystem • Machine Learning • Graph Algorithms • Natural Language Processing Kafka Connect & Kafka Streams/KSQL
Hadoop Clusterd Hadoop Cluster Big Data Cluster Event Hub – taking Velocity into account Location Social Click stream Sensor Data Billing & Ordering CRM / Profile Marketing Campaigns Call Center Mobile Apps Batch Analytics Streaming Analytics Results Parallel Batch Processing Distributed Filesystem Stream Analytics NoSQL Reference / Models SQL Search Dashboard BI Tools Enterprise Data Warehouse Search / Explore Online & Mobile Apps File Import / SQL Import Weather Data Event Hub Event Hub Event Hub Kafka Connect & Kafka Streams/KSQL
Container Hadoop Clusterd Hadoop Cluster Big Data Cluster Event Hub – Asynchronous Microservice Architecture Location Social Click stream Sensor Data Billing & Ordering CRM / Profile Marketing Campaigns Call Center Mobile Apps Parallel Batch ProcessingDistributed Filesystem Microservice NoSQLRDBMS SQL Search BI Tools Enterprise Data Warehouse Search / Explore Online & Mobile Apps File Import / SQL Import Weather Data { } API Event Hub Event Hub Event Hub Kafka Connect & Kafka Streams/KSQL
Kafka Connect & Kafka Streams/KSQL Technology on its own won't help you. You need to know how to use it properly.

More Related Content

PDF
Apache Kafka - Scalable Message Processing and more!
byGuido Schmutz
 
PDF
Apache Kafka - A modern Stream Processing Platform
byGuido Schmutz
 
PDF
Building event-driven (Micro)Services with Apache Kafka Ecosystem
byGuido Schmutz
 
PDF
Microservices with Kafka Ecosystem
byGuido Schmutz
 
PDF
Kafka Connect & Kafka Streams/KSQL - the ecosystem around Kafka
byGuido Schmutz
 
PDF
Kafka Connect & Kafka Streams/KSQL - the ecosystem around Kafka
byGuido Schmutz
 
PDF
Apache Kafka - Scalable Message Processing and more!
byGuido Schmutz
 
PDF
Apache Kafka - Event Sourcing, Monitoring, Librdkafka, Scaling & Partitioning
byGuido Schmutz
 
Apache Kafka - Scalable Message Processing and more!
byGuido Schmutz
 
Apache Kafka - A modern Stream Processing Platform
byGuido Schmutz
 
Building event-driven (Micro)Services with Apache Kafka Ecosystem
byGuido Schmutz
 
Microservices with Kafka Ecosystem
byGuido Schmutz
 
Kafka Connect & Kafka Streams/KSQL - the ecosystem around Kafka
byGuido Schmutz
 
Kafka Connect & Kafka Streams/KSQL - the ecosystem around Kafka
byGuido Schmutz
 
Apache Kafka - Scalable Message Processing and more!
byGuido Schmutz
 
Apache Kafka - Event Sourcing, Monitoring, Librdkafka, Scaling & Partitioning
byGuido Schmutz
 

What's hot

PDF
Ingesting streaming data into Graph Database
byGuido Schmutz
 
PDF
Apache Kafka Scalable Message Processing and more!
byGuido Schmutz
 
PDF
Internet of Things (IoT) - in the cloud or rather on-premises?
byGuido Schmutz
 
PDF
Apache Kafka - Scalable Message-Processing and more !
byGuido Schmutz
 
PDF
Spark (Structured) Streaming vs. Kafka Streams
byGuido Schmutz
 
PDF
Apache Kafka - Scalable Message-Processing and more !
byGuido Schmutz
 
PDF
Building Event Driven (Micro)services with Apache Kafka
byGuido Schmutz
 
PDF
Introduction to Stream Processing
byGuido Schmutz
 
PDF
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
byGuido Schmutz
 
PDF
Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka
byGuido Schmutz
 
PDF
Introduction to Stream Processing
byGuido Schmutz
 
PDF
Apache Kafka - Scalable Message-Processing and more !
byGuido Schmutz
 
PDF
Microservices with Kafka Ecosystem
byGuido Schmutz
 
PDF
Streaming Visualization
byGuido Schmutz
 
PDF
Ingesting and Processing IoT Data - using MQTT, Kafka Connect and KSQL
byGuido Schmutz
 
PDF
Building event-driven (Micro)Services with Apache Kafka
byGuido Schmutz
 
PDF
Streaming Visualization
byGuido Schmutz
 
PDF
Building Event-Driven (Micro) Services with Apache Kafka
byGuido Schmutz
 
PDF
Building Event Driven (Micro)services with Apache Kafka
byGuido Schmutz
 
PDF
Building event-driven (Micro)Services with Apache Kafka
byGuido Schmutz
 
Ingesting streaming data into Graph Database
byGuido Schmutz
 
Apache Kafka Scalable Message Processing and more!
byGuido Schmutz
 
Internet of Things (IoT) - in the cloud or rather on-premises?
byGuido Schmutz
 
Apache Kafka - Scalable Message-Processing and more !
byGuido Schmutz
 
Spark (Structured) Streaming vs. Kafka Streams
byGuido Schmutz
 
Apache Kafka - Scalable Message-Processing and more !
byGuido Schmutz
 
Building Event Driven (Micro)services with Apache Kafka
byGuido Schmutz
 
Introduction to Stream Processing
byGuido Schmutz
 
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
byGuido Schmutz
 
Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka
byGuido Schmutz
 
Introduction to Stream Processing
byGuido Schmutz
 
Apache Kafka - Scalable Message-Processing and more !
byGuido Schmutz
 
Microservices with Kafka Ecosystem
byGuido Schmutz
 
Streaming Visualization
byGuido Schmutz
 
Ingesting and Processing IoT Data - using MQTT, Kafka Connect and KSQL
byGuido Schmutz
 
Building event-driven (Micro)Services with Apache Kafka
byGuido Schmutz
 
Streaming Visualization
byGuido Schmutz
 
Building Event-Driven (Micro) Services with Apache Kafka
byGuido Schmutz
 
Building Event Driven (Micro)services with Apache Kafka
byGuido Schmutz
 
Building event-driven (Micro)Services with Apache Kafka
byGuido Schmutz
 

Similar to Kafka Connect & Kafka Streams/KSQL - powerful ecosystem around Kafka core

PDF
KSQL - Stream Processing simplified!
byGuido Schmutz
 
PDF
Kafka Connect & Streams - the ecosystem around Kafka
byGuido Schmutz
 
PDF
Chti jug - 2018-06-26
byFlorent Ramiere
 
PDF
Jug - ecosystem
byFlorent Ramiere
 
PDF
KSQL - Stream Processing simplified!
byGuido Schmutz
 
PDF
ksqlDB - Stream Processing simplified!
byGuido Schmutz
 
PDF
KSQL Deep Dive - The Open Source Streaming Engine for Apache Kafka
byKai Wähner
 
PDF
Streaming ETL with Apache Kafka and KSQL
byNick Dearden
 
PDF
Unlocking the world of stream processing with KSQL, the streaming SQL engine ...
byMichael Noll
 
PDF
Big, Fast, Easy Data: Distributed Stream Processing for Everyone with KSQL, t...
byMichael Noll
 
PPTX
KSQL and Kafka Streams – When to Use Which, and When to Use Both
byconfluent
 
PDF
Un'introduzione a Kafka Streams e KSQL... and why they matter!
byPaolo Castagna
 
PDF
Big Data LDN 2017: Look Ma, No Code! Building Streaming Data Pipelines With A...
byMatt Stubbs
 
PDF
Kafka Connect and Streams (Concepts, Architecture, Features)
byKai Wähner
 
ODP
KSQL- Streaming Sql for Kafka
byKnoldus Inc.
 
PDF
Real-Time Stream Processing with KSQL and Apache Kafka
byconfluent
 
PDF
Kafka Summit SF 2017 - Kafka Stream Processing for Everyone with KSQL
byconfluent
 
PDF
KSQL Intro
byconfluent
 
PDF
Kai Waehner - KSQL – The Open Source SQL Streaming Engine for Apache Kafka - ...
byCodemotion
 
PDF
Kai Waehner - KSQL – The Open Source SQL Streaming Engine for Apache Kafka - ...
byCodemotion
 
KSQL - Stream Processing simplified!
byGuido Schmutz
 
Kafka Connect & Streams - the ecosystem around Kafka
byGuido Schmutz
 
Chti jug - 2018-06-26
byFlorent Ramiere
 
Jug - ecosystem
byFlorent Ramiere
 
KSQL - Stream Processing simplified!
byGuido Schmutz
 
ksqlDB - Stream Processing simplified!
byGuido Schmutz
 
KSQL Deep Dive - The Open Source Streaming Engine for Apache Kafka
byKai Wähner
 
Streaming ETL with Apache Kafka and KSQL
byNick Dearden
 
Unlocking the world of stream processing with KSQL, the streaming SQL engine ...
byMichael Noll
 
Big, Fast, Easy Data: Distributed Stream Processing for Everyone with KSQL, t...
byMichael Noll
 
KSQL and Kafka Streams – When to Use Which, and When to Use Both
byconfluent
 
Un'introduzione a Kafka Streams e KSQL... and why they matter!
byPaolo Castagna
 
Big Data LDN 2017: Look Ma, No Code! Building Streaming Data Pipelines With A...
byMatt Stubbs
 
Kafka Connect and Streams (Concepts, Architecture, Features)
byKai Wähner
 
KSQL- Streaming Sql for Kafka
byKnoldus Inc.
 
Real-Time Stream Processing with KSQL and Apache Kafka
byconfluent
 
Kafka Summit SF 2017 - Kafka Stream Processing for Everyone with KSQL
byconfluent
 
KSQL Intro
byconfluent
 
Kai Waehner - KSQL – The Open Source SQL Streaming Engine for Apache Kafka - ...
byCodemotion
 
Kai Waehner - KSQL – The Open Source SQL Streaming Engine for Apache Kafka - ...
byCodemotion
 

More from Guido Schmutz

PDF
30 Minutes to the Analytics Platform with Infrastructure as Code
byGuido Schmutz
 
PDF
Event Broker (Kafka) in a Modern Data Architecture
byGuido Schmutz
 
PDF
Big Data, Data Lake, Fast Data - Dataserialiation-Formats
byGuido Schmutz
 
PDF
Kafka as your Data Lake - is it Feasible?
byGuido Schmutz
 
PDF
Event Hub (i.e. Kafka) in Modern Data Architecture
byGuido Schmutz
 
PDF
Event Hub (i.e. Kafka) in Modern Data (Analytics) Architecture
byGuido Schmutz
 
PDF
Location Analytics - Real-Time Geofencing using Apache Kafka
byGuido Schmutz
 
PDF
Solutions for bi-directional integration between Oracle RDBMS and Apache Kafka
byGuido Schmutz
 
PDF
What is Apache Kafka? Why is it so popular? Should I use it?
byGuido Schmutz
 
PDF
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
byGuido Schmutz
 
PDF
Location Analytics Real-Time Geofencing using Kafka
byGuido Schmutz
 
PDF
Streaming Visualisation
byGuido Schmutz
 
PDF
Kafka as an event store - is it good enough?
byGuido Schmutz
 
PDF
Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka
byGuido Schmutz
 
PDF
Fundamentals Big Data and AI Architecture
byGuido Schmutz
 
PDF
Location Analytics - Real-Time Geofencing using Kafka
byGuido Schmutz
 
PDF
Streaming Visualization
byGuido Schmutz
 
PDF
Streaming Visualization
byGuido Schmutz
 
PDF
Location Analytics - Real Time Geofencing using Apache Kafka
byGuido Schmutz
 
PDF
Introduction to Stream Processing
byGuido Schmutz
 
30 Minutes to the Analytics Platform with Infrastructure as Code
byGuido Schmutz
 
Event Broker (Kafka) in a Modern Data Architecture
byGuido Schmutz
 
Big Data, Data Lake, Fast Data - Dataserialiation-Formats
byGuido Schmutz
 
Kafka as your Data Lake - is it Feasible?
byGuido Schmutz
 
Event Hub (i.e. Kafka) in Modern Data Architecture
byGuido Schmutz
 
Event Hub (i.e. Kafka) in Modern Data (Analytics) Architecture
byGuido Schmutz
 
Location Analytics - Real-Time Geofencing using Apache Kafka
byGuido Schmutz
 
Solutions for bi-directional integration between Oracle RDBMS and Apache Kafka
byGuido Schmutz
 
What is Apache Kafka? Why is it so popular? Should I use it?
byGuido Schmutz
 
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
byGuido Schmutz
 
Location Analytics Real-Time Geofencing using Kafka
byGuido Schmutz
 
Streaming Visualisation
byGuido Schmutz
 
Kafka as an event store - is it good enough?
byGuido Schmutz
 
Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka
byGuido Schmutz
 
Fundamentals Big Data and AI Architecture
byGuido Schmutz
 
Location Analytics - Real-Time Geofencing using Kafka
byGuido Schmutz
 
Streaming Visualization
byGuido Schmutz
 
Streaming Visualization
byGuido Schmutz
 
Location Analytics - Real Time Geofencing using Apache Kafka
byGuido Schmutz
 
Introduction to Stream Processing
byGuido Schmutz
 

Recently uploaded

PDF
DIGITAL FORENSICS - Notes for Everything.pdf
bypankajkumavatbeit
 
PDF
eResource Scheduler Enterprise Resource Management and Scheduling Software.pdf
byeResource Scheduler
 
PPTX
The Future of IT Service Management AI Automation & Beyond.pptx
byChetu
 
PPTX
Ethics in AI - Artificial Intelligence Fundamentals.pptx
byemenyiblessing
 
PDF
TrustArc Webinar - Looking Ahead: The 2026 Privacy Landscape
byTrustArc
 
PDF
The year in review - MarvelClient in 2025
bypanagenda
 
PDF
Igniting the Future: Copilot trends, agentic transformation and product roadm...
byUni Systems S.M.S.A.
 
PDF
Accelerating Responsible AI Adoption in Public Sector and Private Organizations.
byYasir Naveed Riaz
 
PPTX
Combining Induction and Transduction for Abstract Reasoning.pptx
byallen578468
 
PDF
Innovative AI Solutions for Business Growth.pdf
byVox Works
 
PDF
8 LLM Surveys https://tinyurl.com/bdz8e6fp
byBob Marcus
 
PDF
TPPmark2025 Kenta Inoue's answer 12/04/2025
byKenta Inoue
 
PDF
Unlocking the Power of Salesforce Architecture: Frameworks for Effective Solu...
byvarsha30tiwari
 
PDF
AI Technology important knowledge ......
byutkarshj2007
 
PDF
Ransomware_Resilience_Strategic_Playbook.pdf
byAfonso Henrique Rodrigues Alves
 
PDF
Top 7 Manufacturing Software for Small Businesses Boosting Growth in 2025
byEnvertis Software Solutions
 
PDF
Usage Control for Process Discovery through a Trusted Execution Environment
byValerioGoretti
 
PDF
Security Technologys: Access Control, Firewall, VPN
byShielaLasala
 
PPTX
Ritesh_kumar_Aatmanirbhar Bharat: Make in India, Make for the World.pptx
byriteshrkgs2008
 
PDF
Greetings All Students Update 3 by Mia Corp
by©LDMMIA, ©Reiki Yoga
 
DIGITAL FORENSICS - Notes for Everything.pdf
bypankajkumavatbeit
 
eResource Scheduler Enterprise Resource Management and Scheduling Software.pdf
byeResource Scheduler
 
The Future of IT Service Management AI Automation & Beyond.pptx
byChetu
 
Ethics in AI - Artificial Intelligence Fundamentals.pptx
byemenyiblessing
 
TrustArc Webinar - Looking Ahead: The 2026 Privacy Landscape
byTrustArc
 
The year in review - MarvelClient in 2025
bypanagenda
 
Igniting the Future: Copilot trends, agentic transformation and product roadm...
byUni Systems S.M.S.A.
 
Accelerating Responsible AI Adoption in Public Sector and Private Organizations.
byYasir Naveed Riaz
 
Combining Induction and Transduction for Abstract Reasoning.pptx
byallen578468
 
Innovative AI Solutions for Business Growth.pdf
byVox Works
 
8 LLM Surveys https://tinyurl.com/bdz8e6fp
byBob Marcus
 
TPPmark2025 Kenta Inoue's answer 12/04/2025
byKenta Inoue
 
Unlocking the Power of Salesforce Architecture: Frameworks for Effective Solu...
byvarsha30tiwari
 
AI Technology important knowledge ......
byutkarshj2007
 
Ransomware_Resilience_Strategic_Playbook.pdf
byAfonso Henrique Rodrigues Alves
 
Top 7 Manufacturing Software for Small Businesses Boosting Growth in 2025
byEnvertis Software Solutions
 
Usage Control for Process Discovery through a Trusted Execution Environment
byValerioGoretti
 
Security Technologys: Access Control, Firewall, VPN
byShielaLasala
 
Ritesh_kumar_Aatmanirbhar Bharat: Make in India, Make for the World.pptx
byriteshrkgs2008
 
Greetings All Students Update 3 by Mia Corp
by©LDMMIA, ©Reiki Yoga
 

Kafka Connect & Kafka Streams/KSQL - powerful ecosystem around Kafka core

  • 1.
    BASEL BERN BRUGGDÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH Kafka Connect & Kafka Streams/KSQL Powerful Ecosystem Around Kafka Core Guido Schmutz – 5.12.2017 @gschmutz guidoschmutz.wordpress.com
  • 2.
    Guido Schmutz Working atTrivadis for more than 20 years Oracle ACE Director for Fusion Middleware and SOA Consultant, Trainer Software Architect for Java, Oracle, SOA and Big Data / Fast Data Head of Trivadis Architecture Board Technology Manager @ Trivadis More than 30 years of software development experience Contact: guido.schmutz@trivadis.com Blog: http://guidoschmutz.wordpress.com Slideshare: http://www.slideshare.net/gschmutz Twitter: gschmutz Kafka Connect & Kafka Streams/KSQL
  • 3.
    Agenda 1. What isApache Kafka? 2. Kafka Connect 3. Kafka Streams 4. KSQL 5. Kafka and "Big Data" / "Fast Data" Ecosystem 6. Kafka in Software Architecture Kafka Connect & Kafka Streams/KSQL
  • 4.
    Demo Example Truck-2 truck/nn/ position Truck-1 Truck-3 mqtt- source truck_ position detect_danger ous_driving dangerous_ driving Truck Driver jdbc-source trucking_ driver join_dangerous _driving_driver dangerous_dri ving_driver console consumer 2016-06-02 14:39:56.605|98|27|803014426| Wichita to LittleRock Route2| Normal|38.65|90.21|5187297736652502631 Kafka Connect & Kafka Streams/KSQL 27, Walter, Ward, Y, 24-JUL-85, 2017-10-02 15:19:00 {"id":27,"firstName":"Walter", "lastName":"Ward","available ":"Y","birthdate":"24-JUL- 85","last_update":150692305 2012}
  • 5.
    What is ApacheKafka? Kafka Connect & Kafka Streams/KSQL
  • 6.
    Apache Kafka History 20122013 2014 2015 2016 2017 Cluster mirroring data compression Intra-cluster replication 0.7 0.8 0.9 Data Processing (Streams API) 0.10 Data Integration (Connect API) 0.11 2018 Exactly Once Semantics Performance Improvements KSQL Developer Preview Kafka Connect & Kafka Streams/KSQL 1.0 JBOD Support Support Java 9
  • 7.
    Apache Kafka -Unix Analogy $ cat < in.txt | grep "kafka" | tr a-z A-Z > out.txt Kafka Connect API Kafka Connect APIKafka Streams API Kafka Core (Cluster) Adapted from: Confluent KSQL Kafka Connect & Kafka Streams/KSQL
  • 8.
    Apache Kafka –A Streaming Platform Kafka Connect & Kafka Streams/KSQL High-Level Architecture Distributed Log at the Core Scale-Out Architecture Logs do not (necessarilys) forget
  • 9.
    How to geta Kafka environent Kafka Connect & Kafka Streams/KSQL On Premises • Bare Metal Installation • Docker • Mesos / Kubernetes • Hadoop Distributions Cloud • Oracle Event Hub Cloud Service • Azure HDInsight Kafka • Confluent Cloud • …
  • 10.
    Demo (I) Truck-2 truck position Truck-1 Truck-3 console consumer 2016-06-02 14:39:56.605|98|27|803014426| Wichita to LittleRock Route2| Normal|38.65|90.21|5187297736652502631 Testdata-Generator by Hortonworks Kafka Connect & Kafka Streams/KSQL
  • 11.
    Demo (I) –Create Kafka Topic $ kafka-topics --zookeeper zookeeper:2181 --create --topic truck_position --partitions 8 --replication-factor 1 $ kafka-topics --zookeeper zookeeper:2181 –list __consumer_offsets _confluent-metrics _schemas docker-connect-configs docker-connect-offsets docker-connect-status truck_position Kafka Connect & Kafka Streams/KSQL
  • 12.
    Demo (I) –Run Producer and Kafka-Console-Consumer Kafka Connect & Kafka Streams/KSQL
  • 13.
    Demo (I) –Java Producer to "truck_position" Constructing a Kafka Producer private Properties kafkaProps = new Properties(); kafkaProps.put("bootstrap.servers","broker-1:9092); kafkaProps.put("key.serializer", "...StringSerializer"); kafkaProps.put("value.serializer", "...StringSerializer"); producer = new KafkaProducer<String, String>(kafkaProps); ProducerRecord<String, String> record = new ProducerRecord<>("truck_position", driverId, eventData); try { metadata = producer.send(record).get(); } catch (Exception e) {} Kafka Connect & Kafka Streams/KSQL
  • 14.
    Demo (II) –devices send to MQTT instead of Kafka Truck-2 truck/nn/ position Truck-1 Truck-3 2016-06-02 14:39:56.605|98|27|803014426| Wichita to Little Rock Route2| Normal|38.65|90.21|5187297736652502631 Kafka Connect & Kafka Streams/KSQL
  • 15.
    Demo (II) –devices send to MQTT instead of Kafka Kafka Connect & Kafka Streams/KSQL
  • 16.
    Demo (II) -devices send to MQTT instead of Kafka – how to get the data into Kafka? Truck-2 truck/nn/ position Truck-1 Truck-3 truck position raw ? 2016-06-02 14:39:56.605|98|27|803014426| Wichita to Little Rock Route2| Normal|38.65|90.21|5187297736652502631 Kafka Connect & Kafka Streams/KSQL
  • 17.
    Kafka Connect Kafka Connect& Kafka Streams/KSQL
  • 18.
    Kafka Connect -Overview Source Connector Sink Connector Kafka Connect & Kafka Streams/KSQL
  • 19.
    Kafka Connect –Single Message Transforms (SMT) Simple Transformations for a single message Defined as part of Kafka Connect • some useful transforms provided out-of-the-box • Easily implement your own Optionally deploy 1+ transforms with each connector • Modify messages produced by source connector • Modify messages sent to sink connectors Makes it much easier to mix and match connectors Some of currently available transforms: • InsertField • ReplaceField • MaskField • ValueToKey • ExtractField • TimestampRouter • RegexRouter • SetSchemaMetaData • Flatten • TimestampConverter Kafka Connect & Kafka Streams/KSQL
  • 20.
    Kafka Connect –Many Connectors 60+ since first release (0.9+) 20+ from Confluent and Partners Source: http://www.confluent.io/product/connectors Confluent supported Connectors Certified Connectors Community Connectors Kafka Connect & Kafka Streams/KSQL
  • 21.
    Demo (III) Truck-2 truck/nn/ position Truck-1 Truck-3 mqtt to kafka truck_ position 2016-06-02 14:39:56.605|98|27|803014426| Wichitato Little Rock Route2| Normal|38.65|90.21|5187297736652502631 console consumer Kafka Connect & Kafka Streams/KSQL
  • 22.
    Demo (III) –Create MQTT Connect through REST API #!/bin/bash curl -X "POST" "http://192.168.69.138:8083/connectors" -H "Content-Type: application/json" -d $'{ "name": "mqtt-source", "config": { "connector.class": "com.datamountaineer.streamreactor.connect.mqtt.source.MqttSourceConnector", "connect.mqtt.connection.timeout": "1000", "tasks.max": "1", "connect.mqtt.kcql": "INSERT INTO truck_position SELECT * FROM truck/+/position", "name": "MqttSourceConnector", "connect.mqtt.service.quality": "0", "connect.mqtt.client.id": "tm-mqtt-connect-01", "connect.mqtt.converter.throw.on.error": "true", "connect.mqtt.hosts": "tcp://mosquitto:1883" } }' Kafka Connect & Kafka Streams/KSQL
  • 23.
    Demo (III) –Call REST API and Kafka Console Consumer Kafka Connect & Kafka Streams/KSQL
  • 24.
    Demo (III) Truck-2 truck/nn/ position Truck-1 Truck-3 mqtt to kafka truck_ position 2016-06-02 14:39:56.605|98|27|803014426| Wichitato Little Rock Route2| Normal|38.65|90.21|5187297736652502631 console consumer what about some analytics ? Kafka Connect & Kafka Streams/KSQL
  • 25.
    Kafka Streams Kafka Connect& Kafka Streams/KSQL
  • 26.
    Kafka Streams -Overview • Designed as a simple and lightweight library in Apache Kafka • no external dependencies on systems other than Apache Kafka • Part of open source Apache Kafka, introduced in 0.10+ • Leverages Kafka as its internal messaging layer • Supports fault-tolerant local state • Event-at-a-time processing (not microbatch) with millisecond latency • Windowing with out-of-order data using a Google DataFlow-like model Kafka Connect & Kafka Streams/KSQL
  • 27.
    Kafka Stream DSLand Processor Topology KStream<Integer, String> stream1 = builder.stream("in-1"); KStream<Integer, String> stream2= builder.stream("in-2"); KStream<Integer, String> joined = stream1.leftJoin(stream2, …); KTable<> aggregated = joined.groupBy(…).count("store"); aggregated.to("out-1"); 1 2 lj a t State Kafka Connect & Kafka Streams/KSQL
  • 28.
    Kafka Stream DSLand Processor Topology KStream<Integer, String> stream1 = builder.stream("in-1"); KStream<Integer, String> stream2= builder.stream("in-2"); KStream<Integer, String> joined = stream1.leftJoin(stream2, …); KTable<> aggregated = joined.groupBy(…).count("store"); aggregated.to("out-1"); 1 2 lj a t State Kafka Connect & Kafka Streams/KSQL
  • 29.
    Kafka Streams Cluster ProcessorTopology Kafka Cluster input-1 input-2 store (changelog) output 1 2 lj a t State Kafka Connect & Kafka Streams/KSQL
  • 30.
  • 31.
  • 32.
    Stream vs. Table EventStream State Stream (Change Log Stream) 2017-10-02T20:18:46 11,Normal,41.87,-87.67 2017-10-02T20:18:55 11,Normal,40.38,-89.17 2017-10-02T20:18:59 21,Normal,42.23,-91.78 2017-10-02T20:19:01 21,Normal,41.71,-91.32 2017-10-02T20:19:02 11,Normal,38.65,-90.2 2017-10-02T20:19:23 21,Normal41.71,-91.32 11 2017-10-02T20:18:46,11,Normal,41.87,-87.67 11 2017-10-02T20:18:55,11,Normal,40.38,-89.17 21 2017-10-02T20:18:59, 21,Normal,42.23,-91.78 21 2017-10-02T20:19:01,21,Normal,41.71,-91.32 11 2017-10-02T20:19:02,11,Normal,38.65,-90.2 21 2017-10-02T20:19:23,21,Normal41.71,-91.32 Kafka Connect & Kafka Streams/KSQL KStream KTable
  • 33.
    Kafka Streams: KeyFeatures Kafka Connect & Kafka Streams/KSQL • Native, 100%-compatible Kafka integration • Secure stream processing using Kafka's security features • Elastic and highly scalable • Fault-tolerant • Stateful and stateless computations • Interactive queries • Time model • Windowing • Supports late-arriving and out-of-order data • Millisecond processing latency, no micro-batching • At-least-once and exactly-once processing guarantees
  • 34.
  • 35.
    Demo (IV) -Create Stream final KStreamBuilder builder = new KStreamBuilder(); KStream<String, String> source = builder.stream(stringSerde, stringSerde, "truck_position"); KStream<String, TruckPosition> positions = source.map((key,value) -> new KeyValue<>(key, TruckPosition.create(value))); KStream<String, TruckPosition> filtered = positions.filter(TruckPosition::filterNonNORMAL); filtered.map((key,value) -> new KeyValue<>(key,value._originalRecord)) .to("dangerous_driving"); Kafka Connect & Kafka Streams/KSQL
  • 36.
    KSQL Kafka Connect &Kafka Streams/KSQL
  • 37.
    KSQL: a StreamingSQL Engine for Apache Kafka • Enables stream processing with zero coding required • The simples way to process streams of data in real-time • Powered by Kafka and Kafka Streams: scalable, distributed, mature • All you need is Kafka – no complex deployments • available as Developer preview! • STREAM and TABLE as first-class citizens • STREAM = data in motion • TABLE = collected state of a stream • join STREAM and TABLE Kafka Connect & Kafka Streams/KSQL
  • 38.
    KSQL Deployment Models StandaloneMode Cluster Mode Source: Confluent Kafka Connect & Kafka Streams/KSQL
  • 39.
  • 40.
    Demo (V) -Start Kafka KSQL $ docker-compose exec ksql-cli ksql-cli local --bootstrap-server broker-1:9092 ====================================== = _ __ _____ ____ _ = = | |/ // ____|/ __ | | = = | ' /| (___ | | | | | = = | < ___ | | | | | = = | . ____) | |__| | |____ = = |_|______/ __________| = = = = Streaming SQL Engine for Kafka = Copyright 2017 Confluent Inc. CLI v0.1, Server v0.1 located at http://localhost:9098 Having trouble? Type 'help' (case-insensitive) for a rundown of how things work! ksql> Kafka Connect & Kafka Streams/KSQL
  • 41.
    Demo (IV) -Create Stream ksql> CREATE STREAM truck_position_s (ts VARCHAR, truckid VARCHAR, driverid BIGINT, routeid BIGINT, routename VARCHAR, eventtype VARCHAR, latitude DOUBLE, longitude DOUBLE, correlationid VARCHAR) WITH (kafka_topic='truck_position', value_format='DELIMITED'); Message ---------------- Stream created Kafka Connect & Kafka Streams/KSQL
  • 42.
    Demo (IV) -Create Stream ksql> describe truck_position_s; Field | Type --------------------------------- ROWTIME | BIGINT ROWKEY | VARCHAR(STRING) TS | VARCHAR(STRING) TRUCKID | VARCHAR(STRING) DRIVERID | BIGINT ROUTEID | BIGINT ROUTENAME | VARCHAR(STRING) EVENTTYPE | VARCHAR(STRING) LATITUDE | DOUBLE LONGITUDE | DOUBLE CORRELATIONID | VARCHAR(STRING) Kafka Connect & Kafka Streams/KSQL
  • 43.
    Demo (IV) -Create Stream ksql> SELECT * FROM truck_position_s; 1506922133306 | "truck/13/position0 | �2017-10-02T07:28:53 | 31 | 13 | 371182829 | Memphis to Little Rock | Normal | 41.76 | -89.6 | -2084263951914664106 1506922133396 | "truck/16/position0 | �2017-10-02T07:28:53 | 19 | 16 | 160405074 | Joplin to Kansas City Route 2 | Normal | 41.48 | -88.07 | -2084263951914664106 1506922133457 | "truck/30/position0 | �2017-10-02T07:28:53 | 26 | 30 | 160779139 | Des Moines to Chicago Route 2 | Normal | 41.85 | -89.29 | -2084263951914664106 1506922133485 | "truck/23/position0 | �2017-10-02T07:28:53 | 32 | 23 | 1090292248 | Peoria to Ceder Rapids Route 2 | Normal | 41.48 | -88.07 | - 2084263951914664106 1506922133497 | "truck/12/position0 | �2017-10-02T07:28:53 | 80 | 12 | 1961634315 | Saint Louis to Memphis | Normal | 41.74 | -91.47 | - 2084263951914664106 1506922133547 | "truck/14/position0 | �2017-10-02T07:28:53 | 73 | 14 | 1927624662 | Springfield to KC Via Columbia | Normal | 35.12 | -90.68 | - 2084263951914664106 Kafka Connect & Kafka Streams/KSQL
  • 44.
    Demo (IV) -Create Stream ksql> SELECT * FROM truck_position_s WHERE eventtype != 'Normal'; 1506922264016 | "truck/11/position0 | �2017-10-02T07:31:04 | 27 | 11 | 1325712174 | Saint Louis to Tulsa Route2 | Lane Departure | 38.5 | -90.69 | - 2084263951914664106 1506922281156 | "truck/11/position0 | �2017-10-02T07:31:21 | 27 | 11 | 1325712174 | Saint Louis to Tulsa Route2 | Unsafe tail distance | 37.81 | -92.31 | -2084263951914664106 1506922284436 | "truck/10/position0 | �2017-10-02T07:31:24 | 93 | 10 | 1384345811 | Joplin to Kansas City | Unsafe following distance | 37.02 | -94.54 | -2084263951914664106 1506922297887 | "truck/11/position0 | �2017-10-02T07:31:37 | 27 | 11 | 1325712174 | Saint Louis to Tulsa Route2 | Unsafe following distance | 37.09 | - 94.23 | -2084263951914664106 Kafka Connect & Kafka Streams/KSQL
  • 45.
    Demo (IV) -Create Stream ksql> CREATE STREAM dangerous_driving_s WITH (kafka_topic= dangerous_driving', value_format='DELIMITED') AS SELECT * FROM truck_position_s WHERE eventtype != 'Normal'; Message ---------------------------- Stream created and running ksql> select * from dangerous_driving_s; 1506922849375 | "truck/11/position0 | �2017-10-02T07:40:49 | 90 | 11 | 160779139 | Des Moines to Chicago Route 2 | Overspeed | 41.48 | -88.07 | 3569183071347898366 1506922866488 | "truck/11/position0 | �2017-10-02T07:41:06 | 90 | 11 | 160779139 | Des Moines to Chicago Route 2 | Overspeed | 40.38 | -89.17 | 3569183071347898366 Kafka Connect & Kafka Streams/KSQL
  • 46.
    Demo (IV) -Create Stream ksql> describe dangerous_driving_s; Field | Type --------------------------------- ROWTIME | BIGINT ROWKEY | VARCHAR(STRING) TS | VARCHAR(STRING) TRUCKID | VARCHAR(STRING) DRIVERID | BIGINT ROUTEID | BIGINT ROUTENAME | VARCHAR(STRING) EVENTTYPE | VARCHAR(STRING) LATITUDE | DOUBLE LONGITUDE | DOUBLE CORRELATIONID | VARCHAR(STRING) Kafka Connect & Kafka Streams/KSQL
  • 47.
  • 48.
    Demo (V) –Create JDBC Connect through REST API #!/bin/bash curl -X "POST" "http://192.168.69.138:8083/connectors" -H "Content-Type: application/json" -d $'{ "name": "jdbc-driver-source", "config": { "connector.class": "JdbcSourceConnector", "connection.url":"jdbc:postgresql://db/sample?user=sample&password=sample", "mode": "timestamp", "timestamp.column.name":"last_update", "table.whitelist":"driver", "validate.non.null":"false", "topic.prefix":"trucking_", "key.converter":"org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enable": "false", "value.converter":"org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": "false", "name": "jdbc-driver-source", "transforms":"createKey,extractInt", "transforms.createKey.type":"org.apache.kafka.connect.transforms.ValueToKey", "transforms.createKey.fields":"id", "transforms.extractInt.type":"org.apache.kafka.connect.transforms.ExtractField$Key", "transforms.extractInt.field":"id" } }' Kafka Connect & Kafka Streams/KSQL
  • 49.
    Demo (V) –Create JDBC Connect through REST API Kafka Connect & Kafka Streams/KSQL
  • 50.
    Demo (V) -Create Table with Driver State ksql> CREATE TABLE driver_t (id BIGINT, first_name VARCHAR, last_name VARCHAR, available VARCHAR) WITH (kafka_topic='trucking_driver', value_format='JSON'); Message ---------------- Table created Kafka Connect & Kafka Streams/KSQL
  • 51.
    Demo (V) -Create Table with Driver State ksql> CREATE STREAM dangerous_driving_and_driver_s WITH (kafka_topic='dangerous_driving_and_driver_s', value_format='JSON') AS SELECT driverid, first_name, last_name, truckid, routeid,routename, eventtype FROM truck_position_s LEFT JOIN driver_t ON dangerous_driving_and_driver_s.driverid = driver_t.id; Message ---------------------------- Stream created and running ksql> select * from dangerous_driving_and_driver_s; 1511173352906 | 21 | 21 | Lila | Page | 58 | 1594289134 | Memphis to Little Rock Route 2 | Unsafe tail distance 1511173353669 | 12 | 12 | Laurence | Lindsey | 93 | 1384345811 | Joplin to Kansas City | Lane Departure 1511173435385 | 11 | 11 | Micky | Isaacson | 22 | 1198242881 | Saint Louis to Chicago Route2 | Unsafe tail distance Kafka Connect & Kafka Streams/KSQL
  • 52.
    Kafka and "BigData" / "Fast Data" Ecosystem Kafka Connect & Kafka Streams/KSQL
  • 53.
    Kafka and theBig Data / Fast Data ecosystem Kafka integrates with many popular products / frameworks • Apache Spark Streaming • Apache Flink • Apache Storm • Apache Apex • Apache NiFi • StreamSets • Oracle Stream Analytics • Oracle Service Bus • Oracle GoldenGate • Oracle Event Hub Cloud Service • Debezium CDC • … Additional Info: https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem Kafka Connect & Kafka Streams/KSQL
  • 54.
    Kafka in SoftwareArchitecture Kafka Connect & Kafka Streams/KSQL
  • 55.
    Hadoop Clusterd Hadoop Cluster BigData Cluster Traditional Big Data Architecture BI Tools Enterprise Data Warehouse Billing & Ordering CRM / Profile Marketing Campaigns File Import / SQL Import SQL Search / Explore Online & Mobile Apps Search NoSQL Parallel Batch Processing Distributed Filesystem • Machine Learning • Graph Algorithms • Natural Language Processing Kafka Connect & Kafka Streams/KSQL
  • 56.
    Event Hub Event Hub Hadoop Clusterd Hadoop Cluster BigData Cluster Event Hub – handle event stream data BI Tools Enterprise Data Warehouse Location Social Click stream Sensor Data Billing & Ordering CRM / Profile Marketing Campaigns Event Hub Call Center Weather Data Mobile Apps SQL Search / Explore Online & Mobile Apps Search Data Flow NoSQL Parallel Batch Processing Distributed Filesystem • Machine Learning • Graph Algorithms • Natural Language Processing Kafka Connect & Kafka Streams/KSQL
  • 57.
    Hadoop Clusterd Hadoop Cluster BigData Cluster Event Hub – taking Velocity into account Location Social Click stream Sensor Data Billing & Ordering CRM / Profile Marketing Campaigns Call Center Mobile Apps Batch Analytics Streaming Analytics Results Parallel Batch Processing Distributed Filesystem Stream Analytics NoSQL Reference / Models SQL Search Dashboard BI Tools Enterprise Data Warehouse Search / Explore Online & Mobile Apps File Import / SQL Import Weather Data Event Hub Event Hub Event Hub Kafka Connect & Kafka Streams/KSQL
  • 58.
    Container Hadoop Clusterd Hadoop Cluster BigData Cluster Event Hub – Asynchronous Microservice Architecture Location Social Click stream Sensor Data Billing & Ordering CRM / Profile Marketing Campaigns Call Center Mobile Apps Parallel Batch ProcessingDistributed Filesystem Microservice NoSQLRDBMS SQL Search BI Tools Enterprise Data Warehouse Search / Explore Online & Mobile Apps File Import / SQL Import Weather Data { } API Event Hub Event Hub Event Hub Kafka Connect & Kafka Streams/KSQL
  • 59.
    Kafka Connect &Kafka Streams/KSQL Technology on its own won't help you. You need to know how to use it properly.