Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Apache Kafka - Scalable Message Processing and more!

496 views

Published on

After a quick overview and introduction of Apache Kafka, this session cover two components which extend the core of Apache Kafka: Kafka Connect and Kafka Streams/KSQL.
Kafka Connects role is to access data from the out-side-world and make it available inside Kafka by publishing it into a Kafka topic. On the other hand, Kafka Connect is also responsible to transport information from inside Kafka to the outside world, which could be a database or a file system. There are many existing connectors for different source and target systems available out-of-the-box, either provided by the community or by Confluent or other vendors. You simply configure these connectors and off you go.
Kafka Streams is a light-weight component which extends Kafka with stream processing functionality. By that, Kafka can now not only reliably and scalable transport events and messages through the Kafka broker but also analyse and process these event in real-time. Interestingly Kafka Streams does not provide its own cluster infrastructure and it is also not meant to run on a Kafka cluster. The idea is to run Kafka Streams where it makes sense, which can be inside a “normal” Java application, inside a Web container or on a more modern containerized (cloud) infrastructure, such as Mesos, Kubernetes or Docker. Kafka Streams has a lot of interesting features, such as reliable state handling, queryable state and much more. KSQL is a streaming engine for Apache Kafka, providing a simple and completely interactive SQL interface for processing data in Kafka.

Published in: Data & Analytics

Apache Kafka - Scalable Message Processing and more!

  1. 1. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH Apache Kafka – Scalable Stream Processing and more! Guido Schmutz – 5.12.2017 @gschmutz guidoschmutz.wordpress.com
  2. 2. Guido Schmutz Working at Trivadis for more than 20 years Oracle ACE Director for Fusion Middleware and SOA Consultant, Trainer Software Architect for Java, Oracle, SOA and Big Data / Fast Data Head of Trivadis Architecture Board Technology Manager @ Trivadis More than 30 years of software development experience Contact: guido.schmutz@trivadis.com Blog: http://guidoschmutz.wordpress.com Slideshare: http://www.slideshare.net/gschmutz Twitter: gschmutz Apache Kafka – Scalable Stream Processing and more!
  3. 3. Agenda 1. What is Apache Kafka? 2. Kafka Connect 3. Kafka Integration with other components 4. Kafka Streams 5. KSQL Apache Kafka – Scalable Stream Processing and more!
  4. 4. What is Apache Kafka? Apache Kafka – Scalable Stream Processing and more!
  5. 5. Apache Kafka History 2012 2013 2014 2015 2016 2017 Cluster mirroring data compression Intra-cluster replication 0.7 0.8 0.9 Data Processing (Streams API) 0.10 Data Integration (Connect API) 0.11 2018 Exactly Once Semantics Performance Improvements KSQL Developer Preview Apache Kafka – Scalable Stream Processing and more! 1.0 JBOD Support Support Java 9
  6. 6. Apache Kafka – A Streaming Platform Apache Kafka – Scalable Stream Processing and more! High-Level Architecture Distributed Log at the Core Scale-Out Architecture Logs do not (necessarily) forget
  7. 7. Strong Ordering Guarantees most business systems need strong ordering guarantees messages that require relative ordering need to be sent to the same partition supply same key for all messages that require a relative order To maintain global ordering use a single partition topic Producer 1 Consumer 1 Broker 1 Broker 2 Broker 3 Consumer 2 Consumer 3 Key-1 Key-2 Key-3 Key-4 Key-5 Key-6 Key-3 Key-1 Apache Kafka – Scalable Stream Processing and more!
  8. 8. Durable and Highly Available Messaging Producer 1 Broker 1 Broker 2 Broker 3 Producer 1 Broker 1 Broker 2 Broker 3 Consumer 1 Consumer 1 Consumer 2Consumer 2 Apache Kafka – Scalable Stream Processing and more!
  9. 9. Durable and Highly Available Messaging (II) Producer 1 Broker 1 Broker 2 Broker 3 Producer 1 Broker 1 Broker 2 Broker 3 Consumer 1 Consumer 1 Consumer 2 Consumer 2 Apache Kafka – Scalable Stream Processing and more!
  10. 10. How to get a Kafka environent Apache Kafka – Scalable Stream Processing and more! On Premises • Bare Metal Installation • Docker • Mesos / Kubernetes • Hadoop Distributions Cloud • Oracle Event Hub Cloud Service • Azure HDInsight Kafka • Confluent Cloud • …
  11. 11. Demo - Kafka Truck-2 truck position Truck-1 Truck-3 console consumer 2016-06-02 14:39:56.605|98|27|803014426| Wichita to Little Rock Route2| Normal|38.65|90.21|5187297736652502631 Testdata-Generator by Hortonworks Apache Kafka – Scalable Stream Processing and more!
  12. 12. Demo – Create Kafka Topic $ kafka-topics --zookeeper zookeeper:2181 --create --topic truck_position --partitions 8 --replication-factor 1 $ kafka-topics --zookeeper zookeeper:2181 –list __consumer_offsets _confluent-metrics _schemas docker-connect-configs docker-connect-offsets docker-connect-status truck_position Apache Kafka – Scalable Stream Processing and more!
  13. 13. Demo – Run Producer and Kafka-Console-Consumer Apache Kafka – Scalable Stream Processing and more!
  14. 14. Demo – Java Producer to "truck_position" Constructing a Kafka Producer private Properties kafkaProps = new Properties(); kafkaProps.put("bootstrap.servers","broker-1:9092); kafkaProps.put("key.serializer", "...StringSerializer"); kafkaProps.put("value.serializer", "...StringSerializer"); producer = new KafkaProducer<String, String>(kafkaProps); ProducerRecord<String, String> record = new ProducerRecord<>("truck_position", driverId, eventData); try { metadata = producer.send(record).get(); } catch (Exception e) {} Apache Kafka – Scalable Stream Processing and more!
  15. 15. Demo - MQTT instead of Kafka Truck-2 truck/nn/ position Truck-1 Truck-3 2016-06-02 14:39:56.605|98|27|803014426| Wichita to Little Rock Route2| Normal|38.65|90.21|5187297736652502631 Apache Kafka – Scalable Stream Processing and more!
  16. 16. Demo –MQTT instead of Kafka Apache Kafka – Scalable Stream Processing and more!
  17. 17. Demo MQTT instead of Kafka – how to get the data into Kafka? Truck-2 truck/nn/ position Truck-1 Truck-3 truck position raw ? 2016-06-02 14:39:56.605|98|27|803014426| Wichita to Little Rock Route2| Normal|38.65|90.21|5187297736652502631 Apache Kafka – Scalable Stream Processing and more!
  18. 18. Apache Kafka – wait there is more! Apache Kafka – Scalable Stream Processing and more! Source Connector trucking_ driver Kafka Broker Sink Connector Stream Processing
  19. 19. Kafka Connect Apache Kafka – Scalable Stream Processing and more!
  20. 20. Kafka Connect - Overview Source Connector Sink Connector Apache Kafka – Scalable Stream Processing and more!
  21. 21. Kafka Connect – Single Message Transforms (SMT) Simple Transformations for a single message Defined as part of Kafka Connect • some useful transforms provided out-of-the-box • Easily implement your own Optionally deploy 1+ transforms with each connector • Modify messages produced by source connector • Modify messages sent to sink connectors Makes it much easier to mix and match connectors Some of currently available transforms: • InsertField • ReplaceField • MaskField • ValueToKey • ExtractField • TimestampRouter • RegexRouter • SetSchemaMetaData • Flatten • TimestampConverter Apache Kafka – Scalable Stream Processing and more!
  22. 22. Kafka Connect – Many Connectors 60+ since first release (0.9+) 20+ from Confluent and Partners Source: http://www.confluent.io/product/connectors Confluent supported Connectors Certified Connectors Community Connectors Apache Kafka – Scalable Stream Processing and more!
  23. 23. Demo – Kafka Connect Truck-2 truck/nn/ position Truck-1 Truck-3 mqtt to kafka truck_ position console consumer Apache Kafka – Scalable Stream Processing and more! 2016-06-02 14:39:56.605|98|27|803014426| Wichita to Little Rock Route2| Normal|38.65|90.21|5187297736652502631
  24. 24. Demo – Create MQTT Connect through REST API #!/bin/bash curl -X "POST" "http://192.168.69.138:8083/connectors" -H "Content-Type: application/json" -d $'{ "name": "mqtt-source", "config": { "connector.class": "com.datamountaineer.streamreactor.connect.mqtt.source.MqttSourceConnector", "connect.mqtt.connection.timeout": "1000", "tasks.max": "1", "connect.mqtt.kcql": "INSERT INTO truck_position SELECT * FROM truck/+/position", "name": "MqttSourceConnector", "connect.mqtt.service.quality": "0", "connect.mqtt.client.id": "tm-mqtt-connect-01", "connect.mqtt.converter.throw.on.error": "true", "connect.mqtt.hosts": "tcp://mosquitto:1883" } }' Apache Kafka – Scalable Stream Processing and more!
  25. 25. Demo – Call REST API and Kafka Console Consumer Apache Kafka – Scalable Stream Processing and more!
  26. 26. Kafka Integration with other components Apache Kafka – Scalable Stream Processing and more!
  27. 27. Kafka and the Big Data / Fast Data ecosystem Kafka integrates with many popular products / frameworks • Apache Spark Streaming • Apache Flink • Apache Storm • Apache Apex • Apache NiFi • StreamSets • Oracle Stream Analytics • Oracle Service Bus • Oracle GoldenGate • Oracle Event Hub Cloud Service • Debezium CDC • … Additional Info: https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem Apache Kafka – Scalable Stream Processing and more!
  28. 28. StreamSets Data Collector • Founded by ex-Cloudera, Informatica employees • Continuous open source, intent-driven, big data ingest • Visible, record-oriented approach fixes combinatorial explosion • Batch or stream processing • Standalone, Spark cluster, MapReduce cluster • IDE for pipeline development by ‘civilians’ • Relatively new - first public release September 2015 • So far, vast majority of commits are from StreamSets staff Apache Kafka – Scalable Stream Processing and more!
  29. 29. Demo StreamSets Data Collector Truck-3 truck position raw truck/nn/ positionTruck-4 Truck-5 Kafka to Cassandra {"truckid":"57","driverid":"15","routeid":"1927624662 ","eventtype":"Normal","latitude":"38.65","longitude": "-90.21","correlationId":"4412891759760421296"} MQTT-2 to Kafka Edge Port: 1883 trucking Apache Kafka – Scalable Stream Processing and more!
  30. 30. Demo StreamSets Data Collector Apache Kafka – Scalable Stream Processing and more!
  31. 31. Demo StreamSets Data Collector Apache Kafka – Scalable Stream Processing and more!
  32. 32. Demo StreamSets Data Collector Apache Kafka – Scalable Stream Processing and more!
  33. 33. Demo StreamSets Data Collector Apache Kafka – Scalable Stream Processing and more!
  34. 34. Demo StreamSets Data Collector Truck-3 truck position raw truck/nn/ positionTruck-4 Truck-5 Kafka to Cassandra {"truckid":"57","driverid":"15","routeid":"1927624662 ","eventtype":"Normal","latitude":"38.65","longitude": "-90.21","correlationId":"4412891759760421296"} MQTT-2 to Kafka Edge Port: 1883 trucking what about some analytics ? Apache Kafka – Scalable Stream Processing and more!
  35. 35. Kafka Streams Apache Kafka – Scalable Stream Processing and more!
  36. 36. Kafka Streams - Overview • Designed as a simple and lightweight library in Apache Kafka • no external dependencies on systems other than Apache Kafka • Part of open source Apache Kafka, introduced in 0.10+ • Leverages Kafka as its internal messaging layer • Supports fault-tolerant local state • Event-at-a-time processing (not microbatch) with millisecond latency • Windowing with out-of-order data using a Google DataFlow-like model Apache Kafka – Scalable Stream Processing and more!
  37. 37. Kafka Stream DSL and Processor Topology KStream<Integer, String> stream1 = builder.stream("in-1"); KStream<Integer, String> stream2= builder.stream("in-2"); KStream<Integer, String> joined = stream1.leftJoin(stream2, …); KTable<> aggregated = joined.groupBy(…).count("store"); aggregated.to("out-1"); 1 2 lj a t State Apache Kafka – Scalable Stream Processing and more!
  38. 38. Kafka Stream DSL and Processor Topology KStream<Integer, String> stream1 = builder.stream("in-1"); KStream<Integer, String> stream2= builder.stream("in-2"); KStream<Integer, String> joined = stream1.leftJoin(stream2, …); KTable<> aggregated = joined.groupBy(…).count("store"); aggregated.to("out-1"); 1 2 lj a t State Apache Kafka – Scalable Stream Processing and more!
  39. 39. Kafka Streams Cluster Processor Topology Kafka Cluster input-1 input-2 store (changelog) output 1 2 lj a t State Apache Kafka – Scalable Stream Processing and more!
  40. 40. Kafka Cluster Processor Topology input-1 Partition 0 Partition 1 Partition 2 Partition 3 input-2 Partition 0 Partition 1 Partition 2 Partition 3 Kafka Streams 1 Kafka Streams 2 Kafka Streams 3 Kafka Streams 4 Apache Kafka – Scalable Stream Processing and more!
  41. 41. Demo – Kafka Streams Truck-2 truck/nn/ position Truck-1 Truck-3 mqtt to kafka truck_ position_s detect_danger ous_driving dangerous_ driving console consumer 2016-06-02 14:39:56.605|98|27|803014426| Wichita to Little Rock Route2| Normal|38.65|90.21|5187297736652502631 Apache Kafka – Scalable Stream Processing and more! Kafka to Cassandra trucking
  42. 42. Demo (IV) - Create Stream final KStreamBuilder builder = new KStreamBuilder(); KStream<String, String> source = builder.stream(stringSerde, stringSerde, "truck_position"); KStream<String, TruckPosition> positions = source.map((key,value) -> new KeyValue<>(key, TruckPosition.create(value))); KStream<String, TruckPosition> filtered = positions.filter(TruckPosition::filterNonNORMAL); filtered.map((key,value) -> new KeyValue<>(key,value._originalRecord)) .to("dangerous_driving"); Apache Kafka – Scalable Stream Processing and more!
  43. 43. KSQL Apache Kafka – Scalable Stream Processing and more!
  44. 44. KSQL: a Streaming SQL Engine for Apache Kafka • Enables stream processing with zero coding required • The simples way to process streams of data in real-time • Powered by Kafka and Kafka Streams: scalable, distributed, mature • All you need is Kafka – no complex deployments • available as Developer preview! • STREAM and TABLE as first-class citizens • STREAM = data in motion • TABLE = collected state of a stream • join STREAM and TABLE Apache Kafka – Scalable Stream Processing and more!
  45. 45. Demo – KSQL Truck-2 truck/nn/ position Truck-1 Truck-3 mqtt to kafka truck_ position detect_danger ous_driving dangerous_ driving console consumer 2016-06-02 14:39:56.605|98|27|803014426| Wichita to Little Rock Route2| Normal|38.65|90.21|5187297736652502631 Apache Kafka – Scalable Stream Processing and more! Kafka to Cassandra trucking
  46. 46. Demo (V) - Start Kafka KSQL $ docker-compose exec ksql-cli ksql-cli local --bootstrap-server broker-1:9092 ====================================== = _ __ _____ ____ _ = = | |/ // ____|/ __ | | = = | ' /| (___ | | | | | = = | < ___ | | | | | = = | . ____) | |__| | |____ = = |_|______/ __________| = = = = Streaming SQL Engine for Kafka = Copyright 2017 Confluent Inc. CLI v0.1, Server v0.1 located at http://localhost:9098 Having trouble? Type 'help' (case-insensitive) for a rundown of how things work! ksql> Apache Kafka – Scalable Stream Processing and more!
  47. 47. Demo (IV) - Create Stream ksql> CREATE STREAM truck_position_s (ts VARCHAR, truckid VARCHAR, driverid BIGINT, routeid BIGINT, routename VARCHAR, eventtype VARCHAR, latitude DOUBLE, longitude DOUBLE, correlationid VARCHAR) WITH (kafka_topic='truck_position', value_format='DELIMITED'); Message ---------------- Stream created Apache Kafka – Scalable Stream Processing and more!
  48. 48. Demo (IV) - Create Stream ksql> describe truck_position_s; Field | Type --------------------------------- ROWTIME | BIGINT ROWKEY | VARCHAR(STRING) TS | VARCHAR(STRING) TRUCKID | VARCHAR(STRING) DRIVERID | BIGINT ROUTEID | BIGINT ROUTENAME | VARCHAR(STRING) EVENTTYPE | VARCHAR(STRING) LATITUDE | DOUBLE LONGITUDE | DOUBLE CORRELATIONID | VARCHAR(STRING) Apache Kafka – Scalable Stream Processing and more!
  49. 49. Demo (IV) - Create Stream ksql> SELECT * FROM truck_position_s; 1506922133306 | "truck/13/position0 | �2017-10-02T07:28:53 | 31 | 13 | 371182829 | Memphis to Little Rock | Normal | 41.76 | -89.6 | -2084263951914664106 1506922133396 | "truck/16/position0 | �2017-10-02T07:28:53 | 19 | 16 | 160405074 | Joplin to Kansas City Route 2 | Normal | 41.48 | -88.07 | -2084263951914664106 1506922133457 | "truck/30/position0 | �2017-10-02T07:28:53 | 26 | 30 | 160779139 | Des Moines to Chicago Route 2 | Normal | 41.85 | -89.29 | -2084263951914664106 1506922133485 | "truck/23/position0 | �2017-10-02T07:28:53 | 32 | 23 | 1090292248 | Peoria to Ceder Rapids Route 2 | Normal | 41.48 | -88.07 | - 2084263951914664106 1506922133497 | "truck/12/position0 | �2017-10-02T07:28:53 | 80 | 12 | 1961634315 | Saint Louis to Memphis | Normal | 41.74 | -91.47 | - 2084263951914664106 1506922133547 | "truck/14/position0 | �2017-10-02T07:28:53 | 73 | 14 | 1927624662 | Springfield to KC Via Columbia | Normal | 35.12 | -90.68 | - 2084263951914664106 Apache Kafka – Scalable Stream Processing and more!
  50. 50. Demo (IV) - Create Stream ksql> SELECT * FROM truck_position_s WHERE eventtype != 'Normal'; 1506922264016 | "truck/11/position0 | �2017-10-02T07:31:04 | 27 | 11 | 1325712174 | Saint Louis to Tulsa Route2 | Lane Departure | 38.5 | -90.69 | - 2084263951914664106 1506922281156 | "truck/11/position0 | �2017-10-02T07:31:21 | 27 | 11 | 1325712174 | Saint Louis to Tulsa Route2 | Unsafe tail distance | 37.81 | -92.31 | -2084263951914664106 1506922284436 | "truck/10/position0 | �2017-10-02T07:31:24 | 93 | 10 | 1384345811 | Joplin to Kansas City | Unsafe following distance | 37.02 | -94.54 | -2084263951914664106 1506922297887 | "truck/11/position0 | �2017-10-02T07:31:37 | 27 | 11 | 1325712174 | Saint Louis to Tulsa Route2 | Unsafe following distance | 37.09 | - 94.23 | -2084263951914664106 Apache Kafka – Scalable Stream Processing and more!
  51. 51. Demo (IV) - Create Stream ksql> CREATE STREAM dangerous_driving_s WITH (kafka_topic= dangerous_driving', value_format='DELIMITED') AS SELECT * FROM truck_position_s WHERE eventtype != 'Normal'; Message ---------------------------- Stream created and running ksql> select * from dangerous_driving_s; 1506922849375 | "truck/11/position0 | �2017-10-02T07:40:49 | 90 | 11 | 160779139 | Des Moines to Chicago Route 2 | Overspeed | 41.48 | -88.07 | 3569183071347898366 1506922866488 | "truck/11/position0 | �2017-10-02T07:41:06 | 90 | 11 | 160779139 | Des Moines to Chicago Route 2 | Overspeed | 40.38 | -89.17 | 3569183071347898366 Apache Kafka – Scalable Stream Processing and more!
  52. 52. Demo (IV) - Create Stream ksql> describe dangerous_driving_s; Field | Type --------------------------------- ROWTIME | BIGINT ROWKEY | VARCHAR(STRING) TS | VARCHAR(STRING) TRUCKID | VARCHAR(STRING) DRIVERID | BIGINT ROUTEID | BIGINT ROUTENAME | VARCHAR(STRING) EVENTTYPE | VARCHAR(STRING) LATITUDE | DOUBLE LONGITUDE | DOUBLE CORRELATIONID | VARCHAR(STRING) Apache Kafka – Scalable Stream Processing and more!
  53. 53. Demo - All Truck-2 truck/nn/ position Truck-1 Truck-3 mqtt- source truck_ position detect_danger ous_driving dangerous_ driving Truck Driver jdbc-source trucking_ driver join_dangerous _driving_driver dangerous_dri ving_driver 27, Walter, Ward, Y, 24-JUL-85, 2017-10-02 15:19:00 console consumer 2016-06-02 14:39:56.605|98|27|803014426| Wichita to Little Rock Route2| Normal|38.65|90.21|5187297736652502631 {"id":27,"firstName":"Walter", "lastName":"Ward","available ":"Y","birthdate":"24-JUL- 85","last_update":150692305 2012} Apache Kafka – Scalable Stream Processing and more! Kafka to Cassandra trucking
  54. 54. Demo (V) – Create JDBC Connect through REST API #!/bin/bash curl -X "POST" "http://192.168.69.138:8083/connectors" -H "Content-Type: application/json" -d $'{ "name": "jdbc-driver-source", "config": { "connector.class": "JdbcSourceConnector", "connection.url":"jdbc:postgresql://db/sample?user=sample&password=sample", "mode": "timestamp", "timestamp.column.name":"last_update", "table.whitelist":"driver", "validate.non.null":"false", "topic.prefix":"trucking_", "key.converter":"org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enable": "false", "value.converter":"org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": "false", "name": "jdbc-driver-source", "transforms":"createKey,extractInt", "transforms.createKey.type":"org.apache.kafka.connect.transforms.ValueToKey", "transforms.createKey.fields":"id", "transforms.extractInt.type":"org.apache.kafka.connect.transforms.ExtractField$Key", "transforms.extractInt.field":"id" } }' Apache Kafka – Scalable Stream Processing and more!
  55. 55. Demo (V) – Create JDBC Connect through REST API Apache Kafka – Scalable Stream Processing and more!
  56. 56. Demo (V) - Create Table with Driver State ksql> CREATE TABLE driver_t (id BIGINT, first_name VARCHAR, last_name VARCHAR, available VARCHAR) WITH (kafka_topic='trucking_driver', value_format='JSON'); Message ---------------- Table created Apache Kafka – Scalable Stream Processing and more!
  57. 57. Demo (V) - Create Table with Driver State ksql> CREATE STREAM dangerous_driving_and_driver_s WITH (kafka_topic='dangerous_driving_and_driver_s', value_format='JSON') AS SELECT driverid, first_name, last_name, truckid, routeid,routename, eventtype FROM truck_position_s LEFT JOIN driver_t ON dangerous_driving_and_driver_s.driverid = driver_t.id; Message ---------------------------- Stream created and running ksql> select * from dangerous_driving_and_driver_s; 1511173352906 | 21 | 21 | Lila | Page | 58 | 1594289134 | Memphis to Little Rock Route 2 | Unsafe tail distance 1511173353669 | 12 | 12 | Laurence | Lindsey | 93 | 1384345811 | Joplin to Kansas City | Lane Departure 1511173435385 | 11 | 11 | Micky | Isaacson | 22 | 1198242881 | Saint Louis to Chicago Route2 | Unsafe tail distance Apache Kafka – Scalable Stream Processing and more!
  58. 58. Apache Kafka – Scalable Stream Processing and more! Technology on its own won't help you. You need to know how to use it properly.

×