Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

... No it's Apache Kafka!

802 views

Published on

Is it a Bird? Is it a Plane? ... No it's Apache Kafka!
An introduction to Apache Kafka.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

... No it's Apache Kafka!

  1. 1. DARWIN IT-PROFESSIONALS IT Driven Evolution ... No it’s Apache Kafka Martien van den Akker 2018 Anintroduction ToApacheKafka Is it a bird? Is it a plane?
  2. 2. martien.van.den.akker@darwin-it.nl @Makker_nl Who I am Copyright © 2018 Darwin IT-Professionals B.V. 2
  3. 3. What is Kafka? Does it handle events? Yes, but what about architecture? Does it handle streaming data? Yes, but what Is Streaming Exactly? So, it supports Event Distribution? Yes, but How about process Visibility? Is it awesome? Yes, but what can it mean to you? 1 2 3 4 Agenda 5 Copyright © 2018 Darwin IT-Professionals B.V. 3
  4. 4. DarwinIT-Professionals WHAT IS KAFKA? Copyright © 2018 Darwin IT-Professionals B.V. 4
  5. 5. Is ita log? • The immutable commit log is the heart of Kafka – Publish and subscribe data to and from any number distributed systems – Fault tolerant – Stored on file system, leveraging the speed of sequential read and append to files. No direct reads. Therefor also fast on older/conventional disks. – Some application systems do use Kafka as a log system (pretty smart actually) Copyright © 2018 Darwin IT-Professionals B.V. https://www.confluent.io/what-is-apache-kafka/ 5
  6. 6. Is ita log? Copyright © 2018 Darwin IT-Professionals B.V. https://www.confluent.io/what-is-apache-kafka/ 6
  7. 7. Is ita queue? (1/3) • No, it is actually a (set of clustered) topics (or multi-consumer queues in AQ terms) • Conventional Queue/Topic implementations ensure once and only once delivery • But Kafka: – Consumer’s position is kept as an integer off-set to the next message (per partition) – A consumer can rewind to older offsets to replay data – A new consumer can read from beginning or just start at the end Copyright © 2018 Darwin IT-Professionals B.V. 7
  8. 8. Is ita queue? (2/3) • Each partition – Ordered imutable sequence of records – Records assigned an integer: offset – Every record durably persisted – Configurable retention period – Partitions are distributed for fault tolerance Copyright © 2018 Darwin IT-Professionals B.V. 8
  9. 9. Is ita queue? (3/3) • Only metadata is offset per consumer • Offset controlled per consumer • Kafka consumers are very cheap Copyright © 2018 Darwin IT-Professionals B.V. 9
  10. 10. Is ita stream? • Think of a stream as a sequence of messages in time, for instance: – Call Detail Records – Transactions on bank accounts • Kafka supports stream processors: – Consume continual streams of data from input topics – Perform processing – Produces continual streams to output topics • KSQL (Kafka SQL Streaming engine) – Perform ‘continuous SQL’ on a set of Streams and/or tables to create new streams or tables Copyright © 2018 Darwin IT-Professionals B.V. 10
  11. 11. Is ita database? • Complex question, but – You can query and join streams • I think it’s not really, but – Kafka stores and does not forget event data (considering retention) – Distributes the events/registrations in order – Consumer can independently process in order Copyright © 2018 Darwin IT-Professionals B.V. https://speakerdeck.com/ept/is-kafka-a-database 11
  12. 12. Is itan integrationplatform? • It is a data/event distribution platform • Connects – Producers with Consumers – applications through Connectors – Stream Processors • But low-level – No content based routing – No transformations – No orchestration • But loads of connectors Copyright © 2018 Darwin IT-Professionals B.V. 12
  13. 13. Oracle Event Hub CS • Fully Managed • Streaming Data platform • Leverages Kafka • Deployments – Topics: fully managed – Dedicated: customer managed Copyright © 2018 Darwin IT-Professionals B.V. 13 Integration Cloud - Automate Processes
  14. 14. Kafka is • Apache Kafka is an open source distributed streaming platform capable of handling trillions of events a day. Initially conceived as a messaging queue, Kafka is based on an abstraction of a distributed commit log. (https://www.confluent.io/what-is-apache-kafka/) Copyright © 2018 Darwin IT-Professionals B.V. 14
  15. 15. DarwinIT-Professionals YES, BUT WHAT ABOUT ARCHITECTURE? Does it handle events? Copyright © 2018 Darwin IT-Professionals B.V. 15
  16. 16. What is an Event • An event is a data object identified by key, on a timestamp • An event is raised by a producer • An event is consumed by a consumer • A producer does not care about consuming • A consumer does not care about the origin of the event Copyright © 2018 Darwin IT-Professionals B.V. 16
  17. 17. EDA: EventDriven Architecture • Message based • EDA: Architecture where – Some functions/Services notify their state changes – Other functions/Services react on notifications of state • In Service Oriented Architecture implemented by a Service Bus • However: OSB controls the reaction and submission of messages based on an invocation Copyright © 2018 Darwin IT-Professionals B.V. 17
  18. 18. EDA importantfor Microservices • Microservices de-coupled by a back-bone Copyright © 2018 Darwin IT-Professionals B.V. Kafka Order Payment Picking Shipment Customer Customer Created Order Created Payment Received Goods Fetched Order Sent 18
  19. 19. DarwinIT-Professionals YES, BUT WHAT IS STREAMING EXACTLY? Does it handle streaming data? Copyright © 2018 Darwin IT-Professionals B.V. 19
  20. 20. Kafka as streamingplatform • An event is a data object identified by key, on a timestamp • Multiple events with same meaning in sequence form streams • Streams can be joined and/or filtered to form new streams with new meaning Copyright © 2018 Darwin IT-Professionals B.V. • Stream filters can result in new events 20
  21. 21. What Can IDo With KSQL? • KSQL Allows you to combine streams and tables • KSQL tables represent a current state of a set of structured records from a stream (based on key) • Streaming ETL – for instance icw. GoldenGate • Real-time Monitoring and Analytics • Data exploration and discovery • Anomaly detection – That can lead to new events • Etc. Copyright © 2018 Darwin IT-Professionals B.V. 21
  22. 22. Announced atCodeOne 2018 22Copyright © 2018 Darwin IT-Professionals B.V. Oracle Database 12.2+ Query Real-Time Kafka Streams with Oracle SQL Melli Anamalai – Sr. Principal Product Manager Version 2 of the connector https://technology.amis.nl/2018/10/25/querying-and-publishing-kafka-events-from-oracle-database-sql-and-pl-sql
  23. 23. DarwinIT-Professionals YES, BUT HOW ABOUT PROCESS VISIBILITY? So, it supports Event Distribution? Copyright © 2018 Darwin IT-Professionals B.V. 23
  24. 24. EDA importantfor Microservices • Lack of visibility of process flow Copyright © 2018 Darwin IT-Professionals B.V. Kafka Order Payment Picking Shipment Customer Customer Created Order Created Payment Received Goods Fetched Order Sent 24 And what about reordering Process Steps?
  25. 25. Visibilityof events? • Types of process support in a services architecture: – Choreography • How services ‘dance’ with each other, move around eachother – Orchestration • A composer and director are responsible for process flow Copyright © 2018 Darwin IT-Professionals B.V. 25
  26. 26. Toolsto consider Copyright © 2018 Darwin IT-Professionals B.V. https://www.slideshare.net/BerndRuecker/ kafka-summit-2018-monitoring-and-orchestration-of-your-microservices-landscape-with-zeebe Integration Cloud - Automate Processes • Orchestrate by Events • Zeebe is a lightweight Kakfa supporting Process engine • Oracle Integration Cloud icw. Event Hub obvious choice for SaaS, etc. 26
  27. 27. DarwinIT-Professionals YES, BUT WHAT CAN IT MEAN TO YOU? Is it awesome? Copyright © 2018 Darwin IT-Professionals B.V. 27
  28. 28. What Kafkacan mean for you • Kafka can join EDA with RealTime BI • Kafka can bridge On-Premise with Cloud • Kafka can log from different sources and let you query them (try that with app server/application logs…) • Kafka can glue a heterogenous environment with COTS, SOA, Microservices, PaaS/SaaS, Realtime-BI • Find anomalies in events and react on them Copyright © 2018 Darwin IT-Professionals B.V. 28
  29. 29. DarwinIT-Professionals THANKYOU FORYOUR ATTENDANCE, PATIENCE AND ATTENTION Copyright © 2018 Darwin IT-Professionals B.V. 29
  30. 30. Q & A Copyright © 2018 Darwin IT-Professionals B.V. 30

×