Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Apache Kafka as Event-Driven Open Source Streaming Platform (Prague Meetup)


Published on

From Prague Kafka Meetup in November 2018.

This session introduces Apache Kafka as event-driven open source streaming platform. Apache Kafka goes far beyond scalable, high volume messaging. In addition, you can leverage Kafka Connect for integration and the Kafka Streams API for building lightweight stream processing microservices in autonomous teams. The open source Confluent Platform adds further components such as a KSQL, Schema Registry, REST Proxy, Clients for different programming languages and Connectors for different technologies and databases.

Published in: Technology

Apache Kafka as Event-Driven Open Source Streaming Platform (Prague Meetup)

  1. 1. 1Confidential Introduction to Apache Kafka as Event-Driven Open Source Streaming Platform Kai Waehner Technology Evangelist LinkedIn @KaiWaehner Apache Kafka, Kafka Streams, KSQL, Schema Registry, and more…
  2. 2. Events What is an event?
  3. 3. Events
  4. 4. Events A Sale An Invoice A Trade A Customer Experience
  5. 5. Events and State User_data = ‘old_value’ User_data = ‘new_value’
  6. 6. Table-Stream Duality
  7. 7. All Your Data is Streams of Events
  8. 8. Events have two functions 1. Triggers 2. Data
  9. 9. Where are they? Events haven’t had a proper home in infrastructure or in code. They are implicit. Here!
  10. 10. Events are everywhere A business is a series of events and the reactions to those events.
  11. 11. Event-driven Retail
  12. 12. What do we have today? Data Stores
  13. 13. You need both these things
  14. 14. Events and Business Services Business Service Business Service Sales Events New Orders to place Processing and Inventory
  15. 15. Events and Data Pipelines Databases Databases Customer Data Updates Unified 360 Merged Customer Profiles
  16. 16. These are equivalent Both are Event Stream Processing
  17. 17. The Streaming Platform
  18. 18. Haven’t we seen all this before?
  19. 19. What’s different this time around? Published in 2009 Published in 2004
  20. 20. What’s different this time around?
  21. 21. Microservices Mobile Machine Learning Internet of Things The World has Changed
  22. 22. Confluents Business Value per Use Case Improve Customer Experience (CX) Increase Revenue (make money) Business Value Decrease Costs (save money) Core Business Platform Increase Operational Efficiency Migrate to Cloud Mitigate Risk (protect money) Key Drivers Strategic Objectives (sample) Fraud Detection IoT sensor ingestion Digital replatforming/ Mainframe Offload Connected Car: Navigation & improved in-car experience: Audi Customer 360 Simplifying Omni-channel Retail at Scale: Target Faster transactional processing / analysis incl. Machine Learning / AI Mainframe Offload: RBC Microservices Architecture Online Fraud Detection Online Security (syslog, log aggregation, Splunk replacement) Middleware replacement Regulatory Digital Transformation Application Modernization: Multiple Examples Website / Core Operations (Central Nervous System) The [Silicon Valley] Digital Natives; LinkedIn, Netflix, Uber, Yelp... Predictive Maintenance: Audi Streaming Platform in a regulated environment (e.g. Electronic Medical Records): Celmatix Real-time app updates Real Time Streaming Platform for Communications and Beyond: Capital One Developer Velocity - Building Stateful Financial Applications with Kafka Streams: Funding Circle Detect Fraud & Prevent Fraud in Real Time: PayPal Kafka as a Service - A Tale of Security and Multi-Tenancy: Apple Example Use Cases $↑ $↓ $ Example Case Studies (of many)
  23. 23. How do we get there? Why Apache Kafka and Confluent?
  24. 24. à It is NOT a Queue! J
  25. 25. • Robustness and scalability • Slow consumers • Reprocessing
  26. 26. • Strict Ordering • Persistence • Exactly-Once Processing
  27. 27. Kafka Streams Execution Model
  28. 28. Apache Kafka + Kafka Connect + Kafka Streams + many additional components
  29. 29. 6969 Kafka Clients Apache Kafka Native Clients Confluent Native Clients Community Supported Clients Proxy http/REST stdin/stdout
  30. 30. 7070 Confluent REST Proxy REST Proxy Non-Java Applications Native Kafka Java Applications Schema Registry REST / HTTP(S) Talking to Non-native Kafka Apps and Outside the Firewall
  31. 31. 7171 Confluent Schema Registry Elastic HDFS Example Consumers Serializer App 1 Serializer App 2 ! Kafka Topic! Schema Registry Deserializer Deserializer
  32. 32. KSQL – A Streaming SQL Engine for Apache Kafka
  33. 33. KSQL – A Streaming SQL Engine for Apache Kafka CREATE STREAM vip_actions AS SELECT userid, page, action FROM clickstream c LEFT JOIN users u ON c.userid = u.user_id WHERE u.level = 'Platinum'; CREATE TABLE possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTES) GROUP BY card_number HAVING count(*) > 3;
  34. 34. Apache Kafka: The Rise of a Streaming Platform The Log ConnectorsConnectors Producer Consumer Streaming Engine Add-ons Schema Registry REST Proxy KSQL Etc.
  35. 35. Event-Streaming Maturity Model / Adoption Journey Pre-Streaming Streaming Awareness and Pilot Early Production Streaming Mission Critical, Integrated Streaming Global Streaming Central Nervous System
  36. 36. Free material for Managers, Project lead, Architects, Developers
  37. 37. 85 Kai Waehner Technology Evangelist @KaiWaehner LinkedIn Questions? Feedback? Please contact me! Download Apache Kafka & Confluent Platform