Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Kafka Vienna Meetup 020719


Published on

Speaker: Patrik Kleindl (
Slides of the introduction to Apache Kafka and some popular use cases.
Slides were provided by Confluent (

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Kafka Vienna Meetup 020719

  1. 1. 1 Apache Kafka Introduction and Use Cases Apache Kafka Meetup Vienna
  2. 2. 2 About Me • Patrik Kleindl • Solution Architect @ BearingPoint • • LinkedIn:
  3. 3. 3 Looks familiar?
  4. 4. 4 Writers Kafka cluster Readers
  5. 5. 5 Enabling event sharing to connect the world's largest professional network of more than 500 million users Providing on-demand digital content for over 130 millions subscribers accessible worldwide on any device Disrupting the transportation industry by connecting providers with consumers in real-time Pushing breaking news in real time while storing all past publishings online to provide a single source of truth From those that were born cloud-native To more traditional ones that continue to adapt The most Successful Digital Businesses are Inherently Event-driven Leveraging IoT sensors to create an intelligent swarm of connected cars with real-time traffic routing Building a microservices architecture to enable a robust ground transportation booking and management offering
  6. 6. 6 Mainframes Hadoop Data Warehouse ... Device Logs ... Splunk ... App App Microservice ... Data Stores Custom Apps/MicroservicesLogs 3rd Party Apps Universal Event Pipeline Real-Time Inventory Real-Time Fraud Detection Real-Time Customer 360 Machine Learning Models Real-Time Data Transformation ... Contextual Event-Driven Apps Apache Kafka® STREAMS CONNECT CLIENTS
  7. 7. 7 Implementing an Event-driven Architecture Requires a Paradigm Shift SaaS appsApps Custom appsMicroservices Relational DB From data represented in static tables... data represented as streams of events Relational DBs Apps Microservices SaaS apps Custom apps Data warehouse Data warehouse
  8. 8. 8 A Streaming Platform is the Underpinning of an Event-driven Architecture Ubiquitous connectivity Globally scalable platform for all event producers and consumers Immediate data access Data accessible to all consumers in real time Single system of record Persistent storage to enable reprocessing of past events Continuous queries Stream processing capabilities for in-line data transformation Microservice s DBs SaaS apps Mobile Customer 360 Real-time fraud detection Data warehouse Producers Consumers Database change Microservices events SaaS data Customer experience s Streams of real time events Stream processing apps
  9. 9. 9 How do I get streams of data into and out of Kafka? Development & Connectivity
  10. 10. 10Producing to Kafka Time C CC
  11. 11. 11 Schema Registry: Make Data Backwards Compatible and Future-Proof ● Define the expected fields for each Kafka topic ● Automatically handle schema changes (e.g. new fields) ● Prevent backwards incompatible changes ● Support multi-data center environments Elastic Cassandra HDFS Example Consumers Serializer App 1 Serializer App 2 ! Kafka Topic! Schema Registry Community Feature
  12. 12. 12 Apache Kafka Connect API: Import and Export Data In & Out of Kafka JDBC Mongo MySQL Elastic Cassandra HDFS Kafka Connect API Kafka Pipeline Connector Connector Connector Connector Connector Connector Sources Sinks Fault tolerant Manage hundreds of data sources and sinks Preserves data schema Integrated within Confluent Control Center Apache Kafka Feature
  13. 13. Confluent Partner Briefing 13 Apache Kafka Kafka Connect API Reliable and scalable integration of Kafka with other systems – no coding required.
  14. 14. 14 REST Proxy Non-Java Applications Native Kafka Java Applications Schema Registry REST / HTTP Simplifies administrative actions Simplifies message creation and consumption Provides a RESTful interface to a Kafka cluster REST Proxy: Talk to Non-native Kafka Apps and Outside the Firewall Community Feature
  15. 15. 15 How do I build real- time applications? Stream Processing
  16. 16. 16 Shoulders of Streaming Giants Consumer, Producer KSQL Kafka Streams powers powers Flexibility Ease of Use CREATE STREAM, CREATE TABLE, SELECT, JOIN, GROUP BY, SUM, … KStream, KTable, filter(), map(), flatMap(), join(), aggregate(), … subscribe(), poll(), send(), flush(), beginTransaction(), …
  17. 17. 17 1 Things Kafka Streams Does Runs everywhere Clustering done for you Exactly-once processing Event-time processing Integrated database Joins, windowing, aggregation S/M/L/XL/XXL/XXXL sizes
  18. 18. 1818C K O 2 0 1 8 J U L Y Improve Customer Experience (CX) Increase Revenue (make money) Business Value Decrease Costs (save money) Core Business Platform Increase Operational Efficiency Migrate to Cloud Mitigate Risk (protect money) Fraud Detection IoT sensor ingestion Digital replatforming/ Mainframe Offload Connected Car: Navigation & improved in-car experience: Audi Customer 360 Simplifying Omni-channel Retail at Scale: Walmart Faster transactional processing / analysis incl. Machine Learning / AI Mainframe Offload: RBC Microservices Architecture Online Fraud Detection Online Security (syslog, log aggregation, Splunk replacement) Middleware replacement Regulatory Application Modernization: Multiple Examples Website / Core Operations (Central Nervous System) The [Silicon Valley] Digital Natives; LinkedIn, Netflix, Uber, Yelp... Predictive Maintenance: Audi Streaming Platform in a regulated environment (e.g. Electronic Medical Records): Celmatix Real-time app updates Real Time Streaming Platform for Communications and Beyond: Capital One Developer Velocity - Building Stateful Financial Applications with Kafka Streams: Funding Circle Detect Fraud & Prevent Fraud in Real Time: ING Kafka as a Service - A Tale of Security and Multi-Tenancy: Apple $↑ $↓ $ Example Case Studies (of many) Example Case Studies Digital Transformation 10 business use case Strategic Driver 20 business use case
  19. 19. 19 Mainframe in Private DC Confluent-Mainframe Offload Architecture z/OS Connect REST / Web services Applications Visualization tools Access Layer Mainframe Confluent REST Proxy CDC Kafka Connect JDBC Kafka Streams & KSQL Transformations Message Queue (MQ) Search Cloud Data Services Infosphere Data Replicator Confluent Kafka on Public Cloud Confluent Schema Registry Kafka Connect Mainframe + Confluent interconnect Integration & Transformation Confluent Replicator Confluent Schema Registry Public Cloud Services Confluent Platform in the Cloud
  20. 20. 20 Oracle Exadata in Private DC Confluent - Oracle Exadata to BigQuery Architecture Oracle Golden Gate Connector Applications Visualization tools Access Layer Confluent Replicator Kafka ConnectJDBC Kafka Streams & KSQL Transformations Search Cloud Data Services Confluent Kafka on Public Cloud Confluent Schema Registry Kafka Connect Oracle Exadata + Confluent interconnect Integration & Transformation Transaction log Public Cloud Services Confluent Platform in the Cloud
  21. 21. 21 Confluent Kafka REST / Web services Transport Layer Integration & Transformation Layer Destinations Data Sources Confluent REST Proxy Network Traffic Confluent Schema Registry Kafka Connect Kafka Connect JDBC/CDC Firewall Logs RDBMS Syslog Application Logs Arcsight, Splunk, etc Modern Monitoring Curated Event Stream Alien Vault ,Protectwise, etc SIEM Accumulo, Graph, etc Modeling Richer insights & archival Curated Event Stream Curated Event Stream Syslog Syslog Legacy Systems KSQL and KStreams • Aggregation • Rules engine • Reference data join • Enrichment • Filtering and Curation Confluent- SIEM Modernization/Offload Architecture Public Cloud Services
  22. 22. 22Event-Driven Analytics & Machine Learning Production ML App KStreams Confluent Kafka Confluent REST Proxy App Kafka Connect Confluent Schema Registry Model Building KStreams AppAppApps AppAppAppKafka Producers AppAppAppDBs AppAppAppLegacy Systems Training Data Model Params Model Params, Features Output
  23. 23. 23 Payment Fraud Detection System with Confluent Kafka Producer Elastic search Grafana Kafka Cluster Kafka Connect KSQL Payment App in Car Emergency Fraud System All Data Potential Fraud Apply Analytic Model Filter Predictions At the edge Otherdata Operational Reporting
  24. 24. 24 Kafka Brokers High retention x/mesg/sec Publish x/mesg/sec Kafka Streams / KSQL part of CEP/Machine Learning workflow Business Applications DB / noSQL / Hadoop / Elastic / Machine Learning Confluent MQTT/Rest Server / ProxyLoad Balancer Firewall Architecture Event-Driven IOT with Confluent Subscribe Kafka Connect