Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Rise Of Event Streaming – Why Apache Kafka Changes Everything

891 views

Published on

Business digitalization trends like microservices, the Internet of Things or Machine Learning are driving the need to process events at a whole new scale, speed and efficiency. Traditional solutions like ETL/data integration or messaging are not build to serve these needs.
Today, the open source project Apache Kafka® is being used by thousands of companies including over 60% of the Fortune 100 to power and innovate their businesses by focusing their data strategies around event-driven architectures leveraging event streaming.We will discuss the market and technology changes that have given rise to Kafka and to Event Streaming, and we will introduce the audience to the key aspects of building an Event streaming platform with Kafka. Examples of productive use cases from the automotive, manufacturing and transportation sector will showcase the power of event streaming.

Published in: Software
  • Be the first to comment

The Rise Of Event Streaming – Why Apache Kafka Changes Everything

  1. 1. 1 The Rise Of Event Streaming – Why Apache Kafka Changes Everything Kai Waehner | Technology Evangelist, Confluent contact@kai-waehner.de | LinkedIn | @KaiWaehner | www.confluent.io | www.kai-waehner.de
  2. 2. 22 The world is changing.
  3. 3. 33 Transformation is happening everywhere...
  4. 4. 4 Transportation & Automotive
  5. 5. 5 Banking
  6. 6. 6 Largest media co - creates no content Largest movie house - owns no cinemas Online bookstore - diversifying across industries Largest taxi company - owns no vehicles Most valuable retailer - has no inventory World’s largest accommodation provider - owns no real estate
  7. 7. 77 The New Enterprise Reality Innovate or be disrupted ● Create new business models ● Deliver new, real-time customer experiences ● Deliver massive internal efficiencies Every company is a software company ● Capital One 10,000 of 40,000 employees are software engineers ● Goldman Sachs 1.5B (billion!) lines of code across 7,000+ applications ● JP Morgan: 50+% of IT budget for new projects and applications, <50% for maintenance Innovation is about events, not “data” ● Digital-first organizations must pick the right foundational technologies ● Established enterprises must implement new technologies without “rip-and-replace”
  8. 8. 8 Every Company is Becoming a Software Company
  9. 9. 9 Every Company is Becoming a Software Company
  10. 10. 1010 What enables this transformation?
  11. 11. 11 The World has Changed Business Digitalization Trends are Driving the Need to Process Events at a whole new Scale, Speed and Efficiency
  12. 12. 1212 What is the challenge?
  13. 13. 1313 A company is built on DATA FLOWS but all we have is DATA STORES
  14. 14. 14 A company built on data stores looks like this
  15. 15. 15 ETL/Data Integration Messaging Highly Scalable Durable Persistent Ordered Real-time Difficult to Scale No Persistence After Consumption No Replay Batch Expensive Time Consuming
  16. 16. 1616 How is it done with an Event Streaming Platform
  17. 17. 1717 Highly Scalable Persistent ETL/Data Integration MessagingETL/Data Integration MessagingMessaging Batch Expensive Time Consuming Difficult to Scale No Persistence After Consumption No Replay Real-timeHighly Scalable Durable Persistent Ordered Real-time Event Streaming
  18. 18. 18 Events A Sale An Invoice A Trade A Customer Experience
  19. 19. 19 An Event Streaming Platform is the Underpinning of an Event-driven Architecture Microservices DBs SaaS apps Mobile Customer 360 Real-time fraud detection Data warehouse Producers Consumers Database change Microservices events SaaS data Customer experience s Streams of real time events Streaming AppsStreaming Apps Streaming Apps
  20. 20. 20
  21. 21. 21 The beginning of a new Era https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying The first use case: Log Analytics. This is why Kafka was created!
  22. 22. 22 ● Global Scale ● Real-time ● Persistent Storage ● Stream Processing Apache Kafka: The De-facto Standard for Real-Time Event Streaming Edge Cloud Data LakeDatabases Datacenter IoT SaaS AppsMobile Microservices Machine Learning Apache Kafka
  23. 23. 23 Apache Kafka at Scale at Tech Giants > 7 trillion messages / day > 6 Petabytes / day “...you name it” * Kafka Is not just used by tech giants ** Kafka is not just used for big data
  24. 24. 24 Producing to Kafka P Time
  25. 25. 25 Consuming from Kafka P Time C2 Near Real Time C3 Real Time C1 Batch
  26. 26. 26 Distributed System with Replication and High Availability Broker 1 Topic1 partition1 Broker 2 Broker 3 Broker 4 Topic1 partition1 Topic1 partition1 Leader Follower Topic1 partition2 Topic1 partition2 Topic1 partition2 Topic1 partition3 Topic1 partition4 Topic1 partition3 Topic1 partition3 Topic1 partition4 Topic1 partition4
  27. 27. 27 Distributed System with Replication and High Availability read, write App (Kafka Streams) Kafka (data) More Apps (KSQL, Connect, Python, REST, “You-name-it”) BookingsTeam FraudTeam … MobileTeam …
  28. 28. 28 Apache Kafka (kafka.apache.org) includes Kafka Connect and Kafka Streams Kafka Streams Your app sinksource KafkaConnect KafkaConnect
  29. 29. 2929 KSQL Continuous queries on infinite streams
  30. 30. 3030 STREAM PROCESSING Create and store materialized views Filter Analyze in-flight Time C CC
  31. 31. 31 TRADITIONAL DATABASE EVENT STREAM PROCESSING SELECT * FROM DB_TABLE CREATE TABLE T AS SELECT * FROM EVENT_STREAM Active Query: Passive Data: DB Table Active Data: Passive Query: Event Stream
  32. 32. 32 TABLES STREAMS USER JAY SUE FRED CREDIT_SCORE 695 430 710V1 V3 V2 PAYMENTS 42 18 65 ... USER JAY SUE FRED ...
  33. 33. 33 PUSH PULL APP Jay’s credit score is 670 Jay’s credit score is 710 Jay’s credit score is 695 What is Jay’s credit score now? 695 APP
  34. 34. 37 The stream is materialized into a table, which is emitted as a stream Payments Stream Credit Score Stream CREATE TABLE credit_scores AS SELECT user, updateScore(p.amount)…
  35. 35. 3838 What is the result
  36. 36. 4040 Event-Driven App (Location Tracking) Only Real-Time Events Messaging Queues can do this Contextual Event-Driven App (ETA) Real-Time combined with stored data Only Event Streaming Platforms can do this Where is my driver? When will my driver get here? Where is my driver? When will my driver get here? Why add the word Contextual?
  37. 37. 41 Front, rear and top view cameras Parking assistant Environment pointer Ultrasonic Sensors Parking assistant with front and rear camera plus environment indicator Crash Sensors Front protection adaptivity Side protection Tail impact protection Infrared Camera Rearview assistance with Pedestrian recognition Front and Rear Radar Sensors ACC with stop and go function Side assist Front Camera Audi Active lane assistant Speed limit indicator Adaptive light The Future of the Automotive Industry is a Real Time Data Cluster
  38. 38. 42 The Future of the Automotive Industry is a Real Time Data Cluster Front, rear and top view cameras Ultrasonic SensorsCrash Sensors Front Camera Infrared Camera Front and Rear Radar Sensors Traffic Alerts Hazard Alerts Personalization Anomaly Detection MQTT MQTT MQTT MQTT MQTTMQTT
  39. 39. 43
  40. 40. 44
  41. 41. 454545
  42. 42. 464646
  43. 43. 474747
  44. 44. 484848
  45. 45. 494949
  46. 46. 52 Real Time Machine Learning with Kafka and TensorFlow
  47. 47. 53 Improve Customer Experience (CX) Increase Revenue (make money) Business Value Decrease Costs (save money) Core Business Platform Increase Operational Efficiency Migrate to Cloud Mitigate Risk (protect money) Key Drivers Strategic Objectives (sample) Fraud Detection IoT sensor ingestion Digital replatforming/ Mainframe Offload Connected Car: Navigation & improved in-car experience: Audi Customer 360 Simplifying Omni-channel Retail at Scale: Target Faster transactional processing / analysis incl. Machine Learning / AI Mainframe Offload: RBC Microservices Architecture Online Fraud Detection Online Security (syslog, log aggregation, Splunk replacement) Middleware replacement Regulatory Digital Transformation Application Modernization: Multiple Examples Website / Core Operations (Central Nervous System) The [Silicon Valley] Digital Natives; LinkedIn, Netflix, Uber, Yelp... Predictive Maintenance: Audi Streaming Platform in a regulated environment (e.g. Electronic Medical Records): Celmatix Real-time app updates Real Time Streaming Platform for Communications and Beyond: Capital One Developer Velocity - Building Stateful Financial Applications with Kafka Streams: Funding Circle Detect Fraud & Prevent Fraud in Real Time: PayPal Kafka as a Service - A Tale of Security and Multi-Tenancy: Apple Example Use Cases $↑ $↓ $ Example Case Studies (of many)
  48. 48. 5454 How does Confluent help
  49. 49. $$€€££ - Making Money - Saving Money - Protecting Money Every Company is Becoming a Software Company
  50. 50. 5656 I N V E S T M E N T & T I M E VALUE 3 4 5 1 2 Event Streaming Maturity Model 56 Initial Awareness / Pilot Start to Build Pipeline / Deliver 1 New Outcome Leverage Stream Processing Build Contextual Event-Driven Apps Central Nervous System Product, Support, Training, Partners, Technical Account Management...
  51. 51. 5757 CONFLUENT PLATFORM EFFICIENT OPERATIONS AT SCALE PRODUCTION-STAGE PREREQUISITES UNRESTRICTED DEVELOPER PRODUCTIVITY Multi-language Development Rich Pre-built Ecosystem SQL-based Stream Processing GUI-driven Mgmt & Monitoring Flexible DevOps Automation Dynamic Performance & Elasticity Enterprise-grade Security Data Compatibility Global Availability APACHE KAFKA Fully Managed Cloud ServiceSelf Managed Software FREEDOM OF CHOICE COMMITTER-LED EXPERTISE PartnersTraining Professional Services Enterprise Support DEVELOPER OPERATOR ARCHITECT Hybrid Infrastructure
  52. 52. 58 Questions? Let’s connect... Kai Waehner Technology Evangelist kai.waehner@confluent.io @KaiWaehner www.confluent.io www.kai-waehner.de LinkedIn

×