Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Jun Rao, Confluent | Kafka Summit SF 2019 Keynote ft. Chris Kasten, Walmart Labs

606 views

Published on

Jun Rao, Confluent Co-Founder discusses the power of Kafka, why it was created at LinkedIn, and what it's used for at Kafka Summit SF 2019's keynote. Featuring Chris Kasten, VP Walmart Cloud.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Jun Rao, Confluent | Kafka Summit SF 2019 Keynote ft. Chris Kasten, Walmart Labs

  1. 1. APACHE KAFKA PAST, PRESENT & FUTURE
  2. 2. Second most active and visited Apache Project 100,000+ organizations globally Trillions of events per day 60% of Fortune 100 companies 1000 brokers per cluster
  3. 3. It’s all about data!
  4. 4. INSIGHTS USER PRODUCT DATA SCIENCE SIGNALS VALUE VIRALITY
  5. 5. Initial Database Driven Architecture DATABASE WEB APP WEB APP
  6. 6. REALIZATION 1 STATEEVENT >
  7. 7. STATEEVENT > I changed my job from LinkedIn to Confluent. I work at Confluent.
  8. 8. JOB CHANGE RECOMMENDATION ENGINE SEARCH INDEX EMAIL SERVICE
  9. 9. REALIZATION 2 LEVERAGE ALL DIGITIZED EVENTS
  10. 10. NON-TRANSACTIONAL EVENTSTRANSACTIONAL EVENTS 0.1% 99.9%
  11. 11. DATABASE IS A MISMATCH FOR REALIZATION 1 & 2
  12. 12. SQL SQL SQL RECOMMENDATION ENGINE SEARCH INDEX EMAIL SERVICE Mismatch 1: No First Class Treatment for Events DATABASE
  13. 13. Mismatch 2: Not Suitable for Volume of All Digitized Events DATABASE 1000x more volume NON-TRANSACTIONAL EVENTS TRANSACTIONAL EVENTS
  14. 14. USER TRACKING APPLICATION LOGS OPERATIONAL METRICS KEY/VALUE STORE KEY/VALUE STORE DATABASE MICROSERVICES ...HADOOP LOG SEARCH MONITORING DATA WAREHOUSE REC. ENGINE SEARCH SECURITY EMAIL SOCIAL GRAPH
  15. 15. LOG USER TRACKING APPLICATION LOGS OPERATIONAL METRICS KEY/VALUE STORE KEY/VALUE STORE DATABASE MICROSERVICES ...HADOOP LOG SEARCH MONITORING DATA WAREHOUSE REC. ENGINE SEARCH SECURITY EMAIL SOCIAL GRAPH
  16. 16. USER TRACKING APPLICATION LOGS OPERATIONAL METRICS KEY/VALUE STORE KEY/VALUE STORE DATABASE MICROSERVICES ...HADOOP LOG SEARCH MONITORING DATA WAREHOUSE REC. ENGINE SEARCH SECURITY EMAIL SOCIAL GRAPH LOG Why not messaging systems?
  17. 17. MESSAGING USER TRACKING APPLICATION LOGS OPERATIONAL METRICS KEY/VALUE STORE KEY/VALUE STORE DATABASE MICROSERVICES ...HADOOP LOG SEARCH MONITORING DATA WAREHOUSE REC. ENGINE SEARCH SECURITY EMAIL SOCIAL GRAPH
  18. 18. MESSAGING USER TRACKING APPLICATION LOGS OPERATIONAL METRICS KEY/VALUE STORE KEY/VALUE STORE DATABASE MICROSERVICES ...HADOOP LOG SEARCH MONITORING DATA WAREHOUSE REC. ENGINE SEARCH SECURITY EMAIL SOCIAL GRAPH Not designed for volume. Weak persistence.
  19. 19. HIGH THROUGHPUT PUB/SUB PHASE 1 OF KAFKA:
  20. 20. Solving Mismatch 1: Persistent Event Log As First Class Citizen Solving Mismatch 1: Persistent Event Log As First Class Citizen DATABASE 0 1 2 3 4 5 6 7 8LOG READS WRITES DESTINATION SYSTEM A (TIME = 2) DESTINATION SYSTEM B (TIME = 6) DATA SOURCE
  21. 21. Solving Mismatch 2: Distributed Architecture BROKER 1 BROKER 2 BROKER 3 BROKER 4 TOPIC 1-PART 1 TOPIC 2-PART 3 TOPIC 3-PART 1 TOPIC 2-PART 1 TOPIC 5-PART 1 TOPIC 6-PART 1 TOPIC 1-PART 2 TOPIC 2-PART 4 TOPIC 3-PART 2 TOPIC 2-PART 2 TOPIC 5-PART 2 TOPIC 6-PART 2 CONSUMER CONSUMERCONSUMER PRODUCER PRODUCERPRODUCER Solving Mismatch 2: Distributed Architecture
  22. 22. DATA DEMOCRACY! BENEFIT OF KAFKA AFTER PHASE 1:
  23. 23. KAFKA JOINED APACHE THANK YOU TO ALL CONTRIBUTORS!
  24. 24. BROKER 1 BROKER 2 BROKER 3 BROKER 4 TOPIC 1-PART 1 Intra-Cluster Replication TOPIC 2-PART 2 TOPIC 2-PART 1 TOPIC 1-PART 2 TOPIC 1-PART 1 TOPIC 2-PART 2 TOPIC 2-PART 1 TOPIC 1-PART 2 TOPIC 1-PART 1 TOPIC 2-PART 2 TOPIC 2-PART 1 TOPIC 1-PART 2
  25. 25. Security ZOOKEEPER 1 ZOOKEEPER 2 ZOOKEEPER 3 ZOOKEEPER ENSEMBLE EXTERNAL CLIENT 2EXTERNAL CLIENT 1 EXTERNAL CLIENT N INTERNAL CLIENT 1 INTERNAL CLIENT 2 INTERNAL CLIENT N BROKER 1 BROKER 2 BROKER N KAFKA CLUSTER
  26. 26. Connect CONNECTORCONNECTOR
  27. 27. Kafka Streams & KSQL & Exactly-Once Semantics EVENT-DRIVEN MICROSERVICE EVENT-DRIVEN MICROSERVICE EVENT-DRIVEN MICROSERVICE CREATE STREAM fraudulent_payments AS SELECT * FROM payments WHERE fraudProbability > 0.8; val fraudulentPayments: KStream[String, Payment] = builder .stream[String, Payment](“payments-kafka-topic”) .filter((_ ,payment) => payment.fraudProbability > 0.8) fraudulentPayments.to(“fraudulent-payments-topic”)
  28. 28. Complete Event Streaming Platform NON-TRANSACTIONAL EVENTS CONNECTOR CONNECTOR CONNECTOR EVENT-DRIVEN MICROSERVICE EVENT-DRIVEN MICROSERVICE KSQL KSTREAMS CONNECTOR CONNECTOR
  29. 29. Royal Bank of Canada — Event Driven Banking 30+ Use Cases 50+ Apps 10+ Different lines of businesses Lowering anomaly detection from weeks to real-time EVENT STREAMING PLATFORM ... DATA WAREHOUSE DIGITAL MARKETING FRAUD SECURITY CONSUMER CREDIT SERVICES CORPORATE REAL ESTATE INVESTOR SERVICES TREASURY SERVICES ...
  30. 30. Carnival Cruise Line
  31. 31. Still Important Work Ahead ZOOKEEPER TOPIC 1-PART 1 BROKER 1 BROKER 2 BROKER 3 BROKER 4 TOPIC 2-PART 2 TOPIC 2-PART 1 TOPIC 1-PART 2 TOPIC 1-PART 1 TOPIC 2-PART 2 TOPIC 2-PART 1 TOPIC 1-PART 2 TOPIC 1-PART 1 TOPIC 2-PART 2 TOPIC 2-PART 1 TOPIC 1-PART 2 METADATA METADATA METADATA
  32. 32. Still Important Work Ahead TOPIC 1-PART 1 BROKER 1 BROKER 2 BROKER 3 BROKER 4 TOPIC 2-PART 2 TOPIC 2-PART 1 TOPIC 1-PART 2 TOPIC 1-PART 1 TOPIC 2-PART 2 TOPIC 2-PART 1 TOPIC 1-PART 2 TOPIC 1-PART 1 TOPIC 2-PART 2 TOPIC 2-PART 1 TOPIC 1-PART 2 METADATA METADATAMETADATA
  33. 33. Success for your business not only depends on software, but how you build software.

×