Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Jay Kreps, Confluent | Kafka Summit SF 2019 Keynote ft. Dev Tagare, Lyft + Priya Shivakumar, Confluent Cloud

412 views

Published on

Jay Kreps, Confluent Co-Founder and Co-Creator of Apache Kafka, delivers the keynote presentation at Kafka Summit San Francisco 2019. He explains modern stream processing, real-time databases, KSQL, and Confluent Cloud's newest offering – a fully managed, serverless Kafka. In an effort to bring event streaming to even more developers, Priya Shivakumar announces Kafka made serverless in Confluent Cloud, with $50 free for the first three months.

Recording includes a Q&A session between Jay Kreps and Devendra Tagare, Engineering Manager at Lyft. They discuss the enhanced features Confluent Cloud offers on top of typical Kafka use cases – from mission critical reliability, scaling billions of messages at under 50ms latency, to multicloud data streaming and 24/7 Kafka support."

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Jay Kreps, Confluent | Kafka Summit SF 2019 Keynote ft. Dev Tagare, Lyft + Priya Shivakumar, Confluent Cloud

  1. 1. DATABASES ARE ONLY HALF DONE
  2. 2. Marc Andreessen: Software is Eating the World
  3. 3. Strong Form Companies are BECOMING SOFTWARE Weak Form Companies are USING MORE SOFTWARE
  4. 4. Loan Application Using Software BORROWER 1 CREDIT OFFICER 3 LOAN OFFICER 5 RISK OFFICER 4 APPROVE DENY 6 APPLICATION FORM 2
  5. 5. Loan Application Using Software BORROWER 1 CREDIT OFFICER 3 LOAN OFFICER 5 RISK OFFICER 4 APPROVE DENY 6 APPLICATION FORM 2 HUMAN CENTRIC 1-2 WEEKS
  6. 6. Loan Application in Software BORROWER 1 APPROVE DENY 3 LOAN APP UI CREDIT SERVICE RISK SERVICE CRM SERVICE $ ! 2
  7. 7. Loan Application in Software BORROWER 1 APPROVE DENY 3 LOAN APP UI CREDIT SERVICE RISK SERVICE CRM SERVICE $ ! 2 SOFTWARE CENTRIC SECONDS
  8. 8. Using Software: Classic Three Tier Architecture USER UI SERVICE DATABASE
  9. 9. Becoming Software: Services Talking To Each Other With APIs SERVICESERVICESERVICE SERVICE
  10. 10. IS MORE SOFTWARE THE USER OF THE SOFTWARE
  11. 11. What does this mean for databases?
  12. 12. We have hundreds of databases...
  13. 13. We have hundreds of databases... FUNDAMENTAL ASSUMPTION: DATA IS PASSIVE
  14. 14. Fundamental Assumption: Data is Passive QUERY UPDATE
  15. 15. Good for Building: CRUD Applications Synchronous Services
  16. 16. Unless there is a user and UI waiting, why should it be synchronous?
  17. 17. The Solution: Event Streams
  18. 18. EVENT STREAMING PLATFORM APP APP APP APP APP APP APP APP DB DB DB APP APP APP APP DB DB DB APP APP
  19. 19. Kafka is a Foundation for Event Streams 0 1 2 3 4 5 6 7 8LOG READS WRITES DESTINATION SYSTEM A DESTINATION SYSTEM B
  20. 20. Solution: Stream Processing INPUT STREAMS OUTPUT STREAMS STREAM PROCESSOR
  21. 21. KSQL Continuous queries on infinite streams
  22. 22. TRADITIONAL DATABASE EVENT STREAM PROCESSING SELECT * FROM DB_TABLE CREATE TABLE T AS SELECT * FROM EVENT_STREAM Active Query: Passive Data: DB Table Active Data: Passive Query: Event Stream
  23. 23. TABLES STREAMS USER JAY SUE FRED CREDIT_SCORE 695 430 710V1 V3 V2 PAYMENTS 42 18 65 ... USER JAY SUE FRED ...
  24. 24. STREAMS CREATE STREAM payments (user VARCHAR, amount INT) WITH (kafka_topic = ’all_payments’, key = ’user’, value_format= ’avro’);
  25. 25. TABLES CREATE TABLE credit_scores AS SELECT user, updateScore(p.amount) AS credit_score FROM payments AS p GROUP BY user;
  26. 26. But wait, some things are still synchronous...
  27. 27. PUSH PULL APP Jay’s credit score is 670 Jay’s credit score is 710 Jay’s credit score is 695 What is Jay’s credit score now? 695 APP
  28. 28. PUSH PULL SELECT user, credit_score FROM credit_history WHERE ROWKEY = ‘jay’;
  29. 29. Today: Integrating all these systems is up to you CONNECTOR CONNECTOR APP DB APP DB STREAM PROCESSING CONNECTOR APPDB
  30. 30. STREAM PROCESSING Today: Integrating all these systems is up to you CONNECTOR CONNECTOR APP DB APP DB CONNECTOR APPDB EXTRACT STORE LOAD STORE TRANSFORM
  31. 31. EASY ⇔ MAINSTREAM
  32. 32. How can we make this simpler? KLIP 7: cnfl.io/ksql-klip-7 KLIP 8: cnfl.io/ksql-klip-8 github.com/confluentinc/ksql
  33. 33. Solution: Fewer moving parts 33 APP DB APP DB APP KSQL
  34. 34. PUSH PULL SELECT user, credit_score FROM credit_history WHERE ROWKEY = ‘jay’ EMIT CHANGES; SELECT user, credit_score FROM credit_history WHERE ROWKEY = ‘jay’;
  35. 35. DEMO
  36. 36. This is a natural generalization of databases.
  37. 37. Architecture of a Distributed Database LOG DB DB DB SERVING NODES
  38. 38. Architecture of a Distributed Stream Processing System LOG DB DB DB STREAM PROCESSORS
  39. 39. KSQL: Streaming SQL Engine => Event Streaming Database LOG STREAM PROCESSORS DB DB DB DB DB DB SERVING NODES KSQL
  40. 40. Data Warehousing JURY RIGGED ETL FACTS DIMS REPORTING
  41. 41. A data warehouse is NOT the central nervous system of a digital business.
  42. 42. EVENT STREAMING PLATFORM APP APP APP APP APP APP APP APP KSQL DB DB APP APP APP APP KSQL DB DB APP APP
  43. 43. We still need all of these!
  44. 44. In fact this makes it easier to use them... CREATE SINK CONNECTOR elasticConnector WITH ( ‘connector.class’ = '...ElasticsearchSinkConnector', ‘topics’ = 'CREDIT_SCORES', ‘connection.url’ = 'http://localhost:9200', ‘type.name’ = 'kafka-connect', ... ); APP KSQL CONNECTOR APP
  45. 45. TARGET FIRST RELEASE: NOVEMBER KLIP 7: cnfl.io/ksql-klip-7 KLIP 8: cnfl.io/ksql-klip-8 github.com/confluentinc/ksql

×