Download as PDF, PPTX
Uploaded byHostedbyConfluent
Making Kafka Cloud Native | Jay Kreps, Co-Founder & CEO, Confluent
The document discusses the transition of companies towards becoming software-driven entities, highlighting the importance of cloud-native architectures and data in motion to enable real-time operations and customer experiences. It emphasizes the role of Kafka and similar technologies in managing data streams and integrating disparate systems while addressing the challenges faced in maintaining traditional databases. The document also explores future trends in cloud-native data services, the need for seamless event streaming solutions, and the benefits of effective data governance in this context.
More Related Content
Similar to Making Kafka Cloud Native | Jay Kreps, Co-Founder & CEO, Confluent
Making Kafka Cloud Native | Jay Kreps, Co-Founder & CEO, Confluent
- 1.
- 2. Companies are becomingsoftware. 2
- 3. Old World ...disconnected usecases around the edges of a company Software is... New World ...a platform for directly transacting business 3
- 4.
- 5.
- 6. What technologies enablethis? 6
- 7. Internet of Things Mobile Microservices Machine Learning 7 Data inMotion Data in Motion Cloud
- 8.
- 9.
- 10. Databases are fundamentallyincomplete. Databases are designed for siloed, UI-centric applications. 10 Data at rest Slow, daily batch processing Simple, static real-time queries Databases
- 11. A software definedcompany requires total connectivity and instant reaction in real-time. 11
- 12. PUBLIC CLOUD With onlydata at rest, a company is fragmented and siloed LINE OF BUSINESS 02 LINE OF BUSINESS 01 12
- 13.
- 14. The Paradigm forData in Motion: Event Streams Data Management = Storage + Flow 14 Rich front-end customer experiences Real-time Data Real-time Stream Processing Real-time backend operations QUERY A Sale A shipment A Trade A Customer Experience
- 15. PUBLIC CLOUD Enterprise Architectureis a Big Mess LINE OF BUSINESS 02 LINE OF BUSINESS 01 15
- 16.
- 17. ksqlDB: Data in Motion+ Data at Rest 17
- 18. User Payments Jay 42 Sue18 Fred 65 ... ... User Jay 695 Sue 430 User Jay 695 Sue 430 Tables Streams User Credit Score Jay 695 Sue 430 Fred 710 V1 V3 V2
- 19. SELECT * FROM DB_TABLE CREATETABLE T AS SELECT * FROM EVENT_STREAM Active Query: Passive Data: DB Table Active Data: Passive Query: Event Stream Traditional Database Stream Processing
- 20. Build a completestreaming app with a few SQL statements 20 Capture events Perform continuous transformations Create materialized views Serve lookups against materialized views 1 2 3 4
- 21.
- 22.
- 23. The Rise ofCloud & Cloud Native Data Services 23
- 24. Over the nextfour years $140B+ in IT spend will move to the cloud. 24
- 25. Cloud Native DataSystems Aurora DynamoDB Kinesis S3 Spanner Snowflake 25
- 26. What is aCloud-Native Data System? Elastic Usage-based Cost Model Infinite Api-driven Operations Secure and Reliable Serverless Global Multitenant 26
- 27.
- 28. Managing Kafka atScale is Hard Tweaking servers Tuning GC Fiddling with ZK Upgrades Patches Rebalances Mirroring 28
- 29. What my manager thinksI do What the Front-end team thinks I do What I think I do What I actually do
- 30. “I know! We’lluse Kubernetes!” 30
- 31. Cloud Native DataSystems Are The Future 31
- 32. Moving up thestack 32
- 33. If running Kafkais so hard, why is Kafka winning? 33 Apache Kafka Amazon Kinesis Microsoft Azure Event Hubs Google Cloud Pub/Sub VS
- 34.
- 35. Community 28,030 Stack Overflow Questions 210 Meetups 58,902 Meetup Attendees 11,315 Jirasfor Apache Kafka 51,335 Emails sent to the Apache Kafka Mailing list 667 KIPs 35
- 36.
- 37. A point ofview
- 38. Open Platforms Win LinuxEmail TCP-IP x86 38
- 39. For Kafka toThrive There Must Be Cloud Native Kafka Services 39
- 40. The Capabilities ofa Cloud-Native Data System Elastic Usage-based Cost Model Infinite Api-driven Operations Secure and Reliable Serverless Global Multitenant 40
- 41.
- 42. Making a CloudNative Kafka Service 42
- 43.
- 44.
- 45.
- 46.
- 47.
- 48.
- 49.
- 50.
- 51. Data Governance meetsData Discovery Self-service Platform Security Data Catalog Data Lineage Data Policies Data Quality 51
- 52. Confluent Cloud DataGovernance Data Quality Increase data trust ● Enterprise ready Schema Registry ● Schemas management UI ● Broker-side schema ID validation Data Catalog Classify, organize, discover ● Search and discover schemas metadata ● Manage data classifications ● Classify schemas with tags Data Lineage Turn data visibility on ● Visualize complex data in motion pipelines ● Audit data movement across systems NOW IN EARLY-ACCESS 52
- 53.
- 54.
- 55.
- 56. ● Create aglobal fabric for event streams that spans the globe ● Span cloud environments and on premises ● Enables dynamic, API-driven replication topologies ● Exact offset mirroring between clusters ● No additional moving parts to manage or monitor 56 Global: Cluster linking Keep Data in Sync Globally to serve regional needs Currently in Preview
- 57. What about onpremises? 57
- 58. Confluent for Kubernetes Introducinga Declarative, API-driven Control Plane to deploy and manage Confluent in Private Infrastructures. AVAILABLE NOW! Runs on Kubernetes: the infrastructure runtime for cloud-native architectures. Declarative API for operating Confluent in production Integrates with Cloud-Native ecosystem for Security, Reliability, DevOps Automation Manage topics and RBAC policies through Infrastructure as Code 58
- 59. Not Just AboutOne Cloud Service, Kafka Itself Improves 59
- 60. KIP-500 Early Access NoZooKeeper 60
- 61. Event-Driven Raft Consensus(AKA Kraft) L L With Zookeeper With Quorum Controller L denotes quorum leader 61
- 62. Simplicity is Sophistication Simpler operations Tighter security 2Million+ partitions per cluster Single process execution 62 Simpler deployment
- 63. Support for upto 2m partitions 0 200 400 600 Controlled Shutdown Time Recovery from Uncontrolled Shutdown Zookeeper Quorum Controller 63
- 64. Event Streaming Platformin Every Company 64 Kafka
- 65. This is theessential architecture for companies that are becoming software. 65
- 66.