Download as PDF, PPTX
![Confluent Cloud
Milliseconds Minutes
Basic, Standard [0-100Mbps]
Do Nothing
Elastic Scaling w/ fully-managed Confluent Cloud
*Even in public clouds provider quotas for VMs, disks, security groups can be encountered causing delays. Confluent has these limits raised already.
Dedicated [Mbps - Gbps]
1 Click—Select CKU from drop
down in cluster management UI and
click Apply Changes
Other Kafka Services
Days - Weeks
Determine how much capacity is needed
Procure capacity*
Configure new brokers
a. Disks b. OS c. Network d. Kafka (application)
Identify partitions on specific brokers to
rebalance & topics they are part of
For each Topic: migrate partitions
a. Increase ISR +1 b. Wait for new replica to sync
c. Failover master d. Reduce ISR -1 e. Delete old replica](https://image.slidesharecdn.com/s061addisonhuddydanrosanovahowconfluentcompletestheeventstreamingplatform-200910020228/75/How-Confluent-Completes-the-Event-Streaming-Platform-Addison-Huddy-Dan-Rosanova-Confluent-Kafka-Summit-2020-6-2048.jpg)
Uploaded byHostedbyConfluent
How Confluent Completes the Event Streaming Platform (Addison Huddy & Dan Rosanova, Confluent) Kafka Summit 2020
Confluent Platform 6.0 and Project Metamorphosis complete the event streaming platform by providing elastic scalability, infinite storage, global access, and transforming Kafka. Key features include self-balancing clusters and dynamic scaling on Confluent Cloud, tiered storage and infinite retention on the platform, and cluster linking to simplify hybrid and multi-cloud deployments. These new capabilities help remove limitations on scale, storage, and deployment that traditionally challenged Kafka applications.
More Related Content
Similar to How Confluent Completes the Event Streaming Platform (Addison Huddy & Dan Rosanova, Confluent) Kafka Summit 2020
How Confluent Completes the Event Streaming Platform (Addison Huddy & Dan Rosanova, Confluent) Kafka Summit 2020
- 1. How Confluent Completesthe Event Streaming Platform With Confluent Platform 6.0 and Project Metamorphosis @DanRosanova @AddisonHuddy
- 2. As our roadmapmay change in the future, the features referred to herein may change, may not be delivered on time or may not be delivered at all. This information is not a commitment to deliver any functionality and customers should make their purchasing decisions based upon features that are currently available.
- 3. Hadoop ... Device Logs... App ...MicroserviceMainframes Data Warehouse Splunk ... Data Stores Logs 3rd Party Apps Custom Apps / Microservices Real-time Inventory Real-time Fraud Detection Real-time Customer 360 Machine Learning Models Real-time Data Transformation ... Contextual Event-Driven Applications Universal Event Pipeline CONFLUENT
- 4. Elastic Confluent Cloud ● Self-serveclusters ● GBps+ scale Confluent Platform 6.0 ● Self-Balancing Clusters Cost-effective Confluent Cloud ● 99.5% Basic SLA ● New connectors Confluent Platform 6.0 ● Admin REST API Infinite Confluent Cloud ● Infinite retention ● Schema registry Confluent Platform 6.0 ● Tiered Storage ● ksqlDB Transforming Kafka with Confluent Platform 6.0 and Project Metamorphosis Global Confluent Cloud ● Cluster linking ● New cloud regions Confluent Platform 6.0 ● Cluster Linking (preview)
- 5.
- 6. Confluent Cloud Milliseconds Minutes Basic,Standard [0-100Mbps] Do Nothing Elastic Scaling w/ fully-managed Confluent Cloud *Even in public clouds provider quotas for VMs, disks, security groups can be encountered causing delays. Confluent has these limits raised already. Dedicated [Mbps - Gbps] 1 Click—Select CKU from drop down in cluster management UI and click Apply Changes Other Kafka Services Days - Weeks Determine how much capacity is needed Procure capacity* Configure new brokers a. Disks b. OS c. Network d. Kafka (application) Identify partitions on specific brokers to rebalance & topics they are part of For each Topic: migrate partitions a. Increase ISR +1 b. Wait for new replica to sync c. Failover master d. Reduce ISR -1 e. Delete old replica
- 7. Dynamic Scaling withConfluent Platform T3, P0 T1, P0 T2, P1 T3, P0 T1, P1 T2, P0 T3, P1 Broker 1 Broker 2 New: Broker 3 T1, P0 T2, P1T1, P1 T2, P0T3, P1 Broker 1 Broker 2 Broker 3 Scaling with OSS Kafka T3, P0T1, P0 T2, P1 Broker 1 Broker 2 New: Broker 3 Object Store Broker 1 Broker 2 Broker 3 Object Store T3, P1T1, P1 T2, P0 T3, P0T1, P0 T2, P1T3, P1 T1, P1 T2, P0 Scaling with Tiered Storage + Self- balancing Clusters on Kubernetes Rebalance in Hours... Rebalance in Seconds...
- 8.
- 9. 9 Historically, the way toscale storage in Kafka is to scale brokers. Broker storage can be expensive and is tightly bound to compute: Brokers
- 10. Confluent Cloud Infinite Retention Scaleup storage without adding brokers by removing storage limits on topics to enable infinite data retention in Kafka Cluster Broker 1 Broker 2 Broker 2 Topic 1 Topic 2 Topic 3 Topic 2 Topic 3 Topic 2 Topic 3 Topic 1Topic 1
- 11. Confluent Platform Tiered Storage Offloadold events to cheaper, 3rd party object storage to reduce costs and enable infinite data retention in Kafka 11 Tiered Storage enables Kafka to recognize two layers of storage: Brokers Object Storage Offload old data to object store
- 12.
- 13. 13 Cluster Linking (preview) Cluster Linking simplifieshybrid-cloud and multi-cloud deployments for Kafka Connecting independent clusters or migrating clusters presents major challenges: 1. Requires deploying a separate Connect cluster, adding architectural complexity and operational burden 1. Offsets are NOT preserved, so consumers are at risk of skipping or rereading messages 1 2
- 14. 14 Cluster Linking (preview) Cluster Linking simplifieshybrid-cloud and multi-cloud deployments for Kafka Cluster Linking requires no additional infrastructure and preserves offsets:
- 16. Confluent Developer developer.confluent.io Learn Kafka. Startbuilding with Apache Kafka at Confluent Developer.
- 17.