[NEW LAUNCH!] Introducing Amazon Managed Streaming for Kafka (Amazon MSK) (ANT398) - AWS re:Invent 2018
•Download as PPTX, PDF•
3 likes•3,049 views
Discover the power of running Apache Kafka on a fully managed AWS service. In this session, we describe how Amazon Managed Streaming for Kafka (Amazon MSK) runs Apache Kafka clusters for you, demo Amazon MSK and a migration, show you how to get started, and walk through other important details about the new service.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to [NEW LAUNCH!] Introducing Amazon Managed Streaming for Kafka (Amazon MSK) (ANT398) - AWS re:Invent 2018
Similar to [NEW LAUNCH!] Introducing Amazon Managed Streaming for Kafka (Amazon MSK) (ANT398) - AWS re:Invent 2018 (20)
More from Amazon Web Services
More from Amazon Web Services (20)
[NEW LAUNCH!] Introducing Amazon Managed Streaming for Kafka (Amazon MSK) (ANT398) - AWS re:Invent 2018
- 2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Damian Wylie Principal Product Manager Amazon Data Streaming A N T 3 9 8 Amazon ManagedStreaming for Kafka:A Fully Managed,HighlyAvailable, andSecureService forApacheKafka
- 3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda • Real-time data • Apache Kafka • Introducing Amazon Managed Streaming for Kafka (Amazon MSK) • How to get started • Comparing Amazon MSK with Amazon Kinesis Data Streams • Demo • Q&A
- 4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Data is produced continuously Mobile Apps Web Clickstream Application Logs Metering Records IoT Sensors Smart Buildings [Wed Oct 11 14:32:52 2018] [error] [client 127.0.0.1] client denied by server configuration: /export/home/live/ap/h tdocs/test
- 5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Data can be transformed continuously
- 6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. The diminishing value of data over time
- 7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Apache Kafka
- 9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Apache Kafka use cases Real-time web and log analytics Messaging Transaction and event sourcing Decoupled microservices Streaming ETL
- 10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Apache Kafka Anatomy 101 Producer Broker Broker Broker Data Consumer Cluster Zookeeper Producer
- 11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Apache Kafka Anatomy – Writes to partitions Newest dataOldest data 50 1 2 3 4 0 1 2 3 0 1 2 3 4 Partition 2 Partition 1 Partition 3 Writes from Producers Topic with 3 partitions
- 12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Apache Kafka Anatomy – Reads from partitions Newest dataOldest data 50 1 2 3 4 0 1 2 3 0 1 2 3 4 Partition 2 Partition 1 Partition 3 Topic with 3 partitions Consumer Consumer Consumer Consumer Group = next consumer offset
- 13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Challenges operating Apache Kafka Difficult to setup Hard to achieve high availability Tricky to scale AWS integrations = development No console, no visible metrics 𝑓 𝑘𝑎𝑓𝑘𝑎 𝑢𝑠𝑎𝑔𝑒 = 𝑛=1 ∞ 𝑆𝑅𝐸
- 14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. A fully managed, highly available, and secure service for Apache Kafka Now available in public preview in the US East (N. Virginia) Region
- 15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Getting started with Amazon MSK is easy • Fully compatible with Apache Kafka v1.1.1 • AWS Management Console and AWS API for provisioning • Clusters are setup automatically • Provision Apache Kafka brokers and storage • Create and tear down clusters on-demand
- 16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Create cluster, producer/consumer demo
- 17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Automation drives higher availability @ Preview • Cluster lifecycle is fully automated Brokers and Apache Zookeeper nodes auto-heal IPs remain intact Patches are applied automatically @ GA • Service level agreement (SLA) • Apache Kafka version upgrades
- 18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Where’s Apache Zookeeper? • Apache Zookeeper is under the hood, highly available, and included with each cluster at no additional cost
- 19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Scalability and configurability @ GA • Scale a cluster • Horizontally (add more of the same) • Vertically (add larger brokers) scale a cluster • Supports Apache Kafka partition reassignment tooling • Define custom cluster configurations • Auto scale storage
- 20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Deeply integrated with AWS services @ Preview • Amazon Virtual Private Cloud (Amazon VPC) for network isolation • AWS Key Management Service (AWS KMS) for at-rest encryption • AWS Identity and Access Management (IAM) for control-plane API control • Amazon CloudWatch for Apache Kafka broker, topic, and ZK metrics • Amazon Elastic Compute Cloud (Amazon EC2) M5 instances as brokers • Amazon EBS GP2 broker storage • Offered in the US-East (N. Virginia) AWS Region @ GA • Tagging • AWS CloudTrail • AWS CloudFormation • Offered worldwide
- 21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What Amazon MSK does for you • Makes Apache Kafka more accessible to your organization • Drives best practices through design, defaults, and automation • Allows developers to focus more on app development, less on infrastructure management • Amazon MSK is committed to improving open-source Apache Kafka 𝑓 𝑘𝑎𝑓𝑘𝑎 𝑢𝑠𝑎𝑔𝑒 = 𝑛=1 ∞ 𝑆𝑡𝑟𝑒𝑎𝑚𝑖𝑛𝑔 𝐴𝑝𝑝𝑠
- 22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. How it works
- 23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon MSK defaults Config Default Setting offsets.topic.replication.factor 3 transaction.state.log.replication.factor 3 transaction.state.log.min.isr 2 auto.create.topics.enable False default.replication.factor 3 min.insync.replicas 2 unclean.leader.election.enable True auto.leader.rebalance.enable True authorizer.class.name kafka.security.auth.SimpleAclAuthorizer group.initial.rebalance.delay.ms 3000 log.retention.hours 168
- 24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. How pricing works • On-demand, hourly pricing prorated to the second • Broker and storage pricing • Broker pricing starts with kafka.m5.large @ $0.21/hr • Storage pricing is $0.10 per GB-month
- 25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Comparing Amazon Kinesis Data Streams to MSK Amazon Kinesis Data Streams Amazon MSK Newest dataOldest data 50 1 2 3 4 0 1 2 3 0 1 2 3 4 Shard 2 Shard 1 Shard 3 Writes from Producers Stream with 3 shards Newest dataOldest data 50 1 2 3 4 0 1 2 3 0 1 2 3 4 Partition 2 Partition 1 Partition 3 Writes from Producers Topic with 3 partitions
- 26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Comparing Amazon Kinesis Data Streams to MSK • AWS API experience • Throughput provisioning model • Seamless scaling • Typically lower costs • Deep AWS integrations • Open-source compatibility • Strong third-party tooling • Cluster provisioning model • Apache Kafka scaling isn’t seamless to clients • Raw performance Amazon Kinesis Data Streams Amazon MSK
- 27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Why add Amazon MSK?
- 28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Cluster resilience demo
- 29. Thank you! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Damian Wylie wylied@amazon.com Follow @DamianWylie on Twitter for live updates
- 30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Editor's Notes
- Narrative: The reality is that most data is produced continuously and is coming at us at lightning speeds due to an explosive growth of real-time data sources. TP: Machine data will make up 40% of our digital universe by 2020 Narrative: Whether it is log data coming from mobile and web applications, purchase data from ecommerce sites, or sensor data from IoT devices, it all delivers information that can help companies learn about what their customers, organization, and business are doing right now. TP: Customer Benefits Improve operational efficiencies, improve customer experiences, new business models Smart building: reduce energy costs, cut maintenance, increase safety and security Smart textiles: monitor skin temperature, monitor stress
- Narrative: So how much is this data worth? Well, it depends… Recent data is highly valuable If you act on it in time Perishable Insights (M. Gualtieri, Forrester) Old + Recent data is more valuable If you have the means to combine them Narrative: Processing real-time data as it arrives can let you make decisions much faster and get the most value from your data. But, building your own custom applications to process streaming data is complicated and resource intensive. You need to train or hire developers with the right skillsets, and then wait for months for the applications to be built and fine-tuned, and the operate and scale the application as the business grows. All of this takes lots of time and money, and, at the end of the day, lots of companies just never get there, settle for the status-quo, and live with information that is hours or days old.
- Zero Admin: Capture and deliver streaming data into S3, Redshift, ElasticCache and other AWS destinations without writing an application or managing infrastructure Direct-to-data store integration: Batch, compress, and encrypt streaming data for delivery into S3, and other destinations in as little as 60 secs, set up in minutes Seamless elasticity: Seamlessly scales to match data throughput (feedback: add bullet to discuss why firehose created. Major use case)