Apache Kafka is a popular stream-processing platform, but it’s no secret that it can be tough to set up, manage, and scale. Amazon Managed Streaming for Kafka (Amazon MSK) can help remove some of that toil for you. In this session, you learn about new Amazon MSK features and capabilities. You also get a glimpse under the hood, giving you a better understanding of how Amazon MSK operationalizes Apache Kafka so you don't have to. We compare and contrast Amazon Kinesis Data Streams and Apache Kafka (with/without MSK) and show how to lift-and-shift your workload into Amazon MSK with minimal downtime.

1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Ditching the overhead: Moving
Apache Kafka workloads into Amazon
MSK
Damian Wylie
Principal product manager
AWS
A B D 3 0 1

2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Agenda
A brief intro to Apache Kafka
The challenges of running Apache Kafka in production
How Amazon MSK addresses these challenges so that you don’t have to
Announcements
Replicating or migrating your workloads using MirrorMaker

3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Related breakouts
ADB206: A deep dive into Amazon MSK
Damian Wylie and Vijay Kistampalli
Chalk Talk, W184a @ 5:00 p.m. on Thursday, May 30

4. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Apache Kafka use cases
Real-time web and log analytics
Messaging
Transaction and event sourcing
Decoupled microservices
Streaming ETL

6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Apache Kafka anatomy 101
Producer
Broker
Broker
Broker
Data consumer
Cluster
Apache
ZooKeeper
Producer

7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Apache Kafka anatomy: Writes to partitions
Newest dataOldest data
50 1 2 3 4
0 1 2 3
0 1 2 3 4
Partition 2
Partition 1
Partition 3
Writes from
producers
Topic with 3 partitions

8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Apache Kafka anatomy: Reads from partitions
Newest dataOldest data
50 1 2 3 4
0 1 2 3
0 1 2 3 4
Partition 2
Partition 1
Partition 3
Topic with 3 partitions
Consumer
Consumer
Consumer
Consumer group
= Next consumer offset

9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Challenges operating Apache Kafka
Difficult to set up
Hard to achieve high
availability
Difficult to scale
AWS integrations = development
No console, no
visible metrics
𝑓 𝑘𝑎𝑓𝑘𝑎 𝑢𝑠𝑎𝑔𝑒 = ෍
𝑛=1
∞
𝑆𝑅𝐸

10. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
What Amazon MSK does for you
• Makes Apache Kafka more accessible to your organization
• Drives best practices through design, defaults, and automation
• Allows developers to focus more on application development and less on
infrastructure management
• Amazon MSK is committed to improving open-source Apache Kafka
𝑓 𝑘𝑎𝑓𝑘𝑎 𝑢𝑠𝑎𝑔𝑒 = ෍
𝑛=1
∞
𝑆𝑡𝑟𝑒𝑎𝑚𝑖𝑛𝑔 𝐴𝑝𝑝𝑠

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Getting started with Amazon MSK is easy
• Fully compatible with Apache Kafka v1.1.1 and v2.1.0
• AWS Management Console and AWS API for provisioning
• Clusters are set up automatically in minutes
• Provision Apache Kafka brokers and storage
• Create and tear down clusters on demand

13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Where’s Apache Zookeeper?
Apache Zookeeper is under the hood
It is highly available, fully managed,
automatically provisioned, and included
with each cluster at no additional cost

14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
How connectivity
works

15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
How pricing works
• On-demand, hourly pricing is prorated to the second
• Broker and storage pricing
• Broker pricing starts with kafka.m5.large at $0.21 per hour
• Storage pricing is $0.10 per GB-month
• Data transfer from replication within the cluster and ZooKeeper nodes are
included at no additional cost

16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Launching now

17. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
New!
New!
New!

18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon MSK customer references
“Reduced maintenance overhead”
“Made it easy to set up, maintain, and scale Kafka clusters”
“Accelerates time to market”
“Ensures data durability, cluster availability, and scalability“
“Significantly increase[s] the efficiency of our teams and
reduce[s] time spent maintaining our clusters”

19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
New security features
Encryption in transit via TLS
inCluster and clientBroker

20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
New security features
Mutual TLS authentication
Certificate-based authentication using AWS Certificate
Manager Private Certificate Authority (AWS PCA)
1. Create PCA with a root certificate within AWS ACM
2. Create Amazon MSK cluster with authentication enabled, selecting PCAs
3. Consumers and producers are configured with a certificate issued by the root CA and trust store
4. Apache Kafka ACLs can now be configured using the certificate dname as the principal user
AWS Certificate Manager

21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
HIPAA eligible
AWS CloudTrail for API auditing
AWS
CloudTrail
New compliance features

22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
New ease of use features
Custom configurations
For new clusters; support for updating existing clusters
coming soon
Cluster-wide storage scaling
Cluster tagging and tag-based IAM polices

23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
New ease of use features
Custom configurations (CLI only)
Console support coming in the next few weeks
auto.create.topics.enable
delete.topic.enable
group.initial.rebalance.delay.ms
group.max.session.timeout.ms
group.min.session.timeout.ms
log.cleaner.delete.retention.ms
log.cleaner.min.cleanable.ratio
log.flush.interval.messages
log.flush.interval.ms
log.retention.bytes
log.retention.hours
log.retention.minutes
log.retention.ms
log.roll.ms
log.segment.bytes
max.incremental.fetch.session.cache.slots
message.max.bytes
min.insync.replicas
num.partitions
offsets.retention.minutes
transaction.max.timeout.ms
unclean.leader.election.enable
zookeeper.connection.timeout.ms

24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
How performance meets cost

25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
How to ditch the overhead

26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
MirrorMaker v1: How it works

27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
MirrorMaker command
bin/kafka-mirror-maker.sh
--consumer.config consumer.properties
--producer.config producer.properties
--num.streams
--num.producers
--whitelist <regex topics>
[--blacklist <regex topics> ]

28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
MirrorMaker v1 best practices
Run the tool in the destination—in this case, in the VPC with your MSK cluster; if encryption is required in
transit, run it in the source
For no data loss and order
For consumer, set auto.commit.enabled=false
For producer
max.in.flight.requests.per.connection=1
retries=Int.Max_Value
acks=all
max.block.ms = Long.Max_Value
For MirrorMaker
set – abortOnSendFail
For high throughput for producer
max.in.flight.requests.per.connection = 1+ (warning: no ordering)
Enable compression (compression.type = gzip)
Buffer messages and fill message batches – tune buffer.memory, batch.size, linger.ms
Tune socket buffers – receive.buffer.bytes, send.buffer.bytes

29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
MirrorMaker v1 best practices
For high throughput of consumer
Increase the number of threads/consumers per MirrorMaker process - num.streams
Increase the number of MirrorMaker processes across machines first before increasing threads to allow for
high availability
Increase the number of MirrorMaker processes first on the same machine and then on different machines
(with same groupid)
Isolate topics that have very high throughput and use separate MirrorMaker instances
For management and configuration
Use AWS CloudFormation and configuration management tools like Chef and Ansible
Use Amazon EFS file system mounts to keep all configuration files accessible from all Amazon EC2 Instances
Use containers for easy scaling and management of MirrorMaker instances

30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
MirrorMaker v1 limitations
Does not replicate topic
configurations
It replicates topics if
auto.create.topics.enable = true in the
destination cluster, but topics are created
with the default configuration in the
destination cluster
Can cause configuration divergence
With auto.create.topics.enable = false,
topics have to be manually created in the
destination
Topic configuration changes have to be
manually replicated
Message offsets might not match
between source and destination
clusters
To avoid duplicates, shut down producers to
the source cluster, confirming that consumers
have consumed all messages, replicating all
messages to the destination cluster, and
starting consumers on the destination cluster
with auto.offset.reset = latest
Failover and disaster recovery scenarios are not
supported
Whitelists and blacklists only support Java-style
regular expressions and cannot be dynamically
updated

31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
MirrorMaker v1 Limitations
• Minimal operational and management support
• Only supports at-least-once guarantees; does not support an idempotent
producer or transactions
• Minimal metrics support
• Any configuration change means the cluster must be bounced
• Rebalancing causes latency spikes, which may trigger further rebalances

32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
MirrorMaker v2
Addresses the limitations of v1
• Leverages the Kafka Connect framework and ecosystem
• Detects new topics, partitions
• Automatically syncs topic configuration between clusters
• Supports active/active cluster pairs, as well as any number of active clusters
• Provides new metrics, including end-to-end replication latency across multiple data centers or clusters
• Emits offsets required to migrate consumers between clusters and tooling for offset translation
• Supports a high-level configuration file for specifying multiple clusters and replication flows in one
place, compared to low-level producer/consumer properties for each MM1 process
• https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0

33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Migration and replication guide
1. Create Amazon MSK destination cluster
2. Start MirrorMaker from an Amazon EC2 instance within the same Amazon VPC
as the destination cluster
3. Inspect MirrorMaker lag
4. If you are migrating, once MirrorMaker has caught up, redirect producers and
consumers to new cluster using the Amazon MSK cluster bootstrap broker
value
5. Shut down MirrorMaker

34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Demo

35. Thank you!
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Damian Wylie
wylied@amazon.com
LinkedIn: wyliedamian
Twitter: @DamianWylie