Achieving a 50% Reduction in Cross-AZ Network Costs from Kafka (Uda...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Achieving a 50% Reduction in
Cross-AZ Network Costs
from Kafka
Uday Sagar Shiramshetty
SignalFx

About Me:
● 4+ years of experience in
Monitoring and Distributed
Systems
● Site Reliability Engineer
● Distributed Systems...

About SignalFx:
● Real-Time Cloud Monitoring
Platform for Infrastructure,
Microservices and Applications
● 20 Kafka cluste...

4
SignalFx is the Only Real-Time Observability Platform
Smart Gateway™
Metricization
Metadata Extraction
TraceStore
COLLEC...

5
Goal and Agenda
Goal: Reduce data transfer across availability zones to lower network costs.
Agenda:
• Motivation: AWS n...

6
Motivation: AWS Network Pricing Model
Cross AZ Data Price ($)
1 GB 0.02
1 TB 20
1 PB 20,000

7
• Kafka brokers are grouped in Racks
• Racks provide fault tolerance and each rack may be in a different physical
locati...

A Brief Overview

9Confidential & Proprietary ©SignalFx 2019

10Confidential & Proprietary ©SignalFx 2019

11Confidential & Proprietary ©SignalFx 2019

12Confidential & Proprietary ©SignalFx 2019

13Confidential & Proprietary ©SignalFx 2019

14Confidential & Proprietary ©SignalFx 2019

15
Simple Kafka
Setup
Kafka Setup
+ AZ aware
routing
Kafka Setup
+Compression*
Kafka Setup
+ AZ aware
routing
+Compression...

Deep Dive on
Kafka AZ
Awareness

17
We need our producers and consumers to talk to leader brokers in the same AZ.

18
• Kafka Producer needs to be aware of leader partitions in an AZ
• Kafka client library can help us find the locations ...

19
On producer initialization:
String myRack = System.getProperty("KAFKA_PRODUCER_RACK", "defaultRack");
TopicDescription ...

20
There are few things to consider before finalizing on desired partitions to produce
messages.
1. What if partitions are...

21
Kafka Cluster Group Coordinator

22
Kafka Consumer Rebalance

23
PartitionAssignor
Kafka Java client library has an interface called PartitionAssignor, that allows us to
define our own...

24
PartitionAssignor
Partitions can be assigned to consumers in the group based on different implementations of
PartitionA...

25
RangeAssignor
An introductory example:
Two consumers C0 and C1 and two topics t0 and t1 with 3 partitions each.
Topic t...

26
RackAwareAssignor
Two consumers C0 (R1) and C1 (R2), two topics t0 and t1 with 3 partitions each.
Topic t0: t0p0 (R1), ...

27
RackAwareAssignor
RackAwareAssignor.class is supplied as partition.assignment.strategy config
value during consumer ini...

28
RackAwareAssignor
Step 1:
maxNumOfPartitions = Math
.ceil(topicPartitionsCount * 1.0 / totalConsumersCount);
Step 2:
As...

29
RackAwareAssignor
private static final String RACK_KEY_NAME = "consumer_rack";
private static final String CONSUMER_RAC...

Kafka AZ
Awareness on
Other Complicated
Use Cases

31
Aggregations / Sequential Processing
Related messages need to be enqueued on same partitions.

32
Fixed Consumer Assignment
Related messages need to be processed on single consumer instance, without group
membership c...

33
Tokens - Partitions Assignment
Partition assignment should be based on large ranges of tokens assigned to racks,
not in...

34
Tokens - Partitions Assignment
Problem: When broker 0 goes down, broker 1 gets all the load
1
1
1
1

35
Tokens - Partitions Assignment
Better Assignment would like:

36
Tokens - Partitions Assignment
When broker 0 goes down, broker 1 and 2 share the load

37
Kafka as Backup Store
Consumer groups in all racks need to consume entire data on all partitions.

38
Fission Reaction
When one update triggers multiple updates, AZ awareness will cause scalability
troubles.

Kafka Message
Compression

40
Kafka Message Compression
• Kafka supports end-to-end compression
• Data is compressed by the Kafka producer client
• D...

41
Kafka Message Compression
• Enabling compression is as simple as setting the config compression.type on Kafka
producer ...

42
Benefits from AZ Awareness
45% decrease in Data Transfer - Regional Bytes

43
Benefits from AZ Awareness
Decrease in average end-to-end
latency over last day

44
Benefits from Message Compression
Decrease in average bytes/message over
last day

Thank You!
Any Questions?

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1010457 views
Storytelling For The Web: Integrate... by Chiara Aliotta 918112 views
Artificial Intelligence, Data and C... by OECD Directorate ... 818086 views
How to Leverage AI to Boost Employe... by SocialHRCamp 341372 views
2024 State of Marketing Report – by... by Marius Sescu 194920 views
Everything You Need To Know About C... by Expeed Software 218230 views

View on Slideshare

1 of 45 Ad

View on Slideshare

1 of 45 Ad