6 Nines: How Stripe keeps Kafka highly-available across the globe w...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Donny Nadolny
6 Nines:
How Stripe keeps Kafka highly-available
across the globe

Kafka at Stripe
● Used by vast majority of services at Stripe
● Stripe:
○ Payments & more for internet businesses
○ Our re...

Typical Cluster Topology
● Brokers (and ZooKeeeper) spread across 3 AWS availability zones
● Topic replication factor: 3 w...

Typical Cluster Topology
4

Typical Cluster Topology
5

Typical Cluster Topology
6

7
Kafka is Highly Available

Kafka is highly available, but…
● Region failure
● 1 AZ failure + 1 machine failure
● 1 machine failure during rolling res...

High(er) availability Kafka - requirements
● Producers always need to be able to produce
● Consumers can fall behind with ...

Higher availability Kafka - classic setup
10

Higher availability Kafka
11

Higher availability Kafka
12

Higher availability Kafka
13

Higher availability Kafka - only for producers!
14

HA-write tradeoﬀs
● Pros:
○ Easy to set up
○ Much better produce availability
● Cons:
○ Worse produce latency due to mirro...

Changing requirements - consumers
● Over time, consumers became more important
16

17
Multicluster Kafka

Multicluster Kafka
18

Multicluster Kafka
19

Multicluster Kafka
20

Multicluster Kafka
21

Kproxy
● Observability
● ACLs
○ “Dark read” conﬁg
● Single place for health checking
22

Kproxy publish availability
● Extremely high publish availability
● Available if even 1 cluster is healthy
23

Kproxy consume availability
● If a cluster goes down, move publishes to another cluster
● Messages unavailable until clust...

Kproxy consume availability - unconsumed messages
● (latest oﬀset - committed oﬀset) is guaranteed upper bound
● Consumer ...

High availability consume
● Publish to all clusters instead of any cluster
● Deduplicate on the consumer
● Similarly ultra...

High availability consume
● Publish to all clusters instead of any cluster
● Deduplicate on the consumer
● Similarly ultra...

More beneﬁts
● Zero-impact: turn oﬀ cluster for maintenance
○ Route produces to other clusters, wait for consumers to catc...

Single point of failure: kproxy
Yes, but:
● Stateless
○ Health checking of clusters is in-memory
● Zero coordination betwe...

Kproxy host sets
● Isolate consumers from producers
● Isolate low-latency / high availability producers from others
● Isol...

Kproxy host sets
31

Kproxy host sets - rollout
32
1. Single host of a low priority
host set
2. A few hosts in that host set
3. Many hosts in t...

Kproxy host sets - rollout
33

Data locality
● Producers and consumers can request that they publish to / consume from
a set of regions
● Common producer...

Failure detection
● Simple or complicated?
○ By topic-partition-cluster or by cluster?
○ Use topology (broker overlap, ava...

Failure detection - our approach
● Use actual publishes as health check
● Initially by topic-cluster, simple circuit break...

37
Multicluster Kafka Limitations

Limitations
● Our implementation:
○ Custom client & API
○ Small amount of unconsumed messages if cluster goes down
○ Keyed...

Custom client & API
● Big tradeoﬀ: other tools (Apache Flink, Kafka Connect, Spark
Streaming, etc) don’t work oﬀ the shelf...

Custom client & API
● Kafka client protocol is very large & changes with versions
● How to express partitions across multi...

Gotchas
● Higher eﬀective number of partitions, but can’t use it for more
consumer parallelism unless you do extra work in...

42
Results

Kafka is highly available, but…
● Region failure
● 1 AZ failure + 1 machine failure
● 1 machine failure during rolling res...

Mitigated by multicluster / Kproxy
● Region failure
● 1 AZ failure + 1 machine failure
● 1 machine failure during rolling ...

Mostly Mitigated by multicluster / Kproxy
● Misbehaving client ﬂooding requests
○ Mostly mitigated - can isolate clients w...

Publish availability
● SLO for other teams at Stripe:
● 99.999% of publishes succeed within 5 seconds for high-priority
pu...

Publish availability
● SLO for other teams at Stripe:
● 99.999% of publishes succeed within 5 seconds for high-priority
pu...

Publish latency
● SLO: p50/p90/p99 of 10ms/15ms/35ms
● Actual: 3.5ms/5ms/9.5ms
48

49
We’re hiring! stripe.com/jobs

50
Thanks!

51
Questions?

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1033581 views
Storytelling For The Web: Integrate... by Chiara Aliotta 945137 views
Artificial Intelligence, Data and C... by OECD Directorate ... 865593 views
How to Leverage AI to Boost Employe... by SocialHRCamp 363144 views
2024 State of Marketing Report – by... by Marius Sescu 206375 views
Everything You Need To Know About C... by Expeed Software 226977 views

View on Slideshare

1 of 51 Ad

View on Slideshare

1 of 51 Ad