The Why Behind the How: Kafka Reliability from First Principles
•
0 likes•87 views
"Apache Kafka is known to be a highly-available system which stores and serves data reliably. Being a distributed system there is no one-config-fits-all and different use-cases require different trade-offs. Many talks cover ""how” to configure Kafka for resilience, but lack to explain the ""why"" behind the configs. Understanding configurations and their interactions from first principles breaks these chains, and unlocks Kafka’s full potential. In this talk we will touch on the how, but mainly peek behind the curtains to explain the why. We will revisit the relevant parameters for brokers, topics, producers, and consumers that allows you to configure Kafka with different reliability and availability properties. We will dig into the glory details of these parameters and their interactions with each other, using building blocks that apply to distributed systems. This knowledge will enable you to understand and reason about the numerous available trade-offs, allowing you to configure Kafka for your use-case. Say goodbye to searching for “tutorial X” that matches your use-case; you might not find one anyway."
Recommended
More Related Content
Similar to The Why Behind the How: Kafka Reliability from First Principles
Similar to The Why Behind the How: Kafka Reliability from First Principles (20)
More from HostedbyConfluent
More from HostedbyConfluent (20)
Recently uploaded
Recently uploaded (20)
The Why Behind the How: Kafka Reliability from First Principles
- 1. The Why Behind the How: Kafka Reliability from First Principles Matthias J. Sax Software Engineer | Apache Kafka Committer and PMC member Twitter/𝕏 @MatthiasJSax
- 2. 2 The How Broker: minimum of 3 nodes Topic: replication.factor = 3 min.insync.replicas = 2 Producer: retries = 2147483647 acks = all
- 3. Data Durability (aka Data Safety) • Successfully written data should never get lost, even in the case of failure • Successfully written data must be available for reader to consume Availability • The system should never be offline - Write Availability: The system must accept writes, and ensure that writes are durable - Read Availability: The system must be able to serve reads 3 Reliability Pillars
- 4. Mean-time to Error • What is the time, until an error is expected? Mean-time to Recovery • How long does it take to recover from a partial failure? Availability SLO: • How much unavailability / down time is acceptable? 4 High-Availability Consideration
- 5. Soft Failures • Server/process crashes but can be restarted Hard Failures • Corrupted disk Disasters Out of Scope: We only consider single cluster / AZ deployments in this talk, and thus, we don’t cover disaster recovery requiring multi-AZ deployments 5 Error Model
- 6. The previously discussed factors in combination determine how much redundancy is required in your system. 6 The Core of Reliability
- 7. How many Brokers do I need? 7
- 8. - Single broker - Producer w/o* acks 8 Let’s Start Simple
- 9. - Single broker - Producer w/o* acks - No delivery guarantee 9 Let’s Start Simple ?
- 10. - Single broker - Producer w/o* acks - No delivery guarantee - Soft durability guarantee 10 Let’s Start Simple ? ?
- 11. - Single broker - Producer w/o* acks - No delivery guarantee - Soft durability guarantee - No HA 11 Let’s Start Simple ? ? ?
- 12. - Single broker - Producer w/ acks 12 First Try: Acks
- 13. - Single broker - Producer w/ acks - Data delivery guarantee 13 First Try: Acks ✓ ✓
- 14. - Single broker - Producer w/ acks - Data delivery guarantee - Soft durability guarantee 14 First Try: Acks ✓ ? ✓
- 15. - Single broker - Producer w/ acks - Data delivery guarantee - Soft durability guarantee - No HA 15 First Try: Acks ✓ ? ? ✓
- 16. - Two brokers - replication.factor = 2 - Producer w/o acks 16 Second Try: Replication
- 17. - Two brokers - replication.factor = 2 - Producer w/o acks - No delivery guarantee 17 Second Try: Replication ?
- 18. - Two brokers - replication.factor = 2 - Producer w/o acks - No delivery guarantee - No durability guarantee 18 Second Try: Replication ? ?
- 19. - Two brokers - replication.factor = 2 - Producer w/o acks - No delivery guarantee - No durability guarantee - Write HA* - Read HA* 19 Second Try: Replication ? ? ✓
- 20. - Two brokers - replication.factor = 2 - Producer w/ acks=1 20 Third Try: Replication and Acks
- 21. - Two brokers - replication.factor = 2 - Producer w/ acks=1 - Delivery guarantee 21 Third Try: Replication and Acks ✓ ✓
- 22. - Two brokers - replication.factor = 2 - Producer w/ acks=1 - Delivery guarantee - No durability guarantee 22 Third Try: Replication and Acks ? ✓ ✓
- 23. - Two brokers - replication.factor = 2 - Producer w/ acks=1 - Delivery guarantee - No durability guarantee 23 Third Try: Replication and Acks ? ? ✓ ✓
- 24. - Two brokers - replication.factor = 2 - Producer w/ acks=1 - Delivery guarantee - No durability guarantee - Write HA* - Read HA* 24 Third Try: Replication and Acks ? ? ✓ ✓ ✓
- 25. - Two brokers - replication.factor = 2 - Producer w/ acks=all 25 Fourth Try: Acks all the way
- 26. - Two brokers - replication.factor = 2 - Producer w/ acks=all - Delivery guarantee - Hard durability guarantee 26 Fourth Try: Acks all the way ✓ ✓ ✓ ✓
- 27. - Two brokers - replication.factor = 2 - Producer w/ acks=all - Delivery guarantee - Hard durability guarantee - No Write HA - Read HA* 27 Fourth Try: Acks all the way ✓ ✓ ✓ ? ✓
- 28. - Three brokers - replication.factor = 3 - Producer w/ acks=all 28 Fifth Try: Higher Replication FTW?
- 29. - Three brokers - replication.factor = 3 - Producer w/ acks=all - Delivery guarantee - Hard durability guarantee 29 Fifth Try: Higher Replication FTW? ✓ ✓ ✓ ✓ ✓
- 30. - Three brokers - replication.factor = 3 - Producer w/ acks=all - Delivery guarantee - Hard durability guarantee - No Write HA - Read HA 30 Fifth Try: Higher Replication FTW? ? ✓ ✓ ✓ ✓ ✓
- 31. - Three brokers - replication.factor = 3 - min.insync.replicas=2 - Producer w/ acks=all 31 Sixth Try: Min.Insync.Replicas
- 32. - Three brokers - replication.factor = 3 - min.insync.replicas=2 - Producer w/ acks=all - Delivery guarantee - Hard durability guarantee - Write HA - Read HA 32 Sixth Try: Min.Insync.Replicas ✓ ✓ ✓ ✓ ✓ ✓
- 33. - Three brokers - replication.factor = 3 - min.insync.replicas=2 - Producer w/ acks=all - Delivery guarantee - Hard durability guarantee - Write HA - Read HA 33 Sixth Try: Min.Insync.Replicas ✓ ✓ ✓ ✓ ✓
- 34. What about read/write HA? 34
- 58. 58 Zookeeper - Need 3 ZK nodes - Number of brokers independent KRaft - Need 3 KRaft nodes + independent number of “data” broker Leader Election & Quorum Consensus protocols used to elect new leader Only in-sync replicase can become new leader
- 65. 65 Can Fetch from Follower help? No. - Designed for multi-AZ setup - Avoid cross-AZ network - Also: clients still connects to leader frequently
- 67. 67 Alternatives - ack = 2,3,4 <any other number> - weakens guarantees Winner takes it all? No. It’s also a dollar-cost and performance (throughput / latency) tradeoff. What do I need to my use case? Take aways Need data replication for hard durability guarantee and HA. Need acks for delivery guarantee. Acks are dynamic to not lose write HA.
- 68. 68 (*) HA requires leader election at any point in time => Need quorum, ie, 3 brokers or ZK nodes for reliability => always odd number of nodes to maintain n/2 quorum on failure Rule of Thumbs replication.factor – 1: => number of concurrent disk errors to guard against data loss => number of concurrent broker failures to keep read HA* replication.factor – min-insync- replicas: => no. of brokers down to keep write HA*
- 69. The Why Behind the How: Kafka Reliability from First Principles Matthias J. Sax Software Engineer | Apache Kafka Committer and PMC member Twitter/𝕏 @MatthiasJSax