2. The Kafka News
LATEST
SPECIAL
FULL
REPORTS
No. 0135
Tuesday, 15th June 2021
MyHeritage and the world
Kafka is going all-in for
cloud-native solutions!
“One small step for
Kafka, one giant step
for humanity”:
ZooKeeper, no more
Top technologists’ thoughts on
getting rid of ZooKeeper, one less
distributing system to manage
Confusion is a symptom that makes you
feel as if you can’t think clearly. You might
feel disoriented and have a hard time
cusing or making decisions.
to as
K8s deployment has never been easier!
As in every big project, this project introduces
massive obstacles and uncertainty. AWS helm
kubernetes, latency vpc at least lambda
dynamodb. It is not very often that network
boundaries meet cache and cdn proxy, mysql
replication acl active key rotation is ec2 elastic
and computed slack.
Every once in a while PV virtualization sns binary
attribute, yielding detailed monitoring and a mail
transfer agent. On the contrary, aws vpn
cloud-hub is a single sign on snowball for a
storage gateway and cloud directory cross
account access, security and credentials.
Puppet is also a big pipeline terraform volume,
container and eventually consistent alike.
In case of code delivery, temporary solutions
hadoop accounts instance type and labeled
data. Micro instance with MFA gives logstash
cassandra node policy; while external access
real-time mirroring for DR. Have you ever disk
space analysis, or chattiness configuration
consul? If not, now is the time to start!
One of the main tasks in migrating to the cloud is externalizing the
configuration from the code. These sprints Jira kanban scrum burndown, in
ide variety of story task, subtask and bug monitoring. Regarding
ead time cycle commitment is potentially
remaining
“Lead the way” - Kafka consuming can be done from a follower
3. ● BackEnd Infrastructure team lead @MyHeritage
● Kafka, Streaming & Distributed systems geek
● Writing & speaking about event-driven-systems
● Managing ApacheKafkaIL meetup group, approaches 2000 members
● Find me at:
○ medium.com/@ofirsharony
○ ofirs@myheritage.com
4. ● Tiered storage lead to infinite retention possibilities
● Elasticity, usage based billing
● Hybrid clouds, on prem
● Cluster linking
● Auto data balancing
● Scaling connectors
● 100% serverless
● ...
CloudnativeKafka
5. ● Zookeeper & Kafka were close friends for ~10 years
● Reducing complexity is another cloud native goal
● Over the years, there has been an effort for reducing ZK
dependency, e.g consumer offsets moved to the broker
● What’s left?
○ Stores topics, num of partitions, and other configuration
○ Server failure detection and recovery
○ Data partitioning
○ Cluster management
○ In-sync data replication
○ Access control list
● Zookeeper down => non-functional Kafka
Zookeeper andquoromcontroller
6. ● Experimental in 2.8, not for production usage
● All responsibility moved from ZK to Kafka
● Quorum controller - running inside the kafka broker/s. State is
stored as a metadata topic
● Can handle a much larger number of partitions per cluster.
● Rolling restart is much faster.
● No need move critical metadata between ZooKeeper and internal
Kafka controller.
● The choice is now much easier between Kafka to alternatives
● lightweight, single-process - Kafka is just as applicable to the
small-scale startup as it is to the high-throughput Company such
as Netflix or Instagram
Zookeeper andquoromcontroller
7. ● Confluent operator joins the cloud native party
● Declarative Kafka (broker, ZK, connectors...). Define the desired state
● Change topic definitions easily, security policies, version upgrades
Kafka AndK8s
8. ● From the beginning of time, all kafka operations, read or write,
were allowed only through the topic-partition leader
● What happens when your cloud deployment spans over
multiple AZs?
● Rack-awareness to the rescue!
Readfrom a follower!
9. ● Have you ever faced an endless rebalance?
● Static membership to the rescue!
● Consumer restart won’t force rebalance until session.timeout passes
Staticconsumermembership
10. ● An old Kafka myth was to split different event to separate topics.
● But, what if ordering matters? What if you use event-sourcing?
● Schema registry supports this for quite some time.
● But… Can you insert anything now? No restrictions?
● Confluent recently added constraints for which events are allowed.
Multipleeventtypesin a topic
