Getting Kafka running on Kubernetes is only step one of a journey to create a production-ready Kafka cluster. This talk walks through the other steps: 1) Monitoring and remediating faults. 2) Updates to Kubernetes nodes for clusters not using shared storage. 3) Automating Kafka updates and restarts. We present how to create fault-tolerant Kafka clusters on Kubernetes without sacrificing availability, durability, or latency. Learn about Lyft's overlay-free Kubernetes networking driver and how we use it to keep performance on par with non-Kubernetes clusters.

1. Agenda
Easy Peasy, Right?
Shared or Local Storage?
DaemonSet Option
StatefulSet Option
Upgrades
Security
Conclusion

2. Easy Peasy, Right?

3. Provisioning team vs. Kafka team
Hi Kafka team, you’re using
the old provisioning system.
Please move Kafka to
Kubernetes.
We’d love to. How do we get
started?

4. Provisioning team vs. Kafka team
Here’s an example of a app,
just copy it.
We got a problem, data
keeps getting lost on each
deploy. What do we do?

5. Provisioning team vs. Kafka team
Doesn’t Kafka have
replication features?

6. Provisioning team vs. Kafka team
Mmmmmmmmmmmaaahhh!
Yeh, but we’ve got terabytes
of data to copy for each
node, it’ll take days to do
updates.

7. Benefits
Common Operating System provisioning system:
Security upgrades
Logs and metrics
Containerization:
Developer control of software dependencies
Automated deploys
Why Kubernetes?

8. Kafka is a “reliability tool”
Move data without lossiness
High stakes usage
Why Kafka?

9. Observability Data

10. Event Streaming
DB

11. Change Data Capture
DB
DB
DB DB

12. Availability
Durability
Latency and Throughput
Cost
Key Objectives

13. Deploy multiple brokers
Restart one replica at a time
Clients reconnect to new leaders
Availability

14. Restarted brokers pick up existing data
Data must match broker id
In case of volume loss, replicate to new volume
Durability

15. Use page cache as much as possible
Low latency disks
Low overhead network
Latency and Throughput

16. Shared or Local Storage?

17. How much?
1GB/
sec
** Representative number for
calculation sake. Not our actual load.

18. How much?
Retention: 4 Days

19. How much?
345.6 TB

20. How much? - With 3x replication
1 PB

21. Options - AWS
EBS vs. Instance Store

22. EBS Option
Monthly Cost for 1PB
End-to-End Latency P99
Consume and Produce
IO1 20k IOPS $372k 9ms
IO1 10k IOPS $256k 11ms
GP2 $111k 12ms
ST1 $50k 22ms

23. EBS Option Compute
m5a.2xlarge
32GB of RAM
$47k for 173 Nodes, 6TB each

24. EBS Option
Monthly Cost for 1PB
with Compute
End-to-End Latency P99
Consume and Produce
IO1 20k IOPS $419k 9ms
IO1 10k IOPS $303k 11ms
GP2 $158k 12ms
ST1 $97k 22ms

25. EBS: Can we reduce replication?
# Brokers Availability Prediction w/ EBS1
One 91.6%
Two 99.82%
Three 99.99%
1. https://bit.ly/2llzyFZ

26. Instance Store Option
d2.xlarge
2TB 2TB 2TB
Also known as “Ephemeral Volumes”

27. Instance Store Option
d2.xlarge
6TB
2TB 2TB 2TB
RAID0

28. Instance Store: Looking better
Monthly Cost for 1PB
End-to-End Latency P99
Consume and Produce
IO1 20k IOPS $419k 9ms
IO1 10k IOPS $303k 11ms
GP2 $158k 12ms
ST1 $97k 22ms
d2.xlarge $83k 14ms
i3.4xlarge $241k 8ms

29. Instance Store and K8s
Not as easy as EBS
Kafka container must re-schedule on same node
Recycle broker ids

30. Reschedule on same node
Active Broker 0 Active Broker 1 Active Broker 2 Idle K8s Node

31. Reschedule on same node
Active Broker 0
kafka-v2.2
Active Broker 1
kafka-v2.2
Active Broker 2
kafka-v2.2
Idle K8s Node

32. Reschedule on same node
Active Broker 0
kafka-v2.2
Active Broker 1
kafka-v2.2
Active Broker 2
kafka-v2.2
Idle K8s Node

33. Reschedule on same node
Active Broker 0
kafka-v2.2
Active Broker 1
kafka-v2.2
Active Broker 2
kafka-v2.2
Active Broker 0
kafka-v2.3

34. Reschedule on same node
Idle K8s Node Active Broker 1
kafka-v2.2
Active Broker 2
kafka-v2.2
Active Broker 0
kafka-v2.3
Wait up-to 24 hours for replication

35. Reschedule on same node
Idle K8s Node Active Broker 1
kafka-v2.2
Active Broker 2
kafka-v2.2
Active Broker 0
kafka-v2.3

36. Reschedule on same node
Active Broker 1
kafka-v2.3
Active Broker 1
kafka-v2.2
Active Broker 2
kafka-v2.2
Active Broker 0
kafka-v2.3

37. Reschedule on same node
Active Broker 1
kafka-v2.3
Idle K8s Node Active Broker 2
kafka-v2.2
Active Broker 0
kafka-v2.3
Wait up-to 24 hours for replication

38. Reschedule on same node
Active Broker 1
kafka-v2.3
Idle K8s Node Active Broker 2
kafka-v2.2
Active Broker 0
kafka-v2.3
Wait up-to 24 hours for replication

39. Several Solutions
DaemonSet
StatefulSet w/ PLV on K8s 1.14+
Local PVs using OpenEBS on K8s 1.12+
Local Path Provisioner on K8s 1.12+

40. DaemonSet Option

41. Kafka as a DaemonSet
DaemonSet declares:
Each broker is placed on every K8s node
Static id assigned to K8s node
Ids mounted into broker container
Only Kafka scheduled on K8s cluster

42. Kafka as a DaemonSet
Terraform
resource "aws_instance" "kubelet" { count = x … }
1
Kafka Id
1
Kafka Id
2
2
Kafka Id
3
3
Kubelet 1 Kubelet 2 Kubelet 3

43. Kafka Network Architecture
Client 1 Client 2
Kafka Node 1
Bootstrap Server - Load Balancer or Round-robin DNS
Kafka Node 2 Kafka Node 3
Talk to node 1, 2, and 3
Talk to node 1, 2, and 3 Talk to node 1, 2, and 3

44. Kubernetes Network w/ DaemonSet
Kafka Id
1
Kafka Id
2
Kafka Id
3
Kubelet 1 Kubelet 2 Kubelet 3
HostPort: 9093 HostPort: 9093 HostPort: 9093
Client 1 Client 2
Load Balancer or Round-robin DNS
hostNetwork:
true

45. StatefulSet Option

46. Kafka as a StatefulSet
StatefulSet declares:
Kafka pod is pinned to specific local disk
StatefulSet id follows broker
Nodes labeled to have only Kafka

47. Kafka as a StatefulSet
Terraform
resource "aws_autoscaling_group" "kubelet" { min-size = x }
Kafka Id
1
Kubelet
Stateful
Set Id 1
Kafka Id
2
Kubelet
Stateful
Set Id 2
Kafka Id
3
Kubelet
Stateful
Set Id 3
Hot spare or other
Pods
Kubelet

48. Kubernetes StatefulSet w/ kube-proxy
Kafka Id
1
Kafka Id
2
Kafka Id
3
Kubelet 1 Kubelet 2 Kubelet 3
Client 1 Client 2
ClusterIP
w/ Iptables
kube-proxy
ClusterIP
w/ Iptables
kube-proxy
ClusterIP
w/ Iptables
kube-proxy

49. Kubernetes Network: amazon-vpc-cni-k8s1
Kafka Id
1
Kafka Id
2
Kafka Id
3
Kubelet 1 Kubelet 2 Kubelet 3
Client 1 Client 2
ENI: eth0
Kafka Container IP:
1.1.2.11
Kublet IP: 10.1.1.11
veth0
Kafka Container IP:
1.1.2.11
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
ENI: eth0
Kafka Container IP:
1.1.2.12
Kublet IP: 10.1.1.12
veth0
Kafka Container IP:
1.1.2.12
ENI: eth0
Kafka Container IP:
1.1.2.13
Kublet IP: 10.1.1.13
veth0
Kafka Container IP:
1.1.2.13
1. https://github.com/aws/amazon-vpc-cni-k8s

50. Kubernetes Network: cni-ipvlan-vpc-k8s1
Kafka Id
1
Kafka Id
2
Kafka Id
3
Kubelet 1 Kubelet 2 Kubelet 3
Client 1 Client 2
ENI:
eth0 - mac:
01-23-45-67-89-11
Kafka Container IP: 1.1.2.11
Kublet IP: 10.1.1.11
ipv1 - mac:
01-23-45-67-89-11
IP: 1.1.2.11
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
ENI:
eth0 - mac:
01-23-45-67-89-12
Kafka Container IP: 1.1.2.12
Kublet IP: 10.1.1.12
ipv1 - mac: 01-23-45-67-89-12
IP: 1.1.2.12
ENI:
eth0 - mac:
01-23-45-67-89-13
Kafka Container IP: 1.1.2.13
Kublet IP: 10.1.1.13
ipv1 - mac: 01-23-45-67-89-13
IP: 1.1.2.13
1. https://github.com/lyft/cni-ipvlan-vpc-k8s

51. Node Death Problem
Manual intervention is required for
StatefulSet
kubectl delete <plv-name>

52. Scale Down
Evacuate partitions *before* delete

53. Upgrades

54. Health Check For Rolling Upgrade
Check URP
Watch jmx agent utilization

55. RollingUpgrade by AZ
Ideal:
Upgrade one AZ at a time
Requires creating a custom controller

56. Immutable Infrastructure Upgrade
Active Kubelet: 30 days old Active Kubelet: 30 days old
POD A POD B
POD C POD D
POD E POD F
POD G POD H

57. Immutable Infrastructure Upgrade
Active Kubelet: 30 days old Active Kubelet: 30 days old
POD A POD B
POD C POD D
POD E POD F
POD G POD H
New Kubelet: 0 days old

58. Immutable Infrastructure Upgrade
Deleting Kubelet: 30 days
old
Active Kubelet: 30 days old
POD E POD F
POD G POD H
Active Kubelet: 0 days old
POD A POD B
POD C POD D

59. Immutable Infrastructure Upgrade
Deleting Kubelet: 30 days
old
Active Kubelet: 30 days old
POD E POD F
POD G POD H
Active Kubelet: 0 days old
POD A POD B
POD C POD D

60. Immutable Infrastructure Upgrade
Active Kubelet: 30 days old
POD E POD F
POD G POD H
Active Kubelet: 0 days old
POD A POD B
POD C POD D
New Kubelet: 0 days old

61. Immutable Infrastructure Upgrade
Deleting Kubelet: 30 days
old
POD E POD F
POD G POD H
Active Kubelet: 0 days old
POD A POD B
POD C POD D
New Kubelet: 0 days old

62. Immutable Infrastructure Upgrade
POD E POD F
POD G POD H
Active Kubelet: 0 days old
POD A POD B
POD C POD D
New Kubelet: 0 days old

63. Immutable Infrastructure Upgrade
Recall the problem with state transfer
Killed Broker 0 Active Broker 1
kafka-v2.3
Active Broker 2
kafka-v2.3
Updated Broker 0
kafka-v2.3
Wait up-to 24 hours for replication

64. Mutable Infrastructure Upgrade
Option 1: Do Rolling Reboots
Kubelet
Kafka Node 2
KubeletKubelet
Kafka Node 1 Kafka Node 3
SSH:
Upgrade and
Reboot

65. Mutable Infrastructure Upgrade
Option 1: Do Rolling Reboots
Kubelet
Kafka Node 2
KubeletKubelet
Kafka Node 1 Kafka Node 3
SSH: Wait for
URP == 0

66. Mutable Infrastructure Upgrade
Option 1: Do Rolling Reboots
Kubelet
Kafka Node 2
KubeletKubelet
Kafka Node 1 Kafka Node 3
SSH: Upgrade
and Reboot

67. Mutable Infrastructure Upgrade
Option 1: Do Rolling Reboots
Kubelet
Kafka Node 2
KubeletKubelet
Kafka Node 1 Kafka Node 3
SSH: Wait for
URP == 0

68. Mutable Infrastructure Upgrade
Option 2: Decentralized Rolling Reboots
Kubelet
Kafka Node 1
Zookeeper or
ETCD
# cron
1. Upgrade
2. If new
kernel -
invoke
restarter
Restarter
Kubelet
Kafka Node 2
# cron
1. Upgrade
2. If new
kernel -
invoke
restarter
Restarter
Kubelet
Kafka Node 3
# cron
1. Upgrade
2. If new
kernel -
invoke
restarter
Restarter
Take and wait
for locks

69. Mutable Infrastructure Upgrade
Option 2: Decentralized Rolling Reboots
Kubelet
Kafka Node 1
Zookeeper or
ETCD
# cron
1. Upgrade
2. If new
kernel -
invoke
restarter
Restarter
Kubelet
Kafka Node 2
# cron
1. Upgrade
2. If new
kernel -
invoke
restarter
Restarter
Kubelet
Kafka Node 3
# cron
1. Upgrade
2. If new
kernel -
invoke
restarter
Restarter
az-1 az-2 az-3
Restart one AZ or
“rack” at a time

70. Security

71. Encryption in Transit
Challenges:
TLS for clients to Kafka
Don’t use wild-card certs
MTLS for inter-broker communications
Don’t check your private certs into VCS

72. Encryption in Transit
KIAM w/ ACM-PCA
# On start:
keytool -genkeypair ...
keytool -certreq ...
aws acm-pca issue-certificate ...
aws acm-pca get-certificate ...
keytool -import ...
Kubelet
Client
Private-ca in truststore
KIAM
Agent
KIAM
Server
Fetch
AWS
ACM-PCA
AWS DNS-
Route53
External-
DNS
ENI
Kafka Node
1
TLS Connection
AWS
IAM Allow Kafka IAM
role

73. Encryption at Rest
On AWS:
Instance Store and EBS both offer
encryption
Encrypt before produce and decrypt on
consume

74. Conclusion

75. StatefulSets preferred, DaemonSet good back up option
Newer versions of Kubernetes support ephemeral disks better
Use AWS VPC Kubernetes CNI driver using IPvlan
With ephemeral disks, do mutable upgrades
Takeaways

76. Summary
K8s Adjustment Reliability Wins Monthly Cost Savings Per 100 nodes
Mutable Upgrades Increased availability+durability $4k
Ephemeral Volumes Improved tail latency $8k-$120k better than EBS
CNI driver using IPvlan Improved throughput+latency $0
Resolve Node Death Increased availability+durability $0

77. Rolling restarts by AZ in K8s
Remove need to intervene with StatefulSet on
node death
Publish comprehensive benchmarks
Other cloud provider benchmarks
Future Work

78. github.com/devtagare
Dev Tagare <dtagare@lyft.com>
linkedin.com/in/andrey-falko/
Thank You
Thank you!