"There are various strategies for securely connecting to Kafka clusters between different networks or over the public internet. Many cloud providers even offer endpoints that privately route traffic between networks and are not exposed to the internet. But, depending on your network setup and how you are running Kafka, these options ... might not be an option! In this session, we’ll discuss how you can use SSH bastions or a self managed PrivateLink endpoint to establish connectivity to your Kafka clusters without exposing brokers directly to the internet. We explain the required network configuration, and show how we at Materialize have contributed to librdkafka to simplify these scenarios and avoid fragile workarounds."

1. Navigating private network connectivity
options for Kafka clusters
Steffen Hausmann • Field Engineer @ Materialize • @sthmmm

2. Private connectivity options
Virtual private cloud (VPC)
client
Peering
connection
Virtual private cloud Private subnet
broker-1 broker-2
1
PrivateLink endpoint
client
Private subnet
broker-1 broker-2
Virtual private cloud
Virtual private cloud (VPC)
Network Load
Balancer
2
Virtual private cloud (VPC)
Private subnet
broker-1 broker-2
Public subnet
SSH bastion
Virtual private cloud (VPC)
client
3

3. Run your own PrivateLink setup
Virtual private cloud (VPC)
Private Kafka subnet
Network Load Balancer
Virtual private cloud (VPC)
broker-1.local:9092
Interface
endpoint
Private client subnet
Listener Target group
port: 9092 broker-1.local:9092
broker-2.local:9092
broker-2.local:9092
producer

4. producer
Run your own PrivateLink setup
Virtual private cloud (VPC)
Private Kafka subnet
Network Load Balancer
Virtual private cloud (VPC)
broker-1.local:9092
Interface
endpoint
Private client subnet
Listener Target group
port: 9092 broker-1.local:9092
broker-2.local:9092
broker-2.local:9092
Private hosted zone
broker-1.local alias vpce-XXX.aws
broker-2.local alias vpce-XXX.aws
Metadata Response
2 brokers:
broker 2 at broker-2.local:9092
broker 1 at broker-1.local:9092 (controller)
1 topics:
topic ”tgif" with 2 partitions:
partition 0, leader 2, replicas: 2,1, isrs: 2,1
...
Metadata Request
topics: tgif
Produce Request
topic_data:
name: tgif
partition_data:
index: 0
...
Produce Response
responses:
name: tgif
partition_response:
error_code: NOT_LEADER_OR_FOLLOWER

5. Run your own PrivateLink setup
Virtual private cloud (VPC)
Private Kafka subnet
Network Load Balancer
Virtual private cloud (VPC)
broker-1.local:9092
advertised: broker-1.local:9001
Interface
endpoint
Private client subnet
Private hosted zone
Listener Target group
port: 9001 broker-1.local:9092
Listener Target group
port: 9002 broker-2.local:9092
broker-2.local:9092
advertised: broker-2.local:9002
broker-1.local alias vpce-XXX.aws
broker-2.local alias vpce-XXX.aws
Metadata Response
2 brokers:
broker 2 at broker-2.local:9002
broker 1 at broker-1.local:9001 (controller)
1 topics:
topic ”tgif" with 2 partitions:
partition 0, leader 2, replicas: 2,1, isrs: 2,1
...
Produce Request
topic_data:
name: tgif
partition_data:
index: 0
...
producer

6. Run your own PrivateLink setup in 3 easy steps?
0 Create a Network Load Balancer and PrivateLink Endpoint
1
Create a Private Hosted Zone to map broker DNS names to the
PrivateLink endpoint
2
Adapt advertised.listeners of all brokers and create a
listener/target group for each individual broker
3
Keep the listener and target group configuration in sync with the cluster
configuration

7. Allow overriding DNS resolution in librdkafka
Virtual private cloud (VPC)
Private Kafka subnet
Network Load Balancer
Virtual private cloud (VPC)
broker-1.local:9092
Interface
endpoint
Private client subnet
Listener Target group
port: 9001 broker-1.local:9092
Listener Target group
port: 9002 broker-2.local:9092
broker-2.local:9092
producer
Metadata Response
2 brokers:
broker 2 at broker-2.local:9092
broker 1 at broker-1.local:9092 (controller)
1 topics:
topic ”tgif" with 2 partitions:
partition 0, leader 2, replicas: 2,1, isrs: 2,1
...
DNS resolution
resolve_cb(broker-2.local, 9092)
→ getaddrinfo(vpce-XXX.aws, 9002)

8. Adding a layer of protection with private connectivity
2
Managed Kafka offerings often provide native PrivateLink support
out of the box
1
PrivateLink and SSH bastions add an additional layer of protection for
connecting to Kafka clusters
3
When running your own setup, use the resolve_cb callback in
librdkafka to avoid fragile changes to brokers and your network

9. References
2
Connecting Kafka Across Multiple AWS VPCs
https://www.confluent.io/kafka-summit-
sf18/connecting-kafka-across-multiple-aws-vpcs/
1
Allow overriding DNS resolution in librdkafka
https://github.com/confluentinc/librdkafka/pull/4051
3
Secure connectivity patterns to access Kafka
https://aws.amazon.com/blogs/big-data/secure-connectivity-
patterns-to-access-amazon-msk-across-aws-regions/

10. Thank you!
Steffen Hausmann • Field Engineer @ Materialize • @sthmmm