1. The document discusses Kafka Connect and its evolution for managing near real-time streaming pipelines. It describes how Kafka Connect can be used for data integration across different systems and challenges around identity when deploying Kafka Connect. 2. It introduces the concept of a managed Kafka Connect which deploys Kafka Connect on the customer's own Kubernetes namespace to avoid security and identity issues. The managed Connect is configured and managed through a centralized control plane. 3. It details how the managed Kafka Connect control plane can be used to provision Kafka Connect clusters, deploy connectors between different systems, override connector configurations, and monitor tasks.

1. 1
1
Changing Landscapes in Data Integration
Kafka Connect for near-real time streaming pipelines
Kafka Summit Americas- 2021
Presenters: Ashok Kadambala & Shreesha Hebbar

2. 2
Agenda
• Kafkascape
• Evolution in Kafka cluster deployments & Kafka use cases
• Motivation for Managed Kafka Connect
• Control Plane Overview & Design Principals
• Managing Connect & Connectors
• Future Ahead!

3. 3
Kafkascape

4. 4
Embracing Regions and AZs
• Kafka Cluster is stretched across 2 AZs within a region.
• Brokers are grouped into racks using Kafka's rack-awareness.
• Each AZ consists of 2 Kafka logical racks.
• Topics are created with Replication factor 4 and Min ISR 3.
• Zookeeper Quorum node is deployed in another region.
Kafka Logical
Rack-1
Kafka Logical
Rack-2
Region1 – AZ1
Placement Group 1 Placement Group 2
ZK ZK
Kafka Logical
Rack-3
Kafka Logical
Rack-4
Region1 – AZ2
Placement Group 1 Placement Group 2
ZK ZK
ZK
Region2 - AZ1
Zookeeper Ensemble
2 . 5 A Z S t r e t c h c l u s t e r
• On AZ failure, Consumers will continue w/o any changes.
• Producers will either
•Reduce Acks to 1 programmatically (or)
•Update Topic Min ISR to 2 using Self-Service API.

5. 5
Evolution in Kafka Use Cases
• High Throughput Pub/Sub
• Event Driven Architectures
• Data-plane for logs, metrics and metadata
aggregation
U s e c a s e s s o f a r. . !
• Near real time streaming & stream processing
• Multi-Cloud workloads
• Data streaming pipelines across cloud environments
E m e r g i n g U s e c a s e s
Database (On Perm) AWS S3
?
A T y p i c a l S t r e a m i n g U s e c a s e – D a t a b a s e t o S 3

6. 6
Kafka Connect – Declarative Data Integration
Connect Connect
Source Systems Sink Systems
• Framework to stream data between Apache Kafka and other systems.
• Flexible and extensible.
• Supports 100+ source/sink connectors and easy to plugin custom connectors.
D e c l a r a t i v e D a t a I n t e g r a t i o n f r a m e w o r k f o r A p a c h e K a f k a

7. 7
Kafka Connect – The Identity Problem
Connect
Sink Systems
Source Systems
• Source and Sink Systems (that Kafka Connect interacts
with) are owned by customers.
• Kafka Connect needs necessary READ/WRITE permissions
on source/sink systems to stream data.
• Deploying Kafka Connect instances on Service Provider's
infrastructure brings Security/Identity Challenges.
K a f k a C o n n e c t S i m p l i f i e d V i e w

8. 8
Managed Kafka Connect
Kafka Connect Control plane
Connect Connect Connect
M a n a g e
C u s t o m e r ’ s D o m a i n
• Kafka Connect is deployed on Customer’s Kubernetes namespace and connect to source/sink systems with Customer’s identity.
• De-centralized and Scalable architecture without security/identity impediments.
• Each Kafka Connect cluster is configured with a unique http(s) URL.
• Connect instances and Connectors are managed through centralized & federated control-plane.
c o n n e c t 1 . j p m c h a s e . n e t c o n n e c t 2 . j p m c h a s e . n e t c o n n e c t 3 . j p m c h a s e . n e t
S e l f S e r v i c e
S e r v i c e P r o v i d e r ’ s N a m e s p a c e
C u s t o m e r N a m e s p a c e s

9. 9
Kafka Connect Use Cases
Data Fan-in, Fan-out
Mirroring at Tenant-level
Enabling Data Mesh Architectures
VS
Multi-Cloud Data Integration

10. 10
Control plane - Design Principles
Secured from the start
Inclusive / Integrated experience for the clients
APIs on top of available REST endpoints to simplify the end-end process
Intuitive UI with guided navigation
License management
Minimal Operational overhead (monitoring, metrics, etc. bundled in)

11. 11
Enhancing Base Image
Security
Extensions
Connectors
Monitoring &
Resiliency
Extensions
Metrics
Reporter
Config
Providers
Kafka Connect base image is layered with
• Commonly Used Connectors based on demand-supply factors
• Security Extensions to enforce AuthX on connect REST
endpoints
• Config providers integrating with external security systems
• JMX Metrics reporter to plug into any TSDB enabled metrics
toolkit
• Monitoring & Resiliency Extensions
Connect
Base Image

12. 12
Kafka Connect - AuthX Extension
public class AuthXRequestFilter implements ContainerRequestFilter {
public AuthXRequestFilter() {
}
@Override
public void filter(ContainerRequestContext requestContext) throws IOException {
//AuthX code block goes here
}
}
public class KafkaConnectAuthXExtension implements ConnectRestExtension {
@Override
public void register(ConnectRestExtensionContext connectRestExtensionContext) {
connectRestExtensionContext.configurable().register(new AuthXRequestFilter());
}
@Override
public void configure(Map<String, ?> map) {
}
}
rest.extension.classes=com.jpmorgan.kafka.security.KafkaConnectAuthXExtension
- name: CONNECT_REST_EXTENSION_CLASSE
value: "com.jpmorgan.kafka.security.KafkaConnectAuthXExtension"

13. 13
Provisioning Kafka Connect
{
“applicationId": 100000,
"kafkaCluster": “AK1000",
"kconnectName": “kafka-summit-connect",
"kubeCluster": "carl-mt-a5",
"kubeNamespace": “ksummit-kube-space",
"identityPrincipal": “someUser@REALM.COM",
"numOfConnectInstances": 2,
"externalSystems": [
{
"name": “source-db",
"hosts": [
“somehost.jpmchase.net"
],
"ports": [
6137
],
"ingress": false,
"egress": true
}
]
}
• Every Connect deployment has its own config, offset and
status topics (<kconnectName>-status for e.g.)
• Tailored Kubernetes deployment config is generated based
on provided connect request.
• Connect deployment is assigned with a unique HTTP(s) URL
based on kconnectName for managing connect
deployment.
• Kafka Connect is always deployed in distributed mode.

14. 14
Deploying Connectors
S o u r c e C o n n e c t o r S i n k C o n n e c t o r
{
"name": "jdbc-source-connector",
"topics": [
{
"topicName": “db-application-topic",
"numOfPartitions": 10,
"retentionInDays": 1,
"topicSizeInGB": 1,
"compactedTopic": false
}
],
"config": {
"topic.prefix": "db-",
"producer.override.compression.type": "zstd",
"producer.override.max.request.size": "2000000",
}
}
{
"name": “s3-sink-connector",
"topics": [
"db-application“,
"db-trade-checkpoint“,
],
"config": {
"consumer.override.group.id": "s3-sink-connector-cg-100000",
"tasks.max": "1",
"topics": "db-application, db-trade-checkpoint",
"s3.bucket.name": “some-bucket-name“,
"s3.credentials.provider.class":"com.jpmc.kconnect.AWSCredentialProvider“
}
}
• Necessary Topic related entities can be created as part of submitting connectors.
• Specific source/sink connectors can be overridden with producer. & consumer. prefixes.

15. 15
Overriding Connector Config
• KIP- 458 connector.client.config.override.policy - Class name or alias of implementation of ConnectorClientConfigOverridePolicy. Defines what client
configurations can be overridden by the connector. The default implementation is `None`. The other possible policies in the framework include `All` and `Principal`.
public class CustomConnectorClientConfigOverridePolicy extends AbstractConnectorClientConfigOverridePolicy {
private static final Set<String> NOT_ALLOWED_CONFIG = Stream.of(CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG).
collect(Collectors.toSet());
@Override
protected String policyName() {
return "Custom";
}
@Override
protected boolean isAllowed(ConfigValue configValue) {
return !NOT_ALLOWED_CONFIG.contains(configValue.name());
}
@Override
public void configure(Map<String, ?> configs) {
}
}
connector.client.config.override.policy = Custom
- name: CONNECT_CONNECTOR_CLIENT_CONFIG_OVERRIDE_POLICY
value: Custom

16. 16
Future ahead..!
Multi-Kube
Automated Task Monitoring
Resiliency Extensions
Supporting more connectors
Guided Navigation Adopting Operators

17. 17
17
Thank You