Technical Enablement Session Partners Q&A

Thanks for joining!
We’ll get started soon!
Technical Enablement Session

@yourtwitterhandle | developer.confluent.io
Our Partner Technical Enablement offering
Scheduled sessions On-demand
Join us for these live sessions
where our experts will guide you
through sessions of different level
and will be available to answer
your questions. Some examples of
sessions are below:
• Confluent 101: for new starters
• Hybrid Cloud Workshop:
learn by doing
• Path to Production series ,
Confluent Cloud workshops
series
• Product Updates
Learn the basics with a guided
experience, at your own pace with
our learning paths on-demand. You
will also find an always growing
repository of more advanced
presentations to dig-deeper. Some
examples are below:
• Aware/Novice/Competent
Learning paths
• Confluent Use Cases
• Positioning Confluent Value
• Confluent Cloud Networking
• … and many more
AskTheExpert
we’ll offer a channel dedicated to
streaming questions
• Build CoE inside partners by
getting people with similar
interest together
• Connect with opportunities
and discover trends at focus
partners
• Build a Technical Community
• Q&A
• Tech Talk

What are the best practices to debug client applications
(producers/consumers in general but also Kafka Streams
applications)?

The Conﬂuent Q3 ‘23 Launch
Announcing the latest updates to our cloud-native data streaming
platform, Conﬂuent Cloud

Conﬂuent Cloud
Cloud native data streaming platform built by the founders of Apache Kafka®
9
Cloud-Native Complete Everywhere
Stream conﬁdently on the world’s most trusted data streaming platform built by the founders of
Apache Kafka©, with resilience, security, compliance, and privacy built-in by default.
Cloud Native
The 10x Apache Kafka®
service: elastic, resilient
and performant, powered
by the Kora Engine
Complete
Go above & beyond Kafka
with all the essential tools
for a complete data
streaming platform
Everywhere
Connect your data in real
time with a platform that
spans from on-prem to
cloud and across clouds

Deliver Intelligent, Secure, and Cost-effective Data Pipelines
10
Storage Price Reduction: Cost-effectively store data at any scale without growing compute at 20% lower prices
CC for Apache Flink®
(Open Preview)
+
Enterprise Clusters
Secure, cost-effective, and serverless Kafka
powered by the Kora Engine
Confluent Terraform Provider updates
+
Enhance security and compliance while
continuing to reduce operational burden
through automated infrastructure
management
HashiCorp
Sentinel
Integration
Resource
Importer
Data
Catalog
Support
Cloud Audit Logs for Kafka Produce
& Consume
Experience full visibility and control of
sensitive data access in Confluent Cloud with
detailed audit events enabling swift response
to unauthorized access.
Cluster Linking updates
Cluster Linking with AWS Private Link:
Easily stream data between regions, teams or
environments within AWS private networks
Bi-directional Cluster Linking Optimize
disaster recovery and increase reliability with
bi-directional cluster linking
Data Portal in
Stream Governance
Safely unlock data and increase developer
productivity with a self-service, data-centric
portal for discovering, accessing, and
enriching real-time data streams flowing
across your organization
(coming soon)
Easily build high-quality, reusable data streams with the industry’s only cloud-native, serverless Flink
service

Data Portal in Stream
Governance
11
Seamlessly and securely request
access to data streams and trigger an
approval workflow that connects the
user with the data owner, all within the
Confluent Cloud UI
Easily build and manage data products
to power streaming pipelines and
applications by understanding,
accessing, and enriching existing data
streams
Complete
Safely unlock data and increase
developer productivity with a
self-service, data-centric portal for
discovering, accessing, and enriching
real-time data streams flowing across
your organization
Search, discover, and explore existing
topics, tags, and metadata across the
organization with end-to-end visibility to
choose the data most relevant for your
projects
Coming Soon

Introducing Data Portal in Stream Governance
Access your data streams through a developer-friendly, self-service UI
Search, discover, and
explore existing topics,
tags, and metadata
across the organization
Seamlessly request
access to data streams
and trigger an approval
workﬂow
Understand, access, & enrich
data streams to power
real-time data streaming
pipelines and applications

Bidirectional Cluster
Linking
13
Optimize disaster recovery and
increase reliability with bi-directional
cluster linking
Facilitate seamless consumer
migration with retained offsets for
consistent data processing with
Bi-directional cluster links
Increase efﬁciency and reduce data
recovery time by eliminating the need
for custom code
Streamline security conﬁguration with
support for DR and active/active
architecture with Bi-directional links
that provides outbound and inbound
connections
Everywhere
**Note - bi-directional cluster linking is available for new cluster links only,
existing cluster link need to be deleted and re-activated to obtain this
functionality.

Enhanced Disaster Recovery Capabilities with
Bidirectional Cluster Linking
14
Cluster Link
bidirectional
Connection and Authentication
Connection and Authentication
Cluster A Cluster B
Applications
in region B
Cluster A Cluster B
Cluster Link
bidirectional
Topics on
Cluster A
Mirror
Topics on
Cluster B
Mirror Topics
on Cluster A
Topics on
Cluster B
ACLs / RBAC for Cluster
B
API Key or OAuth for Cluster
A
API Key or OAuth for Cluster B
ACLs / RBAC for Cluster A
Applications
in region A
Data &
Metadata
Data &
Metadata

Cluster Linking with
AWS Private Link
15
Simplified setup: Utilize Network Link
Service and Endpoint for a reliable
connection between clusters
Enhanced network-level security: AWS
PrivateLink isolates Confluent Cloud
clusters, preventing external resources
and Cluster Linking access
Seamless cluster linking: Establish a
secure networking path between
separate Confluent Cloud networks for
efficient data exchange
Everywhere
Easily stream data between regions,
teams or environments within AWS
private networks

Deliver Intelligent, Secure, and Cost-effective Data Pipelines
Storage Price Reduction: Cost-effectively store data at any scale without growing compute at 20% lower prices
Easily build high-quality, reusable data streams with the industry’s only cloud-native, serverless Flink
service
Apache Flink® on CC
(Open Preview)
+
Enterprise Clusters
Secure, cost-effective, and serverless Kafka
powered by the Kora Engine
Confluent Terraform Provider updates
+
Enhance security and compliance while
continuing to reduce operational burden
through automated infrastructure
management
HashiCorp
Sentinel
Integration
Resource
Importer
Data
Catalog
Support
Cloud Audit Logs for Kafka Produce
& Consume
Experience full visibility and control of
sensitive data access in Confluent Cloud with
detailed audit events enabling swift response
to unauthorized access.
Cluster Linking updates
Cluster Linking with AWS Private Link:
Easily stream data between regions, teams or
environments within AWS private networks
Bi-directional Cluster Linking Optimize
disaster recovery and increase reliability with
bi-directional cluster linking
Data Portal in
Stream Governance
Safely unlock data and increase developer
productivity with a self-service, data-centric
portal for discovering, accessing, and
enriching real-time data streams flowing
across your organization
(coming soon)

Conﬂuent Service Mesh
Roman Schmitz, November 2023

What is the Conﬂuent Service
Mesh (CSM)?

“A service mesh is a tool for adding observability, security,
and reliability features to “cloud native” applications by
transparently inserting this functionality at the platform
layer rather than the application layer. The service mesh is
rapidly becoming a standard part of the cloud native stack,
especially for Kubernetes adopters.”
20
-linkerd.io

21
-linkerd.io

22
-linkerd.io

Life as we know it
Producer Consumer

With CSM in the Mix
Producer Consumer
Pluggable
Code
CSM
Pluggable
Code
CSM

Kafka
Broker
port 9092
Kafka
Broker
port 9092
Kafka
Broker
port 9092
Conﬂuent Service Mesh at a glance
Conﬂuent Service
Mesh
Producer
Consumer
listener
port
30001
Pluggable
Code
listener
port
30002
Pluggable
Code
listener
port
30003
Pluggable
Code

Kafka Startup
Kafka
Broker
Get Metadata
Client
Return Metadata
Metadata Response
{
"Brokers": [
{
"NodeId": 0,
"Host": "broker0.yourdomain.com",
"Port": 9092
},
{
"NodeId": 1,
"Port": 9092
},
{
"NodeId": 2,
"Port": 9092
}
],
"Topics": [],
…
}
Connect to one of the
brokers

Kafka Startup With CSM
Return Metadata
Kafka
Broker
CSM
Get Metadata
Client
Modify Metadata
Return Metadata
Modiﬁed Metadata Response
{
"Brokers": [
{
"NodeId": 0,
"Host": "csm.yourdomain.com",
"Port": 30001
},
{
"NodeId": 1,
"Port": 30002
},
{
"NodeId": 2,
"Port": 30003
}
],
"Topics": [],
…
}
Connect to a CSM port

End-to-end Encryption Features
• Local key management and JKS support
• Gemalto, Hashicorp, many security appliances
• Cloud provider key management service support
• AES, RSA encryption, SHA256 hashing
• AVRO, JSON, Protobuf, XML, String, Byte arrays,
Byte buffer level encryption and tokenization
• Field access control
• Format preserving encryption (NIST SP 800-38G)
• Support for metadata and data classiﬁcation
• Support for master keys (Encryption of a data key
with a wrapping key)
• Support for key rotation
• Support for event digital signature support to
validate producers
Consumer
Protected
Producer
KMS/Tokenizer
Schema
Registry

Kafka Messages and Serialization
Consumer
Producer
Cleartext
{
"name": "Joe Example",
"address": "123 Main St",
"ssn_id": "123-45-6789",
"account": 678900000234,
"Order_time": 1560070133853,
"current_balance": 67,
"itemid": "Item_9"
}
{
"ssn_id": "123-45-6789",
"account": 678900000234,
"Order_time": 1560070133853,
"itemid": "Item_9"
}
Serializer Deserializer
1001001001001000110
1001010100101010001
1001010010010100101
0010101001010010100
1010100101001010101
0101010101001001000
1010011101101001010
1011110

Kafka Messages with encryption
Consumer
Producer
{
"ssn_id": "123-45-6789",
"account": 678900000234,
"Order_time": 1560070133853,
"itemid": "Item_9"
}
{
"ssn_id": "123-45-6789",
"account": 678900000234,
"Order_time": 1560070133853,
"itemid": "Item_9"
}
Serializer Deserializer
1100100100110010001
1010010101001010100
0110010100100101001
0100101010010100101
0010101001010010101
0101010101010010010
0010100111011010010
101011110
Protected
Encryption Decryption

Message-level encryption
{
"ssn_id": "123-45-6789",
"account": 678900000234,
"Order_time": 1560070133853,
"itemid": "Item_9"
}
Generate
Data Key
pPYP7QM+LjMfjJ+QdOrLF3VTjMy1sWPtf
epEXXwqkxXrnIbT1iEuzas2J/aOlUv7md
7YFP4Zq5PbrWWTLKeQDRlBVCOBacD15jl
pcME0EONfErWd/CljAaTtCEnGRtfKsCHx
0zasCvXK3G0v15GdptqEGoREtXpea5f9q
M8nYXc1tQbjX4mKP0nB/aVQSmKLXBeEU3
KaiioyXsT3Vsr+tLSCWO76Tfhfaum8Ue4
F5WKPD3svJA==
{
"ssn_id": "123-45-6789",
"account": 678900000234,
"Order_time": 1560070133853,
"itemid": "Item_9"
}
Info added to Metadata:
Encrypted Data Key, version, hash

Key Exchange Process
Kafka
Broker
Key
Store/KMS
Get Master Key
Key
Store/KMS
Encryption
Decryption
Get Data Key
Secured
Serializer
Encrypt Event
Encrypt Data Key
Send encrypted event and encrypted data key
Encryption
Decryption
Secured
Deserializer
Fetch Events
Get Master Key
Decrypt Data Key
Decrypt Event
Use decrypted data
key for decryption
Use data key for
encryption
Use master key for
decryption
Use master key
for encryption

Data Protection
with Conﬂuent
Service Mesh
and Encryption
accelerator
36
CSM producer sidecar is
responsible for data
protection independently
of the client type.
Protected
Producer Consumer
KMS/Tokenizer
CSM consumer sidecar is
responsible for safely
exposing data in clear and
can also handle ﬁeld
access control.
CSM CSM

Field-level protection
{
"ssn_id": "123-45-6789",
"account": "678900000234",
"Order_time": 1560070133853,
"itemid": "Item_9"
}
Generate
Data Key
{
"name": "Hyt Piqdfggr",
"address": "852 Jdrf Wd",
"ssn_id": "dKI4gflV6r339Q==",
"account": "PrM1vyf/CxwoqQ==",
"Order_time": 1560070133853,
"itemid": "Item_9"
}
{
"ssn_id": "123-45-6789",
"account": "678900000234",
"Order_time": 1560070133853,
"itemid": "Item_9"
}
Protected
Producer Consumer
KMS/Tokenizer
CSM CSM

Data Protection with Access Control via CSM
Original message
Original message
{
"ssn_id": "123-45-6789",
"account": "678900000234",
"Order_time": 1560070133853,
"itemid": "Item_9"
}
{
"name": "Hyt Piqdfggr",
"address": "852 Jdrf Wd",
"Order_time": 1560070133853,
"itemid": "Item_9"
}
Protected
{
"ssn_id": "123-45-6789",
"account": "678900000234",
"Order_time": 1560070133853,
"itemid": "Item_9"
}
{
"Order_time": 1560070133853,
"itemid": "Item_9"
}
Original message
with Access Control

OPA - Open Policy Agent
https://www.openpolicyagent.org/
OPA testing and examples: The Rego Playground

Policy Based Field Level Access Control
Which fields
should be
hidden or
redacted?
Producer Consumer
Open Policy Agent
Pluggable
Code
Confluent Service
Mesh
Pluggable
Code
Confluent Service
Mesh

USA
financial
Policy Based Field Level Access Control
Original message
{
"ssn_id": "123-45-6789",
"account": "678900000234",
"Order_time": 1560070133853,
"itemid": "Item_9"
“country”: “usa”
}
{
"account": "678900000234",
"Order_time": 1560070133853,
"itemid": "Item_9"
}
{
"ssn_id": "123-45-6789",
"account": "678900000234",
"Order_time": 1560070133853,
"itemid": "Item_9"
}
USA
financial
pii
Brazil
financial
pii
Open Policy
Agent
nothing sent
Pluggable
Code
Confluent
Service Mesh

Integration with Data catalogs, classification
Data classification
{
"type":"record",
"name":"DataClassifications",
"classifications":{
"PII":{
"encrypt":{
"key":"SamplePIIKey",
"wrapping.key":"RSAPII"
},
"classifications":{
"Personal":{
"tokenize":{ }
},
"Financial":{
"encrypt":{
"key":"SampleFinancialKey",
"wrapping.key":"RSAPIIFinancial"
}
}
}
},
"Protected": {
"encrypt": {
"authorizer.class": "classNameHere",
"authorizer.deny": false,
"opa.module.name": "classification",
"opa.rego": "/csm/classification.rego",
"opa.query": "data.classification.allow"
}
}
},
"fields":[ ]
Data Catalog
{
"type":"record",
"name":"ADataCatalog",
"namespace":"com.mybusiness",
"fields":[
{
"name":"SSN",
"type":"string",
"classifications": ["PII/Financial",
“Protected”]
},
{
"name":"Name",
"type":"string",
"classifications": ["PII/Personal",
“Protected”]
},
{
"name":"Address",
"type":"string",
"classifications": ["PII/Personal",
“Protected”]
},
{
"name":"Account",
"type":"string",
"classifications": ["PII/Financial",
“Protected”]
PII/Personal Name: Joe Example
PII/Personal Address: 123 Main St
CustID: 12345
PII/Financial SSN: 123-45-6789
Persona: 56A
Credit: 780
PII/Financial Acct #: 3456789
Current Balance: 0
PII/Personal Name: Hyt Piqdfggr
PII/Personal Address: 852 Jdrf Wd
CustID: 12345
PII/Financial SSN: dKI4gflV6r339Q==
Persona: 56A
Credit: 780
PII/Financial Acct #: PrM1vyf/CxwoqQ==
Current Balance: 0

OPA Configuration and Integration
Link OPA Policies in Classifications
Add OPA Policies (rego)
Local OPA module (Session Authorizer)
local path to rego file
rego path (decision,
package)

Mutual TLS (mTLS) or Kerberos
Producer Consumer
MTLS /
Kerberos
MTLS /
Kerberos
O
N
PREM
O
N
LY
🤬
FAIL

With CSM in the Mix
Client
Pluggable
Code
CSM
MTLS
principal
User1 => key/secret
User2 => key/secret
SASL
(key/secret
)
Lookup Auth from Principal
during
SSL
H
andshake

Example CSM MTLS Flow
Extract Principal
from Cert
Some
Database
CSM
SSL Handshake
Client
Lookup key/secret
from DB with Principal
as key
Return key/secret
Conﬂuent
Cloud
Authenticate sasl
with key/secret
Finish Handshake

Example: CSM Auth Swapping Configurations
…
csm.ssl=true
csm.ssl.enabled=true
csm.ssl.truststore.location=${truststore}
csm.ssl.truststore.password=confluent
csm.ssl.keystore.location=${keystore}
csm.ssl.keystore.password=confluent
csm.ssl.key.password=confluent
csm.ssl.client.auth=required
csm.ssl.principal.mapping.rules: RULE:^CN=([a-zA-Z.0-9@-]+).*$/$1/,DEFAULT
…
csm.authorizers=vaultAuth
vaultAuth.class=io.confluent.csid.csm.auth.VaultAuth
vaultAuth.vault.address=http://vault:8200
vaultAuth.vault.auth.token=vault-plaintext-root-token
vaultAuth.vault.store=secret/testing
vaultAuth.vault.split=/
…
mTLS Configuration
…
csm.ssl=true
sasl.enabled.mechanisms=GSSAPI
csm.sasl.mechanism=GSSAPI
…
csm.authorizers=vaultAuth
vaultAuth.class=io.confluent.csid.csm.auth.VaultAuth
vaultAuth.vault.address=http://vault:8200
vaultAuth.vault.auth.token=vault-plaintext-root-token
vaultAuth.vault.store=secret/testing
vaultAuth.vault.split=/
…
Kerberos Configuration
Examples, Documentation:
https://confluentinc.github.io/csid-csm/

Typical Hybrid
CSM-Setup
- hybrid setup
- self-managed connect
- local CSM and clients
- ksqlDB and CP in
Conﬂuent Cloud
- ksqlDB on
ﬁeld-level-encrypted
topics
- AWS KMS for keys (AWS,
Azure, Vault, …)

CSM in a sidecar
- external service writing
to plain-text topic
- kstreams app ﬁltering
data and writing to
encrypted topic
- local client connecting to
CCloud via CSM/directly

CSM as (Gateway)
Service on VMs
- CSM deployed on
containers/VMs
- HA achieved with
multiple CSM-replicas
and LB
- reminder: CSM is
stateless (!)
- Scaling
horizontally/vertically
- load-balancers for
external CSM-access

Client Conﬁguration Examples

Conﬁguration Example: Clients using CSM
bootstrap.servers=pkc-XXXXX.eu-west-1.aws.confluent.cloud:9092
security.protocol=SASL_SSL
bootstrap.servers=csm:30001
security.protocol=SASL_PLAINTEXT
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required
username='<CCLOUD API KEY>' password='<CCLOUD API SECRET>';
sasl.mechanism=PLAIN
# Required for correctness in Apache Kafka clients prior to 2.6
client.dns.lookup=use_all_dns_ips
# Required connection configs for Confluent Cloud Schema Registry
schema.registry.url=https://
basic.auth.credentials.source=USER_INFO
basic.auth.user.info=<SR-KEY>:<SR-SECRET>
bootstrap.servers=csm:30001
security.protocol=SASL_PLAINTEXT
sasl.mechanisms=PLAIN
sasl.username=<CCLOUD API KEY>
sasl.password=<CCLOUD API SECRET>
Java-Client librdkafka (kcat, C#, Python)

Conﬁguration Example: CSM with AWS KMS
csm.ssl=false
broker.ssl=true
host.name=csm
client.dns.lookup=use_all_dns_ips
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required
username="<CCLOUD API KEY>"
password="<CCLOUD API SECRET>";
# Required connection configs for Confluent Cloud Schema Registry
schema.registry.url=https://psrc-XXXXX.eu-central-1.aws.confluent.cloud
basic.auth.credentials.source=USER_INFO
basic.auth.user.info=<SR-KEY>:<SR-SECRET>
csm.get.brokers.on.boot=true
csm.port=30001
csm.request.interceptors=in
csm.response.interceptors=out
in.class=io.confluent.csid.csm.encryption.produce.EncryptInterceptor
in.key=rschmitz-symmetric
in.encryption.provider.name=aws
in.schema.registry.url=https://psrc-XXXXX.eu-central-1.aws.confluent.cloud
in.basic.auth.credentials.source=USER_INFO
in.basic.auth.user.info=<SR-KEY>:<SR-SECRET>
in.aws.provider.class = io.confluent.encryption.common.crypto.cipher.impl.AWSKMSProvider
in.aws.provider.use.default.sdk=true
in.aws.provider.region=eu-west-1
in.aws.provider.access.key.id=<AWS API-KEY>
in.aws.provider.secret.key=<AWS API-SECRET>
…
in.class=io.confluent.csid.csm.encryption.produce.EncryptInterceptor
in.key=rschmitz-symmetric
in.encryption.metadata.policy.class=CatalogPolicy
in.encryption.metadata.name=DataCatalog
in.encryption.classifications.name=DataClassifications
in.encryption.provider.name=aws
…
Example csm.properties Field-Level-Conﬁguration

CSM as a Gateway
to Conﬂuent Cloud
Transparent
end-to-end
encryption
Field-level
authorization and
access-control with
policy-based
ﬁeld-level
encryption
Use existing
authentication
mechanisms in
cloud migrations

CSM Ingress on k8s / SNI:
Formatter for Listener Overrides

Use case: Kubernetes Ingress
Ingress Scenario:
● CSM maps each broker to one port
that is exposed as a k8s service
● Ingress will not allow to open ports
dynamically (or more than a few
speciﬁc ports at all - 80, 8080, 443)

Solution: Formatter for Listener Overrides
Return Metadata
Kafka
Broker
CSM
Get Metadata
Client
Modify Metadata
Return Metadata
Modiﬁed Metadata Response Updated
{
"Brokers": [
{
"NodeId": 0,
"Port": 30001
"Host": "b30001.csm.yourdomain.com",
"Port": 9092
},
{
"NodeId": 1,
"Port": 30002
"Host": "b30002.csm.yourdomain.com",
"Port": 9092
},
…
],
"Topics": [],
…
}
Connect to a CSM port

Solution: SNI Routing
SNI: Server Name Indication - Wikipedia
(https://github.com/Schm1tz1/sni-routing-examples)
● Hosting of multiple (virtual) services
with same (physical) frontend and
different backends
● Used in Ingress for (de)multiplexing
TCP traffic
● Routing to backend services using
information from TLS handshake
(hello)
● Similar pattern based on HTTP
headers very common in for
Web-Servers

Formatter for Listener Overrides and SNI
Changes to "CSM standard setup":
● CSM conﬁgured to return virtual
hostnames that can be mapped
back to internal ports (example:
host.name.formatter=b$p.$h:9092)
● Matching Certiﬁcates (wildcard)
● Ingress with SNI rules / mapping for
these hostnames
● External DNS entries (wildcard)
pointing to ingress IPs

Features Comparison
Client-side Encryption CSM-based Encryption
Field-level encryption ✅ (Java,.NET only) ✅
Payload-level encryption ✅ ✅
Tokenization/Masking ✅ (Java,.NET only) ✅
Format-Preserving Encryption ✅ (Java,.NET only) ✅
Supports Kafka Streams ✅ ✅
Supports Kafka Connect JSON, AVRO only ✅
Supports ksqlDB ✅ ✅
Supports REST Proxy ❌ ✅
Popular KMS integrations ✅ (Java,.NET only) ✅
Supports access control ✅ ✅
Node.js, python, C++ support limited features ✅
Other (Go, Ruby) lang support ❌ ✅
Component-based install ✅ Not required

E2EE Libraries
Features and integrations
✅ Feature
included
❌ Feature
prioritized but
not complete
❌ Feature
not included
or prioritized
na Not
Applicable

Technical Enablement Session Partners Q&A

Technical Enablement Session Partners Q&A

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Technical Enablement Session Partners Q&A

Similar to Technical Enablement Session Partners Q&A (20)

More from confluent

More from confluent (20)

Recently uploaded

Recently uploaded (20)

Technical Enablement Session Partners Q&A