SlideShare a Scribd company logo
Thanks for joining!
We’ll get started soon!
Technical Enablement Session
Partners Q&A
Partners Q&A
@yourtwitterhandle | developer.confluent.io
Our Partner Technical Enablement offering
Scheduled sessions On-demand
Join us for these live sessions
where our experts will guide you
through sessions of different level
and will be available to answer
your questions. Some examples of
sessions are below:
• Confluent 101: for new starters
• Hybrid Cloud Workshop:
learn by doing
• Path to Production series ,
Confluent Cloud workshops
series
• Product Updates
Learn the basics with a guided
experience, at your own pace with
our learning paths on-demand. You
will also find an always growing
repository of more advanced
presentations to dig-deeper. Some
examples are below:
• Aware/Novice/Competent
Learning paths
• Confluent Use Cases
• Positioning Confluent Value
• Confluent Cloud Networking
• … and many more
AskTheExpert
we’ll offer a channel dedicated to
streaming questions
• Build CoE inside partners by
getting people with similar
interest together
• Connect with opportunities
and discover trends at focus
partners
• Build a Technical Community
• Q&A
• Tech Talk
@yourtwitterhandle | developer.confluent.io
What are the best practices to debug client applications
(producers/consumers in general but also Kafka Streams
applications)?
@yourtwitterhandle | developer.confluent.io
@yourtwitterhandle | developer.confluent.io
The Confluent Q3 ‘23 Launch
Announcing the latest updates to our cloud-native data streaming
platform, Confluent Cloud
Confluent Cloud
Cloud native data streaming platform built by the founders of Apache Kafka®
9
Cloud-Native Complete Everywhere
Stream confidently on the world’s most trusted data streaming platform built by the founders of
Apache Kafka©, with resilience, security, compliance, and privacy built-in by default.
Cloud Native
The 10x Apache Kafka®
service: elastic, resilient
and performant, powered
by the Kora Engine
Complete
Go above & beyond Kafka
with all the essential tools
for a complete data
streaming platform
Everywhere
Connect your data in real
time with a platform that
spans from on-prem to
cloud and across clouds
The Confluent Q3 ‘23 Launch
Deliver Intelligent, Secure, and Cost-effective Data Pipelines
10
Cloud-Native Complete Everywhere
Storage Price Reduction: Cost-effectively store data at any scale without growing compute at 20% lower prices
CC for Apache Flink®
(Open Preview)
+
Enterprise Clusters
Secure, cost-effective, and serverless Kafka
powered by the Kora Engine
Confluent Terraform Provider updates
+
Enhance security and compliance while
continuing to reduce operational burden
through automated infrastructure
management
HashiCorp
Sentinel
Integration
Resource
Importer
Data
Catalog
Support
Cloud Audit Logs for Kafka Produce
& Consume
Experience full visibility and control of
sensitive data access in Confluent Cloud with
detailed audit events enabling swift response
to unauthorized access.
Cluster Linking updates
Cluster Linking with AWS Private Link:
Easily stream data between regions, teams or
environments within AWS private networks
Bi-directional Cluster Linking Optimize
disaster recovery and increase reliability with
bi-directional cluster linking
Data Portal in
Stream Governance
Safely unlock data and increase developer
productivity with a self-service, data-centric
portal for discovering, accessing, and
enriching real-time data streams flowing
across your organization
(coming soon)
Easily build high-quality, reusable data streams with the industry’s only cloud-native, serverless Flink
service
Data Portal in Stream
Governance
11
Seamlessly and securely request
access to data streams and trigger an
approval workflow that connects the
user with the data owner, all within the
Confluent Cloud UI
Easily build and manage data products
to power streaming pipelines and
applications by understanding,
accessing, and enriching existing data
streams
Complete
Safely unlock data and increase
developer productivity with a
self-service, data-centric portal for
discovering, accessing, and enriching
real-time data streams flowing across
your organization
Search, discover, and explore existing
topics, tags, and metadata across the
organization with end-to-end visibility to
choose the data most relevant for your
projects
Coming Soon
Introducing Data Portal in Stream Governance
Access your data streams through a developer-friendly, self-service UI
Search, discover, and
explore existing topics,
tags, and metadata
across the organization
Seamlessly request
access to data streams
and trigger an approval
workflow
Understand, access, & enrich
data streams to power
real-time data streaming
pipelines and applications
Bidirectional Cluster
Linking
13
Optimize disaster recovery and
increase reliability with bi-directional
cluster linking
Facilitate seamless consumer
migration with retained offsets for
consistent data processing with
Bi-directional cluster links
Increase efficiency and reduce data
recovery time by eliminating the need
for custom code
Streamline security configuration with
support for DR and active/active
architecture with Bi-directional links
that provides outbound and inbound
connections
Everywhere
**Note - bi-directional cluster linking is available for new cluster links only,
existing cluster link need to be deleted and re-activated to obtain this
functionality.
Enhanced Disaster Recovery Capabilities with
Bidirectional Cluster Linking
14
Cluster Link
bidirectional
Connection and Authentication
Connection and Authentication
Cluster A Cluster B
Applications
in region B
Cluster A Cluster B
Cluster Link
bidirectional
Topics on
Cluster A
Mirror
Topics on
Cluster B
Mirror Topics
on Cluster A
Topics on
Cluster B
ACLs / RBAC for Cluster
B
API Key or OAuth for Cluster
A
API Key or OAuth for Cluster B
ACLs / RBAC for Cluster A
Applications
in region A
Data &
Metadata
Data &
Metadata
Cluster Linking with
AWS Private Link
15
Simplified setup: Utilize Network Link
Service and Endpoint for a reliable
connection between clusters
Enhanced network-level security: AWS
PrivateLink isolates Confluent Cloud
clusters, preventing external resources
and Cluster Linking access
Seamless cluster linking: Establish a
secure networking path between
separate Confluent Cloud networks for
efficient data exchange
Everywhere
Easily stream data between regions,
teams or environments within AWS
private networks
The Confluent Q3 ‘23 Launch
Deliver Intelligent, Secure, and Cost-effective Data Pipelines
Cloud-Native Complete Everywhere
Storage Price Reduction: Cost-effectively store data at any scale without growing compute at 20% lower prices
Easily build high-quality, reusable data streams with the industry’s only cloud-native, serverless Flink
service
Apache Flink® on CC
(Open Preview)
+
Enterprise Clusters
Secure, cost-effective, and serverless Kafka
powered by the Kora Engine
Confluent Terraform Provider updates
+
Enhance security and compliance while
continuing to reduce operational burden
through automated infrastructure
management
HashiCorp
Sentinel
Integration
Resource
Importer
Data
Catalog
Support
Cloud Audit Logs for Kafka Produce
& Consume
Experience full visibility and control of
sensitive data access in Confluent Cloud with
detailed audit events enabling swift response
to unauthorized access.
Cluster Linking updates
Cluster Linking with AWS Private Link:
Easily stream data between regions, teams or
environments within AWS private networks
Bi-directional Cluster Linking Optimize
disaster recovery and increase reliability with
bi-directional cluster linking
Data Portal in
Stream Governance
Safely unlock data and increase developer
productivity with a self-service, data-centric
portal for discovering, accessing, and
enriching real-time data streams flowing
across your organization
(coming soon)
Partners Q&A
Confluent Service Mesh
Roman Schmitz, November 2023
What is the Confluent Service
Mesh (CSM)?
“A service mesh is a tool for adding observability, security,
and reliability features to “cloud native” applications by
transparently inserting this functionality at the platform
layer rather than the application layer. The service mesh is
rapidly becoming a standard part of the cloud native stack,
especially for Kubernetes adopters.”
20
-linkerd.io
“A service mesh is a tool for adding observability, security,
and reliability features to “cloud native” applications by
transparently inserting this functionality at the platform
layer rather than the application layer. The service mesh is
rapidly becoming a standard part of the cloud native stack,
especially for Kubernetes adopters.”
21
-linkerd.io
“A service mesh is a tool for adding observability, security,
and reliability features to “cloud native” applications by
transparently inserting this functionality at the platform
layer rather than the application layer. The service mesh is
rapidly becoming a standard part of the cloud native stack,
especially for Kubernetes adopters.”
22
-linkerd.io
Life as we know it
Producer Consumer
With CSM in the Mix
Producer Consumer
Pluggable
Code
CSM
Pluggable
Code
CSM
Kafka
Broker
port 9092
Kafka
Broker
port 9092
Kafka
Broker
port 9092
Confluent Service Mesh at a glance
Confluent Service
Mesh
Producer
Consumer
listener
port
30001
Pluggable
Code
listener
port
30002
Pluggable
Code
listener
port
30003
Pluggable
Code
Kafka Startup
Kafka
Broker
Get Metadata
Client
Return Metadata
Metadata Response
{
"Brokers": [
{
"NodeId": 0,
"Host": "broker0.yourdomain.com",
"Port": 9092
},
{
"NodeId": 1,
"Host": "broker1.yourdomain.com",
"Port": 9092
},
{
"NodeId": 2,
"Host": "broker2.yourdomain.com",
"Port": 9092
}
],
"Topics": [],
…
}
Connect to one of the
brokers
Kafka Startup With CSM
Return Metadata
Kafka
Broker
CSM
Get Metadata
Client
Modify Metadata
Return Metadata
Modified Metadata Response
{
"Brokers": [
{
"NodeId": 0,
"Host": "csm.yourdomain.com",
"Port": 30001
},
{
"NodeId": 1,
"Host": "csm.yourdomain.com",
"Port": 30002
},
{
"NodeId": 2,
"Host": "csm.yourdomain.com",
"Port": 30003
}
],
"Topics": [],
…
}
Connect to a CSM port
What’s the Pluggable Code?
End-to-End Encryption
Payload-Level Encryption
End-to-end Encryption Features
• Local key management and JKS support
• Gemalto, Hashicorp, many security appliances
• Cloud provider key management service support
• AES, RSA encryption, SHA256 hashing
• AVRO, JSON, Protobuf, XML, String, Byte arrays,
Byte buffer level encryption and tokenization
• Field access control
• Format preserving encryption (NIST SP 800-38G)
• Support for metadata and data classification
• Support for master keys (Encryption of a data key
with a wrapping key)
• Support for key rotation
• Support for event digital signature support to
validate producers
Consumer
Protected
Producer
KMS/Tokenizer
Schema
Registry
Kafka Messages and Serialization
Consumer
Producer
Cleartext
{
"name": "Joe Example",
"address": "123 Main St",
"ssn_id": "123-45-6789",
"account": 678900000234,
"Order_time": 1560070133853,
"current_balance": 67,
"itemid": "Item_9"
}
{
"name": "Joe Example",
"address": "123 Main St",
"ssn_id": "123-45-6789",
"account": 678900000234,
"Order_time": 1560070133853,
"current_balance": 67,
"itemid": "Item_9"
}
Serializer Deserializer
1001001001001000110
1001010100101010001
1001010010010100101
0010101001010010100
1010100101001010101
0101010101001001000
1010011101101001010
1011110
Kafka Messages with encryption
Consumer
Producer
{
"name": "Joe Example",
"address": "123 Main St",
"ssn_id": "123-45-6789",
"account": 678900000234,
"Order_time": 1560070133853,
"current_balance": 67,
"itemid": "Item_9"
}
{
"name": "Joe Example",
"address": "123 Main St",
"ssn_id": "123-45-6789",
"account": 678900000234,
"Order_time": 1560070133853,
"current_balance": 67,
"itemid": "Item_9"
}
Serializer Deserializer
1100100100110010001
1010010101001010100
0110010100100101001
0100101010010100101
0010101001010010101
0101010101010010010
0010100111011010010
101011110
Protected
Encryption Decryption
Message-level encryption
{
"name": "Joe Example",
"address": "123 Main St",
"ssn_id": "123-45-6789",
"account": 678900000234,
"Order_time": 1560070133853,
"current_balance": 67,
"itemid": "Item_9"
}
Generate
Data Key
pPYP7QM+LjMfjJ+QdOrLF3VTjMy1sWPtf
epEXXwqkxXrnIbT1iEuzas2J/aOlUv7md
7YFP4Zq5PbrWWTLKeQDRlBVCOBacD15jl
pcME0EONfErWd/CljAaTtCEnGRtfKsCHx
0zasCvXK3G0v15GdptqEGoREtXpea5f9q
M8nYXc1tQbjX4mKP0nB/aVQSmKLXBeEU3
KaiioyXsT3Vsr+tLSCWO76Tfhfaum8Ue4
F5WKPD3svJA==
{
"name": "Joe Example",
"address": "123 Main St",
"ssn_id": "123-45-6789",
"account": 678900000234,
"Order_time": 1560070133853,
"current_balance": 67,
"itemid": "Item_9"
}
Info added to Metadata:
Encrypted Data Key, version, hash
Key Exchange Process
Kafka
Broker
Key
Store/KMS
Get Master Key
Key
Store/KMS
Encryption
Decryption
Get Data Key
Secured
Serializer
Encrypt Event
Encrypt Data Key
Send encrypted event and encrypted data key
Encryption
Decryption
Secured
Deserializer
Fetch Events
Get Master Key
Decrypt Data Key
Decrypt Event
Use decrypted data
key for decryption
Use data key for
encryption
Use master key for
decryption
Use master key
for encryption
Data Protection
with Confluent
Service Mesh
and Encryption
accelerator
36
CSM producer sidecar is
responsible for data
protection independently
of the client type.
Protected
Producer Consumer
KMS/Tokenizer
CSM consumer sidecar is
responsible for safely
exposing data in clear and
can also handle field
access control.
CSM CSM
Field-Level Encryption
Field-level protection
{
"name": "Joe Example",
"address": "123 Main St",
"ssn_id": "123-45-6789",
"account": "678900000234",
"Order_time": 1560070133853,
"current_balance": 67,
"itemid": "Item_9"
}
Generate
Data Key
{
"name": "Hyt Piqdfggr",
"address": "852 Jdrf Wd",
"ssn_id": "dKI4gflV6r339Q==",
"account": "PrM1vyf/CxwoqQ==",
"Order_time": 1560070133853,
"current_balance": 67,
"itemid": "Item_9"
}
{
"name": "Joe Example",
"address": "123 Main St",
"ssn_id": "123-45-6789",
"account": "678900000234",
"Order_time": 1560070133853,
"current_balance": 67,
"itemid": "Item_9"
}
Protected
Producer Consumer
KMS/Tokenizer
CSM CSM
Data Protection with Access Control via CSM
Original message
Original message
{
"name": "Joe Example",
"address": "123 Main St",
"ssn_id": "123-45-6789",
"account": "678900000234",
"Order_time": 1560070133853,
"current_balance": 67,
"itemid": "Item_9"
}
{
"name": "Hyt Piqdfggr",
"address": "852 Jdrf Wd",
"ssn_id": "dKI4gflV6r339Q==",
"account": "PrM1vyf/CxwoqQ==",
"Order_time": 1560070133853,
"current_balance": 67,
"itemid": "Item_9"
}
Protected
{
"name": "Joe Example",
"address": "123 Main St",
"ssn_id": "123-45-6789",
"account": "678900000234",
"Order_time": 1560070133853,
"current_balance": 67,
"itemid": "Item_9"
}
{
"name": "Joe Example",
"address": "123 Main St",
"ssn_id": "dKI4gflV6r339Q==",
"account": "PrM1vyf/CxwoqQ==",
"Order_time": 1560070133853,
"current_balance": 67,
"itemid": "Item_9"
}
Original message
with Access Control
OPA - Open Policy Agent
https://www.openpolicyagent.org/
OPA testing and examples: The Rego Playground
Policy Based Field Level Access Control
Which fields
should be
hidden or
redacted?
Producer Consumer
Open Policy Agent
Pluggable
Code
Confluent Service
Mesh
Pluggable
Code
Confluent Service
Mesh
USA
financial
Policy Based Field Level Access Control
Original message
{
"name": "Joe Example",
"address": "123 Main St",
"ssn_id": "123-45-6789",
"account": "678900000234",
"Order_time": 1560070133853,
"current_balance": 67,
"itemid": "Item_9"
“country”: “usa”
}
{
"account": "678900000234",
"Order_time": 1560070133853,
"itemid": "Item_9"
}
{
"name": "Joe Example",
"address": "123 Main St",
"ssn_id": "123-45-6789",
"account": "678900000234",
"Order_time": 1560070133853,
"current_balance": 67,
"itemid": "Item_9"
}
USA
financial
pii
Brazil
financial
pii
Open Policy
Agent
nothing sent
Pluggable
Code
Confluent
Service Mesh
Integration with Data catalogs, classification
Data classification
{
"type":"record",
"name":"DataClassifications",
"classifications":{
"PII":{
"encrypt":{
"key":"SamplePIIKey",
"wrapping.key":"RSAPII"
},
"classifications":{
"Personal":{
"tokenize":{ }
},
"Financial":{
"encrypt":{
"key":"SampleFinancialKey",
"wrapping.key":"RSAPIIFinancial"
}
}
}
},
"Protected": {
"encrypt": {
"authorizer.class": "classNameHere",
"authorizer.deny": false,
"opa.module.name": "classification",
"opa.rego": "/csm/classification.rego",
"opa.query": "data.classification.allow"
}
}
},
"fields":[ ]
Data Catalog
{
"type":"record",
"name":"ADataCatalog",
"namespace":"com.mybusiness",
"fields":[
{
"name":"SSN",
"type":"string",
"classifications": ["PII/Financial",
“Protected”]
},
{
"name":"Name",
"type":"string",
"classifications": ["PII/Personal",
“Protected”]
},
{
"name":"Address",
"type":"string",
"classifications": ["PII/Personal",
“Protected”]
},
{
"name":"Account",
"type":"string",
"classifications": ["PII/Financial",
“Protected”]
PII/Personal Name: Joe Example
PII/Personal Address: 123 Main St
CustID: 12345
PII/Financial SSN: 123-45-6789
Persona: 56A
Credit: 780
PII/Financial Acct #: 3456789
Current Balance: 0
PII/Personal Name: Hyt Piqdfggr
PII/Personal Address: 852 Jdrf Wd
CustID: 12345
PII/Financial SSN: dKI4gflV6r339Q==
Persona: 56A
Credit: 780
PII/Financial Acct #: PrM1vyf/CxwoqQ==
Current Balance: 0
OPA Configuration and Integration
Link OPA Policies in Classifications
Add OPA Policies (rego)
Local OPA module (Session Authorizer)
local path to rego file
rego path (decision,
package)
Authentication Swapping
Mutual TLS (mTLS) or Kerberos
Producer Consumer
MTLS /
Kerberos
MTLS /
Kerberos
O
N
PREM
O
N
LY
🤬
FAIL
With CSM in the Mix
Client
Pluggable
Code
CSM
MTLS
principal
User1 => key/secret
User2 => key/secret
SASL
(key/secret
)
Lookup Auth from Principal
during
SSL
H
andshake
Example CSM MTLS Flow
Extract Principal
from Cert
Some
Database
CSM
SSL Handshake
Client
Lookup key/secret
from DB with Principal
as key
Return key/secret
Confluent
Cloud
Authenticate sasl
with key/secret
Finish Handshake
Example: CSM Auth Swapping Configurations
…
csm.ssl=true
csm.ssl.enabled=true
csm.ssl.truststore.location=${truststore}
csm.ssl.truststore.password=confluent
csm.ssl.keystore.location=${keystore}
csm.ssl.keystore.password=confluent
csm.ssl.key.password=confluent
csm.ssl.client.auth=required
csm.ssl.principal.mapping.rules: RULE:^CN=([a-zA-Z.0-9@-]+).*$/$1/,DEFAULT
…
csm.authorizers=vaultAuth
vaultAuth.class=io.confluent.csid.csm.auth.VaultAuth
vaultAuth.vault.address=http://vault:8200
vaultAuth.vault.auth.token=vault-plaintext-root-token
vaultAuth.vault.store=secret/testing
vaultAuth.vault.split=/
…
mTLS Configuration
…
csm.ssl=true
sasl.enabled.mechanisms=GSSAPI
csm.sasl.mechanism=GSSAPI
…
csm.authorizers=vaultAuth
vaultAuth.class=io.confluent.csid.csm.auth.VaultAuth
vaultAuth.vault.address=http://vault:8200
vaultAuth.vault.auth.token=vault-plaintext-root-token
vaultAuth.vault.store=secret/testing
vaultAuth.vault.split=/
…
Kerberos Configuration
Examples, Documentation:
https://confluentinc.github.io/csid-csm/
CSM Deployment Options
Typical Hybrid
CSM-Setup
- hybrid setup
- self-managed connect
- local CSM and clients
- ksqlDB and CP in
Confluent Cloud
- ksqlDB on
field-level-encrypted
topics
- AWS KMS for keys (AWS,
Azure, Vault, …)
CSM in a sidecar
- external service writing
to plain-text topic
- kstreams app filtering
data and writing to
encrypted topic
- local client connecting to
CCloud via CSM/directly
CSM as (Gateway)
Service on VMs
- CSM deployed on
containers/VMs
- HA achieved with
multiple CSM-replicas
and LB
- reminder: CSM is
stateless (!)
- Scaling
horizontally/vertically
- load-balancers for
external CSM-access
Client Configuration Examples
Configuration Example: Clients using CSM
bootstrap.servers=pkc-XXXXX.eu-west-1.aws.confluent.cloud:9092
security.protocol=SASL_SSL
bootstrap.servers=csm:30001
security.protocol=SASL_PLAINTEXT
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required
username='<CCLOUD API KEY>' password='<CCLOUD API SECRET>';
sasl.mechanism=PLAIN
# Required for correctness in Apache Kafka clients prior to 2.6
client.dns.lookup=use_all_dns_ips
# Required connection configs for Confluent Cloud Schema Registry
schema.registry.url=https://
basic.auth.credentials.source=USER_INFO
basic.auth.user.info=<SR-KEY>:<SR-SECRET>
bootstrap.servers=pkc-XXXXX.eu-west-1.aws.confluent.cloud:9092
security.protocol=SASL_SSL
bootstrap.servers=csm:30001
security.protocol=SASL_PLAINTEXT
sasl.mechanisms=PLAIN
sasl.username=<CCLOUD API KEY>
sasl.password=<CCLOUD API SECRET>
Java-Client librdkafka (kcat, C#, Python)
Configuration Example: CSM with AWS KMS
csm.ssl=false
broker.ssl=true
bootstrap.servers=pkc-XXXXX.eu-west-1.aws.confluent.cloud:9092
host.name=csm
client.dns.lookup=use_all_dns_ips
sasl.mechanism=PLAIN
security.protocol=SASL_SSL
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required 
username="<CCLOUD API KEY>" 
password="<CCLOUD API SECRET>";
# Required connection configs for Confluent Cloud Schema Registry
schema.registry.url=https://psrc-XXXXX.eu-central-1.aws.confluent.cloud
basic.auth.credentials.source=USER_INFO
basic.auth.user.info=<SR-KEY>:<SR-SECRET>
csm.get.brokers.on.boot=true
csm.port=30001
csm.request.interceptors=in
csm.response.interceptors=out
in.class=io.confluent.csid.csm.encryption.produce.EncryptInterceptor
in.key=rschmitz-symmetric
in.encryption.provider.name=aws
in.schema.registry.url=https://psrc-XXXXX.eu-central-1.aws.confluent.cloud
in.basic.auth.credentials.source=USER_INFO
in.basic.auth.user.info=<SR-KEY>:<SR-SECRET>
in.aws.provider.class = io.confluent.encryption.common.crypto.cipher.impl.AWSKMSProvider
in.aws.provider.use.default.sdk=true
in.aws.provider.region=eu-west-1
in.aws.provider.access.key.id=<AWS API-KEY>
in.aws.provider.secret.key=<AWS API-SECRET>
…
in.class=io.confluent.csid.csm.encryption.produce.EncryptInterceptor
in.key=rschmitz-symmetric
in.encryption.metadata.policy.class=CatalogPolicy
in.encryption.metadata.name=DataCatalog
in.encryption.classifications.name=DataClassifications
in.encryption.provider.name=aws
…
Example csm.properties Field-Level-Configuration
CSM Demo
CSM as a Gateway
to Confluent Cloud
Transparent
end-to-end
encryption
Field-level
authorization and
access-control with
policy-based
field-level
encryption
Use existing
authentication
mechanisms in
cloud migrations
Backup Slides
CSM Ingress on k8s / SNI:
Formatter for Listener Overrides
Use case: Kubernetes Ingress
Ingress Scenario:
● CSM maps each broker to one port
that is exposed as a k8s service
● Ingress will not allow to open ports
dynamically (or more than a few
specific ports at all - 80, 8080, 443)
Solution: Formatter for Listener Overrides
Return Metadata
Kafka
Broker
CSM
Get Metadata
Client
Modify Metadata
Return Metadata
Modified Metadata Response Updated
{
"Brokers": [
{
"NodeId": 0,
"Host": "csm.yourdomain.com",
"Port": 30001
"Host": "b30001.csm.yourdomain.com",
"Port": 9092
},
{
"NodeId": 1,
"Host": "csm.yourdomain.com",
"Port": 30002
"Host": "b30002.csm.yourdomain.com",
"Port": 9092
},
…
],
"Topics": [],
…
}
Connect to a CSM port
Solution: SNI Routing
SNI: Server Name Indication - Wikipedia
(https://github.com/Schm1tz1/sni-routing-examples)
● Hosting of multiple (virtual) services
with same (physical) frontend and
different backends
● Used in Ingress for (de)multiplexing
TCP traffic
● Routing to backend services using
information from TLS handshake
(hello)
● Similar pattern based on HTTP
headers very common in for
Web-Servers
Formatter for Listener Overrides and SNI
Changes to "CSM standard setup":
● CSM configured to return virtual
hostnames that can be mapped
back to internal ports (example:
host.name.formatter=b$p.$h:9092)
● Matching Certificates (wildcard)
● Ingress with SNI rules / mapping for
these hostnames
● External DNS entries (wildcard)
pointing to ingress IPs
Features and KMS E2EE/CSM
Features Comparison
Client-side Encryption CSM-based Encryption
Field-level encryption ✅ (Java,.NET only) ✅
Payload-level encryption ✅ ✅
Tokenization/Masking ✅ (Java,.NET only) ✅
Format-Preserving Encryption ✅ (Java,.NET only) ✅
Supports Kafka Streams ✅ ✅
Supports Kafka Connect JSON, AVRO only ✅
Supports ksqlDB ✅ ✅
Supports REST Proxy ❌ ✅
Popular KMS integrations ✅ (Java,.NET only) ✅
Supports access control ✅ ✅
Node.js, python, C++ support limited features ✅
Other (Go, Ruby) lang support ❌ ✅
Component-based install ✅ Not required
E2EE Libraries
Features and integrations
✅ Feature
included
❌ Feature
prioritized but
not complete
❌ Feature
not included
or prioritized
na Not
Applicable
Q&A with Confluent Professional Services: Confluent Service Mesh

More Related Content

What's hot

Salesforce overview
Salesforce overviewSalesforce overview
Salesforce overview
Ratchata Ardchawuthikulawong
 
Introducing the Salesforce platform
Introducing the Salesforce platformIntroducing the Salesforce platform
Introducing the Salesforce platform
John Stevenson
 
Performing a successful technical debt assessment in Salesforce
Performing a successful technical debt assessment in SalesforcePerforming a successful technical debt assessment in Salesforce
Performing a successful technical debt assessment in Salesforce
Coforge (Erstwhile WHISHWORKS)
 
Salesforce intro session_for_students_v2
Salesforce intro session_for_students_v2Salesforce intro session_for_students_v2
Salesforce intro session_for_students_v2
Stephen Edache Paul
 
Salesforce Sales Cloud: Best Practices to Win More Deals
Salesforce Sales Cloud: Best Practices to Win More DealsSalesforce Sales Cloud: Best Practices to Win More Deals
Salesforce Sales Cloud: Best Practices to Win More Deals
Cloud Analogy
 
Streaming all over the world Real life use cases with Kafka Streams
Streaming all over the world  Real life use cases with Kafka StreamsStreaming all over the world  Real life use cases with Kafka Streams
Streaming all over the world Real life use cases with Kafka Streams
confluent
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023
confluent
 
Salesforce introduction
Salesforce introductionSalesforce introduction
Salesforce introduction
Anas Anas
 
How to govern and secure a Data Mesh?
How to govern and secure a Data Mesh?How to govern and secure a Data Mesh?
How to govern and secure a Data Mesh?
confluent
 
Evolution from EDA to Data Mesh: Data in Motion
Evolution from EDA to Data Mesh: Data in MotionEvolution from EDA to Data Mesh: Data in Motion
Evolution from EDA to Data Mesh: Data in Motion
confluent
 
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka
Real-Life Use Cases & Architectures for Event Streaming with Apache KafkaReal-Life Use Cases & Architectures for Event Streaming with Apache Kafka
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka
Kai Wähner
 
The Top 5 Apache Kafka Use Cases and Architectures in 2022
The Top 5 Apache Kafka Use Cases and Architectures in 2022The Top 5 Apache Kafka Use Cases and Architectures in 2022
The Top 5 Apache Kafka Use Cases and Architectures in 2022
Kai Wähner
 
Cloud Scale Analytics Pitch Deck
Cloud Scale Analytics Pitch DeckCloud Scale Analytics Pitch Deck
Cloud Scale Analytics Pitch Deck
Nicholas Vossburg
 
Introduction to Salesforce Platform - Basic
Introduction to Salesforce Platform - BasicIntroduction to Salesforce Platform - Basic
Introduction to Salesforce Platform - Basic
sanskriti agarwal
 
Extended ECM for SAP Solutions
Extended ECM for SAP SolutionsExtended ECM for SAP Solutions
Extended ECM for SAP Solutions
OpenText
 
Learn to Use Databricks for the Full ML Lifecycle
Learn to Use Databricks for the Full ML LifecycleLearn to Use Databricks for the Full ML Lifecycle
Learn to Use Databricks for the Full ML Lifecycle
Databricks
 
Two-Way Integration with Writable External Objects
Two-Way Integration with Writable External ObjectsTwo-Way Integration with Writable External Objects
Two-Way Integration with Writable External Objects
Salesforce Developers
 
MuleSoft Surat Meetup#45 - Anypoint Flex Gateway as a Kubernetes Ingress Cont...
MuleSoft Surat Meetup#45 - Anypoint Flex Gateway as a Kubernetes Ingress Cont...MuleSoft Surat Meetup#45 - Anypoint Flex Gateway as a Kubernetes Ingress Cont...
MuleSoft Surat Meetup#45 - Anypoint Flex Gateway as a Kubernetes Ingress Cont...
Jitendra Bafna
 
Azure Administrator
Azure AdministratorAzure Administrator
Azure Administrator
Viknaraj Manogararajah
 
Apache Kafka® Use Cases for Financial Services
Apache Kafka® Use Cases for Financial ServicesApache Kafka® Use Cases for Financial Services
Apache Kafka® Use Cases for Financial Services
confluent
 

What's hot (20)

Salesforce overview
Salesforce overviewSalesforce overview
Salesforce overview
 
Introducing the Salesforce platform
Introducing the Salesforce platformIntroducing the Salesforce platform
Introducing the Salesforce platform
 
Performing a successful technical debt assessment in Salesforce
Performing a successful technical debt assessment in SalesforcePerforming a successful technical debt assessment in Salesforce
Performing a successful technical debt assessment in Salesforce
 
Salesforce intro session_for_students_v2
Salesforce intro session_for_students_v2Salesforce intro session_for_students_v2
Salesforce intro session_for_students_v2
 
Salesforce Sales Cloud: Best Practices to Win More Deals
Salesforce Sales Cloud: Best Practices to Win More DealsSalesforce Sales Cloud: Best Practices to Win More Deals
Salesforce Sales Cloud: Best Practices to Win More Deals
 
Streaming all over the world Real life use cases with Kafka Streams
Streaming all over the world  Real life use cases with Kafka StreamsStreaming all over the world  Real life use cases with Kafka Streams
Streaming all over the world Real life use cases with Kafka Streams
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023
 
Salesforce introduction
Salesforce introductionSalesforce introduction
Salesforce introduction
 
How to govern and secure a Data Mesh?
How to govern and secure a Data Mesh?How to govern and secure a Data Mesh?
How to govern and secure a Data Mesh?
 
Evolution from EDA to Data Mesh: Data in Motion
Evolution from EDA to Data Mesh: Data in MotionEvolution from EDA to Data Mesh: Data in Motion
Evolution from EDA to Data Mesh: Data in Motion
 
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka
Real-Life Use Cases & Architectures for Event Streaming with Apache KafkaReal-Life Use Cases & Architectures for Event Streaming with Apache Kafka
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka
 
The Top 5 Apache Kafka Use Cases and Architectures in 2022
The Top 5 Apache Kafka Use Cases and Architectures in 2022The Top 5 Apache Kafka Use Cases and Architectures in 2022
The Top 5 Apache Kafka Use Cases and Architectures in 2022
 
Cloud Scale Analytics Pitch Deck
Cloud Scale Analytics Pitch DeckCloud Scale Analytics Pitch Deck
Cloud Scale Analytics Pitch Deck
 
Introduction to Salesforce Platform - Basic
Introduction to Salesforce Platform - BasicIntroduction to Salesforce Platform - Basic
Introduction to Salesforce Platform - Basic
 
Extended ECM for SAP Solutions
Extended ECM for SAP SolutionsExtended ECM for SAP Solutions
Extended ECM for SAP Solutions
 
Learn to Use Databricks for the Full ML Lifecycle
Learn to Use Databricks for the Full ML LifecycleLearn to Use Databricks for the Full ML Lifecycle
Learn to Use Databricks for the Full ML Lifecycle
 
Two-Way Integration with Writable External Objects
Two-Way Integration with Writable External ObjectsTwo-Way Integration with Writable External Objects
Two-Way Integration with Writable External Objects
 
MuleSoft Surat Meetup#45 - Anypoint Flex Gateway as a Kubernetes Ingress Cont...
MuleSoft Surat Meetup#45 - Anypoint Flex Gateway as a Kubernetes Ingress Cont...MuleSoft Surat Meetup#45 - Anypoint Flex Gateway as a Kubernetes Ingress Cont...
MuleSoft Surat Meetup#45 - Anypoint Flex Gateway as a Kubernetes Ingress Cont...
 
Azure Administrator
Azure AdministratorAzure Administrator
Azure Administrator
 
Apache Kafka® Use Cases for Financial Services
Apache Kafka® Use Cases for Financial ServicesApache Kafka® Use Cases for Financial Services
Apache Kafka® Use Cases for Financial Services
 

Similar to Q&A with Confluent Professional Services: Confluent Service Mesh

Confluent Partner Tech Talk with Reply
Confluent Partner Tech Talk with ReplyConfluent Partner Tech Talk with Reply
Confluent Partner Tech Talk with Reply
confluent
 
Microsoft Windows Azure - Platfrom Appfabric Service Bus And Access Control P...
Microsoft Windows Azure - Platfrom Appfabric Service Bus And Access Control P...Microsoft Windows Azure - Platfrom Appfabric Service Bus And Access Control P...
Microsoft Windows Azure - Platfrom Appfabric Service Bus And Access Control P...
Microsoft Private Cloud
 
Cloud Circle Talk - Enterprise Architecture, Cloud Computing and Integrations
Cloud Circle Talk - Enterprise Architecture, Cloud Computing and IntegrationsCloud Circle Talk - Enterprise Architecture, Cloud Computing and Integrations
Cloud Circle Talk - Enterprise Architecture, Cloud Computing and Integrations
paulfallon
 
(NET303) Optimizing Your Cloud Architecture With Network Strategy
(NET303) Optimizing Your Cloud Architecture With Network Strategy(NET303) Optimizing Your Cloud Architecture With Network Strategy
(NET303) Optimizing Your Cloud Architecture With Network Strategy
Amazon Web Services
 
DIMT '23 Session_Demo_ Latest Innovations Breakout.pdf
DIMT '23 Session_Demo_ Latest Innovations Breakout.pdfDIMT '23 Session_Demo_ Latest Innovations Breakout.pdf
DIMT '23 Session_Demo_ Latest Innovations Breakout.pdf
confluent
 
遷移過程中建置混和雲架構的最佳實踐分享
遷移過程中建置混和雲架構的最佳實踐分享遷移過程中建置混和雲架構的最佳實踐分享
遷移過程中建置混和雲架構的最佳實踐分享
Amazon Web Services
 
XCloudLabs- AWS Overview
XCloudLabs- AWS Overview XCloudLabs- AWS Overview
XCloudLabs- AWS Overview
sangam biradar
 
Running Hybrid Cloud Patterns on AWS
Running Hybrid Cloud Patterns on AWSRunning Hybrid Cloud Patterns on AWS
Running Hybrid Cloud Patterns on AWS
Shiva Narayanaswamy
 
Deep Dive: Hybrid Architectures
Deep Dive: Hybrid ArchitecturesDeep Dive: Hybrid Architectures
Deep Dive: Hybrid Architectures
Amazon Web Services
 
Hybridní cloud s F5 v prostředí kontejnerů
Hybridní cloud s F5 v prostředí kontejnerůHybridní cloud s F5 v prostředí kontejnerů
Hybridní cloud s F5 v prostředí kontejnerů
MarketingArrowECS_CZ
 
IaaS Cloud Providers: A comparative analysis
IaaS Cloud Providers: A comparative analysisIaaS Cloud Providers: A comparative analysis
IaaS Cloud Providers: A comparative analysis
Graisy Biswal
 
Kaleido Platform Overview and Full-stack Blockchain Services
Kaleido Platform Overview and Full-stack Blockchain ServicesKaleido Platform Overview and Full-stack Blockchain Services
Kaleido Platform Overview and Full-stack Blockchain Services
Peter Broadhurst
 
Azure Express Route
Azure Express RouteAzure Express Route
Azure Express Route
Mustafa
 
DIMT 2023 SG - Hands-on Workshop_ Getting started with Confluent Cloud.pdf
DIMT 2023 SG - Hands-on Workshop_ Getting started with Confluent Cloud.pdfDIMT 2023 SG - Hands-on Workshop_ Getting started with Confluent Cloud.pdf
DIMT 2023 SG - Hands-on Workshop_ Getting started with Confluent Cloud.pdf
confluent
 
(NET208) Enable & Secure Your Business Apps via the Hybrid Cloud on AWS
(NET208) Enable & Secure Your Business Apps via the Hybrid Cloud on AWS(NET208) Enable & Secure Your Business Apps via the Hybrid Cloud on AWS
(NET208) Enable & Secure Your Business Apps via the Hybrid Cloud on AWS
Amazon Web Services
 
Bridge to Cloud: Using Apache Kafka to Migrate to AWS
Bridge to Cloud: Using Apache Kafka to Migrate to AWSBridge to Cloud: Using Apache Kafka to Migrate to AWS
Bridge to Cloud: Using Apache Kafka to Migrate to AWS
confluent
 
Cloud Native Apps
Cloud Native AppsCloud Native Apps
Cloud Native Apps
David Chou
 
How a National Transportation Software Provider Migrated a Mission-Critical T...
How a National Transportation Software Provider Migrated a Mission-Critical T...How a National Transportation Software Provider Migrated a Mission-Critical T...
How a National Transportation Software Provider Migrated a Mission-Critical T...
Amazon Web Services
 
Migrating Your Windows Datacenter to AWS
Migrating Your Windows Datacenter to AWSMigrating Your Windows Datacenter to AWS
Migrating Your Windows Datacenter to AWS
2nd Watch
 
Cloud for Kubernetes : Session4
Cloud for Kubernetes : Session4Cloud for Kubernetes : Session4
Cloud for Kubernetes : Session4
WhaTap Labs
 

Similar to Q&A with Confluent Professional Services: Confluent Service Mesh (20)

Confluent Partner Tech Talk with Reply
Confluent Partner Tech Talk with ReplyConfluent Partner Tech Talk with Reply
Confluent Partner Tech Talk with Reply
 
Microsoft Windows Azure - Platfrom Appfabric Service Bus And Access Control P...
Microsoft Windows Azure - Platfrom Appfabric Service Bus And Access Control P...Microsoft Windows Azure - Platfrom Appfabric Service Bus And Access Control P...
Microsoft Windows Azure - Platfrom Appfabric Service Bus And Access Control P...
 
Cloud Circle Talk - Enterprise Architecture, Cloud Computing and Integrations
Cloud Circle Talk - Enterprise Architecture, Cloud Computing and IntegrationsCloud Circle Talk - Enterprise Architecture, Cloud Computing and Integrations
Cloud Circle Talk - Enterprise Architecture, Cloud Computing and Integrations
 
(NET303) Optimizing Your Cloud Architecture With Network Strategy
(NET303) Optimizing Your Cloud Architecture With Network Strategy(NET303) Optimizing Your Cloud Architecture With Network Strategy
(NET303) Optimizing Your Cloud Architecture With Network Strategy
 
DIMT '23 Session_Demo_ Latest Innovations Breakout.pdf
DIMT '23 Session_Demo_ Latest Innovations Breakout.pdfDIMT '23 Session_Demo_ Latest Innovations Breakout.pdf
DIMT '23 Session_Demo_ Latest Innovations Breakout.pdf
 
遷移過程中建置混和雲架構的最佳實踐分享
遷移過程中建置混和雲架構的最佳實踐分享遷移過程中建置混和雲架構的最佳實踐分享
遷移過程中建置混和雲架構的最佳實踐分享
 
XCloudLabs- AWS Overview
XCloudLabs- AWS Overview XCloudLabs- AWS Overview
XCloudLabs- AWS Overview
 
Running Hybrid Cloud Patterns on AWS
Running Hybrid Cloud Patterns on AWSRunning Hybrid Cloud Patterns on AWS
Running Hybrid Cloud Patterns on AWS
 
Deep Dive: Hybrid Architectures
Deep Dive: Hybrid ArchitecturesDeep Dive: Hybrid Architectures
Deep Dive: Hybrid Architectures
 
Hybridní cloud s F5 v prostředí kontejnerů
Hybridní cloud s F5 v prostředí kontejnerůHybridní cloud s F5 v prostředí kontejnerů
Hybridní cloud s F5 v prostředí kontejnerů
 
IaaS Cloud Providers: A comparative analysis
IaaS Cloud Providers: A comparative analysisIaaS Cloud Providers: A comparative analysis
IaaS Cloud Providers: A comparative analysis
 
Kaleido Platform Overview and Full-stack Blockchain Services
Kaleido Platform Overview and Full-stack Blockchain ServicesKaleido Platform Overview and Full-stack Blockchain Services
Kaleido Platform Overview and Full-stack Blockchain Services
 
Azure Express Route
Azure Express RouteAzure Express Route
Azure Express Route
 
DIMT 2023 SG - Hands-on Workshop_ Getting started with Confluent Cloud.pdf
DIMT 2023 SG - Hands-on Workshop_ Getting started with Confluent Cloud.pdfDIMT 2023 SG - Hands-on Workshop_ Getting started with Confluent Cloud.pdf
DIMT 2023 SG - Hands-on Workshop_ Getting started with Confluent Cloud.pdf
 
(NET208) Enable & Secure Your Business Apps via the Hybrid Cloud on AWS
(NET208) Enable & Secure Your Business Apps via the Hybrid Cloud on AWS(NET208) Enable & Secure Your Business Apps via the Hybrid Cloud on AWS
(NET208) Enable & Secure Your Business Apps via the Hybrid Cloud on AWS
 
Bridge to Cloud: Using Apache Kafka to Migrate to AWS
Bridge to Cloud: Using Apache Kafka to Migrate to AWSBridge to Cloud: Using Apache Kafka to Migrate to AWS
Bridge to Cloud: Using Apache Kafka to Migrate to AWS
 
Cloud Native Apps
Cloud Native AppsCloud Native Apps
Cloud Native Apps
 
How a National Transportation Software Provider Migrated a Mission-Critical T...
How a National Transportation Software Provider Migrated a Mission-Critical T...How a National Transportation Software Provider Migrated a Mission-Critical T...
How a National Transportation Software Provider Migrated a Mission-Critical T...
 
Migrating Your Windows Datacenter to AWS
Migrating Your Windows Datacenter to AWSMigrating Your Windows Datacenter to AWS
Migrating Your Windows Datacenter to AWS
 
Cloud for Kubernetes : Session4
Cloud for Kubernetes : Session4Cloud for Kubernetes : Session4
Cloud for Kubernetes : Session4
 

More from confluent

Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutes
confluent
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
confluent
 
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
confluent
 
Santander Stream Processing with Apache Flink
Santander Stream Processing with Apache FlinkSantander Stream Processing with Apache Flink
Santander Stream Processing with Apache Flink
confluent
 
Unlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insightsUnlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insights
confluent
 
Workshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con FlinkWorkshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con Flink
confluent
 
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
confluent
 
AWS Immersion Day Mapfre - Confluent
AWS Immersion Day Mapfre   -   ConfluentAWS Immersion Day Mapfre   -   Confluent
AWS Immersion Day Mapfre - Confluent
confluent
 
Eventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalkEventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalk
confluent
 
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent CloudQ&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
confluent
 
Citi TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep DiveCiti TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep Dive
confluent
 
Build real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with ConfluentBuild real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with Confluent
confluent
 
Citi Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka MicroservicesCiti Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka Microservices
confluent
 
Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3
confluent
 
Citi Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging ModernizationCiti Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging Modernization
confluent
 
Citi Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time dataCiti Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time data
confluent
 
Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2
confluent
 
Confluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with SynthesisConfluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with Synthesis
confluent
 
The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023
confluent
 
The Playful Bond Between REST And Data Streams
The Playful Bond Between REST And Data StreamsThe Playful Bond Between REST And Data Streams
The Playful Bond Between REST And Data Streams
confluent
 

More from confluent (20)

Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutes
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
 
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
 
Santander Stream Processing with Apache Flink
Santander Stream Processing with Apache FlinkSantander Stream Processing with Apache Flink
Santander Stream Processing with Apache Flink
 
Unlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insightsUnlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insights
 
Workshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con FlinkWorkshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con Flink
 
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
 
AWS Immersion Day Mapfre - Confluent
AWS Immersion Day Mapfre   -   ConfluentAWS Immersion Day Mapfre   -   Confluent
AWS Immersion Day Mapfre - Confluent
 
Eventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalkEventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalk
 
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent CloudQ&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
 
Citi TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep DiveCiti TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep Dive
 
Build real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with ConfluentBuild real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with Confluent
 
Citi Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka MicroservicesCiti Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka Microservices
 
Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3
 
Citi Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging ModernizationCiti Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging Modernization
 
Citi Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time dataCiti Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time data
 
Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2
 
Confluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with SynthesisConfluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with Synthesis
 
The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023
 
The Playful Bond Between REST And Data Streams
The Playful Bond Between REST And Data StreamsThe Playful Bond Between REST And Data Streams
The Playful Bond Between REST And Data Streams
 

Recently uploaded

Using Query Store in Azure PostgreSQL to Understand Query Performance
Using Query Store in Azure PostgreSQL to Understand Query PerformanceUsing Query Store in Azure PostgreSQL to Understand Query Performance
Using Query Store in Azure PostgreSQL to Understand Query Performance
Grant Fritchey
 
Preparing Non - Technical Founders for Engaging a Tech Agency
Preparing Non - Technical Founders for Engaging  a  Tech AgencyPreparing Non - Technical Founders for Engaging  a  Tech Agency
Preparing Non - Technical Founders for Engaging a Tech Agency
ISH Technologies
 
Liberarsi dai framework con i Web Component.pptx
Liberarsi dai framework con i Web Component.pptxLiberarsi dai framework con i Web Component.pptx
Liberarsi dai framework con i Web Component.pptx
Massimo Artizzu
 
zOS Mainframe JES2-JES3 JCL-JECL Differences
zOS Mainframe JES2-JES3 JCL-JECL DifferenceszOS Mainframe JES2-JES3 JCL-JECL Differences
zOS Mainframe JES2-JES3 JCL-JECL Differences
YousufSait3
 
Energy consumption of Database Management - Florina Jonuzi
Energy consumption of Database Management - Florina JonuziEnergy consumption of Database Management - Florina Jonuzi
Energy consumption of Database Management - Florina Jonuzi
Green Software Development
 
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
gapen1
 
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CDKuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
rodomar2
 
Hand Rolled Applicative User Validation Code Kata
Hand Rolled Applicative User ValidationCode KataHand Rolled Applicative User ValidationCode Kata
Hand Rolled Applicative User Validation Code Kata
Philip Schwarz
 
What next after learning python programming basics
What next after learning python programming basicsWhat next after learning python programming basics
What next after learning python programming basics
Rakesh Kumar R
 
fiscal year variant fiscal year variant.
fiscal year variant fiscal year variant.fiscal year variant fiscal year variant.
fiscal year variant fiscal year variant.
AnkitaPandya11
 
ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
ALGIT - Assembly Line for Green IT - Numbers, Data, FactsALGIT - Assembly Line for Green IT - Numbers, Data, Facts
ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
Green Software Development
 
How to write a program in any programming language
How to write a program in any programming languageHow to write a program in any programming language
How to write a program in any programming language
Rakesh Kumar R
 
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s EcosystemUI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
Peter Muessig
 
SQL Accounting Software Brochure Malaysia
SQL Accounting Software Brochure MalaysiaSQL Accounting Software Brochure Malaysia
SQL Accounting Software Brochure Malaysia
GohKiangHock
 
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
safelyiotech
 
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling ExtensionsUI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
Peter Muessig
 
Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdfUnveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
brainerhub1
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
Drona Infotech
 
Malibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed RoundMalibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed Round
sjcobrien
 
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
kalichargn70th171
 

Recently uploaded (20)

Using Query Store in Azure PostgreSQL to Understand Query Performance
Using Query Store in Azure PostgreSQL to Understand Query PerformanceUsing Query Store in Azure PostgreSQL to Understand Query Performance
Using Query Store in Azure PostgreSQL to Understand Query Performance
 
Preparing Non - Technical Founders for Engaging a Tech Agency
Preparing Non - Technical Founders for Engaging  a  Tech AgencyPreparing Non - Technical Founders for Engaging  a  Tech Agency
Preparing Non - Technical Founders for Engaging a Tech Agency
 
Liberarsi dai framework con i Web Component.pptx
Liberarsi dai framework con i Web Component.pptxLiberarsi dai framework con i Web Component.pptx
Liberarsi dai framework con i Web Component.pptx
 
zOS Mainframe JES2-JES3 JCL-JECL Differences
zOS Mainframe JES2-JES3 JCL-JECL DifferenceszOS Mainframe JES2-JES3 JCL-JECL Differences
zOS Mainframe JES2-JES3 JCL-JECL Differences
 
Energy consumption of Database Management - Florina Jonuzi
Energy consumption of Database Management - Florina JonuziEnergy consumption of Database Management - Florina Jonuzi
Energy consumption of Database Management - Florina Jonuzi
 
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
 
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CDKuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
 
Hand Rolled Applicative User Validation Code Kata
Hand Rolled Applicative User ValidationCode KataHand Rolled Applicative User ValidationCode Kata
Hand Rolled Applicative User Validation Code Kata
 
What next after learning python programming basics
What next after learning python programming basicsWhat next after learning python programming basics
What next after learning python programming basics
 
fiscal year variant fiscal year variant.
fiscal year variant fiscal year variant.fiscal year variant fiscal year variant.
fiscal year variant fiscal year variant.
 
ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
ALGIT - Assembly Line for Green IT - Numbers, Data, FactsALGIT - Assembly Line for Green IT - Numbers, Data, Facts
ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
 
How to write a program in any programming language
How to write a program in any programming languageHow to write a program in any programming language
How to write a program in any programming language
 
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s EcosystemUI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
 
SQL Accounting Software Brochure Malaysia
SQL Accounting Software Brochure MalaysiaSQL Accounting Software Brochure Malaysia
SQL Accounting Software Brochure Malaysia
 
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
 
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling ExtensionsUI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
 
Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdfUnveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
 
Malibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed RoundMalibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed Round
 
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
 

Q&A with Confluent Professional Services: Confluent Service Mesh

  • 1. Thanks for joining! We’ll get started soon! Technical Enablement Session
  • 4. @yourtwitterhandle | developer.confluent.io Our Partner Technical Enablement offering Scheduled sessions On-demand Join us for these live sessions where our experts will guide you through sessions of different level and will be available to answer your questions. Some examples of sessions are below: • Confluent 101: for new starters • Hybrid Cloud Workshop: learn by doing • Path to Production series , Confluent Cloud workshops series • Product Updates Learn the basics with a guided experience, at your own pace with our learning paths on-demand. You will also find an always growing repository of more advanced presentations to dig-deeper. Some examples are below: • Aware/Novice/Competent Learning paths • Confluent Use Cases • Positioning Confluent Value • Confluent Cloud Networking • … and many more AskTheExpert we’ll offer a channel dedicated to streaming questions • Build CoE inside partners by getting people with similar interest together • Connect with opportunities and discover trends at focus partners • Build a Technical Community • Q&A • Tech Talk
  • 5. @yourtwitterhandle | developer.confluent.io What are the best practices to debug client applications (producers/consumers in general but also Kafka Streams applications)?
  • 8. The Confluent Q3 ‘23 Launch Announcing the latest updates to our cloud-native data streaming platform, Confluent Cloud
  • 9. Confluent Cloud Cloud native data streaming platform built by the founders of Apache Kafka® 9 Cloud-Native Complete Everywhere Stream confidently on the world’s most trusted data streaming platform built by the founders of Apache Kafka©, with resilience, security, compliance, and privacy built-in by default. Cloud Native The 10x Apache Kafka® service: elastic, resilient and performant, powered by the Kora Engine Complete Go above & beyond Kafka with all the essential tools for a complete data streaming platform Everywhere Connect your data in real time with a platform that spans from on-prem to cloud and across clouds
  • 10. The Confluent Q3 ‘23 Launch Deliver Intelligent, Secure, and Cost-effective Data Pipelines 10 Cloud-Native Complete Everywhere Storage Price Reduction: Cost-effectively store data at any scale without growing compute at 20% lower prices CC for Apache Flink® (Open Preview) + Enterprise Clusters Secure, cost-effective, and serverless Kafka powered by the Kora Engine Confluent Terraform Provider updates + Enhance security and compliance while continuing to reduce operational burden through automated infrastructure management HashiCorp Sentinel Integration Resource Importer Data Catalog Support Cloud Audit Logs for Kafka Produce & Consume Experience full visibility and control of sensitive data access in Confluent Cloud with detailed audit events enabling swift response to unauthorized access. Cluster Linking updates Cluster Linking with AWS Private Link: Easily stream data between regions, teams or environments within AWS private networks Bi-directional Cluster Linking Optimize disaster recovery and increase reliability with bi-directional cluster linking Data Portal in Stream Governance Safely unlock data and increase developer productivity with a self-service, data-centric portal for discovering, accessing, and enriching real-time data streams flowing across your organization (coming soon) Easily build high-quality, reusable data streams with the industry’s only cloud-native, serverless Flink service
  • 11. Data Portal in Stream Governance 11 Seamlessly and securely request access to data streams and trigger an approval workflow that connects the user with the data owner, all within the Confluent Cloud UI Easily build and manage data products to power streaming pipelines and applications by understanding, accessing, and enriching existing data streams Complete Safely unlock data and increase developer productivity with a self-service, data-centric portal for discovering, accessing, and enriching real-time data streams flowing across your organization Search, discover, and explore existing topics, tags, and metadata across the organization with end-to-end visibility to choose the data most relevant for your projects Coming Soon
  • 12. Introducing Data Portal in Stream Governance Access your data streams through a developer-friendly, self-service UI Search, discover, and explore existing topics, tags, and metadata across the organization Seamlessly request access to data streams and trigger an approval workflow Understand, access, & enrich data streams to power real-time data streaming pipelines and applications
  • 13. Bidirectional Cluster Linking 13 Optimize disaster recovery and increase reliability with bi-directional cluster linking Facilitate seamless consumer migration with retained offsets for consistent data processing with Bi-directional cluster links Increase efficiency and reduce data recovery time by eliminating the need for custom code Streamline security configuration with support for DR and active/active architecture with Bi-directional links that provides outbound and inbound connections Everywhere **Note - bi-directional cluster linking is available for new cluster links only, existing cluster link need to be deleted and re-activated to obtain this functionality.
  • 14. Enhanced Disaster Recovery Capabilities with Bidirectional Cluster Linking 14 Cluster Link bidirectional Connection and Authentication Connection and Authentication Cluster A Cluster B Applications in region B Cluster A Cluster B Cluster Link bidirectional Topics on Cluster A Mirror Topics on Cluster B Mirror Topics on Cluster A Topics on Cluster B ACLs / RBAC for Cluster B API Key or OAuth for Cluster A API Key or OAuth for Cluster B ACLs / RBAC for Cluster A Applications in region A Data & Metadata Data & Metadata
  • 15. Cluster Linking with AWS Private Link 15 Simplified setup: Utilize Network Link Service and Endpoint for a reliable connection between clusters Enhanced network-level security: AWS PrivateLink isolates Confluent Cloud clusters, preventing external resources and Cluster Linking access Seamless cluster linking: Establish a secure networking path between separate Confluent Cloud networks for efficient data exchange Everywhere Easily stream data between regions, teams or environments within AWS private networks
  • 16. The Confluent Q3 ‘23 Launch Deliver Intelligent, Secure, and Cost-effective Data Pipelines Cloud-Native Complete Everywhere Storage Price Reduction: Cost-effectively store data at any scale without growing compute at 20% lower prices Easily build high-quality, reusable data streams with the industry’s only cloud-native, serverless Flink service Apache Flink® on CC (Open Preview) + Enterprise Clusters Secure, cost-effective, and serverless Kafka powered by the Kora Engine Confluent Terraform Provider updates + Enhance security and compliance while continuing to reduce operational burden through automated infrastructure management HashiCorp Sentinel Integration Resource Importer Data Catalog Support Cloud Audit Logs for Kafka Produce & Consume Experience full visibility and control of sensitive data access in Confluent Cloud with detailed audit events enabling swift response to unauthorized access. Cluster Linking updates Cluster Linking with AWS Private Link: Easily stream data between regions, teams or environments within AWS private networks Bi-directional Cluster Linking Optimize disaster recovery and increase reliability with bi-directional cluster linking Data Portal in Stream Governance Safely unlock data and increase developer productivity with a self-service, data-centric portal for discovering, accessing, and enriching real-time data streams flowing across your organization (coming soon)
  • 18. Confluent Service Mesh Roman Schmitz, November 2023
  • 19. What is the Confluent Service Mesh (CSM)?
  • 20. “A service mesh is a tool for adding observability, security, and reliability features to “cloud native” applications by transparently inserting this functionality at the platform layer rather than the application layer. The service mesh is rapidly becoming a standard part of the cloud native stack, especially for Kubernetes adopters.” 20 -linkerd.io
  • 21. “A service mesh is a tool for adding observability, security, and reliability features to “cloud native” applications by transparently inserting this functionality at the platform layer rather than the application layer. The service mesh is rapidly becoming a standard part of the cloud native stack, especially for Kubernetes adopters.” 21 -linkerd.io
  • 22. “A service mesh is a tool for adding observability, security, and reliability features to “cloud native” applications by transparently inserting this functionality at the platform layer rather than the application layer. The service mesh is rapidly becoming a standard part of the cloud native stack, especially for Kubernetes adopters.” 22 -linkerd.io
  • 23. Life as we know it Producer Consumer
  • 24. With CSM in the Mix Producer Consumer Pluggable Code CSM Pluggable Code CSM
  • 25. Kafka Broker port 9092 Kafka Broker port 9092 Kafka Broker port 9092 Confluent Service Mesh at a glance Confluent Service Mesh Producer Consumer listener port 30001 Pluggable Code listener port 30002 Pluggable Code listener port 30003 Pluggable Code
  • 26. Kafka Startup Kafka Broker Get Metadata Client Return Metadata Metadata Response { "Brokers": [ { "NodeId": 0, "Host": "broker0.yourdomain.com", "Port": 9092 }, { "NodeId": 1, "Host": "broker1.yourdomain.com", "Port": 9092 }, { "NodeId": 2, "Host": "broker2.yourdomain.com", "Port": 9092 } ], "Topics": [], … } Connect to one of the brokers
  • 27. Kafka Startup With CSM Return Metadata Kafka Broker CSM Get Metadata Client Modify Metadata Return Metadata Modified Metadata Response { "Brokers": [ { "NodeId": 0, "Host": "csm.yourdomain.com", "Port": 30001 }, { "NodeId": 1, "Host": "csm.yourdomain.com", "Port": 30002 }, { "NodeId": 2, "Host": "csm.yourdomain.com", "Port": 30003 } ], "Topics": [], … } Connect to a CSM port
  • 31. End-to-end Encryption Features • Local key management and JKS support • Gemalto, Hashicorp, many security appliances • Cloud provider key management service support • AES, RSA encryption, SHA256 hashing • AVRO, JSON, Protobuf, XML, String, Byte arrays, Byte buffer level encryption and tokenization • Field access control • Format preserving encryption (NIST SP 800-38G) • Support for metadata and data classification • Support for master keys (Encryption of a data key with a wrapping key) • Support for key rotation • Support for event digital signature support to validate producers Consumer Protected Producer KMS/Tokenizer Schema Registry
  • 32. Kafka Messages and Serialization Consumer Producer Cleartext { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": 678900000234, "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": 678900000234, "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } Serializer Deserializer 1001001001001000110 1001010100101010001 1001010010010100101 0010101001010010100 1010100101001010101 0101010101001001000 1010011101101001010 1011110
  • 33. Kafka Messages with encryption Consumer Producer { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": 678900000234, "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": 678900000234, "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } Serializer Deserializer 1100100100110010001 1010010101001010100 0110010100100101001 0100101010010100101 0010101001010010101 0101010101010010010 0010100111011010010 101011110 Protected Encryption Decryption
  • 34. Message-level encryption { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": 678900000234, "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } Generate Data Key pPYP7QM+LjMfjJ+QdOrLF3VTjMy1sWPtf epEXXwqkxXrnIbT1iEuzas2J/aOlUv7md 7YFP4Zq5PbrWWTLKeQDRlBVCOBacD15jl pcME0EONfErWd/CljAaTtCEnGRtfKsCHx 0zasCvXK3G0v15GdptqEGoREtXpea5f9q M8nYXc1tQbjX4mKP0nB/aVQSmKLXBeEU3 KaiioyXsT3Vsr+tLSCWO76Tfhfaum8Ue4 F5WKPD3svJA== { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": 678900000234, "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } Info added to Metadata: Encrypted Data Key, version, hash
  • 35. Key Exchange Process Kafka Broker Key Store/KMS Get Master Key Key Store/KMS Encryption Decryption Get Data Key Secured Serializer Encrypt Event Encrypt Data Key Send encrypted event and encrypted data key Encryption Decryption Secured Deserializer Fetch Events Get Master Key Decrypt Data Key Decrypt Event Use decrypted data key for decryption Use data key for encryption Use master key for decryption Use master key for encryption
  • 36. Data Protection with Confluent Service Mesh and Encryption accelerator 36 CSM producer sidecar is responsible for data protection independently of the client type. Protected Producer Consumer KMS/Tokenizer CSM consumer sidecar is responsible for safely exposing data in clear and can also handle field access control. CSM CSM
  • 38. Field-level protection { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": "678900000234", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } Generate Data Key { "name": "Hyt Piqdfggr", "address": "852 Jdrf Wd", "ssn_id": "dKI4gflV6r339Q==", "account": "PrM1vyf/CxwoqQ==", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": "678900000234", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } Protected Producer Consumer KMS/Tokenizer CSM CSM
  • 39. Data Protection with Access Control via CSM Original message Original message { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": "678900000234", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } { "name": "Hyt Piqdfggr", "address": "852 Jdrf Wd", "ssn_id": "dKI4gflV6r339Q==", "account": "PrM1vyf/CxwoqQ==", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } Protected { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": "678900000234", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } { "name": "Joe Example", "address": "123 Main St", "ssn_id": "dKI4gflV6r339Q==", "account": "PrM1vyf/CxwoqQ==", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } Original message with Access Control
  • 40. OPA - Open Policy Agent https://www.openpolicyagent.org/ OPA testing and examples: The Rego Playground
  • 41. Policy Based Field Level Access Control Which fields should be hidden or redacted? Producer Consumer Open Policy Agent Pluggable Code Confluent Service Mesh Pluggable Code Confluent Service Mesh
  • 42. USA financial Policy Based Field Level Access Control Original message { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": "678900000234", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" “country”: “usa” } { "account": "678900000234", "Order_time": 1560070133853, "itemid": "Item_9" } { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": "678900000234", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } USA financial pii Brazil financial pii Open Policy Agent nothing sent Pluggable Code Confluent Service Mesh
  • 43. Integration with Data catalogs, classification Data classification { "type":"record", "name":"DataClassifications", "classifications":{ "PII":{ "encrypt":{ "key":"SamplePIIKey", "wrapping.key":"RSAPII" }, "classifications":{ "Personal":{ "tokenize":{ } }, "Financial":{ "encrypt":{ "key":"SampleFinancialKey", "wrapping.key":"RSAPIIFinancial" } } } }, "Protected": { "encrypt": { "authorizer.class": "classNameHere", "authorizer.deny": false, "opa.module.name": "classification", "opa.rego": "/csm/classification.rego", "opa.query": "data.classification.allow" } } }, "fields":[ ] Data Catalog { "type":"record", "name":"ADataCatalog", "namespace":"com.mybusiness", "fields":[ { "name":"SSN", "type":"string", "classifications": ["PII/Financial", “Protected”] }, { "name":"Name", "type":"string", "classifications": ["PII/Personal", “Protected”] }, { "name":"Address", "type":"string", "classifications": ["PII/Personal", “Protected”] }, { "name":"Account", "type":"string", "classifications": ["PII/Financial", “Protected”] PII/Personal Name: Joe Example PII/Personal Address: 123 Main St CustID: 12345 PII/Financial SSN: 123-45-6789 Persona: 56A Credit: 780 PII/Financial Acct #: 3456789 Current Balance: 0 PII/Personal Name: Hyt Piqdfggr PII/Personal Address: 852 Jdrf Wd CustID: 12345 PII/Financial SSN: dKI4gflV6r339Q== Persona: 56A Credit: 780 PII/Financial Acct #: PrM1vyf/CxwoqQ== Current Balance: 0
  • 44. OPA Configuration and Integration Link OPA Policies in Classifications Add OPA Policies (rego) Local OPA module (Session Authorizer) local path to rego file rego path (decision, package)
  • 46. Mutual TLS (mTLS) or Kerberos Producer Consumer MTLS / Kerberos MTLS / Kerberos O N PREM O N LY 🤬 FAIL
  • 47. With CSM in the Mix Client Pluggable Code CSM MTLS principal User1 => key/secret User2 => key/secret SASL (key/secret ) Lookup Auth from Principal during SSL H andshake
  • 48. Example CSM MTLS Flow Extract Principal from Cert Some Database CSM SSL Handshake Client Lookup key/secret from DB with Principal as key Return key/secret Confluent Cloud Authenticate sasl with key/secret Finish Handshake
  • 49. Example: CSM Auth Swapping Configurations … csm.ssl=true csm.ssl.enabled=true csm.ssl.truststore.location=${truststore} csm.ssl.truststore.password=confluent csm.ssl.keystore.location=${keystore} csm.ssl.keystore.password=confluent csm.ssl.key.password=confluent csm.ssl.client.auth=required csm.ssl.principal.mapping.rules: RULE:^CN=([a-zA-Z.0-9@-]+).*$/$1/,DEFAULT … csm.authorizers=vaultAuth vaultAuth.class=io.confluent.csid.csm.auth.VaultAuth vaultAuth.vault.address=http://vault:8200 vaultAuth.vault.auth.token=vault-plaintext-root-token vaultAuth.vault.store=secret/testing vaultAuth.vault.split=/ … mTLS Configuration … csm.ssl=true sasl.enabled.mechanisms=GSSAPI csm.sasl.mechanism=GSSAPI … csm.authorizers=vaultAuth vaultAuth.class=io.confluent.csid.csm.auth.VaultAuth vaultAuth.vault.address=http://vault:8200 vaultAuth.vault.auth.token=vault-plaintext-root-token vaultAuth.vault.store=secret/testing vaultAuth.vault.split=/ … Kerberos Configuration Examples, Documentation: https://confluentinc.github.io/csid-csm/
  • 51. Typical Hybrid CSM-Setup - hybrid setup - self-managed connect - local CSM and clients - ksqlDB and CP in Confluent Cloud - ksqlDB on field-level-encrypted topics - AWS KMS for keys (AWS, Azure, Vault, …)
  • 52. CSM in a sidecar - external service writing to plain-text topic - kstreams app filtering data and writing to encrypted topic - local client connecting to CCloud via CSM/directly
  • 53. CSM as (Gateway) Service on VMs - CSM deployed on containers/VMs - HA achieved with multiple CSM-replicas and LB - reminder: CSM is stateless (!) - Scaling horizontally/vertically - load-balancers for external CSM-access
  • 55. Configuration Example: Clients using CSM bootstrap.servers=pkc-XXXXX.eu-west-1.aws.confluent.cloud:9092 security.protocol=SASL_SSL bootstrap.servers=csm:30001 security.protocol=SASL_PLAINTEXT sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='<CCLOUD API KEY>' password='<CCLOUD API SECRET>'; sasl.mechanism=PLAIN # Required for correctness in Apache Kafka clients prior to 2.6 client.dns.lookup=use_all_dns_ips # Required connection configs for Confluent Cloud Schema Registry schema.registry.url=https:// basic.auth.credentials.source=USER_INFO basic.auth.user.info=<SR-KEY>:<SR-SECRET> bootstrap.servers=pkc-XXXXX.eu-west-1.aws.confluent.cloud:9092 security.protocol=SASL_SSL bootstrap.servers=csm:30001 security.protocol=SASL_PLAINTEXT sasl.mechanisms=PLAIN sasl.username=<CCLOUD API KEY> sasl.password=<CCLOUD API SECRET> Java-Client librdkafka (kcat, C#, Python)
  • 56. Configuration Example: CSM with AWS KMS csm.ssl=false broker.ssl=true bootstrap.servers=pkc-XXXXX.eu-west-1.aws.confluent.cloud:9092 host.name=csm client.dns.lookup=use_all_dns_ips sasl.mechanism=PLAIN security.protocol=SASL_SSL sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<CCLOUD API KEY>" password="<CCLOUD API SECRET>"; # Required connection configs for Confluent Cloud Schema Registry schema.registry.url=https://psrc-XXXXX.eu-central-1.aws.confluent.cloud basic.auth.credentials.source=USER_INFO basic.auth.user.info=<SR-KEY>:<SR-SECRET> csm.get.brokers.on.boot=true csm.port=30001 csm.request.interceptors=in csm.response.interceptors=out in.class=io.confluent.csid.csm.encryption.produce.EncryptInterceptor in.key=rschmitz-symmetric in.encryption.provider.name=aws in.schema.registry.url=https://psrc-XXXXX.eu-central-1.aws.confluent.cloud in.basic.auth.credentials.source=USER_INFO in.basic.auth.user.info=<SR-KEY>:<SR-SECRET> in.aws.provider.class = io.confluent.encryption.common.crypto.cipher.impl.AWSKMSProvider in.aws.provider.use.default.sdk=true in.aws.provider.region=eu-west-1 in.aws.provider.access.key.id=<AWS API-KEY> in.aws.provider.secret.key=<AWS API-SECRET> … in.class=io.confluent.csid.csm.encryption.produce.EncryptInterceptor in.key=rschmitz-symmetric in.encryption.metadata.policy.class=CatalogPolicy in.encryption.metadata.name=DataCatalog in.encryption.classifications.name=DataClassifications in.encryption.provider.name=aws … Example csm.properties Field-Level-Configuration
  • 58. CSM as a Gateway to Confluent Cloud Transparent end-to-end encryption Field-level authorization and access-control with policy-based field-level encryption Use existing authentication mechanisms in cloud migrations
  • 59.
  • 61. CSM Ingress on k8s / SNI: Formatter for Listener Overrides
  • 62. Use case: Kubernetes Ingress Ingress Scenario: ● CSM maps each broker to one port that is exposed as a k8s service ● Ingress will not allow to open ports dynamically (or more than a few specific ports at all - 80, 8080, 443)
  • 63. Solution: Formatter for Listener Overrides Return Metadata Kafka Broker CSM Get Metadata Client Modify Metadata Return Metadata Modified Metadata Response Updated { "Brokers": [ { "NodeId": 0, "Host": "csm.yourdomain.com", "Port": 30001 "Host": "b30001.csm.yourdomain.com", "Port": 9092 }, { "NodeId": 1, "Host": "csm.yourdomain.com", "Port": 30002 "Host": "b30002.csm.yourdomain.com", "Port": 9092 }, … ], "Topics": [], … } Connect to a CSM port
  • 64. Solution: SNI Routing SNI: Server Name Indication - Wikipedia (https://github.com/Schm1tz1/sni-routing-examples) ● Hosting of multiple (virtual) services with same (physical) frontend and different backends ● Used in Ingress for (de)multiplexing TCP traffic ● Routing to backend services using information from TLS handshake (hello) ● Similar pattern based on HTTP headers very common in for Web-Servers
  • 65. Formatter for Listener Overrides and SNI Changes to "CSM standard setup": ● CSM configured to return virtual hostnames that can be mapped back to internal ports (example: host.name.formatter=b$p.$h:9092) ● Matching Certificates (wildcard) ● Ingress with SNI rules / mapping for these hostnames ● External DNS entries (wildcard) pointing to ingress IPs
  • 66. Features and KMS E2EE/CSM
  • 67. Features Comparison Client-side Encryption CSM-based Encryption Field-level encryption ✅ (Java,.NET only) ✅ Payload-level encryption ✅ ✅ Tokenization/Masking ✅ (Java,.NET only) ✅ Format-Preserving Encryption ✅ (Java,.NET only) ✅ Supports Kafka Streams ✅ ✅ Supports Kafka Connect JSON, AVRO only ✅ Supports ksqlDB ✅ ✅ Supports REST Proxy ❌ ✅ Popular KMS integrations ✅ (Java,.NET only) ✅ Supports access control ✅ ✅ Node.js, python, C++ support limited features ✅ Other (Go, Ruby) lang support ❌ ✅ Component-based install ✅ Not required
  • 68. E2EE Libraries Features and integrations ✅ Feature included ❌ Feature prioritized but not complete ❌ Feature not included or prioritized na Not Applicable