confluent, profile picture
Uploaded byconfluent
PDF, PPTX361 views

Q&A with Confluent Professional Services: Confluent Service Mesh

The document outlines the Confluent technical enablement sessions and Q&A offerings for partners, detailing various workshops, learning paths, and product updates focused on data streaming with Apache Kafka. It highlights upcoming features like bi-directional cluster linking and data governance tools to enhance security, reliability, and operational efficiency in cloud-native environments. Additionally, it introduces the Confluent Service Mesh, emphasizing data protection, observability, and encryption standards for cloud-native applications.

Embed presentation

Download as PDF, PPTX
Thanks for joining! We’ll get started soon! Technical Enablement Session
Partners Q&A
Partners Q&A
@yourtwitterhandle | developer.confluent.io Our Partner Technical Enablement offering Scheduled sessions On-demand Join us for these live sessions where our experts will guide you through sessions of different level and will be available to answer your questions. Some examples of sessions are below: • Confluent 101: for new starters • Hybrid Cloud Workshop: learn by doing • Path to Production series , Confluent Cloud workshops series • Product Updates Learn the basics with a guided experience, at your own pace with our learning paths on-demand. You will also find an always growing repository of more advanced presentations to dig-deeper. Some examples are below: • Aware/Novice/Competent Learning paths • Confluent Use Cases • Positioning Confluent Value • Confluent Cloud Networking • … and many more AskTheExpert we’ll offer a channel dedicated to streaming questions • Build CoE inside partners by getting people with similar interest together • Connect with opportunities and discover trends at focus partners • Build a Technical Community • Q&A • Tech Talk
@yourtwitterhandle | developer.confluent.io What are the best practices to debug client applications (producers/consumers in general but also Kafka Streams applications)?
@yourtwitterhandle | developer.confluent.io
@yourtwitterhandle | developer.confluent.io
The Confluent Q3 ‘23 Launch Announcing the latest updates to our cloud-native data streaming platform, Confluent Cloud
Confluent Cloud Cloud native data streaming platform built by the founders of Apache Kafka® 9 Cloud-Native Complete Everywhere Stream confidently on the world’s most trusted data streaming platform built by the founders of Apache Kafka©, with resilience, security, compliance, and privacy built-in by default. Cloud Native The 10x Apache Kafka® service: elastic, resilient and performant, powered by the Kora Engine Complete Go above & beyond Kafka with all the essential tools for a complete data streaming platform Everywhere Connect your data in real time with a platform that spans from on-prem to cloud and across clouds
The Confluent Q3 ‘23 Launch Deliver Intelligent, Secure, and Cost-effective Data Pipelines 10 Cloud-Native Complete Everywhere Storage Price Reduction: Cost-effectively store data at any scale without growing compute at 20% lower prices CC for Apache Flink® (Open Preview) + Enterprise Clusters Secure, cost-effective, and serverless Kafka powered by the Kora Engine Confluent Terraform Provider updates + Enhance security and compliance while continuing to reduce operational burden through automated infrastructure management HashiCorp Sentinel Integration Resource Importer Data Catalog Support Cloud Audit Logs for Kafka Produce & Consume Experience full visibility and control of sensitive data access in Confluent Cloud with detailed audit events enabling swift response to unauthorized access. Cluster Linking updates Cluster Linking with AWS Private Link: Easily stream data between regions, teams or environments within AWS private networks Bi-directional Cluster Linking Optimize disaster recovery and increase reliability with bi-directional cluster linking Data Portal in Stream Governance Safely unlock data and increase developer productivity with a self-service, data-centric portal for discovering, accessing, and enriching real-time data streams flowing across your organization (coming soon) Easily build high-quality, reusable data streams with the industry’s only cloud-native, serverless Flink service
Data Portal in Stream Governance 11 Seamlessly and securely request access to data streams and trigger an approval workflow that connects the user with the data owner, all within the Confluent Cloud UI Easily build and manage data products to power streaming pipelines and applications by understanding, accessing, and enriching existing data streams Complete Safely unlock data and increase developer productivity with a self-service, data-centric portal for discovering, accessing, and enriching real-time data streams flowing across your organization Search, discover, and explore existing topics, tags, and metadata across the organization with end-to-end visibility to choose the data most relevant for your projects Coming Soon
Introducing Data Portal in Stream Governance Access your data streams through a developer-friendly, self-service UI Search, discover, and explore existing topics, tags, and metadata across the organization Seamlessly request access to data streams and trigger an approval workflow Understand, access, & enrich data streams to power real-time data streaming pipelines and applications
Bidirectional Cluster Linking 13 Optimize disaster recovery and increase reliability with bi-directional cluster linking Facilitate seamless consumer migration with retained offsets for consistent data processing with Bi-directional cluster links Increase efficiency and reduce data recovery time by eliminating the need for custom code Streamline security configuration with support for DR and active/active architecture with Bi-directional links that provides outbound and inbound connections Everywhere **Note - bi-directional cluster linking is available for new cluster links only, existing cluster link need to be deleted and re-activated to obtain this functionality.
Enhanced Disaster Recovery Capabilities with Bidirectional Cluster Linking 14 Cluster Link bidirectional Connection and Authentication Connection and Authentication Cluster A Cluster B Applications in region B Cluster A Cluster B Cluster Link bidirectional Topics on Cluster A Mirror Topics on Cluster B Mirror Topics on Cluster A Topics on Cluster B ACLs / RBAC for Cluster B API Key or OAuth for Cluster A API Key or OAuth for Cluster B ACLs / RBAC for Cluster A Applications in region A Data & Metadata Data & Metadata
Cluster Linking with AWS Private Link 15 Simplified setup: Utilize Network Link Service and Endpoint for a reliable connection between clusters Enhanced network-level security: AWS PrivateLink isolates Confluent Cloud clusters, preventing external resources and Cluster Linking access Seamless cluster linking: Establish a secure networking path between separate Confluent Cloud networks for efficient data exchange Everywhere Easily stream data between regions, teams or environments within AWS private networks
The Confluent Q3 ‘23 Launch Deliver Intelligent, Secure, and Cost-effective Data Pipelines Cloud-Native Complete Everywhere Storage Price Reduction: Cost-effectively store data at any scale without growing compute at 20% lower prices Easily build high-quality, reusable data streams with the industry’s only cloud-native, serverless Flink service Apache Flink® on CC (Open Preview) + Enterprise Clusters Secure, cost-effective, and serverless Kafka powered by the Kora Engine Confluent Terraform Provider updates + Enhance security and compliance while continuing to reduce operational burden through automated infrastructure management HashiCorp Sentinel Integration Resource Importer Data Catalog Support Cloud Audit Logs for Kafka Produce & Consume Experience full visibility and control of sensitive data access in Confluent Cloud with detailed audit events enabling swift response to unauthorized access. Cluster Linking updates Cluster Linking with AWS Private Link: Easily stream data between regions, teams or environments within AWS private networks Bi-directional Cluster Linking Optimize disaster recovery and increase reliability with bi-directional cluster linking Data Portal in Stream Governance Safely unlock data and increase developer productivity with a self-service, data-centric portal for discovering, accessing, and enriching real-time data streams flowing across your organization (coming soon)
Partners Q&A
Confluent Service Mesh Roman Schmitz, November 2023
What is the Confluent Service Mesh (CSM)?
“A service mesh is a tool for adding observability, security, and reliability features to “cloud native” applications by transparently inserting this functionality at the platform layer rather than the application layer. The service mesh is rapidly becoming a standard part of the cloud native stack, especially for Kubernetes adopters.” 20 -linkerd.io
“A service mesh is a tool for adding observability, security, and reliability features to “cloud native” applications by transparently inserting this functionality at the platform layer rather than the application layer. The service mesh is rapidly becoming a standard part of the cloud native stack, especially for Kubernetes adopters.” 21 -linkerd.io
“A service mesh is a tool for adding observability, security, and reliability features to “cloud native” applications by transparently inserting this functionality at the platform layer rather than the application layer. The service mesh is rapidly becoming a standard part of the cloud native stack, especially for Kubernetes adopters.” 22 -linkerd.io
Life as we know it Producer Consumer
With CSM in the Mix Producer Consumer Pluggable Code CSM Pluggable Code CSM
Kafka Broker port 9092 Kafka Broker port 9092 Kafka Broker port 9092 Confluent Service Mesh at a glance Confluent Service Mesh Producer Consumer listener port 30001 Pluggable Code listener port 30002 Pluggable Code listener port 30003 Pluggable Code
Kafka Startup Kafka Broker Get Metadata Client Return Metadata Metadata Response { "Brokers": [ { "NodeId": 0, "Host": "broker0.yourdomain.com", "Port": 9092 }, { "NodeId": 1, "Host": "broker1.yourdomain.com", "Port": 9092 }, { "NodeId": 2, "Host": "broker2.yourdomain.com", "Port": 9092 } ], "Topics": [], … } Connect to one of the brokers
Kafka Startup With CSM Return Metadata Kafka Broker CSM Get Metadata Client Modify Metadata Return Metadata Modified Metadata Response { "Brokers": [ { "NodeId": 0, "Host": "csm.yourdomain.com", "Port": 30001 }, { "NodeId": 1, "Host": "csm.yourdomain.com", "Port": 30002 }, { "NodeId": 2, "Host": "csm.yourdomain.com", "Port": 30003 } ], "Topics": [], … } Connect to a CSM port
What’s the Pluggable Code?
End-to-End Encryption
Payload-Level Encryption
End-to-end Encryption Features • Local key management and JKS support • Gemalto, Hashicorp, many security appliances • Cloud provider key management service support • AES, RSA encryption, SHA256 hashing • AVRO, JSON, Protobuf, XML, String, Byte arrays, Byte buffer level encryption and tokenization • Field access control • Format preserving encryption (NIST SP 800-38G) • Support for metadata and data classification • Support for master keys (Encryption of a data key with a wrapping key) • Support for key rotation • Support for event digital signature support to validate producers Consumer Protected Producer KMS/Tokenizer Schema Registry
Kafka Messages and Serialization Consumer Producer Cleartext { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": 678900000234, "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": 678900000234, "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } Serializer Deserializer 1001001001001000110 1001010100101010001 1001010010010100101 0010101001010010100 1010100101001010101 0101010101001001000 1010011101101001010 1011110
Kafka Messages with encryption Consumer Producer { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": 678900000234, "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": 678900000234, "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } Serializer Deserializer 1100100100110010001 1010010101001010100 0110010100100101001 0100101010010100101 0010101001010010101 0101010101010010010 0010100111011010010 101011110 Protected Encryption Decryption
Message-level encryption { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": 678900000234, "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } Generate Data Key pPYP7QM+LjMfjJ+QdOrLF3VTjMy1sWPtf epEXXwqkxXrnIbT1iEuzas2J/aOlUv7md 7YFP4Zq5PbrWWTLKeQDRlBVCOBacD15jl pcME0EONfErWd/CljAaTtCEnGRtfKsCHx 0zasCvXK3G0v15GdptqEGoREtXpea5f9q M8nYXc1tQbjX4mKP0nB/aVQSmKLXBeEU3 KaiioyXsT3Vsr+tLSCWO76Tfhfaum8Ue4 F5WKPD3svJA== { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": 678900000234, "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } Info added to Metadata: Encrypted Data Key, version, hash
Key Exchange Process Kafka Broker Key Store/KMS Get Master Key Key Store/KMS Encryption Decryption Get Data Key Secured Serializer Encrypt Event Encrypt Data Key Send encrypted event and encrypted data key Encryption Decryption Secured Deserializer Fetch Events Get Master Key Decrypt Data Key Decrypt Event Use decrypted data key for decryption Use data key for encryption Use master key for decryption Use master key for encryption
Data Protection with Confluent Service Mesh and Encryption accelerator 36 CSM producer sidecar is responsible for data protection independently of the client type. Protected Producer Consumer KMS/Tokenizer CSM consumer sidecar is responsible for safely exposing data in clear and can also handle field access control. CSM CSM
Field-Level Encryption
Field-level protection { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": "678900000234", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } Generate Data Key { "name": "Hyt Piqdfggr", "address": "852 Jdrf Wd", "ssn_id": "dKI4gflV6r339Q==", "account": "PrM1vyf/CxwoqQ==", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": "678900000234", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } Protected Producer Consumer KMS/Tokenizer CSM CSM
Data Protection with Access Control via CSM Original message Original message { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": "678900000234", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } { "name": "Hyt Piqdfggr", "address": "852 Jdrf Wd", "ssn_id": "dKI4gflV6r339Q==", "account": "PrM1vyf/CxwoqQ==", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } Protected { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": "678900000234", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } { "name": "Joe Example", "address": "123 Main St", "ssn_id": "dKI4gflV6r339Q==", "account": "PrM1vyf/CxwoqQ==", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } Original message with Access Control
OPA - Open Policy Agent https://www.openpolicyagent.org/ OPA testing and examples: The Rego Playground
Policy Based Field Level Access Control Which fields should be hidden or redacted? Producer Consumer Open Policy Agent Pluggable Code Confluent Service Mesh Pluggable Code Confluent Service Mesh
USA financial Policy Based Field Level Access Control Original message { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": "678900000234", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" “country”: “usa” } { "account": "678900000234", "Order_time": 1560070133853, "itemid": "Item_9" } { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": "678900000234", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } USA financial pii Brazil financial pii Open Policy Agent nothing sent Pluggable Code Confluent Service Mesh
Integration with Data catalogs, classification Data classification { "type":"record", "name":"DataClassifications", "classifications":{ "PII":{ "encrypt":{ "key":"SamplePIIKey", "wrapping.key":"RSAPII" }, "classifications":{ "Personal":{ "tokenize":{ } }, "Financial":{ "encrypt":{ "key":"SampleFinancialKey", "wrapping.key":"RSAPIIFinancial" } } } }, "Protected": { "encrypt": { "authorizer.class": "classNameHere", "authorizer.deny": false, "opa.module.name": "classification", "opa.rego": "/csm/classification.rego", "opa.query": "data.classification.allow" } } }, "fields":[ ] Data Catalog { "type":"record", "name":"ADataCatalog", "namespace":"com.mybusiness", "fields":[ { "name":"SSN", "type":"string", "classifications": ["PII/Financial", “Protected”] }, { "name":"Name", "type":"string", "classifications": ["PII/Personal", “Protected”] }, { "name":"Address", "type":"string", "classifications": ["PII/Personal", “Protected”] }, { "name":"Account", "type":"string", "classifications": ["PII/Financial", “Protected”] PII/Personal Name: Joe Example PII/Personal Address: 123 Main St CustID: 12345 PII/Financial SSN: 123-45-6789 Persona: 56A Credit: 780 PII/Financial Acct #: 3456789 Current Balance: 0 PII/Personal Name: Hyt Piqdfggr PII/Personal Address: 852 Jdrf Wd CustID: 12345 PII/Financial SSN: dKI4gflV6r339Q== Persona: 56A Credit: 780 PII/Financial Acct #: PrM1vyf/CxwoqQ== Current Balance: 0
OPA Configuration and Integration Link OPA Policies in Classifications Add OPA Policies (rego) Local OPA module (Session Authorizer) local path to rego file rego path (decision, package)
Authentication Swapping
Mutual TLS (mTLS) or Kerberos Producer Consumer MTLS / Kerberos MTLS / Kerberos O N PREM O N LY 🤬 FAIL
With CSM in the Mix Client Pluggable Code CSM MTLS principal User1 => key/secret User2 => key/secret SASL (key/secret ) Lookup Auth from Principal during SSL H andshake
Example CSM MTLS Flow Extract Principal from Cert Some Database CSM SSL Handshake Client Lookup key/secret from DB with Principal as key Return key/secret Confluent Cloud Authenticate sasl with key/secret Finish Handshake
Example: CSM Auth Swapping Configurations … csm.ssl=true csm.ssl.enabled=true csm.ssl.truststore.location=${truststore} csm.ssl.truststore.password=confluent csm.ssl.keystore.location=${keystore} csm.ssl.keystore.password=confluent csm.ssl.key.password=confluent csm.ssl.client.auth=required csm.ssl.principal.mapping.rules: RULE:^CN=([a-zA-Z.0-9@-]+).*$/$1/,DEFAULT … csm.authorizers=vaultAuth vaultAuth.class=io.confluent.csid.csm.auth.VaultAuth vaultAuth.vault.address=http://vault:8200 vaultAuth.vault.auth.token=vault-plaintext-root-token vaultAuth.vault.store=secret/testing vaultAuth.vault.split=/ … mTLS Configuration … csm.ssl=true sasl.enabled.mechanisms=GSSAPI csm.sasl.mechanism=GSSAPI … csm.authorizers=vaultAuth vaultAuth.class=io.confluent.csid.csm.auth.VaultAuth vaultAuth.vault.address=http://vault:8200 vaultAuth.vault.auth.token=vault-plaintext-root-token vaultAuth.vault.store=secret/testing vaultAuth.vault.split=/ … Kerberos Configuration Examples, Documentation: https://confluentinc.github.io/csid-csm/
CSM Deployment Options
Typical Hybrid CSM-Setup - hybrid setup - self-managed connect - local CSM and clients - ksqlDB and CP in Confluent Cloud - ksqlDB on field-level-encrypted topics - AWS KMS for keys (AWS, Azure, Vault, …)
CSM in a sidecar - external service writing to plain-text topic - kstreams app filtering data and writing to encrypted topic - local client connecting to CCloud via CSM/directly
CSM as (Gateway) Service on VMs - CSM deployed on containers/VMs - HA achieved with multiple CSM-replicas and LB - reminder: CSM is stateless (!) - Scaling horizontally/vertically - load-balancers for external CSM-access
Client Configuration Examples
Configuration Example: Clients using CSM bootstrap.servers=pkc-XXXXX.eu-west-1.aws.confluent.cloud:9092 security.protocol=SASL_SSL bootstrap.servers=csm:30001 security.protocol=SASL_PLAINTEXT sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='<CCLOUD API KEY>' password='<CCLOUD API SECRET>'; sasl.mechanism=PLAIN # Required for correctness in Apache Kafka clients prior to 2.6 client.dns.lookup=use_all_dns_ips # Required connection configs for Confluent Cloud Schema Registry schema.registry.url=https:// basic.auth.credentials.source=USER_INFO basic.auth.user.info=<SR-KEY>:<SR-SECRET> bootstrap.servers=pkc-XXXXX.eu-west-1.aws.confluent.cloud:9092 security.protocol=SASL_SSL bootstrap.servers=csm:30001 security.protocol=SASL_PLAINTEXT sasl.mechanisms=PLAIN sasl.username=<CCLOUD API KEY> sasl.password=<CCLOUD API SECRET> Java-Client librdkafka (kcat, C#, Python)
Configuration Example: CSM with AWS KMS csm.ssl=false broker.ssl=true bootstrap.servers=pkc-XXXXX.eu-west-1.aws.confluent.cloud:9092 host.name=csm client.dns.lookup=use_all_dns_ips sasl.mechanism=PLAIN security.protocol=SASL_SSL sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<CCLOUD API KEY>" password="<CCLOUD API SECRET>"; # Required connection configs for Confluent Cloud Schema Registry schema.registry.url=https://psrc-XXXXX.eu-central-1.aws.confluent.cloud basic.auth.credentials.source=USER_INFO basic.auth.user.info=<SR-KEY>:<SR-SECRET> csm.get.brokers.on.boot=true csm.port=30001 csm.request.interceptors=in csm.response.interceptors=out in.class=io.confluent.csid.csm.encryption.produce.EncryptInterceptor in.key=rschmitz-symmetric in.encryption.provider.name=aws in.schema.registry.url=https://psrc-XXXXX.eu-central-1.aws.confluent.cloud in.basic.auth.credentials.source=USER_INFO in.basic.auth.user.info=<SR-KEY>:<SR-SECRET> in.aws.provider.class = io.confluent.encryption.common.crypto.cipher.impl.AWSKMSProvider in.aws.provider.use.default.sdk=true in.aws.provider.region=eu-west-1 in.aws.provider.access.key.id=<AWS API-KEY> in.aws.provider.secret.key=<AWS API-SECRET> … in.class=io.confluent.csid.csm.encryption.produce.EncryptInterceptor in.key=rschmitz-symmetric in.encryption.metadata.policy.class=CatalogPolicy in.encryption.metadata.name=DataCatalog in.encryption.classifications.name=DataClassifications in.encryption.provider.name=aws … Example csm.properties Field-Level-Configuration
CSM Demo
CSM as a Gateway to Confluent Cloud Transparent end-to-end encryption Field-level authorization and access-control with policy-based field-level encryption Use existing authentication mechanisms in cloud migrations
Backup Slides
CSM Ingress on k8s / SNI: Formatter for Listener Overrides
Use case: Kubernetes Ingress Ingress Scenario: ● CSM maps each broker to one port that is exposed as a k8s service ● Ingress will not allow to open ports dynamically (or more than a few specific ports at all - 80, 8080, 443)
Solution: Formatter for Listener Overrides Return Metadata Kafka Broker CSM Get Metadata Client Modify Metadata Return Metadata Modified Metadata Response Updated { "Brokers": [ { "NodeId": 0, "Host": "csm.yourdomain.com", "Port": 30001 "Host": "b30001.csm.yourdomain.com", "Port": 9092 }, { "NodeId": 1, "Host": "csm.yourdomain.com", "Port": 30002 "Host": "b30002.csm.yourdomain.com", "Port": 9092 }, … ], "Topics": [], … } Connect to a CSM port
Solution: SNI Routing SNI: Server Name Indication - Wikipedia (https://github.com/Schm1tz1/sni-routing-examples) ● Hosting of multiple (virtual) services with same (physical) frontend and different backends ● Used in Ingress for (de)multiplexing TCP traffic ● Routing to backend services using information from TLS handshake (hello) ● Similar pattern based on HTTP headers very common in for Web-Servers
Formatter for Listener Overrides and SNI Changes to "CSM standard setup": ● CSM configured to return virtual hostnames that can be mapped back to internal ports (example: host.name.formatter=b$p.$h:9092) ● Matching Certificates (wildcard) ● Ingress with SNI rules / mapping for these hostnames ● External DNS entries (wildcard) pointing to ingress IPs
Features and KMS E2EE/CSM
Features Comparison Client-side Encryption CSM-based Encryption Field-level encryption ✅ (Java,.NET only) ✅ Payload-level encryption ✅ ✅ Tokenization/Masking ✅ (Java,.NET only) ✅ Format-Preserving Encryption ✅ (Java,.NET only) ✅ Supports Kafka Streams ✅ ✅ Supports Kafka Connect JSON, AVRO only ✅ Supports ksqlDB ✅ ✅ Supports REST Proxy ❌ ✅ Popular KMS integrations ✅ (Java,.NET only) ✅ Supports access control ✅ ✅ Node.js, python, C++ support limited features ✅ Other (Go, Ruby) lang support ❌ ✅ Component-based install ✅ Not required
E2EE Libraries Features and integrations ✅ Feature included ❌ Feature prioritized but not complete ❌ Feature not included or prioritized na Not Applicable
Q&A with Confluent Professional Services: Confluent Service Mesh

More Related Content

PPTX
Kafka 101
byClement Demonchy
 
PDF
Apache Kafka Fundamentals for Architects, Admins and Developers
byconfluent
 
PPTX
Kafka presentation
byMohammed Fazuluddin
 
PDF
Kafka 101 and Developer Best Practices
byconfluent
 
PPTX
A visual introduction to Apache Kafka
byPaul Brebner
 
PPTX
Deep Dive into Apache Kafka
byconfluent
 
PPTX
Microservices in the Apache Kafka Ecosystem
byconfluent
 
PDF
Getting Started with Confluent Schema Registry
byconfluent
 
Kafka 101
byClement Demonchy
 
Apache Kafka Fundamentals for Architects, Admins and Developers
byconfluent
 
Kafka presentation
byMohammed Fazuluddin
 
Kafka 101 and Developer Best Practices
byconfluent
 
A visual introduction to Apache Kafka
byPaul Brebner
 
Deep Dive into Apache Kafka
byconfluent
 
Microservices in the Apache Kafka Ecosystem
byconfluent
 
Getting Started with Confluent Schema Registry
byconfluent
 

What's hot

PPTX
Introduction to Apache Kafka
byJeff Holoman
 
PPTX
Prometheus and Grafana
byLhouceine OUHAMZA
 
PDF
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka
byKai Wähner
 
PDF
Kafka Streams: What it is, and how to use it?
byconfluent
 
PPTX
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
byJean-Paul Azar
 
PDF
Introduction to Kafka Streams
byGuozhang Wang
 
PDF
Apache Kafka - Martin Podval
byMartin Podval
 
PDF
Producer Performance Tuning for Apache Kafka
byJiangjie Qin
 
PDF
Uber: Kafka Consumer Proxy
byconfluent
 
PDF
Stream Processing with Flink and Stream Sharing
byconfluent
 
PPTX
Apache Kafka Best Practices
byDataWorks Summit/Hadoop Summit
 
PDF
Apache Kafka Architecture & Fundamentals Explained
byconfluent
 
PPTX
Introduction to Kafka Cruise Control
byJiangjie Qin
 
PDF
Introducing Confluent Cloud: Apache Kafka as a Service
byconfluent
 
ODP
Stream processing using Kafka
byKnoldus Inc.
 
PDF
Cloud Native Application
byVMUG IT
 
PDF
Everything You Always Wanted to Know About Kafka’s Rebalance Protocol but Wer...
byconfluent
 
PPSX
Apache Flink, AWS Kinesis, Analytics
byAraf Karsh Hamid
 
PPTX
Centralized Logging System Using ELK Stack
byRohit Sharma
 
PDF
A Deep Dive into Kafka Controller
byconfluent
 
Introduction to Apache Kafka
byJeff Holoman
 
Prometheus and Grafana
byLhouceine OUHAMZA
 
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka
byKai Wähner
 
Kafka Streams: What it is, and how to use it?
byconfluent
 
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
byJean-Paul Azar
 
Introduction to Kafka Streams
byGuozhang Wang
 
Apache Kafka - Martin Podval
byMartin Podval
 
Producer Performance Tuning for Apache Kafka
byJiangjie Qin
 
Uber: Kafka Consumer Proxy
byconfluent
 
Stream Processing with Flink and Stream Sharing
byconfluent
 
Apache Kafka Best Practices
byDataWorks Summit/Hadoop Summit
 
Apache Kafka Architecture & Fundamentals Explained
byconfluent
 
Introduction to Kafka Cruise Control
byJiangjie Qin
 
Introducing Confluent Cloud: Apache Kafka as a Service
byconfluent
 
Stream processing using Kafka
byKnoldus Inc.
 
Cloud Native Application
byVMUG IT
 
Everything You Always Wanted to Know About Kafka’s Rebalance Protocol but Wer...
byconfluent
 
Apache Flink, AWS Kinesis, Analytics
byAraf Karsh Hamid
 
Centralized Logging System Using ELK Stack
byRohit Sharma
 
A Deep Dive into Kafka Controller
byconfluent
 

Similar to Q&A with Confluent Professional Services: Confluent Service Mesh

PDF
DIMT '23 Session_Demo_ Latest Innovations Breakout.pdf
byconfluent
 
PPTX
AWS Immersion Day Mapfre - Confluent
byconfluent
 
PDF
Confluent Partner Tech Talk with Reply
byconfluent
 
PDF
Application Modernisation through Event-Driven Microservices
byconfluent
 
PDF
Why Cloud-Native Kafka Matters: 4 Reasons to Stop Managing it Yourself
byDATAVERSITY
 
PDF
DIMT 2023 SG - Hands-on Workshop_ Getting started with Confluent Cloud.pdf
byconfluent
 
PDF
App modernization on AWS with Apache Kafka and Confluent Cloud
byKai Wähner
 
PDF
Partner Connect APAC - 2022 - April
byconfluent
 
PDF
Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022
byHostedbyConfluent
 
PDF
What's new in confluent platform 5.4 online talk
byconfluent
 
PDF
Citi Tech Talk: Hybrid Cloud
byconfluent
 
PPTX
Best Practices for Building Hybrid-Cloud Architectures | Hans Jespersen
byconfluent
 
PDF
Reinventing Kafka in the Data Streaming Era - Jun Rao
byconfluent
 
PDF
3 Ways to Deliver an Elastic, Cost-Effective Cloud Architecture
byconfluent
 
PDF
The Bridge to Cloud (Peter Gustafsson, Confluent) London 2019 Confluent Strea...
byconfluent
 
PDF
Break data silos with real-time connectivity using Confluent Cloud Connectors
byconfluent
 
PDF
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
byHostedbyConfluent
 
PDF
Au delà des brokers, un tour de l’environnement Kafka | Florent Ramière
byconfluent
 
PDF
Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...
byconfluent
 
PDF
3 Ways to Deliver an Elastic, Cost-Effective Cloud Architecture (ANZ)
byconfluent
 
DIMT '23 Session_Demo_ Latest Innovations Breakout.pdf
byconfluent
 
AWS Immersion Day Mapfre - Confluent
byconfluent
 
Confluent Partner Tech Talk with Reply
byconfluent
 
Application Modernisation through Event-Driven Microservices
byconfluent
 
Why Cloud-Native Kafka Matters: 4 Reasons to Stop Managing it Yourself
byDATAVERSITY
 
DIMT 2023 SG - Hands-on Workshop_ Getting started with Confluent Cloud.pdf
byconfluent
 
App modernization on AWS with Apache Kafka and Confluent Cloud
byKai Wähner
 
Partner Connect APAC - 2022 - April
byconfluent
 
Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022
byHostedbyConfluent
 
What's new in confluent platform 5.4 online talk
byconfluent
 
Citi Tech Talk: Hybrid Cloud
byconfluent
 
Best Practices for Building Hybrid-Cloud Architectures | Hans Jespersen
byconfluent
 
Reinventing Kafka in the Data Streaming Era - Jun Rao
byconfluent
 
3 Ways to Deliver an Elastic, Cost-Effective Cloud Architecture
byconfluent
 
The Bridge to Cloud (Peter Gustafsson, Confluent) London 2019 Confluent Strea...
byconfluent
 
Break data silos with real-time connectivity using Confluent Cloud Connectors
byconfluent
 
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
byHostedbyConfluent
 
Au delà des brokers, un tour de l’environnement Kafka | Florent Ramière
byconfluent
 
Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...
byconfluent
 
3 Ways to Deliver an Elastic, Cost-Effective Cloud Architecture (ANZ)
byconfluent
 

More from confluent

PDF
Stream Processing Handson Workshop - Flink SQL Hands-on Workshop (Korean)
byconfluent
 
PPTX
Webinar Think Right - Shift Left - 19-03-2025.pptx
byconfluent
 
PDF
Migration, backup and restore made easy using Kannika
byconfluent
 
PDF
Five Things You Need to Know About Data Streaming in 2025
byconfluent
 
PDF
Data in Motion Tour Seoul 2024 - Keynote
byconfluent
 
PDF
Data in Motion Tour Seoul 2024 - Roadmap Demo
byconfluent
 
PDF
From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...
byconfluent
 
PDF
Data in Motion Tour 2024 Riyadh, Saudi Arabia
byconfluent
 
PDF
Build a Real-Time Decision Support Application for Financial Market Traders w...
byconfluent
 
PDF
Strumenti e Strategie di Stream Governance con Confluent Platform
byconfluent
 
PDF
Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeks
byconfluent
 
PDF
Building Real-Time Gen AI Applications with SingleStore and Confluent
byconfluent
 
PDF
Unlocking value with event-driven architecture by Confluent
byconfluent
 
PDF
Il Data Streaming per un’AI real-time di nuova generazione
byconfluent
 
PDF
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
byconfluent
 
PDF
Building API data products on top of your real-time data infrastructure
byconfluent
 
PDF
Speed Wins: From Kafka to APIs in Minutes
byconfluent
 
PDF
Evolving Data Governance for the Real-time Streaming and AI Era
byconfluent
 
PDF
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
byconfluent
 
PDF
Santander Stream Processing with Apache Flink
byconfluent
 
Stream Processing Handson Workshop - Flink SQL Hands-on Workshop (Korean)
byconfluent
 
Webinar Think Right - Shift Left - 19-03-2025.pptx
byconfluent
 
Migration, backup and restore made easy using Kannika
byconfluent
 
Five Things You Need to Know About Data Streaming in 2025
byconfluent
 
Data in Motion Tour Seoul 2024 - Keynote
byconfluent
 
Data in Motion Tour Seoul 2024 - Roadmap Demo
byconfluent
 
From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...
byconfluent
 
Data in Motion Tour 2024 Riyadh, Saudi Arabia
byconfluent
 
Build a Real-Time Decision Support Application for Financial Market Traders w...
byconfluent
 
Strumenti e Strategie di Stream Governance con Confluent Platform
byconfluent
 
Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeks
byconfluent
 
Building Real-Time Gen AI Applications with SingleStore and Confluent
byconfluent
 
Unlocking value with event-driven architecture by Confluent
byconfluent
 
Il Data Streaming per un’AI real-time di nuova generazione
byconfluent
 
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
byconfluent
 
Building API data products on top of your real-time data infrastructure
byconfluent
 
Speed Wins: From Kafka to APIs in Minutes
byconfluent
 
Evolving Data Governance for the Real-time Streaming and AI Era
byconfluent
 
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
byconfluent
 
Santander Stream Processing with Apache Flink
byconfluent
 

Recently uploaded

PPTX
Road_Quality_Indicator using deeplearning algorithms and computer vision
byDHARUNPRASHOBMM
 
PPTX
List on Python Programming Language.pptx
byRICHARDALONSAGAY1
 
PDF
Virtual Study Circles Innovative Ways to Collaborate Online.pdf
byExplain Learning
 
PDF
BCS-FACS Peter Landin Semantics Seminar - Formal Methods: Whence and Whither?
byJonathan Bowen
 
PDF
Design and Analysis of Algorithms(DAA): Unit-II Asymptotic Notations and Basi...
byKuvempu University
 
PPTX
Computer_Virus_Presentation_With_Slide7.pptx
byHEVISLIVE
 
PDF
Database Management Systems(DBMS):UNIT-I Introduction to Database(DBMS) BCA S...
byKuvempu University
 
PPTX
Modern P&C Insurance Software for Smarter, Faster Operations.pptx
byInsurance Tech Services
 
PPTX
Code Assessment Tools .pptx
byCodexpro
 
PPTX
Week 7 Introduction to HTML PowerPoint Notes.pptx
bylesandawelgens
 
PDF
Constraints First - Why Our On-Prem Ticketing System Starts With Limits, Not ...
bySpirit Face
 
PDF
TNG_Architecting Green Software - An Introduction_Nils-Oliver Linden&Alexande...
byQAware GmbH
 
PDF
How Modern Custom Software is Revolutionizing Mortgage Lending Processes -Ma...
byMatellio
 
PPTX
Membershipanywhere - Digital Membership Card - USA , Canada, Australia.pptx
bymembershipanywhere
 
PDF
Database Management Systems(DBMS):UNIT-II Relational Data Model BCA SEP SEM ...
byKuvempu University
 
PDF
Microservices-Final !!!!!!!!!!!!!!!!.pdf
bydoquangminh962004
 
PDF
How NetSuite Cloud ERP Helps Businesses Overcome Legacy System Downtime.
byTechWize
 
PPTX
Lecture 3 - Scheduling - Operating System
bynurulalomador
 
PDF
DSD-INT 2025 Hydrological Modelling in Society Assumptions, Impacts, and Appl...
byDeltares
 
PPTX
1_MIL_Introduction_Influence-of-Media-and-Information-on-Communication (1).pptx
byDeltaDomsMamas1
 
Road_Quality_Indicator using deeplearning algorithms and computer vision
byDHARUNPRASHOBMM
 
List on Python Programming Language.pptx
byRICHARDALONSAGAY1
 
Virtual Study Circles Innovative Ways to Collaborate Online.pdf
byExplain Learning
 
BCS-FACS Peter Landin Semantics Seminar - Formal Methods: Whence and Whither?
byJonathan Bowen
 
Design and Analysis of Algorithms(DAA): Unit-II Asymptotic Notations and Basi...
byKuvempu University
 
Computer_Virus_Presentation_With_Slide7.pptx
byHEVISLIVE
 
Database Management Systems(DBMS):UNIT-I Introduction to Database(DBMS) BCA S...
byKuvempu University
 
Modern P&C Insurance Software for Smarter, Faster Operations.pptx
byInsurance Tech Services
 
Code Assessment Tools .pptx
byCodexpro
 
Week 7 Introduction to HTML PowerPoint Notes.pptx
bylesandawelgens
 
Constraints First - Why Our On-Prem Ticketing System Starts With Limits, Not ...
bySpirit Face
 
TNG_Architecting Green Software - An Introduction_Nils-Oliver Linden&Alexande...
byQAware GmbH
 
How Modern Custom Software is Revolutionizing Mortgage Lending Processes -Ma...
byMatellio
 
Membershipanywhere - Digital Membership Card - USA , Canada, Australia.pptx
bymembershipanywhere
 
Database Management Systems(DBMS):UNIT-II Relational Data Model BCA SEP SEM ...
byKuvempu University
 
Microservices-Final !!!!!!!!!!!!!!!!.pdf
bydoquangminh962004
 
How NetSuite Cloud ERP Helps Businesses Overcome Legacy System Downtime.
byTechWize
 
Lecture 3 - Scheduling - Operating System
bynurulalomador
 
DSD-INT 2025 Hydrological Modelling in Society Assumptions, Impacts, and Appl...
byDeltares
 
1_MIL_Introduction_Influence-of-Media-and-Information-on-Communication (1).pptx
byDeltaDomsMamas1
 

Q&A with Confluent Professional Services: Confluent Service Mesh

  • 1.
    Thanks for joining! We’llget started soon! Technical Enablement Session
  • 2.
  • 3.
  • 4.
    @yourtwitterhandle | developer.confluent.io OurPartner Technical Enablement offering Scheduled sessions On-demand Join us for these live sessions where our experts will guide you through sessions of different level and will be available to answer your questions. Some examples of sessions are below: • Confluent 101: for new starters • Hybrid Cloud Workshop: learn by doing • Path to Production series , Confluent Cloud workshops series • Product Updates Learn the basics with a guided experience, at your own pace with our learning paths on-demand. You will also find an always growing repository of more advanced presentations to dig-deeper. Some examples are below: • Aware/Novice/Competent Learning paths • Confluent Use Cases • Positioning Confluent Value • Confluent Cloud Networking • … and many more AskTheExpert we’ll offer a channel dedicated to streaming questions • Build CoE inside partners by getting people with similar interest together • Connect with opportunities and discover trends at focus partners • Build a Technical Community • Q&A • Tech Talk
  • 5.
    @yourtwitterhandle | developer.confluent.io Whatare the best practices to debug client applications (producers/consumers in general but also Kafka Streams applications)?
  • 6.
  • 7.
  • 8.
    The Confluent Q3‘23 Launch Announcing the latest updates to our cloud-native data streaming platform, Confluent Cloud
  • 9.
    Confluent Cloud Cloud nativedata streaming platform built by the founders of Apache Kafka® 9 Cloud-Native Complete Everywhere Stream confidently on the world’s most trusted data streaming platform built by the founders of Apache Kafka©, with resilience, security, compliance, and privacy built-in by default. Cloud Native The 10x Apache Kafka® service: elastic, resilient and performant, powered by the Kora Engine Complete Go above & beyond Kafka with all the essential tools for a complete data streaming platform Everywhere Connect your data in real time with a platform that spans from on-prem to cloud and across clouds
  • 10.
    The Confluent Q3‘23 Launch Deliver Intelligent, Secure, and Cost-effective Data Pipelines 10 Cloud-Native Complete Everywhere Storage Price Reduction: Cost-effectively store data at any scale without growing compute at 20% lower prices CC for Apache Flink® (Open Preview) + Enterprise Clusters Secure, cost-effective, and serverless Kafka powered by the Kora Engine Confluent Terraform Provider updates + Enhance security and compliance while continuing to reduce operational burden through automated infrastructure management HashiCorp Sentinel Integration Resource Importer Data Catalog Support Cloud Audit Logs for Kafka Produce & Consume Experience full visibility and control of sensitive data access in Confluent Cloud with detailed audit events enabling swift response to unauthorized access. Cluster Linking updates Cluster Linking with AWS Private Link: Easily stream data between regions, teams or environments within AWS private networks Bi-directional Cluster Linking Optimize disaster recovery and increase reliability with bi-directional cluster linking Data Portal in Stream Governance Safely unlock data and increase developer productivity with a self-service, data-centric portal for discovering, accessing, and enriching real-time data streams flowing across your organization (coming soon) Easily build high-quality, reusable data streams with the industry’s only cloud-native, serverless Flink service
  • 11.
    Data Portal inStream Governance 11 Seamlessly and securely request access to data streams and trigger an approval workflow that connects the user with the data owner, all within the Confluent Cloud UI Easily build and manage data products to power streaming pipelines and applications by understanding, accessing, and enriching existing data streams Complete Safely unlock data and increase developer productivity with a self-service, data-centric portal for discovering, accessing, and enriching real-time data streams flowing across your organization Search, discover, and explore existing topics, tags, and metadata across the organization with end-to-end visibility to choose the data most relevant for your projects Coming Soon
  • 12.
    Introducing Data Portalin Stream Governance Access your data streams through a developer-friendly, self-service UI Search, discover, and explore existing topics, tags, and metadata across the organization Seamlessly request access to data streams and trigger an approval workflow Understand, access, & enrich data streams to power real-time data streaming pipelines and applications
  • 13.
    Bidirectional Cluster Linking 13 Optimize disasterrecovery and increase reliability with bi-directional cluster linking Facilitate seamless consumer migration with retained offsets for consistent data processing with Bi-directional cluster links Increase efficiency and reduce data recovery time by eliminating the need for custom code Streamline security configuration with support for DR and active/active architecture with Bi-directional links that provides outbound and inbound connections Everywhere **Note - bi-directional cluster linking is available for new cluster links only, existing cluster link need to be deleted and re-activated to obtain this functionality.
  • 14.
    Enhanced Disaster RecoveryCapabilities with Bidirectional Cluster Linking 14 Cluster Link bidirectional Connection and Authentication Connection and Authentication Cluster A Cluster B Applications in region B Cluster A Cluster B Cluster Link bidirectional Topics on Cluster A Mirror Topics on Cluster B Mirror Topics on Cluster A Topics on Cluster B ACLs / RBAC for Cluster B API Key or OAuth for Cluster A API Key or OAuth for Cluster B ACLs / RBAC for Cluster A Applications in region A Data & Metadata Data & Metadata
  • 15.
    Cluster Linking with AWSPrivate Link 15 Simplified setup: Utilize Network Link Service and Endpoint for a reliable connection between clusters Enhanced network-level security: AWS PrivateLink isolates Confluent Cloud clusters, preventing external resources and Cluster Linking access Seamless cluster linking: Establish a secure networking path between separate Confluent Cloud networks for efficient data exchange Everywhere Easily stream data between regions, teams or environments within AWS private networks
  • 16.
    The Confluent Q3‘23 Launch Deliver Intelligent, Secure, and Cost-effective Data Pipelines Cloud-Native Complete Everywhere Storage Price Reduction: Cost-effectively store data at any scale without growing compute at 20% lower prices Easily build high-quality, reusable data streams with the industry’s only cloud-native, serverless Flink service Apache Flink® on CC (Open Preview) + Enterprise Clusters Secure, cost-effective, and serverless Kafka powered by the Kora Engine Confluent Terraform Provider updates + Enhance security and compliance while continuing to reduce operational burden through automated infrastructure management HashiCorp Sentinel Integration Resource Importer Data Catalog Support Cloud Audit Logs for Kafka Produce & Consume Experience full visibility and control of sensitive data access in Confluent Cloud with detailed audit events enabling swift response to unauthorized access. Cluster Linking updates Cluster Linking with AWS Private Link: Easily stream data between regions, teams or environments within AWS private networks Bi-directional Cluster Linking Optimize disaster recovery and increase reliability with bi-directional cluster linking Data Portal in Stream Governance Safely unlock data and increase developer productivity with a self-service, data-centric portal for discovering, accessing, and enriching real-time data streams flowing across your organization (coming soon)
  • 17.
  • 18.
    Confluent Service Mesh RomanSchmitz, November 2023
  • 19.
    What is theConfluent Service Mesh (CSM)?
  • 20.
    “A service meshis a tool for adding observability, security, and reliability features to “cloud native” applications by transparently inserting this functionality at the platform layer rather than the application layer. The service mesh is rapidly becoming a standard part of the cloud native stack, especially for Kubernetes adopters.” 20 -linkerd.io
  • 21.
    “A service meshis a tool for adding observability, security, and reliability features to “cloud native” applications by transparently inserting this functionality at the platform layer rather than the application layer. The service mesh is rapidly becoming a standard part of the cloud native stack, especially for Kubernetes adopters.” 21 -linkerd.io
  • 22.
    “A service meshis a tool for adding observability, security, and reliability features to “cloud native” applications by transparently inserting this functionality at the platform layer rather than the application layer. The service mesh is rapidly becoming a standard part of the cloud native stack, especially for Kubernetes adopters.” 22 -linkerd.io
  • 23.
    Life as weknow it Producer Consumer
  • 24.
    With CSM inthe Mix Producer Consumer Pluggable Code CSM Pluggable Code CSM
  • 25.
    Kafka Broker port 9092 Kafka Broker port 9092 Kafka Broker port9092 Confluent Service Mesh at a glance Confluent Service Mesh Producer Consumer listener port 30001 Pluggable Code listener port 30002 Pluggable Code listener port 30003 Pluggable Code
  • 26.
    Kafka Startup Kafka Broker Get Metadata Client ReturnMetadata Metadata Response { "Brokers": [ { "NodeId": 0, "Host": "broker0.yourdomain.com", "Port": 9092 }, { "NodeId": 1, "Host": "broker1.yourdomain.com", "Port": 9092 }, { "NodeId": 2, "Host": "broker2.yourdomain.com", "Port": 9092 } ], "Topics": [], … } Connect to one of the brokers
  • 27.
    Kafka Startup WithCSM Return Metadata Kafka Broker CSM Get Metadata Client Modify Metadata Return Metadata Modified Metadata Response { "Brokers": [ { "NodeId": 0, "Host": "csm.yourdomain.com", "Port": 30001 }, { "NodeId": 1, "Host": "csm.yourdomain.com", "Port": 30002 }, { "NodeId": 2, "Host": "csm.yourdomain.com", "Port": 30003 } ], "Topics": [], … } Connect to a CSM port
  • 28.
  • 29.
  • 30.
  • 31.
    End-to-end Encryption Features •Local key management and JKS support • Gemalto, Hashicorp, many security appliances • Cloud provider key management service support • AES, RSA encryption, SHA256 hashing • AVRO, JSON, Protobuf, XML, String, Byte arrays, Byte buffer level encryption and tokenization • Field access control • Format preserving encryption (NIST SP 800-38G) • Support for metadata and data classification • Support for master keys (Encryption of a data key with a wrapping key) • Support for key rotation • Support for event digital signature support to validate producers Consumer Protected Producer KMS/Tokenizer Schema Registry
  • 32.
    Kafka Messages andSerialization Consumer Producer Cleartext { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": 678900000234, "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": 678900000234, "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } Serializer Deserializer 1001001001001000110 1001010100101010001 1001010010010100101 0010101001010010100 1010100101001010101 0101010101001001000 1010011101101001010 1011110
  • 33.
    Kafka Messages withencryption Consumer Producer { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": 678900000234, "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": 678900000234, "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } Serializer Deserializer 1100100100110010001 1010010101001010100 0110010100100101001 0100101010010100101 0010101001010010101 0101010101010010010 0010100111011010010 101011110 Protected Encryption Decryption
  • 34.
    Message-level encryption { "name": "JoeExample", "address": "123 Main St", "ssn_id": "123-45-6789", "account": 678900000234, "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } Generate Data Key pPYP7QM+LjMfjJ+QdOrLF3VTjMy1sWPtf epEXXwqkxXrnIbT1iEuzas2J/aOlUv7md 7YFP4Zq5PbrWWTLKeQDRlBVCOBacD15jl pcME0EONfErWd/CljAaTtCEnGRtfKsCHx 0zasCvXK3G0v15GdptqEGoREtXpea5f9q M8nYXc1tQbjX4mKP0nB/aVQSmKLXBeEU3 KaiioyXsT3Vsr+tLSCWO76Tfhfaum8Ue4 F5WKPD3svJA== { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": 678900000234, "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } Info added to Metadata: Encrypted Data Key, version, hash
  • 35.
    Key Exchange Process Kafka Broker Key Store/KMS GetMaster Key Key Store/KMS Encryption Decryption Get Data Key Secured Serializer Encrypt Event Encrypt Data Key Send encrypted event and encrypted data key Encryption Decryption Secured Deserializer Fetch Events Get Master Key Decrypt Data Key Decrypt Event Use decrypted data key for decryption Use data key for encryption Use master key for decryption Use master key for encryption
  • 36.
    Data Protection with Confluent ServiceMesh and Encryption accelerator 36 CSM producer sidecar is responsible for data protection independently of the client type. Protected Producer Consumer KMS/Tokenizer CSM consumer sidecar is responsible for safely exposing data in clear and can also handle field access control. CSM CSM
  • 37.
  • 38.
    Field-level protection { "name": "JoeExample", "address": "123 Main St", "ssn_id": "123-45-6789", "account": "678900000234", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } Generate Data Key { "name": "Hyt Piqdfggr", "address": "852 Jdrf Wd", "ssn_id": "dKI4gflV6r339Q==", "account": "PrM1vyf/CxwoqQ==", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": "678900000234", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } Protected Producer Consumer KMS/Tokenizer CSM CSM
  • 39.
    Data Protection withAccess Control via CSM Original message Original message { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": "678900000234", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } { "name": "Hyt Piqdfggr", "address": "852 Jdrf Wd", "ssn_id": "dKI4gflV6r339Q==", "account": "PrM1vyf/CxwoqQ==", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } Protected { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": "678900000234", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } { "name": "Joe Example", "address": "123 Main St", "ssn_id": "dKI4gflV6r339Q==", "account": "PrM1vyf/CxwoqQ==", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } Original message with Access Control
  • 40.
    OPA - OpenPolicy Agent https://www.openpolicyagent.org/ OPA testing and examples: The Rego Playground
  • 41.
    Policy Based FieldLevel Access Control Which fields should be hidden or redacted? Producer Consumer Open Policy Agent Pluggable Code Confluent Service Mesh Pluggable Code Confluent Service Mesh
  • 42.
    USA financial Policy Based FieldLevel Access Control Original message { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": "678900000234", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" “country”: “usa” } { "account": "678900000234", "Order_time": 1560070133853, "itemid": "Item_9" } { "name": "Joe Example", "address": "123 Main St", "ssn_id": "123-45-6789", "account": "678900000234", "Order_time": 1560070133853, "current_balance": 67, "itemid": "Item_9" } USA financial pii Brazil financial pii Open Policy Agent nothing sent Pluggable Code Confluent Service Mesh
  • 43.
    Integration with Datacatalogs, classification Data classification { "type":"record", "name":"DataClassifications", "classifications":{ "PII":{ "encrypt":{ "key":"SamplePIIKey", "wrapping.key":"RSAPII" }, "classifications":{ "Personal":{ "tokenize":{ } }, "Financial":{ "encrypt":{ "key":"SampleFinancialKey", "wrapping.key":"RSAPIIFinancial" } } } }, "Protected": { "encrypt": { "authorizer.class": "classNameHere", "authorizer.deny": false, "opa.module.name": "classification", "opa.rego": "/csm/classification.rego", "opa.query": "data.classification.allow" } } }, "fields":[ ] Data Catalog { "type":"record", "name":"ADataCatalog", "namespace":"com.mybusiness", "fields":[ { "name":"SSN", "type":"string", "classifications": ["PII/Financial", “Protected”] }, { "name":"Name", "type":"string", "classifications": ["PII/Personal", “Protected”] }, { "name":"Address", "type":"string", "classifications": ["PII/Personal", “Protected”] }, { "name":"Account", "type":"string", "classifications": ["PII/Financial", “Protected”] PII/Personal Name: Joe Example PII/Personal Address: 123 Main St CustID: 12345 PII/Financial SSN: 123-45-6789 Persona: 56A Credit: 780 PII/Financial Acct #: 3456789 Current Balance: 0 PII/Personal Name: Hyt Piqdfggr PII/Personal Address: 852 Jdrf Wd CustID: 12345 PII/Financial SSN: dKI4gflV6r339Q== Persona: 56A Credit: 780 PII/Financial Acct #: PrM1vyf/CxwoqQ== Current Balance: 0
  • 44.
    OPA Configuration andIntegration Link OPA Policies in Classifications Add OPA Policies (rego) Local OPA module (Session Authorizer) local path to rego file rego path (decision, package)
  • 45.
  • 46.
    Mutual TLS (mTLS)or Kerberos Producer Consumer MTLS / Kerberos MTLS / Kerberos O N PREM O N LY 🤬 FAIL
  • 47.
    With CSM inthe Mix Client Pluggable Code CSM MTLS principal User1 => key/secret User2 => key/secret SASL (key/secret ) Lookup Auth from Principal during SSL H andshake
  • 48.
    Example CSM MTLSFlow Extract Principal from Cert Some Database CSM SSL Handshake Client Lookup key/secret from DB with Principal as key Return key/secret Confluent Cloud Authenticate sasl with key/secret Finish Handshake
  • 49.
    Example: CSM AuthSwapping Configurations … csm.ssl=true csm.ssl.enabled=true csm.ssl.truststore.location=${truststore} csm.ssl.truststore.password=confluent csm.ssl.keystore.location=${keystore} csm.ssl.keystore.password=confluent csm.ssl.key.password=confluent csm.ssl.client.auth=required csm.ssl.principal.mapping.rules: RULE:^CN=([a-zA-Z.0-9@-]+).*$/$1/,DEFAULT … csm.authorizers=vaultAuth vaultAuth.class=io.confluent.csid.csm.auth.VaultAuth vaultAuth.vault.address=http://vault:8200 vaultAuth.vault.auth.token=vault-plaintext-root-token vaultAuth.vault.store=secret/testing vaultAuth.vault.split=/ … mTLS Configuration … csm.ssl=true sasl.enabled.mechanisms=GSSAPI csm.sasl.mechanism=GSSAPI … csm.authorizers=vaultAuth vaultAuth.class=io.confluent.csid.csm.auth.VaultAuth vaultAuth.vault.address=http://vault:8200 vaultAuth.vault.auth.token=vault-plaintext-root-token vaultAuth.vault.store=secret/testing vaultAuth.vault.split=/ … Kerberos Configuration Examples, Documentation: https://confluentinc.github.io/csid-csm/
  • 50.
  • 51.
    Typical Hybrid CSM-Setup - hybridsetup - self-managed connect - local CSM and clients - ksqlDB and CP in Confluent Cloud - ksqlDB on field-level-encrypted topics - AWS KMS for keys (AWS, Azure, Vault, …)
  • 52.
    CSM in asidecar - external service writing to plain-text topic - kstreams app filtering data and writing to encrypted topic - local client connecting to CCloud via CSM/directly
  • 53.
    CSM as (Gateway) Serviceon VMs - CSM deployed on containers/VMs - HA achieved with multiple CSM-replicas and LB - reminder: CSM is stateless (!) - Scaling horizontally/vertically - load-balancers for external CSM-access
  • 54.
  • 55.
    Configuration Example: Clientsusing CSM bootstrap.servers=pkc-XXXXX.eu-west-1.aws.confluent.cloud:9092 security.protocol=SASL_SSL bootstrap.servers=csm:30001 security.protocol=SASL_PLAINTEXT sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='<CCLOUD API KEY>' password='<CCLOUD API SECRET>'; sasl.mechanism=PLAIN # Required for correctness in Apache Kafka clients prior to 2.6 client.dns.lookup=use_all_dns_ips # Required connection configs for Confluent Cloud Schema Registry schema.registry.url=https:// basic.auth.credentials.source=USER_INFO basic.auth.user.info=<SR-KEY>:<SR-SECRET> bootstrap.servers=pkc-XXXXX.eu-west-1.aws.confluent.cloud:9092 security.protocol=SASL_SSL bootstrap.servers=csm:30001 security.protocol=SASL_PLAINTEXT sasl.mechanisms=PLAIN sasl.username=<CCLOUD API KEY> sasl.password=<CCLOUD API SECRET> Java-Client librdkafka (kcat, C#, Python)
  • 56.
    Configuration Example: CSMwith AWS KMS csm.ssl=false broker.ssl=true bootstrap.servers=pkc-XXXXX.eu-west-1.aws.confluent.cloud:9092 host.name=csm client.dns.lookup=use_all_dns_ips sasl.mechanism=PLAIN security.protocol=SASL_SSL sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<CCLOUD API KEY>" password="<CCLOUD API SECRET>"; # Required connection configs for Confluent Cloud Schema Registry schema.registry.url=https://psrc-XXXXX.eu-central-1.aws.confluent.cloud basic.auth.credentials.source=USER_INFO basic.auth.user.info=<SR-KEY>:<SR-SECRET> csm.get.brokers.on.boot=true csm.port=30001 csm.request.interceptors=in csm.response.interceptors=out in.class=io.confluent.csid.csm.encryption.produce.EncryptInterceptor in.key=rschmitz-symmetric in.encryption.provider.name=aws in.schema.registry.url=https://psrc-XXXXX.eu-central-1.aws.confluent.cloud in.basic.auth.credentials.source=USER_INFO in.basic.auth.user.info=<SR-KEY>:<SR-SECRET> in.aws.provider.class = io.confluent.encryption.common.crypto.cipher.impl.AWSKMSProvider in.aws.provider.use.default.sdk=true in.aws.provider.region=eu-west-1 in.aws.provider.access.key.id=<AWS API-KEY> in.aws.provider.secret.key=<AWS API-SECRET> … in.class=io.confluent.csid.csm.encryption.produce.EncryptInterceptor in.key=rschmitz-symmetric in.encryption.metadata.policy.class=CatalogPolicy in.encryption.metadata.name=DataCatalog in.encryption.classifications.name=DataClassifications in.encryption.provider.name=aws … Example csm.properties Field-Level-Configuration
  • 57.
  • 58.
    CSM as aGateway to Confluent Cloud Transparent end-to-end encryption Field-level authorization and access-control with policy-based field-level encryption Use existing authentication mechanisms in cloud migrations
  • 60.
  • 61.
    CSM Ingress onk8s / SNI: Formatter for Listener Overrides
  • 62.
    Use case: KubernetesIngress Ingress Scenario: ● CSM maps each broker to one port that is exposed as a k8s service ● Ingress will not allow to open ports dynamically (or more than a few specific ports at all - 80, 8080, 443)
  • 63.
    Solution: Formatter forListener Overrides Return Metadata Kafka Broker CSM Get Metadata Client Modify Metadata Return Metadata Modified Metadata Response Updated { "Brokers": [ { "NodeId": 0, "Host": "csm.yourdomain.com", "Port": 30001 "Host": "b30001.csm.yourdomain.com", "Port": 9092 }, { "NodeId": 1, "Host": "csm.yourdomain.com", "Port": 30002 "Host": "b30002.csm.yourdomain.com", "Port": 9092 }, … ], "Topics": [], … } Connect to a CSM port
  • 64.
    Solution: SNI Routing SNI:Server Name Indication - Wikipedia (https://github.com/Schm1tz1/sni-routing-examples) ● Hosting of multiple (virtual) services with same (physical) frontend and different backends ● Used in Ingress for (de)multiplexing TCP traffic ● Routing to backend services using information from TLS handshake (hello) ● Similar pattern based on HTTP headers very common in for Web-Servers
  • 65.
    Formatter for ListenerOverrides and SNI Changes to "CSM standard setup": ● CSM configured to return virtual hostnames that can be mapped back to internal ports (example: host.name.formatter=b$p.$h:9092) ● Matching Certificates (wildcard) ● Ingress with SNI rules / mapping for these hostnames ● External DNS entries (wildcard) pointing to ingress IPs
  • 66.
  • 67.
    Features Comparison Client-side EncryptionCSM-based Encryption Field-level encryption ✅ (Java,.NET only) ✅ Payload-level encryption ✅ ✅ Tokenization/Masking ✅ (Java,.NET only) ✅ Format-Preserving Encryption ✅ (Java,.NET only) ✅ Supports Kafka Streams ✅ ✅ Supports Kafka Connect JSON, AVRO only ✅ Supports ksqlDB ✅ ✅ Supports REST Proxy ❌ ✅ Popular KMS integrations ✅ (Java,.NET only) ✅ Supports access control ✅ ✅ Node.js, python, C++ support limited features ✅ Other (Go, Ruby) lang support ❌ ✅ Component-based install ✅ Not required
  • 68.
    E2EE Libraries Features andintegrations ✅ Feature included ❌ Feature prioritized but not complete ❌ Feature not included or prioritized na Not Applicable