SlideShare a Scribd company logo
1 of 44
Download to read offline
@yourtwitterhandle | developer.confluent.io
What are the best practices to debug client applications
(producers/consumers in general but also Kafka Streams
applications)?
Starting soon…
STARTING SOOOOON..
STARTING SOON…
S
Starting Soon…
STARTING SOON..
@yourtwitterhandle | developer.confluent.io
What are the best practices to debug client applications
(producers/consumers in general but also Kafka Streams
applications)?
Starting soon…
STARTING SOOOOON..
STARTING SOON…
S
Starting Soon…
Tech Talk Q4: Synthesis Software
Technologies
The Chronicles of Kafka:
A Migration Journey
Confluent Cloud Free Trial
New signups receive $400
to spend during their first 30 days.
Our Partner Technical Sales Enablement offering
Scheduled sessions On-demand
Join us for these live sessions
where our experts will guide you
through sessions of different level
and will be available to answer your
questions. Some examples of
sessions are below:
● Confluent 101: for new starters
● Workshops
● Path to production series
Learn the basics with a guided
experience, at your own pace with our
learning paths on-demand. You will
also find an always growing repository
of more advanced presentations to
dig-deeper. Some examples are below:
● Confluent 10
● Confluent Use Cases
● Positioning Confluent Value
● Confluent Cloud Networking
● … and many more
AskTheExpert /
Workshops
For selected partners, we’ll offer
additional support to:
● Technical Sales workshop
● JIT coaching on spotlight
opportunity
● Build CoE inside partners by
getting people with similar
interest together
● Solution discovery
● Tech Talk
● Q&A
Certifications + Accreditations
5
1. Certified Developer for Apache Kafka®
Qualifications: Completion of On-Demand/ Instructor-Led Course
Course Alignment: Confluent Developer Skills for Building Apache
Kafka® Duration: 60 multiple choice questions in 90 minutes
4. Fundamentals Accredited
Know the Basics
Prework Apache and Quiz
Demonstrates Kafka and Confluent Solution awareness
5. Technical Sales Associate Accredited
Confluent Positioning
Technical Sales Bootcamp Pre-work and Quiz
Enablement to equip sales and pre-sales to articulate the
Confluent value proposition
6. Technical Sales Professional Accredited
Hands on, Coaching Sales Pitch. A deeper dive into important
Confluent IP and differentiators in a way that allows attendees to
demonstrate their knowledge of the product, its value, and how to
demo it.
2. Certified Administrator for Apache Kafka® (Operations)
Qualifications: Completion of On-Demand/ Instructor-Led Course
Course Alignment: “Apache Kafka®” Administration by
Confluent Kafka® Duration: 60 multiple choice questions in 90
minutes
3. PS Delivery Qualification COMING SOON
Delivery-focused enablement, prescriptive delivery kits and work
on a real customer use case with Confluent SME’s. Our subject
matter experts will review the architecture and solution and
provide real-time feedback as well as recommendations for
additional enablement or shadowing as needed.
NEW
@yourtwitterhandle | developer.confluent.io
What are the best practices to debug client applications
(producers/consumers in general but also Kafka Streams
applications)?
Goal
Partners Tech Talks are webinars where subject matter experts from a Partner talk about a
specific use case or project. The goal of Tech Talks is to provide best practices and
applications insights, along with inspiration, and help you stay up to date about innovations
in confluent ecosystem.
Why Confluent Cloud
8
Today’s customers expect data in motion
Initiate an action
Instant confirmation
Source: https://www.gigaspaces.com/blog/amazon-found-every-100ms-of-latency-cost-them-1-in-sales
~100 ms
in latency
can cost
you…
… 20% of
digital
traffic
… $400M in
revenue
9
Finance & Banking Insurance Telecom
Travel & Retail
10 OUT
OF 10 8 OUT
OF 8
Fortune 500 Companies
Using Apache Kafka
70+%
Transportation Energy & Utilities Entertainment
Technology
8 OUT
OF 10 9 OUT
OF 10
10 OUT
OF 10
10 OUT
OF 10
10 OUT
OF 10 8 OUT
OF 10
Kafka is the de facto standard for Data in Motion
Operationalizing Kafka on your own is difficult
Kafka is hard in experimentation. It gets harder (and riskier) as you add
mission-critical data and use cases.
“We are in the business of selling and renting clothes. We are not
in the business of managing an event streaming platform… If we
had to manage everything ourselves, I would’ve had to hire at
least 10 more people to keep the systems up and running."
● Architecture planning
● Cluster sizing
● Cluster provisioning
● Broker settings
● Zookeeper management
● Partition placement & data
durability
● Source/sink connectors
development & maintenance
● Monitoring & reporting tools
setup
● Software patches and upgrades
● Security controls and
integrations
● Failover design & planning
● Mirroring & geo-replication
● Streaming data governance
● Load rebalancing & monitoring
● Expansion planning & execution
● Utilization optimization &
visibility
● Cluster migrations
● Infrastructure & performance
upgrades / enhancements
I N V E S T M E N T & T I M E
V
A
L
U
E
1
2
3
4
5
Experimentation
/ Early Interest
Central Nervous
System
Mission critical,
disparate LOBs
Identify a
Project
Mission-critical,
connected LOBs
Key challenges:
Operational burden & resources
Manage and scale platform to support
ever-growing demand
Security & governance
Ensure streaming data is as safe & secure as
data-at-rest as Kafka usage scales
Real-time connectivity & processing
Leverage valuable legacy data to power
modern, cloud-based apps & experiences
Global availability
Maintain high availability across environments
with minimal downtime
Business Risk
● Increasing risk of costly downtime and security breaches
as Kafka spans more use cases, apps / data systems, teams,
and environments
● Valuable resources diverted to address unplanned
downtime and breaches
Unplanned downtime
and breaches
Substantial hidden costs incl. lost
revenue, reputation damage, fines /
audits, reduced CSAT, and data loss
Infra Spend
● Over-provisioning required to support fluctuating demand
● Inability to scale storage w/o growing compute
● Performance degradations with manual configs,
re-balancing, and upgrades / patching
What does this all mean for your business?
Unpacking the direct and indirect costs of self-supporting Open Source Kafka
FTE Costs
It’s hard because... Which results in...
● Costly time & resources spent building / maintaining
low level infra tools vs. differentiating the business
● Increasingly difficult to hire and retain Kafka talent
(job listings & compensation +40% YoY)
Delayed time-to-value
2+ years to reach production at scale
with ~$3-5M+ in ongoing platform
development & operations costs
Increased Total Cost
of Ownership
Hundreds of thousands of $$$’s
spent annually on underutilized &
sub-optimal infra
Confluent Cloud
Cloud-native data streaming platform built by the founders of Apache Kafka®
Everywhere
Connect your data in real
time with a platform that
spans from on-prem to
cloud and across clouds
Complete
Go above & beyond Kafka
with all the essential tools
for a complete data
streaming platform
Cloud-Native
The 10x Apache Kafka®
service: elastic, resilient
and performant, powered
by the Kora Engine
Stream confidently on the world’s most trusted data streaming platform built by the founders of
Apache Kafka©, with resilience, security, compliance, and privacy built-in by default. 13
“Before Confluent, when we had broker outages that
required rebuilds, it could take up to three days of
developer time to resolve. Now, Confluent takes care
of everything for us, so our developers can focus on
building new features and applications.”
Elastic scalability: Scale and shrink to handle GBps+
workloads and petabyte storage 10x faster and easier
without any operational burden
Guaranteed resiliency: Leave Kafka reliability worries
behind with the industries most comprehensive 99.99%
uptime SLA and built-in data durability
Low latency: Speed up real-time analytics and
customer experiences with predictable low latency,
sustained over time
Cloud-Native
The 10x Apache Kafka® service: elastic,
resilient and performant, powered by the
Kora Engine
Kora Engine: The Apache Kafka® engine
built for the cloud
Complete
Go above & beyond Kafka with all the
essential tools for a complete data
streaming platform
Connectors & Stream Processing: Connect to
and from any app / system and process your
data streams in-flight
Reduce TCO and architectural complexity with our
portfolio of 120+ pre-built connectors and stream
processing powered by ksqlDB, all available fully
managed and built-in with Confluent Cloud
Stream Designer: Quickly build and deploy
streaming apps & pipelines
Rapidly build, test, and deploy streaming data
pipelines with Stream Designer, extensible with SQL,
while reducing the need to write boilerplate code
Security & Governance: Secure, discover, and
organize your data streams
Build trust and put your data streams to work with
enterprise-grade security and the only Stream
Governance suite for data in motion
“BHG is a fast-moving company, and Confluent is
quickly becoming not only a central highway for
our data with their vast connector portfolio, but a
streaming transformation engine as well for a vast
number of use cases… We are making Confluent the
true backbone of BHG, including leveraging 20+
Confluent connectors across both modern,
cloud-based technologies & legacy systems, to help
integrate our critical apps & data systems together.”
15
Connectors
Security
Data
Governance
Stream
Processing
Monitoring
Global
Resilience
Stream
Designer
Everywhere
Connect your data in real time with a
platform that spans from on-prem to
cloud and across clouds
Run Anywhere: Deploy across any environment
Provision Confluent as a fully managed service on AWS,
Azure, and Google Cloud across 60+ regions w/
Confluent Cloud, or on-premises w/ Confluent Platform
Unified: Unify data across hybrid and
multi-clouds
Provide consistent, self-service access to real-time data
across all your environments with Cluster Linking and
globally connected clusters that perfectly mirror data
Consistent: Learn one platform for all
environments
Remove the burden of learning new tools for each
environment with a consistent experience spanning
across cloud, on-prem, and hybrid / multicloud
“Our transformation to a cloud-native, agile
company required a large-scale migration from
open source Apache Kafka. With Confluent, we now
support real-time data sharing across all of our
environments, and see a clear path forward for our
hybrid cloud roadmap.”
16
…enabling you to deploy, operate, and scale in
minutes instead of months
Kafka re-architected to be truly Cloud-Native
17
Months Minutes
Weeks
Open Source
Apache Kafka
In-house development and
maintenance without support
Self-managed
Kafka Services
Manual operations with basic
tooling and/or support
Confluent
Cloud
Fully managed, elastic,
and automated product
capabilities with zero overhead
Confluent Cloud for
Apache Flink®
Simple, Serverless Stream Processing
Easily build high-quality,
reusable data streams with
the industry’s only cloud-
native, serverless Flink service
Effortlessly filter, join, and enrich your
data streams with Flink, the de facto
standard for stream processing
Enable high-performance and efficient
stream processing at any scale, without
the complexities of infrastructure
management
Experience Kafka and Flink as a
unified platform, with fully integrated
monitoring, security, and governance
Available for preview in select regions – see the docs for regional availability
C O N F I D E N T I A L
Process data streams in-flight to maximize actionability,
fidelity, and portability
Blob storage
3rd party app
Databases Data Warehouse
Database
SaaS app
Low latency apps
and data pipelines
Consistent, reusable
data products
Optimized resource
utilization
Build streaming
data pipelines to
inform real-time
decision making
Create new enriched and curated
streams of higher value using:
● Data transformations
● Streaming joins, temporal joins, lookup
joins, and versioned joins
● Fan out queries, multi-cluster queries
20
t1, 21.5 USD
t3, 55 EUR
t5, 35.3 EUR
t0, EUR:USD=1.00
t2,
EUR:USD=1.05
t4: EUR:USD=1.10
t1, 21.5 USD
t3, 57.75 USD
t5, 38.83 USD
Currency rate
Orders
STREAMING DATA PIPELINES
How to: Migrate to Confluent Cloud
Migration Overview
Kafka Clients &
Streaming Apps
Kafka Platform
Managed
Self-Managed
Migrate to Confluent Cloud involves moving the underlying Kafka Platform, Kafka client & Streaming
Applications.
Depending on the situation, some Kafka Clients could transition into managed world and some Kafka Clients
might remain as self-managed. One of the key decision is driven by whether private network is required or
not.
Connector
Apps
(Producer/Consumer)
ksqlDB
Schema
Connector
ksqlDB
Schema
Confluent Platform Confluent Cloud
Repoint
Self-Managed
Applications to
Confluent Cloud
23
Real-time
connectivity
& processing
Security
& governance
Operational
burden &
resources
Apache Kafka Confluent
Tedious configuration. Manual & resource-
intensive day-to-day operations to deploy,
manage, and scale clusters.
Custom-built security tools & integrations that
creates tech & operational debt. Lacks
governance for streaming data.
Instant cluster provisioning and scaling.
Automated capacity mgmt, load balancing, and
upgrades.
Battle-tested and enterprise-grade security &
governance tools ready out-of-the-box. (e.g., RBAC,
Audit Logs, Encryption, Stream Governance)
Siloed solutions and custom integrations,
escalating complexity & costs to manage and
maintain platform as it scales.
Enterprise-wide data fabric across apps & data
systems with a rich, pre-built connector
ecosystem and stream processing.
Global
availability
Resource-intensive failover design & operations,
exponentially increasing in complexity & risk as
volume and variety of environments grows.
Highly available and consistent access to data
across all of your environments, spanning all
clouds & on-prem.
“Confluent has become a lynchpin to harness Apache Kafka for improved operational
transparency and timely data-driven decisions. We are delighted with the improvements we have
seen in the monitoring of our business flows, speeding up lending approvals, and providing better and
more timely fraud analytics, and look forward to expanding services to leverage the platform to
simplify, automate, and drive business decisions.”
These capabilities make us the perfect partner to
address the key challenges with Open Source Kafka
@yourtwitterhandle | developer.confluent.io
What are the best practices to debug client applications
(producers/consumers in general but also Kafka Streams
applications)?
Starting soon…
STARTING SOOOOON..
STARTING SOON…
S
Starting Soon…
The Chronicles of
Kafka: A Migration
Story
Bhavesh Sooka and Jonathan Lew
Who we are
Kafkateer
Bhavesh Sooka
Kafkateer
Jonathan Lew
Who we are
Our
Goal
Our Goal
For the audience to learn
something from our
migration experience
and take it back to their
migration journey.
Contents
•Prologue
•Chronicles
•Epilogue
Gamified Fitness
VDP & V1
- 19 clusters
- 40 Markets Globally
using 8 AWS Regions
- 27 million monthly
users
- 20 Device
Manufacturers
- 10 million
workouts/day
- 124m messages
in a day
- 1000s of
microservices
Timeline
Prologue
2015 – V1:
Kafka 0.10 on
EC2
2019 – V1: Upgrade
Apache Kafka 0.10
-> CP 5.0.1
2021: - VDP:
Tibco EMS -> CP
6.0.1
2014 – V1:
Kafka on K8s
2015 – VDP: Tibco
EMS (Event
Messaging Service)
2020 – V1 Upgrade
(x2) CP 5.0.1 ->
5.4.1 -> CP 6.0.1
2021 – The
Migration: Enter your
speakers
1
2
3 5 7
6
4
7 Years
Timeline
Nov 2021 – Feb 2022:
ARB x2
June – Sept: Execution
4 years: Lead
up
Feb - June: Planning Oct: Hyper Care
4
5
3
2
1
Migration Timeline
11 Months
help!
VS
Winner!
ARB
Planning
Planning took 4
Months
Planning Outcomes
- Design decisions
- Configuration
- Secret Storage
- RBAC
- Networking
- Automation
- Deployment plan
Planning:
Design Decisions
Two DD’s we want to
mention in specific are:
Code and config split
Vault Module
Planning
Planning: Networking
Private Link TG
W
VPC
Peering
SSO enabled and LDAP
connectivity
Planning: RBAC
Split into 2 problems to
Solve
People Applications
Planning: Automation
Topic creation
Cluster and Networking
Planning: Deployment Day
Goal: 100% Uptime During Migration
• Move Producer
• Drain
• Move Consumer
• Repeat
Line Manager
• Decision Maker
• Autonomy
• On our Side
Operational Requests
SNAG: CIO
- Flight plan
- Commercials
Business Decisions/Red Tape
This was child’s
play!!!!!
4 Environments
=
4 Months
1 month of Hyper
Care
Execution
• Agile vs Waterfall
• Stakeholders buy in and trust
• Contextualized planning
• Team dynamic
• Business vs IT
• Industry Experts in Kafka
Learnings
Happy
Thank You
Confluent, Jack and
Dan
Vitality
Group/Vitality
Global
Synthesis
You
Happy
And so, they lived happily ever after…
#TagYourIt
Q&A
Q&A

More Related Content

Similar to Confluent Partner Tech Talk with Synthesis

Achieve Sub-Second Analytics on Apache Kafka with Confluent and Imply
Achieve Sub-Second Analytics on Apache Kafka with Confluent and ImplyAchieve Sub-Second Analytics on Apache Kafka with Confluent and Imply
Achieve Sub-Second Analytics on Apache Kafka with Confluent and Imply
confluent
 

Similar to Confluent Partner Tech Talk with Synthesis (20)

Partner Connect APAC - 2022 - April
Partner Connect APAC - 2022 - AprilPartner Connect APAC - 2022 - April
Partner Connect APAC - 2022 - April
 
Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022
Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022
Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022
 
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with K...
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with K...Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with K...
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with K...
 
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
 
Confluent Partner Tech Talk with SVA
Confluent Partner Tech Talk with SVAConfluent Partner Tech Talk with SVA
Confluent Partner Tech Talk with SVA
 
Data Streaming with Apache Kafka & MongoDB
Data Streaming with Apache Kafka & MongoDBData Streaming with Apache Kafka & MongoDB
Data Streaming with Apache Kafka & MongoDB
 
Unlock value with Confluent and AWS.pptx
Unlock value with Confluent and AWS.pptxUnlock value with Confluent and AWS.pptx
Unlock value with Confluent and AWS.pptx
 
Transform Your Organization with Real Real-Time Monitoring
Transform Your Organization with Real Real-Time MonitoringTransform Your Organization with Real Real-Time Monitoring
Transform Your Organization with Real Real-Time Monitoring
 
Choosing the Right Infrastructure Provider
Choosing the Right Infrastructure ProviderChoosing the Right Infrastructure Provider
Choosing the Right Infrastructure Provider
 
Build & Deploy Scalable Cloud Applications in Record Time
Build & Deploy Scalable Cloud Applications in Record TimeBuild & Deploy Scalable Cloud Applications in Record Time
Build & Deploy Scalable Cloud Applications in Record Time
 
Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...
Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...
Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...
 
Kubernetes Connectivity to Cloud Native Kafka | Christina Lin and Evan Shorti...
Kubernetes Connectivity to Cloud Native Kafka | Christina Lin and Evan Shorti...Kubernetes Connectivity to Cloud Native Kafka | Christina Lin and Evan Shorti...
Kubernetes Connectivity to Cloud Native Kafka | Christina Lin and Evan Shorti...
 
Kafka summit apac session
Kafka summit apac sessionKafka summit apac session
Kafka summit apac session
 
Best Practices for Building Hybrid-Cloud Architectures | Hans Jespersen
Best Practices for Building Hybrid-Cloud Architectures | Hans JespersenBest Practices for Building Hybrid-Cloud Architectures | Hans Jespersen
Best Practices for Building Hybrid-Cloud Architectures | Hans Jespersen
 
Message Driven and Event Sourcing
Message Driven and Event SourcingMessage Driven and Event Sourcing
Message Driven and Event Sourcing
 
Achieve Sub-Second Analytics on Apache Kafka with Confluent and Imply
Achieve Sub-Second Analytics on Apache Kafka with Confluent and ImplyAchieve Sub-Second Analytics on Apache Kafka with Confluent and Imply
Achieve Sub-Second Analytics on Apache Kafka with Confluent and Imply
 
IT Modernization For Process Modernization
IT Modernization For Process ModernizationIT Modernization For Process Modernization
IT Modernization For Process Modernization
 
Pivotal CloudFoundry on Google cloud platform
Pivotal CloudFoundry on Google cloud platformPivotal CloudFoundry on Google cloud platform
Pivotal CloudFoundry on Google cloud platform
 
Modernizing Testing as Apps Re-Architect
Modernizing Testing as Apps Re-ArchitectModernizing Testing as Apps Re-Architect
Modernizing Testing as Apps Re-Architect
 
James Watters Kafka Summit NYC 2019 Keynote
James Watters Kafka Summit NYC 2019 KeynoteJames Watters Kafka Summit NYC 2019 Keynote
James Watters Kafka Summit NYC 2019 Keynote
 

More from confluent

More from confluent (20)

Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutes
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
 
Unlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insightsUnlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insights
 
Workshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con FlinkWorkshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con Flink
 
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
 
AWS Immersion Day Mapfre - Confluent
AWS Immersion Day Mapfre   -   ConfluentAWS Immersion Day Mapfre   -   Confluent
AWS Immersion Day Mapfre - Confluent
 
Eventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalkEventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalk
 
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent CloudQ&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
 
Citi TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep DiveCiti TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep Dive
 
Build real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with ConfluentBuild real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with Confluent
 
Q&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service MeshQ&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service Mesh
 
Citi Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka MicroservicesCiti Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka Microservices
 
Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3
 
Citi Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging ModernizationCiti Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging Modernization
 
Citi Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time dataCiti Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time data
 
Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2
 
The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023
 
The Playful Bond Between REST And Data Streams
The Playful Bond Between REST And Data StreamsThe Playful Bond Between REST And Data Streams
The Playful Bond Between REST And Data Streams
 
The Journey to Data Mesh with Confluent
The Journey to Data Mesh with ConfluentThe Journey to Data Mesh with Confluent
The Journey to Data Mesh with Confluent
 
Citi Tech Talk: Monitoring and Performance
Citi Tech Talk: Monitoring and PerformanceCiti Tech Talk: Monitoring and Performance
Citi Tech Talk: Monitoring and Performance
 

Recently uploaded

Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdfMastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
mbmh111980
 
JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)
Max Lee
 
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
Alluxio, Inc.
 

Recently uploaded (20)

10 Essential Software Testing Tools You Need to Know About.pdf
10 Essential Software Testing Tools You Need to Know About.pdf10 Essential Software Testing Tools You Need to Know About.pdf
10 Essential Software Testing Tools You Need to Know About.pdf
 
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAGAI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
 
AI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning FrameworkAI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning Framework
 
Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdfMastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
 
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product UpdatesGraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
 
5 Reasons Driving Warehouse Management Systems Demand
5 Reasons Driving Warehouse Management Systems Demand5 Reasons Driving Warehouse Management Systems Demand
5 Reasons Driving Warehouse Management Systems Demand
 
AI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in MichelangeloAI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in Michelangelo
 
Secure Software Ecosystem Teqnation 2024
Secure Software Ecosystem Teqnation 2024Secure Software Ecosystem Teqnation 2024
Secure Software Ecosystem Teqnation 2024
 
JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)
 
IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024
 
OpenChain @ LF Japan Executive Briefing - May 2024
OpenChain @ LF Japan Executive Briefing - May 2024OpenChain @ LF Japan Executive Briefing - May 2024
OpenChain @ LF Japan Executive Briefing - May 2024
 
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
KLARNA -  Language Models and Knowledge Graphs: A Systems ApproachKLARNA -  Language Models and Knowledge Graphs: A Systems Approach
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
 
how-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdfhow-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdf
 
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
 
Crafting the Perfect Measurement Sheet with PLM Integration
Crafting the Perfect Measurement Sheet with PLM IntegrationCrafting the Perfect Measurement Sheet with PLM Integration
Crafting the Perfect Measurement Sheet with PLM Integration
 
CompTIA Security+ (Study Notes) for cs.pdf
CompTIA Security+ (Study Notes) for cs.pdfCompTIA Security+ (Study Notes) for cs.pdf
CompTIA Security+ (Study Notes) for cs.pdf
 
Agnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in KrakówAgnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in Kraków
 
How to pick right visual testing tool.pdf
How to pick right visual testing tool.pdfHow to pick right visual testing tool.pdf
How to pick right visual testing tool.pdf
 
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
 
Workforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfWorkforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdf
 

Confluent Partner Tech Talk with Synthesis

  • 1. @yourtwitterhandle | developer.confluent.io What are the best practices to debug client applications (producers/consumers in general but also Kafka Streams applications)? Starting soon… STARTING SOOOOON.. STARTING SOON… S Starting Soon… STARTING SOON..
  • 2. @yourtwitterhandle | developer.confluent.io What are the best practices to debug client applications (producers/consumers in general but also Kafka Streams applications)? Starting soon… STARTING SOOOOON.. STARTING SOON… S Starting Soon…
  • 3. Tech Talk Q4: Synthesis Software Technologies The Chronicles of Kafka: A Migration Journey Confluent Cloud Free Trial New signups receive $400 to spend during their first 30 days.
  • 4. Our Partner Technical Sales Enablement offering Scheduled sessions On-demand Join us for these live sessions where our experts will guide you through sessions of different level and will be available to answer your questions. Some examples of sessions are below: ● Confluent 101: for new starters ● Workshops ● Path to production series Learn the basics with a guided experience, at your own pace with our learning paths on-demand. You will also find an always growing repository of more advanced presentations to dig-deeper. Some examples are below: ● Confluent 10 ● Confluent Use Cases ● Positioning Confluent Value ● Confluent Cloud Networking ● … and many more AskTheExpert / Workshops For selected partners, we’ll offer additional support to: ● Technical Sales workshop ● JIT coaching on spotlight opportunity ● Build CoE inside partners by getting people with similar interest together ● Solution discovery ● Tech Talk ● Q&A
  • 5. Certifications + Accreditations 5 1. Certified Developer for Apache Kafka® Qualifications: Completion of On-Demand/ Instructor-Led Course Course Alignment: Confluent Developer Skills for Building Apache Kafka® Duration: 60 multiple choice questions in 90 minutes 4. Fundamentals Accredited Know the Basics Prework Apache and Quiz Demonstrates Kafka and Confluent Solution awareness 5. Technical Sales Associate Accredited Confluent Positioning Technical Sales Bootcamp Pre-work and Quiz Enablement to equip sales and pre-sales to articulate the Confluent value proposition 6. Technical Sales Professional Accredited Hands on, Coaching Sales Pitch. A deeper dive into important Confluent IP and differentiators in a way that allows attendees to demonstrate their knowledge of the product, its value, and how to demo it. 2. Certified Administrator for Apache Kafka® (Operations) Qualifications: Completion of On-Demand/ Instructor-Led Course Course Alignment: “Apache Kafka®” Administration by Confluent Kafka® Duration: 60 multiple choice questions in 90 minutes 3. PS Delivery Qualification COMING SOON Delivery-focused enablement, prescriptive delivery kits and work on a real customer use case with Confluent SME’s. Our subject matter experts will review the architecture and solution and provide real-time feedback as well as recommendations for additional enablement or shadowing as needed. NEW
  • 6. @yourtwitterhandle | developer.confluent.io What are the best practices to debug client applications (producers/consumers in general but also Kafka Streams applications)?
  • 7. Goal Partners Tech Talks are webinars where subject matter experts from a Partner talk about a specific use case or project. The goal of Tech Talks is to provide best practices and applications insights, along with inspiration, and help you stay up to date about innovations in confluent ecosystem.
  • 9. Today’s customers expect data in motion Initiate an action Instant confirmation Source: https://www.gigaspaces.com/blog/amazon-found-every-100ms-of-latency-cost-them-1-in-sales ~100 ms in latency can cost you… … 20% of digital traffic … $400M in revenue 9
  • 10. Finance & Banking Insurance Telecom Travel & Retail 10 OUT OF 10 8 OUT OF 8 Fortune 500 Companies Using Apache Kafka 70+% Transportation Energy & Utilities Entertainment Technology 8 OUT OF 10 9 OUT OF 10 10 OUT OF 10 10 OUT OF 10 10 OUT OF 10 8 OUT OF 10 Kafka is the de facto standard for Data in Motion
  • 11. Operationalizing Kafka on your own is difficult Kafka is hard in experimentation. It gets harder (and riskier) as you add mission-critical data and use cases. “We are in the business of selling and renting clothes. We are not in the business of managing an event streaming platform… If we had to manage everything ourselves, I would’ve had to hire at least 10 more people to keep the systems up and running." ● Architecture planning ● Cluster sizing ● Cluster provisioning ● Broker settings ● Zookeeper management ● Partition placement & data durability ● Source/sink connectors development & maintenance ● Monitoring & reporting tools setup ● Software patches and upgrades ● Security controls and integrations ● Failover design & planning ● Mirroring & geo-replication ● Streaming data governance ● Load rebalancing & monitoring ● Expansion planning & execution ● Utilization optimization & visibility ● Cluster migrations ● Infrastructure & performance upgrades / enhancements I N V E S T M E N T & T I M E V A L U E 1 2 3 4 5 Experimentation / Early Interest Central Nervous System Mission critical, disparate LOBs Identify a Project Mission-critical, connected LOBs Key challenges: Operational burden & resources Manage and scale platform to support ever-growing demand Security & governance Ensure streaming data is as safe & secure as data-at-rest as Kafka usage scales Real-time connectivity & processing Leverage valuable legacy data to power modern, cloud-based apps & experiences Global availability Maintain high availability across environments with minimal downtime
  • 12. Business Risk ● Increasing risk of costly downtime and security breaches as Kafka spans more use cases, apps / data systems, teams, and environments ● Valuable resources diverted to address unplanned downtime and breaches Unplanned downtime and breaches Substantial hidden costs incl. lost revenue, reputation damage, fines / audits, reduced CSAT, and data loss Infra Spend ● Over-provisioning required to support fluctuating demand ● Inability to scale storage w/o growing compute ● Performance degradations with manual configs, re-balancing, and upgrades / patching What does this all mean for your business? Unpacking the direct and indirect costs of self-supporting Open Source Kafka FTE Costs It’s hard because... Which results in... ● Costly time & resources spent building / maintaining low level infra tools vs. differentiating the business ● Increasingly difficult to hire and retain Kafka talent (job listings & compensation +40% YoY) Delayed time-to-value 2+ years to reach production at scale with ~$3-5M+ in ongoing platform development & operations costs Increased Total Cost of Ownership Hundreds of thousands of $$$’s spent annually on underutilized & sub-optimal infra
  • 13. Confluent Cloud Cloud-native data streaming platform built by the founders of Apache Kafka® Everywhere Connect your data in real time with a platform that spans from on-prem to cloud and across clouds Complete Go above & beyond Kafka with all the essential tools for a complete data streaming platform Cloud-Native The 10x Apache Kafka® service: elastic, resilient and performant, powered by the Kora Engine Stream confidently on the world’s most trusted data streaming platform built by the founders of Apache Kafka©, with resilience, security, compliance, and privacy built-in by default. 13
  • 14. “Before Confluent, when we had broker outages that required rebuilds, it could take up to three days of developer time to resolve. Now, Confluent takes care of everything for us, so our developers can focus on building new features and applications.” Elastic scalability: Scale and shrink to handle GBps+ workloads and petabyte storage 10x faster and easier without any operational burden Guaranteed resiliency: Leave Kafka reliability worries behind with the industries most comprehensive 99.99% uptime SLA and built-in data durability Low latency: Speed up real-time analytics and customer experiences with predictable low latency, sustained over time Cloud-Native The 10x Apache Kafka® service: elastic, resilient and performant, powered by the Kora Engine Kora Engine: The Apache Kafka® engine built for the cloud
  • 15. Complete Go above & beyond Kafka with all the essential tools for a complete data streaming platform Connectors & Stream Processing: Connect to and from any app / system and process your data streams in-flight Reduce TCO and architectural complexity with our portfolio of 120+ pre-built connectors and stream processing powered by ksqlDB, all available fully managed and built-in with Confluent Cloud Stream Designer: Quickly build and deploy streaming apps & pipelines Rapidly build, test, and deploy streaming data pipelines with Stream Designer, extensible with SQL, while reducing the need to write boilerplate code Security & Governance: Secure, discover, and organize your data streams Build trust and put your data streams to work with enterprise-grade security and the only Stream Governance suite for data in motion “BHG is a fast-moving company, and Confluent is quickly becoming not only a central highway for our data with their vast connector portfolio, but a streaming transformation engine as well for a vast number of use cases… We are making Confluent the true backbone of BHG, including leveraging 20+ Confluent connectors across both modern, cloud-based technologies & legacy systems, to help integrate our critical apps & data systems together.” 15 Connectors Security Data Governance Stream Processing Monitoring Global Resilience Stream Designer
  • 16. Everywhere Connect your data in real time with a platform that spans from on-prem to cloud and across clouds Run Anywhere: Deploy across any environment Provision Confluent as a fully managed service on AWS, Azure, and Google Cloud across 60+ regions w/ Confluent Cloud, or on-premises w/ Confluent Platform Unified: Unify data across hybrid and multi-clouds Provide consistent, self-service access to real-time data across all your environments with Cluster Linking and globally connected clusters that perfectly mirror data Consistent: Learn one platform for all environments Remove the burden of learning new tools for each environment with a consistent experience spanning across cloud, on-prem, and hybrid / multicloud “Our transformation to a cloud-native, agile company required a large-scale migration from open source Apache Kafka. With Confluent, we now support real-time data sharing across all of our environments, and see a clear path forward for our hybrid cloud roadmap.” 16
  • 17. …enabling you to deploy, operate, and scale in minutes instead of months Kafka re-architected to be truly Cloud-Native 17 Months Minutes Weeks Open Source Apache Kafka In-house development and maintenance without support Self-managed Kafka Services Manual operations with basic tooling and/or support Confluent Cloud Fully managed, elastic, and automated product capabilities with zero overhead
  • 18. Confluent Cloud for Apache Flink® Simple, Serverless Stream Processing Easily build high-quality, reusable data streams with the industry’s only cloud- native, serverless Flink service Effortlessly filter, join, and enrich your data streams with Flink, the de facto standard for stream processing Enable high-performance and efficient stream processing at any scale, without the complexities of infrastructure management Experience Kafka and Flink as a unified platform, with fully integrated monitoring, security, and governance Available for preview in select regions – see the docs for regional availability
  • 19. C O N F I D E N T I A L Process data streams in-flight to maximize actionability, fidelity, and portability Blob storage 3rd party app Databases Data Warehouse Database SaaS app Low latency apps and data pipelines Consistent, reusable data products Optimized resource utilization
  • 20. Build streaming data pipelines to inform real-time decision making Create new enriched and curated streams of higher value using: ● Data transformations ● Streaming joins, temporal joins, lookup joins, and versioned joins ● Fan out queries, multi-cluster queries 20 t1, 21.5 USD t3, 55 EUR t5, 35.3 EUR t0, EUR:USD=1.00 t2, EUR:USD=1.05 t4: EUR:USD=1.10 t1, 21.5 USD t3, 57.75 USD t5, 38.83 USD Currency rate Orders STREAMING DATA PIPELINES
  • 21. How to: Migrate to Confluent Cloud
  • 22. Migration Overview Kafka Clients & Streaming Apps Kafka Platform Managed Self-Managed Migrate to Confluent Cloud involves moving the underlying Kafka Platform, Kafka client & Streaming Applications. Depending on the situation, some Kafka Clients could transition into managed world and some Kafka Clients might remain as self-managed. One of the key decision is driven by whether private network is required or not. Connector Apps (Producer/Consumer) ksqlDB Schema Connector ksqlDB Schema Confluent Platform Confluent Cloud Repoint Self-Managed Applications to Confluent Cloud
  • 23. 23 Real-time connectivity & processing Security & governance Operational burden & resources Apache Kafka Confluent Tedious configuration. Manual & resource- intensive day-to-day operations to deploy, manage, and scale clusters. Custom-built security tools & integrations that creates tech & operational debt. Lacks governance for streaming data. Instant cluster provisioning and scaling. Automated capacity mgmt, load balancing, and upgrades. Battle-tested and enterprise-grade security & governance tools ready out-of-the-box. (e.g., RBAC, Audit Logs, Encryption, Stream Governance) Siloed solutions and custom integrations, escalating complexity & costs to manage and maintain platform as it scales. Enterprise-wide data fabric across apps & data systems with a rich, pre-built connector ecosystem and stream processing. Global availability Resource-intensive failover design & operations, exponentially increasing in complexity & risk as volume and variety of environments grows. Highly available and consistent access to data across all of your environments, spanning all clouds & on-prem. “Confluent has become a lynchpin to harness Apache Kafka for improved operational transparency and timely data-driven decisions. We are delighted with the improvements we have seen in the monitoring of our business flows, speeding up lending approvals, and providing better and more timely fraud analytics, and look forward to expanding services to leverage the platform to simplify, automate, and drive business decisions.” These capabilities make us the perfect partner to address the key challenges with Open Source Kafka
  • 24. @yourtwitterhandle | developer.confluent.io What are the best practices to debug client applications (producers/consumers in general but also Kafka Streams applications)? Starting soon… STARTING SOOOOON.. STARTING SOON… S Starting Soon…
  • 25. The Chronicles of Kafka: A Migration Story Bhavesh Sooka and Jonathan Lew
  • 26. Who we are Kafkateer Bhavesh Sooka Kafkateer Jonathan Lew Who we are
  • 27. Our Goal Our Goal For the audience to learn something from our migration experience and take it back to their migration journey.
  • 29. Gamified Fitness VDP & V1 - 19 clusters - 40 Markets Globally using 8 AWS Regions - 27 million monthly users - 20 Device Manufacturers - 10 million workouts/day - 124m messages in a day - 1000s of microservices
  • 30. Timeline Prologue 2015 – V1: Kafka 0.10 on EC2 2019 – V1: Upgrade Apache Kafka 0.10 -> CP 5.0.1 2021: - VDP: Tibco EMS -> CP 6.0.1 2014 – V1: Kafka on K8s 2015 – VDP: Tibco EMS (Event Messaging Service) 2020 – V1 Upgrade (x2) CP 5.0.1 -> 5.4.1 -> CP 6.0.1 2021 – The Migration: Enter your speakers 1 2 3 5 7 6 4 7 Years
  • 31. Timeline Nov 2021 – Feb 2022: ARB x2 June – Sept: Execution 4 years: Lead up Feb - June: Planning Oct: Hyper Care 4 5 3 2 1 Migration Timeline 11 Months
  • 33. Planning Planning took 4 Months Planning Outcomes - Design decisions - Configuration - Secret Storage - RBAC - Networking - Automation - Deployment plan
  • 34. Planning: Design Decisions Two DD’s we want to mention in specific are: Code and config split Vault Module
  • 36. SSO enabled and LDAP connectivity Planning: RBAC Split into 2 problems to Solve People Applications
  • 38. Planning: Deployment Day Goal: 100% Uptime During Migration • Move Producer • Drain • Move Consumer • Repeat
  • 39. Line Manager • Decision Maker • Autonomy • On our Side Operational Requests SNAG: CIO - Flight plan - Commercials Business Decisions/Red Tape
  • 40. This was child’s play!!!!! 4 Environments = 4 Months 1 month of Hyper Care Execution
  • 41. • Agile vs Waterfall • Stakeholders buy in and trust • Contextualized planning • Team dynamic • Business vs IT • Industry Experts in Kafka Learnings
  • 42. Happy Thank You Confluent, Jack and Dan Vitality Group/Vitality Global Synthesis You
  • 43. Happy And so, they lived happily ever after… #TagYourIt