SlideShare a Scribd company logo
© 2022, Amazon Web Services, Inc. or its affiliates.
© 2022, Amazon Web Services, Inc. or its affiliates.
AWS + Confluent Immersion
Day
Building Secure, Event-Driven
Microservices with Confluent
Cloud on AWS
November 15, 2023
© 2022, Amazon Web Services, Inc. or its affiliates.
© 2023, Amazon Web Services, Inc. or its affiliates.
Today’s hosts:
2
Ahmed Zamzam
Senior Partner Solutions Architect,
Confluent
Nuno Barreto
Partner Sales Solutions Architect, AWS
© 2022, Amazon Web Services, Inc. or its affiliates.
Agenda
• Building Modern Streaming Analytics with Confluent on AWS
with Nuno Barreto, Partner Solutions Architect, AWS
• Event Streaming made easy with Confluent
with Ahmed Zamzam, Sr Partner Solutions Architect, Confluent
• Lab: Building end-to-end streaming data pipelines with Confluent Cloud
with Ahmed Zamzam, Sr Partner Solutions Architect, Confluent
3
© 2022, Amazon Web Services, Inc. or its affiliates.
© 2022, Amazon Web Services, Inc. or its affiliates.
Building Modern Streaming Analytics
with Confluent on AWS
Nuno Barreto (he/him)
Partner Sales Solutions Architect
Amazon Web Services (AWS)
#DatainMotionTour
© 2022, Amazon Web Services, Inc. or its affiliates.
Agenda
Develop a modern data strategy
Act on real time with data streaming
Build seamless streaming with AWS and Confluent
Innovate together to power customer success
Key take aways
© 2022, Amazon Web Services, Inc. or its affiliates.
Data volume and velocity
Multiple analytics needs
Variety of sources and data types
Difficult to manage systems
Complex to scale
Inflexible tools
Security, compliance
Slow performance Increasing and unpredictable cost
Challenges of data analytics at scale
© 2022, Amazon Web Services, Inc. or its affiliates.
Catalog
Governance
Analytics
Machine
learning Databases
Data
lakes
Modern data strategy
People,
apps and
devices
Data
sources
© 2022, Amazon Web Services, Inc. or its affiliates.
Modern data architecture on AWS
Key Pillars
Data at any
scale
Seamless data access and
movement
Purpose-built for best price performance
Built-in ML to solve business
challenges
Unified
governance
© 2022, Amazon Web Services, Inc. or its affiliates.
The benefits of data lakes
Catalog
Store all your data in open formats
Decouple storage from compute
Cost-effectively scale storage to exabytes
Process data in place
Choice of analytical and ML engines
© 2022, Amazon Web Services, Inc. or its affiliates.
Sharing across data movement with Data Mesh
Data producers Data mesh Data consumers
Unique modern data architecture
Suited to business function
Teams that want to share data
Unique modern data architecture
Suited to business function
Team that runs the marketplace Teams that want to use data
© 2022, Amazon Web Services, Inc. or its affiliates.
Source: Perishable insights, Mike Gualtieri, Forrester
Real time Seconds Minutes Hours Days Months
Value
of
data
to
decision-making
Preventive/predictive
Actionable Reactive Historical
Time-critical decisions Traditional “batch” business intelligence
Information half-life
in decision-making
Why real-time data streaming analytics ?
Data loses value quickly over time
© 2022, Amazon Web Services, Inc. or its affiliates.
Common real-time analytics use cases
Anomaly and fraud detection
Empowering IoT analytics
Nourishing marketing campaigns
Real-time personalization
Tailoring customer experience in real time
Supporting healthcare and emergency services
© 2022, Amazon Web Services, Inc. or its affiliates.
Challenges of data streaming
Difficult to set up Tricky to scale
Hard to achieve high availability Integration requires development
Error prone and complex to manage Expensive to maintain
© 2022, Amazon Web Services, Inc. or its affiliates.
Source
Devices and/or
applications
that produce
real-time
data at high
velocity
Stream ingestion
Data from tens of
thousands of data
sources can be written to
a single stream
In-stream storage
Data are stored in the
order they were received
for a set duration
of time and can be
replayed indefinitely
during that time
Stream processing
Records are read in
the order they are
produced, enabling real-
time analytics or streaming
ETL
Destination/
Storage
Database (NoSQL
most common)
Data lake
Data warehouse
Event driven
Applications
`
Visualize
Analytics
dashboard
Real-time streaming analytics pipeline
Ingest, Process & Analyze High Volumes of High-Velocity Data from Various Sources in Real
Time
© 2022, Amazon Web Services, Inc. or its affiliates.
Stream processing Visualize
Source
Mobile device
Metering
Click streams
IoT sensors
AWS SDKs
Kinesis Agent/KPL
Stream ingestion
`
Apache Kafka
Amazon Kinesis Data
Streams
Amazon Kinesis Data
Firehose
In-stream storage
Apache Kafka
Amazon Kinesis Data
Streams
Amazon Kinesis
Data Analytics
AWS Glue Streaming
Amazon EMR
Destination/Storage
Amazon DynamoDB
Amazon EMR
Amazon S3
Amazon Redshift Amazon OpenSearch
Amazon QuickSight
Apache Kafka
Streaming analytics on AWS
© 2022, Amazon Web Services, Inc. or its affiliates.
Rich front-end
customer
experiences
Real-time
Event Streams and Analysis
A Sale A shipment
A Trade
A Customer
Experience
Real-time backend
operations
Event streaming with Kafka to set Data in Motion:
Continuously processing evolving streams of data in real
time
© 2022, Amazon Web Services, Inc. or its affiliates.
Out-of-box integration with
popular services
Certified and validated by
AWS
AWS Native Services
Top-5 Global ISV for S3 Data Volume
3rd-Party ISV Services
• Amazon RDS Ready
• AWS Lambda Ready
• Amazon Redshift Ready
• AWS PrivateLink Ready
• AWS Outposts Ready
Validated Service Designations
Confluent integrations with AWS
© 2022, Amazon Web Services, Inc. or its affiliates.
S
Confluent Cloud on AWS – reference architecture
S3
© 2022, Amazon Web Services, Inc. or its affiliates.
Confluent + AWS: Accelerate Customer Business Outcomes
Topline Impacting New
Experiences
● Event-driven & real-time
● Unify data across org. w/ Kafka
data fabric (Schema Reg,..)
● AWS Analytics, Redshift, ML
connectors
Mitigate Risk
● Higher Service Quality &
Resilience with 99.99% SLA
● Deep Kafka expertise & innovation
● Elastic billing/pricing
Developer Agility
● Focus on innovation (not data
infrastructure)
● Leverage full Kafka OSS
ecosystem + AWS services
Faster Time to Market
● ~50-75% faster time to market*
● Streamline hybrid cloud
migration with no complex lift-n-
shift
● Maintain business continuity
Lower Kafka TCO
● ~25-50% lower TCO *
● GBps-scale & fast deployments
for global expansion
● Deploy Kafka at scale in 1 week
Maximize ROI
● ~200% ROI per Forrester study
● Save 10s of $Ms with legacy
offload to AWS with Confluent
Replicator
* For customers that don’t already have Kafka based system in-market
* TCO assessment to be analyzed for specific customer scenarios
© 2022, Amazon Web Services, Inc. or its affiliates.
Accelerate modernization from on-prem to AWS
Redshift Sink
Lambda Sink
AWS Direct
Connect
Replicator
LEGACY EDW
MAINFRAME
LEGACY DB
JDBC / CDC
connectors
Connect
Leverage 100+ Confluent pre-built connectors
Modernize
Value added apps, increase agility, reduce
TCO
On-prem AWS Cloud
Bridge
Hybrid cloud streaming
Amazon Athena
AWS Glue
SageMaker
Lake Formation
Amazon
DynamoDB
Amazon
Aurora
S3 Sink
Data Streams
Apps
ksqlDB
© 2022, Amazon Web Services, Inc. or its affiliates.
Confluent Wavelength solution with AWS
© 2022, Amazon Web Services, Inc. or its affiliates.
Increase developer agility & speed of innovation
Serverless integration
Connect for effortless integrations with Lambda & data stores
AWS serverless platform
Free up backend operation & Infras. management
Apps
Microservices
ksqlDB
Schema
Registry
COMPUTE
AWS
Lambda
Data stores
REST Proxy
& Clients
Source
Connectors
Lambda
Sink
DATA STORES
Amazon
DynamoDB
Amazon
Aurora
STORAGE
Amazon
S3
S3 Sink
ANALYTICS
Amazon
Athena
Amazon
Redshift
© 2022, Amazon Web Services, Inc. or its affiliates.
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Confluent
Data in Motion
23
Ahmed Zamzam
Sr Partner Solutions Engineer
© 2022, Amazon Web Services, Inc. or its affiliates.
...many more
Other
systems
Other
systems
Kafka Connect
Kafka Cluster
Kafka Connect
Apache Kafka is an event streaming platform
24
© 2022, Amazon Web Services, Inc. or its affiliates.
Core Kafka features
01
Publish & Subscribe
to streams of events
02
Store
your event streams
03
Process & Analyze
your events streams
26
© 2022, Amazon Web Services, Inc. or its affiliates.
ksqlDB
27
© 2022, Amazon Web Services, Inc. or its affiliates.
70%
of Fortune 500 companies
use Apache Kafka
(majority are Confluent customers)
The rise of data in motion
28
© 2022, Amazon Web Services, Inc. or its affiliates.
32,232
Stack overflow
questions
210
meetups
69,000
meetup attendees
10,286
Jira tickets for
Apache Kafka
46,626
Emails sent to
Apache Kafka mailing list
988
KIPs
Community
29
© 2022, Amazon Web Services, Inc. or its affiliates.
Hall of Innovation
CTO Innovation
Award Winner
2019
Enterprise Technology Innovation
AWARDS
Vision
● Original Kafka creators
founded Confluent
● Data in motion pioneers
Category leadership
● 80% of Kafka commits
● 5M+ hours of Kafka
technical experience
● Operate 35K+ clusters
Value
● Remove risk
● Deploy at scale
● Accelerate time to market
Product
● Extends Kafka to be
secure and
enterprise-ready
● Software or
cloud-native service
Confluent is the only company
focused on data in motion
30
© 2022, Amazon Web Services, Inc. or its affiliates.
Confluent: Everywhere
Confluent Platform
The Enterprise Distribution of
Apache Kafka
Confluent Cloud
Apache Kafka Re-engineered
for the Cloud
Self-Managed Software
Fully-Managed Service
VM
Deploy on any platform, on-prem or cloud
Available on the leading public clouds
31
© 2022, Amazon Web Services, Inc. or its affiliates.
Federated streaming, hybrid
and multi-cloud.
Data syndication and replication
across and between clouds and on-
premises, with self-service APIs, data
governance, and visual tooling.
Reliable & real-time data streams
between all customer sites, so you
can run always-on streaming
analytics on the data of the entire
enterprise, despite regional or cloud
provider outages.
Everywhere:
Cluster Linking Global Central Nervous System
© 2022, Amazon Web Services, Inc. or its affiliates.
K A F K A C L O U D
33
© 2022, Amazon Web Services, Inc. or its affiliates.
We Transformed Kafka for the Cloud, Ground Up!
Resilient
with automated
operations to ensure
high availability and
reliability
Performant
with networking
service decoupling
and replication
optimization
Elastic
to seamlessly
expand and shrink
based on customer
demands
KORA ENGINE
The Apache Kafka® Engine Built for the Cloud
Cost efficient
with multi-tenancy,
data tiering, cloud
optimizations and
hands-off operations
© 2022, Amazon Web Services, Inc. or its affiliates.
Event streaming as a cloud native service
Deploy in Minutes
Ensure
service levels
Integrate with cloud
services
Eliminate operational
burden
35
© 2022, Amazon Web Services, Inc. or its affiliates.
Scale faster with serverless development
Stream with Kafka
in minutes
Provision and scale
clusters on-demand
Pay only for
what you use
Start from $0 and pay based
on actual usage
36
Access latest
Kafka version
Always work in the most recent
stable version of Kafka
© 2022, Amazon Web Services, Inc. or its affiliates.
Go to market in 6 months vs. 2+ years
Traditional Kafka development
Development with Confluent Cloud
6-9 months to hire
Kafka resources
3 months
to ramp
Project
Kickoff
9-12 months build the production-grade
Kafka platform and develop application
Start
in 1
week
3-6 months to focus on app
development, not managing
Kafka
Project
Kickoff
Launch your application in months
and grow your business
Go to market in 2 years
Go to market in 6 months
37
© 2022, Amazon Web Services, Inc. or its affiliates.
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
38
Infinite Data
© 2022, Amazon Web Services, Inc. or its affiliates.
Infinite Apache Kafka
Scalable
Quickly scale up storage
resources without
overprovisioning infra
Efficient
Decouple storage from
compute resources to keep
storage costs low
Infinite
Retain as much data in Kafka
as you need to meet any use
case
39
Decouples the storage and compute layers to create highly scalable,
cost-effective storage without limits to enable infinite retention
© 2022, Amazon Web Services, Inc. or its affiliates.
Which use cases or
concerns does
Infinite address?
New customer experiences -
reimagine what you can do
with streaming through
infinite retention of events
Ease operational overhead -
stop worrying about running
out of disk space and simplify
topic configuration
40
© 2022, Amazon Web Services, Inc. or its affiliates.
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
41
Getting Data
into Kafka
© 2022, Amazon Web Services, Inc. or its affiliates.
Make Kafka
Widely Accessible
to Developers
Let developers build with their most
productive language and enable use
cases and architectures that require
multiple programming languages
Confluent Clients
Battle-tested and high performing
© 2022, Amazon Web Services, Inc. or its affiliates.
Apache Kafka Connect API:
Fault tolerant
Manage hundreds of
data sources and sinks
Preserves data schema
Integrated within
Confluent Control Center
JDBC
Mongo
MySQL
Elastic
HDFS
Kafka Connect API
Kafka Pipeline
Connector
Connector
Connector
Connector
Connector
Connector
Sources
Sinks
S3
43
Import and export data in and out of Kafka
© 2022, Amazon Web Services, Inc. or its affiliates.
140+
prebuilt
connectors
100+ Confluent supported 30+ partner supported, Confluent verified
AWS
Lambda
Instantly connect popular data sources & sinks
44
© 2022, Amazon Web Services, Inc. or its affiliates.
Confluent has deep AWS service integrations
© 2022, Amazon Web Services, Inc. or its affiliates.
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
46
Real-time
Transformations
© 2022, Amazon Web Services, Inc. or its affiliates.
Kafka Streams: Write standard Java apps and
microservices to process your data in real time
KStream<User, PageViewEvent> pageViews = builder.stream("pageviews-topic");
KTable<Windowed<User>, Long> viewsPerUserSession = pageViews
.groupByKey()
.count(SessionWindows.with(TimeUnit.MINUTES.toMillis(5)), "session-views");
docs.confluent.io/current/streams/
No separate processing cluster required
Develop on Mac, Linux, and Windows
Deploy to containers, VMs, bare metal, cloud
Powered by Kafka: elastic, scalable,
distributed, and reliable
Perfect for small, medium, and large use cases
Fully integrated with Kafka security
Exactly-once processing semantics
Part of Apache Kafka
47
© 2022, Amazon Web Services, Inc. or its affiliates.
ksqlDB at a glance
What is it?
ksqlDB is an event-streaming
database for working with
streams and tables of data
All the key features of a modern
streaming solution
Aggregations Joins
Windowing
Event-time
Dual query
support
Exactly-once
semantics
Out-of-order
handling
User-defined
functions
CREATE TABLE activePromotions AS
SELECT rideId,
qualifyPromotion(distanceToDst) AS
promotion
FROM locations
GROUP BY rideId
EMIT CHANGES
How does it work?
It separates compute from storage, and scales
elastically in a fault-tolerant manner
It remains highly available during disruption, even in
the face of failure to a quorum of its servers
© 2022, Amazon Web Services, Inc. or its affiliates.
3 modalities of stream processing with Confluent
Kafka clients Kafka Streams ksqlDB
ConsumerRecords<String, String> records = consumer.poll(100);
Map<String, Integer> counts = new DefaultMap<String, Integer>();
for (ConsumerRecord<String, Integer> record : records) {
String key = record.key();
int c = counts.get(key)
c += record.value()
counts.put(key, c)
}
for (Map.Entry<String, Integer> entry : counts.entrySet()) {
int stateCount;
int attempts;
while (attempts++ < MAX_RETRIES) {
try {
stateCount = stateStore.getValue(entry.getKey())
stateStore.setValue(entry.getKey(), entry.getValue() + stateCount)
break;
} catch (StateStoreException e) {
RetryUtils.backoff(attempts);
}
}
}
builder
.stream("input-stream",
Consumed.with(Serdes.String(), Serdes.String()))
.groupBy((key, value) -> value)
.count()
.toStream()
.to("counts", Produced.with(Serdes.String(), Serdes.Long()));
SELECT x, count(*) FROM stream GROUP BY x EMIT CHANGES;
Flexibility Simplicity
confluent.awsworkshop.io
49
© 2022, Amazon Web Services, Inc. or its affiliates.
Easily build event streaming applications
Use one, lightweight SQL
syntax to build a complete
real-time application
CREATE STREAM payments(user VARCHAR,
payment_amount INT)
WITH (kafka_topic = ’all_payments’,
key = ’user’,
value_format = ’avro’);
Create aggregations of
event data that can serve
queries to applications
Enrich Kafka data with a
robust stream processing
framework
USER Payment
Jay $10
Sue $15
Fred $5
... ...
USER Credit Score
Jay 660
Sue 710
Fred 595
USER Credit Score
Jay 660
Sue 710
Fred 595
USER Credit Score
Jay 660
Sue 710
Fred 595
50
© 2022, Amazon Web Services, Inc. or its affiliates.
How to get started
1. Subscribe to Confluent Cloud on the AWS Marketplace and start
with a free $400 (to be used within 60 days)
2. Try out the Confluent/AWS workshop found at
https://confluent.awsworkshop.io/
3. Email us at awsteam@confluent.io if you have any questions
51
© 2022, Amazon Web Services, Inc. or its affiliates.
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Hands-on workshop,
and Q&A
52
Ahmed Zamzam
Sr Partner Solutions Engineer, Confluent
© 2022, Amazon Web Services, Inc. or its affiliates.
Conclusion/Next steps
• Thank you for attending.
• Please fill out the AWS event survey and let
us know what you thought.
• If you have any further questions or wish to
schedule a call with an AWS and Confluent
expert, please contact
awsteam@confluent.io
53
© 2022, Amazon Web Services, Inc. or its affiliates.
Thank you!
© 2022, Amazon Web Services, Inc. or its affiliates. 54

More Related Content

What's hot

Databricks on AWS.pptx
Databricks on AWS.pptxDatabricks on AWS.pptx
Databricks on AWS.pptx
Wasm1953
 
Cloud Migration: Moving Data and Infrastructure to the Cloud
Cloud Migration: Moving Data and Infrastructure to the CloudCloud Migration: Moving Data and Infrastructure to the Cloud
Cloud Migration: Moving Data and Infrastructure to the Cloud
Safe Software
 
Cloud Migration Workshop
Cloud Migration WorkshopCloud Migration Workshop
Cloud Migration Workshop
Amazon Web Services
 
Data Catalog as the Platform for Data Intelligence
Data Catalog as the Platform for Data IntelligenceData Catalog as the Platform for Data Intelligence
Data Catalog as the Platform for Data Intelligence
Alation
 
The ABCs of Treating Data as Product
The ABCs of Treating Data as ProductThe ABCs of Treating Data as Product
The ABCs of Treating Data as Product
DATAVERSITY
 
Power up! Mass Migrations at Speed and Scale - Accenture
Power up! Mass Migrations at Speed and Scale - AccenturePower up! Mass Migrations at Speed and Scale - Accenture
Power up! Mass Migrations at Speed and Scale - Accenture
Amazon Web Services
 
2022 Trends in Enterprise Analytics
2022 Trends in Enterprise Analytics2022 Trends in Enterprise Analytics
2022 Trends in Enterprise Analytics
DATAVERSITY
 
Building Modern Data Platform with Microsoft Azure
Building Modern Data Platform with Microsoft AzureBuilding Modern Data Platform with Microsoft Azure
Building Modern Data Platform with Microsoft Azure
Dmitry Anoshin
 
Azure Cloud Adoption Framework + Governance - Sana Khan and Jay Kumar
Azure Cloud Adoption Framework + Governance - Sana Khan and Jay Kumar Azure Cloud Adoption Framework + Governance - Sana Khan and Jay Kumar
Azure Cloud Adoption Framework + Governance - Sana Khan and Jay Kumar
Timothy McAliley
 
Time to Talk about Data Mesh
Time to Talk about Data MeshTime to Talk about Data Mesh
Time to Talk about Data Mesh
LibbySchulze
 
The Ideal Approach to Application Modernization; Which Way to the Cloud?
The Ideal Approach to Application Modernization; Which Way to the Cloud?The Ideal Approach to Application Modernization; Which Way to the Cloud?
The Ideal Approach to Application Modernization; Which Way to the Cloud?
Codit
 
Microsoft Cloud Adoption Framework for Azure: Governance Conversation
Microsoft Cloud Adoption Framework for Azure: Governance ConversationMicrosoft Cloud Adoption Framework for Azure: Governance Conversation
Microsoft Cloud Adoption Framework for Azure: Governance Conversation
Nicholas Vossburg
 
Executing a Large-Scale Migration to AWS
Executing a Large-Scale Migration to AWSExecuting a Large-Scale Migration to AWS
Executing a Large-Scale Migration to AWS
Amazon Web Services
 
Building Open Data Lakes on AWS with Debezium and Apache Hudi
Building Open Data Lakes on AWS with Debezium and Apache HudiBuilding Open Data Lakes on AWS with Debezium and Apache Hudi
Building Open Data Lakes on AWS with Debezium and Apache Hudi
Gary Stafford
 
AWS Migration Planning Roadmap
AWS Migration Planning RoadmapAWS Migration Planning Roadmap
AWS Migration Planning Roadmap
Amazon Web Services
 
AWS Cloud Assessment
AWS Cloud AssessmentAWS Cloud Assessment
AWS Cloud Assessment
Michael Cronan
 
Data Catalog & ETL - Glue & Athena
Data Catalog & ETL - Glue & AthenaData Catalog & ETL - Glue & Athena
Data Catalog & ETL - Glue & Athena
Amazon Web Services
 
Data Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to MeshData Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to Mesh
Jeffrey T. Pollock
 
Accelerate Cloud Migration to AWS Cloud with Cognizant Cloud Steps
Accelerate Cloud Migration to AWS Cloud with Cognizant Cloud StepsAccelerate Cloud Migration to AWS Cloud with Cognizant Cloud Steps
Accelerate Cloud Migration to AWS Cloud with Cognizant Cloud Steps
Amazon Web Services
 
Getting started on your AWS migration journey
Getting started on your AWS migration journeyGetting started on your AWS migration journey
Getting started on your AWS migration journey
Amazon Web Services
 

What's hot (20)

Databricks on AWS.pptx
Databricks on AWS.pptxDatabricks on AWS.pptx
Databricks on AWS.pptx
 
Cloud Migration: Moving Data and Infrastructure to the Cloud
Cloud Migration: Moving Data and Infrastructure to the CloudCloud Migration: Moving Data and Infrastructure to the Cloud
Cloud Migration: Moving Data and Infrastructure to the Cloud
 
Cloud Migration Workshop
Cloud Migration WorkshopCloud Migration Workshop
Cloud Migration Workshop
 
Data Catalog as the Platform for Data Intelligence
Data Catalog as the Platform for Data IntelligenceData Catalog as the Platform for Data Intelligence
Data Catalog as the Platform for Data Intelligence
 
The ABCs of Treating Data as Product
The ABCs of Treating Data as ProductThe ABCs of Treating Data as Product
The ABCs of Treating Data as Product
 
Power up! Mass Migrations at Speed and Scale - Accenture
Power up! Mass Migrations at Speed and Scale - AccenturePower up! Mass Migrations at Speed and Scale - Accenture
Power up! Mass Migrations at Speed and Scale - Accenture
 
2022 Trends in Enterprise Analytics
2022 Trends in Enterprise Analytics2022 Trends in Enterprise Analytics
2022 Trends in Enterprise Analytics
 
Building Modern Data Platform with Microsoft Azure
Building Modern Data Platform with Microsoft AzureBuilding Modern Data Platform with Microsoft Azure
Building Modern Data Platform with Microsoft Azure
 
Azure Cloud Adoption Framework + Governance - Sana Khan and Jay Kumar
Azure Cloud Adoption Framework + Governance - Sana Khan and Jay Kumar Azure Cloud Adoption Framework + Governance - Sana Khan and Jay Kumar
Azure Cloud Adoption Framework + Governance - Sana Khan and Jay Kumar
 
Time to Talk about Data Mesh
Time to Talk about Data MeshTime to Talk about Data Mesh
Time to Talk about Data Mesh
 
The Ideal Approach to Application Modernization; Which Way to the Cloud?
The Ideal Approach to Application Modernization; Which Way to the Cloud?The Ideal Approach to Application Modernization; Which Way to the Cloud?
The Ideal Approach to Application Modernization; Which Way to the Cloud?
 
Microsoft Cloud Adoption Framework for Azure: Governance Conversation
Microsoft Cloud Adoption Framework for Azure: Governance ConversationMicrosoft Cloud Adoption Framework for Azure: Governance Conversation
Microsoft Cloud Adoption Framework for Azure: Governance Conversation
 
Executing a Large-Scale Migration to AWS
Executing a Large-Scale Migration to AWSExecuting a Large-Scale Migration to AWS
Executing a Large-Scale Migration to AWS
 
Building Open Data Lakes on AWS with Debezium and Apache Hudi
Building Open Data Lakes on AWS with Debezium and Apache HudiBuilding Open Data Lakes on AWS with Debezium and Apache Hudi
Building Open Data Lakes on AWS with Debezium and Apache Hudi
 
AWS Migration Planning Roadmap
AWS Migration Planning RoadmapAWS Migration Planning Roadmap
AWS Migration Planning Roadmap
 
AWS Cloud Assessment
AWS Cloud AssessmentAWS Cloud Assessment
AWS Cloud Assessment
 
Data Catalog & ETL - Glue & Athena
Data Catalog & ETL - Glue & AthenaData Catalog & ETL - Glue & Athena
Data Catalog & ETL - Glue & Athena
 
Data Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to MeshData Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to Mesh
 
Accelerate Cloud Migration to AWS Cloud with Cognizant Cloud Steps
Accelerate Cloud Migration to AWS Cloud with Cognizant Cloud StepsAccelerate Cloud Migration to AWS Cloud with Cognizant Cloud Steps
Accelerate Cloud Migration to AWS Cloud with Cognizant Cloud Steps
 
Getting started on your AWS migration journey
Getting started on your AWS migration journeyGetting started on your AWS migration journey
Getting started on your AWS migration journey
 

Similar to Build real-time streaming data pipelines to AWS with Confluent

Building Modern Streaming Analytics with Confluent on AWS
Building Modern Streaming Analytics with Confluent on AWSBuilding Modern Streaming Analytics with Confluent on AWS
Building Modern Streaming Analytics with Confluent on AWS
confluent
 
Building real-time serverless data applications with Confluent and AWS.pptx
Building real-time serverless data applications with Confluent and AWS.pptxBuilding real-time serverless data applications with Confluent and AWS.pptx
Building real-time serverless data applications with Confluent and AWS.pptx
Ahmed791434
 
Building real-time serverless data applications with Confluent and AWS - Lond...
Building real-time serverless data applications with Confluent and AWS - Lond...Building real-time serverless data applications with Confluent and AWS - Lond...
Building real-time serverless data applications with Confluent and AWS - Lond...
Ahmed791434
 
Single View of Data
Single View of DataSingle View of Data
Single View of Data
confluent
 
Serverless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake ArchitectureServerless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Kai Wähner
 
Confluent:AWS - GameDay.pptx
 Confluent:AWS - GameDay.pptx Confluent:AWS - GameDay.pptx
Confluent:AWS - GameDay.pptx
Ahmed791434
 
Best Practices for Cloud Migrations with Zero Disruption with AWS Marketplace
Best Practices for Cloud Migrations with Zero Disruption with AWS MarketplaceBest Practices for Cloud Migrations with Zero Disruption with AWS Marketplace
Best Practices for Cloud Migrations with Zero Disruption with AWS Marketplace
Denodo
 
Serverless Architecture 101 ⚡
Serverless Architecture 101 ⚡Serverless Architecture 101 ⚡
Serverless Architecture 101 ⚡
Thanh Nguyen
 
Get More from your Data: Accelerate Time-to-Value and Reduce TCO with Conflue...
Get More from your Data: Accelerate Time-to-Value and Reduce TCO with Conflue...Get More from your Data: Accelerate Time-to-Value and Reduce TCO with Conflue...
Get More from your Data: Accelerate Time-to-Value and Reduce TCO with Conflue...
HostedbyConfluent
 
Building Real-Time Serverless Data Applications With Joseph Morais and Adam W...
Building Real-Time Serverless Data Applications With Joseph Morais and Adam W...Building Real-Time Serverless Data Applications With Joseph Morais and Adam W...
Building Real-Time Serverless Data Applications With Joseph Morais and Adam W...
HostedbyConfluent
 
SAP Modernization with AWS
SAP Modernization with AWSSAP Modernization with AWS
SAP Modernization with AWS
Amazon Web Services
 
Data & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdf
Data & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdfData & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdf
Data & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdf
Chris Bingham
 
AWSome Day Singapore Keynote 2015
AWSome Day Singapore Keynote 2015AWSome Day Singapore Keynote 2015
AWSome Day Singapore Keynote 2015
Hwee Bee Tan
 
Speed up data preparation for ML pipelines on AWS
Speed up data preparation for ML pipelines on AWSSpeed up data preparation for ML pipelines on AWS
Speed up data preparation for ML pipelines on AWS
Data Science Milan
 
Saurabh_Shanbhag - Building_SaaS_on_AWS.pptx
Saurabh_Shanbhag - Building_SaaS_on_AWS.pptxSaurabh_Shanbhag - Building_SaaS_on_AWS.pptx
Saurabh_Shanbhag - Building_SaaS_on_AWS.pptx
AWS Chicago
 
Slides: Proven Strategies for Hybrid Cloud Computing with Mainframes — From A...
Slides: Proven Strategies for Hybrid Cloud Computing with Mainframes — From A...Slides: Proven Strategies for Hybrid Cloud Computing with Mainframes — From A...
Slides: Proven Strategies for Hybrid Cloud Computing with Mainframes — From A...
DATAVERSITY
 
Sederhanakan_integrasi_data_anda_dengan_AWS_Glue_handout.pdf
Sederhanakan_integrasi_data_anda_dengan_AWS_Glue_handout.pdfSederhanakan_integrasi_data_anda_dengan_AWS_Glue_handout.pdf
Sederhanakan_integrasi_data_anda_dengan_AWS_Glue_handout.pdf
Jazzy44
 
Realize Value of Your Microsoft Investments- Transformation Day Philadelphia ...
Realize Value of Your Microsoft Investments- Transformation Day Philadelphia ...Realize Value of Your Microsoft Investments- Transformation Day Philadelphia ...
Realize Value of Your Microsoft Investments- Transformation Day Philadelphia ...
Amazon Web Services
 
Realize Value of Your Microsoft Investments - Transformation Day Montreal 2018
Realize Value of Your Microsoft Investments - Transformation Day Montreal 2018Realize Value of Your Microsoft Investments - Transformation Day Montreal 2018
Realize Value of Your Microsoft Investments - Transformation Day Montreal 2018
Amazon Web Services
 
Enabling Your Organization’s Amazon Redshift Adoption – Going from Zero to He...
Enabling Your Organization’s Amazon Redshift Adoption – Going from Zero to He...Enabling Your Organization’s Amazon Redshift Adoption – Going from Zero to He...
Enabling Your Organization’s Amazon Redshift Adoption – Going from Zero to He...
Amazon Web Services
 

Similar to Build real-time streaming data pipelines to AWS with Confluent (20)

Building Modern Streaming Analytics with Confluent on AWS
Building Modern Streaming Analytics with Confluent on AWSBuilding Modern Streaming Analytics with Confluent on AWS
Building Modern Streaming Analytics with Confluent on AWS
 
Building real-time serverless data applications with Confluent and AWS.pptx
Building real-time serverless data applications with Confluent and AWS.pptxBuilding real-time serverless data applications with Confluent and AWS.pptx
Building real-time serverless data applications with Confluent and AWS.pptx
 
Building real-time serverless data applications with Confluent and AWS - Lond...
Building real-time serverless data applications with Confluent and AWS - Lond...Building real-time serverless data applications with Confluent and AWS - Lond...
Building real-time serverless data applications with Confluent and AWS - Lond...
 
Single View of Data
Single View of DataSingle View of Data
Single View of Data
 
Serverless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake ArchitectureServerless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
 
Confluent:AWS - GameDay.pptx
 Confluent:AWS - GameDay.pptx Confluent:AWS - GameDay.pptx
Confluent:AWS - GameDay.pptx
 
Best Practices for Cloud Migrations with Zero Disruption with AWS Marketplace
Best Practices for Cloud Migrations with Zero Disruption with AWS MarketplaceBest Practices for Cloud Migrations with Zero Disruption with AWS Marketplace
Best Practices for Cloud Migrations with Zero Disruption with AWS Marketplace
 
Serverless Architecture 101 ⚡
Serverless Architecture 101 ⚡Serverless Architecture 101 ⚡
Serverless Architecture 101 ⚡
 
Get More from your Data: Accelerate Time-to-Value and Reduce TCO with Conflue...
Get More from your Data: Accelerate Time-to-Value and Reduce TCO with Conflue...Get More from your Data: Accelerate Time-to-Value and Reduce TCO with Conflue...
Get More from your Data: Accelerate Time-to-Value and Reduce TCO with Conflue...
 
Building Real-Time Serverless Data Applications With Joseph Morais and Adam W...
Building Real-Time Serverless Data Applications With Joseph Morais and Adam W...Building Real-Time Serverless Data Applications With Joseph Morais and Adam W...
Building Real-Time Serverless Data Applications With Joseph Morais and Adam W...
 
SAP Modernization with AWS
SAP Modernization with AWSSAP Modernization with AWS
SAP Modernization with AWS
 
Data & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdf
Data & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdfData & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdf
Data & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdf
 
AWSome Day Singapore Keynote 2015
AWSome Day Singapore Keynote 2015AWSome Day Singapore Keynote 2015
AWSome Day Singapore Keynote 2015
 
Speed up data preparation for ML pipelines on AWS
Speed up data preparation for ML pipelines on AWSSpeed up data preparation for ML pipelines on AWS
Speed up data preparation for ML pipelines on AWS
 
Saurabh_Shanbhag - Building_SaaS_on_AWS.pptx
Saurabh_Shanbhag - Building_SaaS_on_AWS.pptxSaurabh_Shanbhag - Building_SaaS_on_AWS.pptx
Saurabh_Shanbhag - Building_SaaS_on_AWS.pptx
 
Slides: Proven Strategies for Hybrid Cloud Computing with Mainframes — From A...
Slides: Proven Strategies for Hybrid Cloud Computing with Mainframes — From A...Slides: Proven Strategies for Hybrid Cloud Computing with Mainframes — From A...
Slides: Proven Strategies for Hybrid Cloud Computing with Mainframes — From A...
 
Sederhanakan_integrasi_data_anda_dengan_AWS_Glue_handout.pdf
Sederhanakan_integrasi_data_anda_dengan_AWS_Glue_handout.pdfSederhanakan_integrasi_data_anda_dengan_AWS_Glue_handout.pdf
Sederhanakan_integrasi_data_anda_dengan_AWS_Glue_handout.pdf
 
Realize Value of Your Microsoft Investments- Transformation Day Philadelphia ...
Realize Value of Your Microsoft Investments- Transformation Day Philadelphia ...Realize Value of Your Microsoft Investments- Transformation Day Philadelphia ...
Realize Value of Your Microsoft Investments- Transformation Day Philadelphia ...
 
Realize Value of Your Microsoft Investments - Transformation Day Montreal 2018
Realize Value of Your Microsoft Investments - Transformation Day Montreal 2018Realize Value of Your Microsoft Investments - Transformation Day Montreal 2018
Realize Value of Your Microsoft Investments - Transformation Day Montreal 2018
 
Enabling Your Organization’s Amazon Redshift Adoption – Going from Zero to He...
Enabling Your Organization’s Amazon Redshift Adoption – Going from Zero to He...Enabling Your Organization’s Amazon Redshift Adoption – Going from Zero to He...
Enabling Your Organization’s Amazon Redshift Adoption – Going from Zero to He...
 

More from confluent

Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutes
confluent
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
confluent
 
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
confluent
 
Santander Stream Processing with Apache Flink
Santander Stream Processing with Apache FlinkSantander Stream Processing with Apache Flink
Santander Stream Processing with Apache Flink
confluent
 
Unlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insightsUnlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insights
confluent
 
Workshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con FlinkWorkshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con Flink
confluent
 
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
confluent
 
AWS Immersion Day Mapfre - Confluent
AWS Immersion Day Mapfre   -   ConfluentAWS Immersion Day Mapfre   -   Confluent
AWS Immersion Day Mapfre - Confluent
confluent
 
Eventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalkEventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalk
confluent
 
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent CloudQ&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
confluent
 
Citi TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep DiveCiti TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep Dive
confluent
 
Q&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service MeshQ&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service Mesh
confluent
 
Citi Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka MicroservicesCiti Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka Microservices
confluent
 
Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3
confluent
 
Citi Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging ModernizationCiti Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging Modernization
confluent
 
Citi Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time dataCiti Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time data
confluent
 
Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2
confluent
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023
confluent
 
Confluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with SynthesisConfluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with Synthesis
confluent
 
The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023
confluent
 

More from confluent (20)

Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutes
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
 
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
 
Santander Stream Processing with Apache Flink
Santander Stream Processing with Apache FlinkSantander Stream Processing with Apache Flink
Santander Stream Processing with Apache Flink
 
Unlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insightsUnlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insights
 
Workshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con FlinkWorkshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con Flink
 
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
 
AWS Immersion Day Mapfre - Confluent
AWS Immersion Day Mapfre   -   ConfluentAWS Immersion Day Mapfre   -   Confluent
AWS Immersion Day Mapfre - Confluent
 
Eventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalkEventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalk
 
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent CloudQ&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
 
Citi TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep DiveCiti TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep Dive
 
Q&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service MeshQ&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service Mesh
 
Citi Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka MicroservicesCiti Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka Microservices
 
Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3
 
Citi Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging ModernizationCiti Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging Modernization
 
Citi Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time dataCiti Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time data
 
Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023
 
Confluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with SynthesisConfluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with Synthesis
 
The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023
 

Recently uploaded

E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian CompaniesE-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
Quickdice ERP
 
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s EcosystemUI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
Peter Muessig
 
Malibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed RoundMalibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed Round
sjcobrien
 
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdfBaha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
Baha Majid
 
Energy consumption of Database Management - Florina Jonuzi
Energy consumption of Database Management - Florina JonuziEnergy consumption of Database Management - Florina Jonuzi
Energy consumption of Database Management - Florina Jonuzi
Green Software Development
 
14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision
ShulagnaSarkar2
 
How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?
ToXSL Technologies
 
Quarter 3 SLRP grade 9.. gshajsbhhaheabh
Quarter 3 SLRP grade 9.. gshajsbhhaheabhQuarter 3 SLRP grade 9.. gshajsbhhaheabh
Quarter 3 SLRP grade 9.. gshajsbhhaheabh
aisafed42
 
E-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet DynamicsE-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet Dynamics
Hornet Dynamics
 
一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理
dakas1
 
Preparing Non - Technical Founders for Engaging a Tech Agency
Preparing Non - Technical Founders for Engaging  a  Tech AgencyPreparing Non - Technical Founders for Engaging  a  Tech Agency
Preparing Non - Technical Founders for Engaging a Tech Agency
ISH Technologies
 
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
The Third Creative Media
 
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
kalichargn70th171
 
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
Bert Jan Schrijver
 
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
kgyxske
 
Kubernetes at Scale: Going Multi-Cluster with Istio
Kubernetes at Scale:  Going Multi-Cluster  with IstioKubernetes at Scale:  Going Multi-Cluster  with Istio
Kubernetes at Scale: Going Multi-Cluster with Istio
Severalnines
 
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
8 Best Automated Android App Testing Tool and Framework in 2024.pdf8 Best Automated Android App Testing Tool and Framework in 2024.pdf
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
kalichargn70th171
 
Oracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptxOracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptx
Remote DBA Services
 
UI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design SystemUI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design System
Peter Muessig
 
Enhanced Screen Flows UI/UX using SLDS with Tom Kitt
Enhanced Screen Flows UI/UX using SLDS with Tom KittEnhanced Screen Flows UI/UX using SLDS with Tom Kitt
Enhanced Screen Flows UI/UX using SLDS with Tom Kitt
Peter Caitens
 

Recently uploaded (20)

E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian CompaniesE-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
 
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s EcosystemUI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
 
Malibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed RoundMalibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed Round
 
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdfBaha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
 
Energy consumption of Database Management - Florina Jonuzi
Energy consumption of Database Management - Florina JonuziEnergy consumption of Database Management - Florina Jonuzi
Energy consumption of Database Management - Florina Jonuzi
 
14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision
 
How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?
 
Quarter 3 SLRP grade 9.. gshajsbhhaheabh
Quarter 3 SLRP grade 9.. gshajsbhhaheabhQuarter 3 SLRP grade 9.. gshajsbhhaheabh
Quarter 3 SLRP grade 9.. gshajsbhhaheabh
 
E-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet DynamicsE-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet Dynamics
 
一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理
 
Preparing Non - Technical Founders for Engaging a Tech Agency
Preparing Non - Technical Founders for Engaging  a  Tech AgencyPreparing Non - Technical Founders for Engaging  a  Tech Agency
Preparing Non - Technical Founders for Engaging a Tech Agency
 
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
 
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
 
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
 
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
 
Kubernetes at Scale: Going Multi-Cluster with Istio
Kubernetes at Scale:  Going Multi-Cluster  with IstioKubernetes at Scale:  Going Multi-Cluster  with Istio
Kubernetes at Scale: Going Multi-Cluster with Istio
 
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
8 Best Automated Android App Testing Tool and Framework in 2024.pdf8 Best Automated Android App Testing Tool and Framework in 2024.pdf
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
 
Oracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptxOracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptx
 
UI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design SystemUI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design System
 
Enhanced Screen Flows UI/UX using SLDS with Tom Kitt
Enhanced Screen Flows UI/UX using SLDS with Tom KittEnhanced Screen Flows UI/UX using SLDS with Tom Kitt
Enhanced Screen Flows UI/UX using SLDS with Tom Kitt
 

Build real-time streaming data pipelines to AWS with Confluent

  • 1. © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. AWS + Confluent Immersion Day Building Secure, Event-Driven Microservices with Confluent Cloud on AWS November 15, 2023
  • 2. © 2022, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. Today’s hosts: 2 Ahmed Zamzam Senior Partner Solutions Architect, Confluent Nuno Barreto Partner Sales Solutions Architect, AWS
  • 3. © 2022, Amazon Web Services, Inc. or its affiliates. Agenda • Building Modern Streaming Analytics with Confluent on AWS with Nuno Barreto, Partner Solutions Architect, AWS • Event Streaming made easy with Confluent with Ahmed Zamzam, Sr Partner Solutions Architect, Confluent • Lab: Building end-to-end streaming data pipelines with Confluent Cloud with Ahmed Zamzam, Sr Partner Solutions Architect, Confluent 3
  • 4. © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. Building Modern Streaming Analytics with Confluent on AWS Nuno Barreto (he/him) Partner Sales Solutions Architect Amazon Web Services (AWS) #DatainMotionTour
  • 5. © 2022, Amazon Web Services, Inc. or its affiliates. Agenda Develop a modern data strategy Act on real time with data streaming Build seamless streaming with AWS and Confluent Innovate together to power customer success Key take aways
  • 6. © 2022, Amazon Web Services, Inc. or its affiliates. Data volume and velocity Multiple analytics needs Variety of sources and data types Difficult to manage systems Complex to scale Inflexible tools Security, compliance Slow performance Increasing and unpredictable cost Challenges of data analytics at scale
  • 7. © 2022, Amazon Web Services, Inc. or its affiliates. Catalog Governance Analytics Machine learning Databases Data lakes Modern data strategy People, apps and devices Data sources
  • 8. © 2022, Amazon Web Services, Inc. or its affiliates. Modern data architecture on AWS Key Pillars Data at any scale Seamless data access and movement Purpose-built for best price performance Built-in ML to solve business challenges Unified governance
  • 9. © 2022, Amazon Web Services, Inc. or its affiliates. The benefits of data lakes Catalog Store all your data in open formats Decouple storage from compute Cost-effectively scale storage to exabytes Process data in place Choice of analytical and ML engines
  • 10. © 2022, Amazon Web Services, Inc. or its affiliates. Sharing across data movement with Data Mesh Data producers Data mesh Data consumers Unique modern data architecture Suited to business function Teams that want to share data Unique modern data architecture Suited to business function Team that runs the marketplace Teams that want to use data
  • 11. © 2022, Amazon Web Services, Inc. or its affiliates. Source: Perishable insights, Mike Gualtieri, Forrester Real time Seconds Minutes Hours Days Months Value of data to decision-making Preventive/predictive Actionable Reactive Historical Time-critical decisions Traditional “batch” business intelligence Information half-life in decision-making Why real-time data streaming analytics ? Data loses value quickly over time
  • 12. © 2022, Amazon Web Services, Inc. or its affiliates. Common real-time analytics use cases Anomaly and fraud detection Empowering IoT analytics Nourishing marketing campaigns Real-time personalization Tailoring customer experience in real time Supporting healthcare and emergency services
  • 13. © 2022, Amazon Web Services, Inc. or its affiliates. Challenges of data streaming Difficult to set up Tricky to scale Hard to achieve high availability Integration requires development Error prone and complex to manage Expensive to maintain
  • 14. © 2022, Amazon Web Services, Inc. or its affiliates. Source Devices and/or applications that produce real-time data at high velocity Stream ingestion Data from tens of thousands of data sources can be written to a single stream In-stream storage Data are stored in the order they were received for a set duration of time and can be replayed indefinitely during that time Stream processing Records are read in the order they are produced, enabling real- time analytics or streaming ETL Destination/ Storage Database (NoSQL most common) Data lake Data warehouse Event driven Applications ` Visualize Analytics dashboard Real-time streaming analytics pipeline Ingest, Process & Analyze High Volumes of High-Velocity Data from Various Sources in Real Time
  • 15. © 2022, Amazon Web Services, Inc. or its affiliates. Stream processing Visualize Source Mobile device Metering Click streams IoT sensors AWS SDKs Kinesis Agent/KPL Stream ingestion ` Apache Kafka Amazon Kinesis Data Streams Amazon Kinesis Data Firehose In-stream storage Apache Kafka Amazon Kinesis Data Streams Amazon Kinesis Data Analytics AWS Glue Streaming Amazon EMR Destination/Storage Amazon DynamoDB Amazon EMR Amazon S3 Amazon Redshift Amazon OpenSearch Amazon QuickSight Apache Kafka Streaming analytics on AWS
  • 16. © 2022, Amazon Web Services, Inc. or its affiliates. Rich front-end customer experiences Real-time Event Streams and Analysis A Sale A shipment A Trade A Customer Experience Real-time backend operations Event streaming with Kafka to set Data in Motion: Continuously processing evolving streams of data in real time
  • 17. © 2022, Amazon Web Services, Inc. or its affiliates. Out-of-box integration with popular services Certified and validated by AWS AWS Native Services Top-5 Global ISV for S3 Data Volume 3rd-Party ISV Services • Amazon RDS Ready • AWS Lambda Ready • Amazon Redshift Ready • AWS PrivateLink Ready • AWS Outposts Ready Validated Service Designations Confluent integrations with AWS
  • 18. © 2022, Amazon Web Services, Inc. or its affiliates. S Confluent Cloud on AWS – reference architecture S3
  • 19. © 2022, Amazon Web Services, Inc. or its affiliates. Confluent + AWS: Accelerate Customer Business Outcomes Topline Impacting New Experiences ● Event-driven & real-time ● Unify data across org. w/ Kafka data fabric (Schema Reg,..) ● AWS Analytics, Redshift, ML connectors Mitigate Risk ● Higher Service Quality & Resilience with 99.99% SLA ● Deep Kafka expertise & innovation ● Elastic billing/pricing Developer Agility ● Focus on innovation (not data infrastructure) ● Leverage full Kafka OSS ecosystem + AWS services Faster Time to Market ● ~50-75% faster time to market* ● Streamline hybrid cloud migration with no complex lift-n- shift ● Maintain business continuity Lower Kafka TCO ● ~25-50% lower TCO * ● GBps-scale & fast deployments for global expansion ● Deploy Kafka at scale in 1 week Maximize ROI ● ~200% ROI per Forrester study ● Save 10s of $Ms with legacy offload to AWS with Confluent Replicator * For customers that don’t already have Kafka based system in-market * TCO assessment to be analyzed for specific customer scenarios
  • 20. © 2022, Amazon Web Services, Inc. or its affiliates. Accelerate modernization from on-prem to AWS Redshift Sink Lambda Sink AWS Direct Connect Replicator LEGACY EDW MAINFRAME LEGACY DB JDBC / CDC connectors Connect Leverage 100+ Confluent pre-built connectors Modernize Value added apps, increase agility, reduce TCO On-prem AWS Cloud Bridge Hybrid cloud streaming Amazon Athena AWS Glue SageMaker Lake Formation Amazon DynamoDB Amazon Aurora S3 Sink Data Streams Apps ksqlDB
  • 21. © 2022, Amazon Web Services, Inc. or its affiliates. Confluent Wavelength solution with AWS
  • 22. © 2022, Amazon Web Services, Inc. or its affiliates. Increase developer agility & speed of innovation Serverless integration Connect for effortless integrations with Lambda & data stores AWS serverless platform Free up backend operation & Infras. management Apps Microservices ksqlDB Schema Registry COMPUTE AWS Lambda Data stores REST Proxy & Clients Source Connectors Lambda Sink DATA STORES Amazon DynamoDB Amazon Aurora STORAGE Amazon S3 S3 Sink ANALYTICS Amazon Athena Amazon Redshift
  • 23. © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Confluent Data in Motion 23 Ahmed Zamzam Sr Partner Solutions Engineer
  • 24. © 2022, Amazon Web Services, Inc. or its affiliates. ...many more Other systems Other systems Kafka Connect Kafka Cluster Kafka Connect Apache Kafka is an event streaming platform 24
  • 25. © 2022, Amazon Web Services, Inc. or its affiliates. Core Kafka features 01 Publish & Subscribe to streams of events 02 Store your event streams 03 Process & Analyze your events streams 26
  • 26. © 2022, Amazon Web Services, Inc. or its affiliates. ksqlDB 27
  • 27. © 2022, Amazon Web Services, Inc. or its affiliates. 70% of Fortune 500 companies use Apache Kafka (majority are Confluent customers) The rise of data in motion 28
  • 28. © 2022, Amazon Web Services, Inc. or its affiliates. 32,232 Stack overflow questions 210 meetups 69,000 meetup attendees 10,286 Jira tickets for Apache Kafka 46,626 Emails sent to Apache Kafka mailing list 988 KIPs Community 29
  • 29. © 2022, Amazon Web Services, Inc. or its affiliates. Hall of Innovation CTO Innovation Award Winner 2019 Enterprise Technology Innovation AWARDS Vision ● Original Kafka creators founded Confluent ● Data in motion pioneers Category leadership ● 80% of Kafka commits ● 5M+ hours of Kafka technical experience ● Operate 35K+ clusters Value ● Remove risk ● Deploy at scale ● Accelerate time to market Product ● Extends Kafka to be secure and enterprise-ready ● Software or cloud-native service Confluent is the only company focused on data in motion 30
  • 30. © 2022, Amazon Web Services, Inc. or its affiliates. Confluent: Everywhere Confluent Platform The Enterprise Distribution of Apache Kafka Confluent Cloud Apache Kafka Re-engineered for the Cloud Self-Managed Software Fully-Managed Service VM Deploy on any platform, on-prem or cloud Available on the leading public clouds 31
  • 31. © 2022, Amazon Web Services, Inc. or its affiliates. Federated streaming, hybrid and multi-cloud. Data syndication and replication across and between clouds and on- premises, with self-service APIs, data governance, and visual tooling. Reliable & real-time data streams between all customer sites, so you can run always-on streaming analytics on the data of the entire enterprise, despite regional or cloud provider outages. Everywhere: Cluster Linking Global Central Nervous System
  • 32. © 2022, Amazon Web Services, Inc. or its affiliates. K A F K A C L O U D 33
  • 33. © 2022, Amazon Web Services, Inc. or its affiliates. We Transformed Kafka for the Cloud, Ground Up! Resilient with automated operations to ensure high availability and reliability Performant with networking service decoupling and replication optimization Elastic to seamlessly expand and shrink based on customer demands KORA ENGINE The Apache Kafka® Engine Built for the Cloud Cost efficient with multi-tenancy, data tiering, cloud optimizations and hands-off operations
  • 34. © 2022, Amazon Web Services, Inc. or its affiliates. Event streaming as a cloud native service Deploy in Minutes Ensure service levels Integrate with cloud services Eliminate operational burden 35
  • 35. © 2022, Amazon Web Services, Inc. or its affiliates. Scale faster with serverless development Stream with Kafka in minutes Provision and scale clusters on-demand Pay only for what you use Start from $0 and pay based on actual usage 36 Access latest Kafka version Always work in the most recent stable version of Kafka
  • 36. © 2022, Amazon Web Services, Inc. or its affiliates. Go to market in 6 months vs. 2+ years Traditional Kafka development Development with Confluent Cloud 6-9 months to hire Kafka resources 3 months to ramp Project Kickoff 9-12 months build the production-grade Kafka platform and develop application Start in 1 week 3-6 months to focus on app development, not managing Kafka Project Kickoff Launch your application in months and grow your business Go to market in 2 years Go to market in 6 months 37
  • 37. © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 38 Infinite Data
  • 38. © 2022, Amazon Web Services, Inc. or its affiliates. Infinite Apache Kafka Scalable Quickly scale up storage resources without overprovisioning infra Efficient Decouple storage from compute resources to keep storage costs low Infinite Retain as much data in Kafka as you need to meet any use case 39 Decouples the storage and compute layers to create highly scalable, cost-effective storage without limits to enable infinite retention
  • 39. © 2022, Amazon Web Services, Inc. or its affiliates. Which use cases or concerns does Infinite address? New customer experiences - reimagine what you can do with streaming through infinite retention of events Ease operational overhead - stop worrying about running out of disk space and simplify topic configuration 40
  • 40. © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 41 Getting Data into Kafka
  • 41. © 2022, Amazon Web Services, Inc. or its affiliates. Make Kafka Widely Accessible to Developers Let developers build with their most productive language and enable use cases and architectures that require multiple programming languages Confluent Clients Battle-tested and high performing
  • 42. © 2022, Amazon Web Services, Inc. or its affiliates. Apache Kafka Connect API: Fault tolerant Manage hundreds of data sources and sinks Preserves data schema Integrated within Confluent Control Center JDBC Mongo MySQL Elastic HDFS Kafka Connect API Kafka Pipeline Connector Connector Connector Connector Connector Connector Sources Sinks S3 43 Import and export data in and out of Kafka
  • 43. © 2022, Amazon Web Services, Inc. or its affiliates. 140+ prebuilt connectors 100+ Confluent supported 30+ partner supported, Confluent verified AWS Lambda Instantly connect popular data sources & sinks 44
  • 44. © 2022, Amazon Web Services, Inc. or its affiliates. Confluent has deep AWS service integrations
  • 45. © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 46 Real-time Transformations
  • 46. © 2022, Amazon Web Services, Inc. or its affiliates. Kafka Streams: Write standard Java apps and microservices to process your data in real time KStream<User, PageViewEvent> pageViews = builder.stream("pageviews-topic"); KTable<Windowed<User>, Long> viewsPerUserSession = pageViews .groupByKey() .count(SessionWindows.with(TimeUnit.MINUTES.toMillis(5)), "session-views"); docs.confluent.io/current/streams/ No separate processing cluster required Develop on Mac, Linux, and Windows Deploy to containers, VMs, bare metal, cloud Powered by Kafka: elastic, scalable, distributed, and reliable Perfect for small, medium, and large use cases Fully integrated with Kafka security Exactly-once processing semantics Part of Apache Kafka 47
  • 47. © 2022, Amazon Web Services, Inc. or its affiliates. ksqlDB at a glance What is it? ksqlDB is an event-streaming database for working with streams and tables of data All the key features of a modern streaming solution Aggregations Joins Windowing Event-time Dual query support Exactly-once semantics Out-of-order handling User-defined functions CREATE TABLE activePromotions AS SELECT rideId, qualifyPromotion(distanceToDst) AS promotion FROM locations GROUP BY rideId EMIT CHANGES How does it work? It separates compute from storage, and scales elastically in a fault-tolerant manner It remains highly available during disruption, even in the face of failure to a quorum of its servers
  • 48. © 2022, Amazon Web Services, Inc. or its affiliates. 3 modalities of stream processing with Confluent Kafka clients Kafka Streams ksqlDB ConsumerRecords<String, String> records = consumer.poll(100); Map<String, Integer> counts = new DefaultMap<String, Integer>(); for (ConsumerRecord<String, Integer> record : records) { String key = record.key(); int c = counts.get(key) c += record.value() counts.put(key, c) } for (Map.Entry<String, Integer> entry : counts.entrySet()) { int stateCount; int attempts; while (attempts++ < MAX_RETRIES) { try { stateCount = stateStore.getValue(entry.getKey()) stateStore.setValue(entry.getKey(), entry.getValue() + stateCount) break; } catch (StateStoreException e) { RetryUtils.backoff(attempts); } } } builder .stream("input-stream", Consumed.with(Serdes.String(), Serdes.String())) .groupBy((key, value) -> value) .count() .toStream() .to("counts", Produced.with(Serdes.String(), Serdes.Long())); SELECT x, count(*) FROM stream GROUP BY x EMIT CHANGES; Flexibility Simplicity confluent.awsworkshop.io 49
  • 49. © 2022, Amazon Web Services, Inc. or its affiliates. Easily build event streaming applications Use one, lightweight SQL syntax to build a complete real-time application CREATE STREAM payments(user VARCHAR, payment_amount INT) WITH (kafka_topic = ’all_payments’, key = ’user’, value_format = ’avro’); Create aggregations of event data that can serve queries to applications Enrich Kafka data with a robust stream processing framework USER Payment Jay $10 Sue $15 Fred $5 ... ... USER Credit Score Jay 660 Sue 710 Fred 595 USER Credit Score Jay 660 Sue 710 Fred 595 USER Credit Score Jay 660 Sue 710 Fred 595 50
  • 50. © 2022, Amazon Web Services, Inc. or its affiliates. How to get started 1. Subscribe to Confluent Cloud on the AWS Marketplace and start with a free $400 (to be used within 60 days) 2. Try out the Confluent/AWS workshop found at https://confluent.awsworkshop.io/ 3. Email us at awsteam@confluent.io if you have any questions 51
  • 51. © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Hands-on workshop, and Q&A 52 Ahmed Zamzam Sr Partner Solutions Engineer, Confluent
  • 52. © 2022, Amazon Web Services, Inc. or its affiliates. Conclusion/Next steps • Thank you for attending. • Please fill out the AWS event survey and let us know what you thought. • If you have any further questions or wish to schedule a call with an AWS and Confluent expert, please contact awsteam@confluent.io 53
  • 53. © 2022, Amazon Web Services, Inc. or its affiliates. Thank you! © 2022, Amazon Web Services, Inc. or its affiliates. 54