Confluent, Ably and AWS Better Together

1. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Confluent/Ably/AWS Immersion Day
Building 2048
Ahmed Zamzam(he/him)
Senior AWS Partner Solutions Architect
Confluent
Roman De Oliveira(he/him)
Lead Solutions Engineer
Ably

2. © 2023, Amazon Web Services, Inc. or its affiliates. 2
Agenda
10:30-10:45 Introduction
10:45-11:30 Confluent and Ably
201
11:30-12:15 Workshop Set-up (Steps
1-6)
12:15-13:00 Lunch
13:00-14:00 Workshop (Steps 7-
10)

3. © 2023, Amazon Web Services, Inc. or its affiliates.
Today’s hosts:
3
Ahmed Zamzam
Senior Partner Solutions Architect,
Confluent
Roman De Oliveira
Lead Solutions Engineer, Ably

4. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Confluent Cloud
Cloud-native data streaming platform built by the founders of Apache Kafka®
Everywhere
Connect your data in real
time with a platform that
spans from on-prem to
cloud and across clouds
Complete
Go above & beyond Kafka
with all the essential tools
for a complete data
streaming platform
Cloud-Native
The 10x Apache Kafka®
service: elastic, resilient
and performant, powered
by the Kora Engine
Stream confidently on the world’s most trusted data streaming platform built by the founders of
Apache Kafka©, with resilience, security, compliance, and privacy built-in by default.
4

5. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why real-time?
Real-time Seconds Minutes Hours Days Months
Value
of
data
to
decision-making
Preventive/Predictive
Actionable Reactive Historical
Source: Perishable insights, Mike Gualtieri, Forrester
Data loses value quickly over time
Time critical
decisions
Traditional “batch” business
intelligence
Information half-life
in decision-making

6. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Typical real-time data pipeline
Data continuously
generated at a high
velocity from different
sources like IoT devices,
Application logs, Online
transactions, etc..
Source
Data captured and
stored in the order it
was received for set
duration of time and
can be replayed
indefinitely.
Event Streaming
Process, analyse and
action on the data as
soon as it is generated
and, in the order, it was
received
Stream Processing
Sink data different
destinations. Data Lakes
(most common) and/or
different Databases
Presentation
Governance

7. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data continuously
generated at a high
velocity from different
sources like IoT devices,
Application logs, Online
transactions, etc..
Source
Data captured and
stored in the order it
was received for set
duration of time and
can be replayed
indefinitely.
Event Streaming
Process, analyse and
action on the data as
soon as it is generated
and, in the order, it was
received
Stream Processing
Sink data different
destinations. Data Lakes
(most common) and/or
different Databases
Presentation
Governance
Typical real-time data pipeline

8. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
8
Kora Engine
The Apache Kafka® engine built
for the cloud
Elastic scalability
Scale and shrink to handle GBps+ workloads
and petabyte storage more than 10x faster
and easier without any operational burden
Guaranteed resiliency
Leave Kafka reliability worries behind with
the industries most comprehensive 99.99%
uptime SLA and built-in data durability
Low latency
Speed up real-time analytics and customer
experiences with predictable low latency,
sustained over time
Multi-tenancy
and serverless
Automated
operations
Compute, storage
and network
decoupling
Global
availability
Kora Engine
Note: Kora is the underlying engine of Confluent Cloud and cannot be downloaded or
run outside Confluent Cloud’s control plane. It is 100% based on Kafka protocol and is
not a replacement of open source Kafka.

9. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Leave Kafka reliability worries behind with
99.99% uptime SLA and 10x built-in durability
Never worry about Kafka storage limits again
with Infinite Storage that’s 10x more scalable
and performant
Scale and shrink to handle 0 to GBps+
workloads and peak customer demands
10x faster and easier
10x Kafka
Confluent Cloud offers a truly
fully managed, cloud-native
data streaming platform for
Apache Kafka, with 10x faster
scaling, infinitely more storage,
and built-in resilience
Resiliency
Storage
Elasticity

10. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
10
ksqlDB
140+
pre-built connectors

11. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Confluent has deep integrations with AWS
11

12. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Together Confluent and AWS empower Endless Use Cases across many
Industries
Retail
Healthcare
Finance &
Banking
Transportation
Common in all
Industries
Inventory
Management
Personalized
Promotions
Product
Development
& Introduction
Sentiment
Analysis
Streaming
Enterprise
Messaging
Systems of
Scale for High
Traffic Periods
Connected
Health
Records
Data
Confidentiality
& Accessibility
Dynamic Staff
Allocation
Optimization
Integrated
Treatment
Proactive Patient
Care
Real-Time
Monitoring
Early-On
Fraud
Detection
Capital
Management
Market Risk
Recognition &
Investigation
Preventive
Regulatory
Scanning
Real-Time What-
If
Analysis
Trade Flow
Monitoring
Advanced
Navigation
Environmental
Factor
Processing
Fleet
Management
Predictive
Maintenance
Threat Detection
& Real-Time
Response
Traffic
Distribution
Optimization
Data Pipelines
Hybrid Cloud
Integration
Microservices
Security and
Fraud
Customer 360 Streaming ETL

13. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data continuously
generated at a high
velocity from different
sources like IoT devices,
Application logs, Online
transactions, etc..
Source
Data captured and
stored in the order it
was received for set
duration of time and
can be replayed
indefinitely.
Event Streaming
Process, analyse and
action on the data as
soon as it is generated
and, in the order, it was
received
Stream Processing
Sink data different
destinations. Data Lakes
(most common) and/or
different Databases
Presentation
Governance
Typical real-time data pipeline

14. ksqlDB at a glance
What is it?
ksqlDB is an event-streaming
database for working with
streams and tables of data
All the key features of a
modern streaming solution
Aggregations Joins
Windowing
Event-time
Dual query
support
Exactly-once
semantics
Out-of-order
handling
User-defined
functions
CREATE TABLE activePromotions AS
SELECT rideId,
qualifyPromotion(distanceToDst) AS
promotion
FROM locations
GROUP BY rideId
EMIT CHANGES
How does it work?
It separates compute from storage, and scales
elastically in a fault-tolerant manner
It remains highly available during disruption, even in
the face of failure to a quorum of its servers

15. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
15
Streams and Tables
{ "event_ts": "2020-02-17T15:22:00Z",
"person" : "robin",
"location": "Leeds"
}
{ "event_ts": "2020-02-17T17:23:00Z",
"person" : "robin",
"location": "London"
}
{ "event_ts": "2020-02-17T22:23:00Z",
"person" : "robin",
"location": "Wakefield"
}
{ "event_ts": "2020-02-18T09:00:00Z",
"person" : "robin",
"location": "Leeds"
+--------------------+-------+---------+
|EVENT_TS |PERSON |LOCATION |
+--------------------+-------+---------+
|2020-02-17 15:22:00 |robin |Leeds |
|2020-02-17 17:23:00 |robin |London |
|2020-02-17 22:23:00 |robin |Wakefield|
|2020-02-18 09:00:00 |robin |Leeds |
+-------+---------+
|PERSON |LOCATION |
+-------+---------+
|robin |Leeds |
Kafka topic
+-------+---------+
|PERSON |LOCATION |
+-------+---------+
|robin |London |
+-------+---------+
|PERSON |LOCATION |
+-------+---------+
|robin |Wakefield|
+-------+---------+
|PERSON |LOCATION |
+-------+---------+
|robin |Leeds |
ksqlDB Table
ksqlDB Stream
Stream (append-only series of
events):
Topic + Schema
Table: state for
given key
Topic + Schema

16. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
16
Pull and Push queries in ksqlDB
Push query
Pull query
Tells you:
Exits:
Point in time value
Immediately
All value changes
Never
16
Pull queries have some limitations currently:
- only available against tables
- require the object to be materialised (i.e. a
TABLE built using an aggregate)
- Currently only available for single key
lookup with optional window range
Push queries are applicable to all
stream and table objects, and
indicated with an EMIT CHANGES
clause.

17. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
17
Stateful aggregations in ksqlDB
{ "event_ts": "2020-02-17T15:22:00Z",
"person" : "robin",
"location": "Leeds"
}
{ "event_ts": "2020-02-17T17:23:00Z",
"person" : "robin",
"location": "London"
}
{ "event_ts": "2020-02-17T22:23:00Z",
"person" : "robin",
"location": "Wakefield"
}
{ "event_ts": "2020-02-18T09:00:00Z",
"person" : "robin",
"location": "Leeds"
Kafka topic
SELECT PERSON,
COUNT(*) AS
LOCATION_CHANGES
FROM MOVEMENTS
GROUP BY PERSON;
+-------+------------------+
|PERSON | LOCATION_CHANGES |
+-------+------------------+
|robin | 1 |
+-------+------------------+
|PERSON | UNIQUE_LOCATIONS |
+-------+------------------+
|robin | 1 |
SELECT PERSON,
COUNT_DISTINCT(LOCATION)AS
UNIQUE_LOCATIONS
FROM MOVEMENTS
GROUP BY PERSON;
+-------+------------------+
|PERSON | LOCATION_CHANGES |
+-------+------------------+
|robin | 2 |
+-------+------------------+
|PERSON | UNIQUE_LOCATIONS |
+-------+------------------+
|robin | 2 |
+-------+------------------+
|PERSON | LOCATION_CHANGES |
+-------+------------------+
|robin | 3 |
+-------+------------------+
|PERSON | UNIQUE_LOCATIONS |
+-------+------------------+
|robin | 3 |
+-------+------------------+
|PERSON | LOCATION_CHANGES |
+-------+------------------+
|robin | 4 |
+-------+------------------+
|PERSON | UNIQUE_LOCATIONS |
+-------+------------------+
|robin | 3 |
Aggregations can be across the entire input, or
windowed (TUMBLING, HOPPING, SESSION)
17

18. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
18
Stateful aggregations in ksqlDB
{ "event_ts": "2020-02-17T15:22:00Z",
"person" : "robin",
"location": "Leeds"
}
{ "event_ts": "2020-02-17T17:23:00Z",
"person" : "robin",
"location": "London"
}
{ "event_ts": "2020-02-17T22:23:00Z",
"person" : "robin",
"location": "Wakefield"
}
{ "event_ts": "2020-02-18T09:00:00Z",
"person" : "robin",
"location": "Leeds"
Kafka topic
CREATE TABLE PERSON_MOVEMENTS AS
SELECT PERSON,
COUNT_DISTINCT(LOCATION) AS
UNIQUE_LOCATIONS,
COUNT(*) AS LOCATION_CHANGES
FROM MOVEMENTS
GROUP BY PERSON;
PERSON_
MOVEMENTS
Internal
ksqlDB
state
store
18

19. Filters CREATE STREAM high_readings AS
SELECT sensor,
reading,
FROM readings
WHERE reading > 41
EMIT CHANGES;
** The above “Create Stream As Select (CSAS)” statement is a Persistent Query that runs
indefinitely and produces the resulting stream backed by a new topic “high readings”.

20. Joins
CREATE STREAM enriched_readings AS
SELECT reading, area, brand_name,
FROM readings
INNER JOIN brands b
ON b.sensor = readings.sensor
EMIT CHANGES;

21. Aggregate CREATE TABLE avg_readings AS
SELECT sensor,
AVG(reading) AS location
FROM readings
GROUP BY sensor
EMIT CHANGES;
** The above “Create Table As Select (CTAS)” statement is a Persistent Query that runs indefinitely
and produces the resulting table backed by a new topic “avg_readings”.

22. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
22
Kafka clients Kafka streams ksqlDB
ConsumerRecords<String, String> records =
consumer.poll(100);
Map<String, Integer> counts = new DefaultMap<String,
Integer>();
for (ConsumerRecord<String, Integer> record : records) {
String key = record.key();
int c = counts.get(key)
c += record.value()
counts.put(key, c)
}
for (Map.Entry<String, Integer> entry : counts.entrySet())
{
int stateCount;
int attempts;
while (attempts++ < MAX_RETRIES) {
try {
stateCount = stateStore.getValue(entry.getKey())
stateStore.setValue(entry.getKey(), entry.getValue() +
stateCount)
break;
} catch (StateStoreException e) {
RetryUtils.backoff(attempts);
}
}
}
builder
.stream("input-stream",
Consumed.with(Serdes.String(), Serdes.String()))
.groupBy((key, value) -> value)
.count()
.toStream()
.to("counts", Produced.with(Serdes.String(),
Serdes.Long()));
SELECT x, count(*) FROM stream GROUP BY x EMIT
CHANGES;
Flexibility Simplicity
3 modalities of stream processing with Confluent

23. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Database
● Full ACID Compliance
● Batch Analytics workloads
But good enough for
transactional queries and
relatively simple aggregations.
IoT Platform
● With features such as
device management
But can natively integrate
with IoT Core or a other
services that can connect to
IoT devices like Ably.
Connect to
web/mobile clients
● Connecting directly to
millions of clients. Use-
cases like gaming and data
sync
But can natively integrate
with Ably
Finding the Best Fit: When Confluent Isn't the Answer

24. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
So, where does Ably fit in?

25. Ably = Kafka for the public Internet
Ably is a realtime experiences infrastructure platform
designed for streaming large volumes of event data
over the public Internet.

26. 26
Realtime APIs & infrastructure to:
- Ship in days
- Perform at scale

27. 27

28. What does Ably provide?
28
Predictable performance
Low-latency global edge network across 385 PoPs. Global median
latencies of <65ms.
Guaranteed ordering & delivery
We guarantee in-order and exactly once data delivery, even after
disconnections.
Fault tolerant infrastructure
Redundant at regional and global levels with 99.999% uptime SLAs.
High scalability & availability
Elastic, battle-tested global infrastructure for massive, instant scale.

29. Event distribution from Confluent Kafka to the edge and back
29

30. How to publish to Ably channels
How to subscribe to Ably channels

31. 2028 Workshop

32. Copyright 2020, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. 33
Play the Game!!!

33. Copyright 2020, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.

34. Copyright 2020, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.

35. Let’s start!!
go.ably.com/workshop

37. Extra Slides