Building Serverless EDA w_ AWS Lambda (1).pptx

Serverless Stream Processing for
Financial Services
Ahmed Zamzam
AWS Partner Solutions
Architect - Confluent
Veda Raman
Specialist Solutions
Architect - AWS
Jason Demby
Senior Business
Development Leader - AWS

Agenda
Introducing Confluent
Rearchitected Kafka, together with the features
you need to rapidly deploy production use cases
2
Serverless stream processing
Building event streaming applications using
ksqlDB and AWS Lambda
Best Practices
Best practices when using AWS Lambda as a
stateless stream processor
The shift towards data streaming and Apache
Kafka and the value it provides
The rise of data streaming

Confluent for Financial Services: Use Cases
Deliver differentiated
customer experiences
Increase digital engagement &
improve omni-channel experience
Secure your enterprise
Detect and respond to fraud,
threats and attacks in real-time
Modernize your infrastructure
Drive massive operational efficiency,
developer velocity and reduce costs
Automate business resiliency
Mitigate risks in service offerings and
market risk exposure
Drive regulatory compliance
Stay compliant across global banking
regulations such as open banking, FINRA,
trade reporting, payments and more
Enable a sharing economy
Decentralize asset ownership and
increase market opportunity to deliver
profitable services in the future

What is streaming data?
Typical characteristics
Low-latency
Continuous Ordered,
incremental
High volume

Why streaming data?
Source: Perishable insights, Mike Gualtieri, Forrester
Data loses value quickly over time
Real-time Seconds Minutes Hours Days Months
Value
of
data
to
decision-making
Preventive/Predictive
Actionable Reactive Historical
Time critical
decisions
Traditional “batch” business intelligence
Information half-life
in decision-making

Event Streaming is the
Central Nervous System
for today’s enterprises.
Apache Kafka®
is the technology.

...many more
Other
Systems
Other
Systems
Kafka
Connect
Kafka Cluster
Kafka
Connect
Apache Kafka is an Event Streaming Platform

© 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Core Kafka Features
01
Publish & Subscribe
to Streams of Events
02
Store
your Event Streams
03
Process & Analyze
your Events Streams

Real-time
Data
Trades
Market data
Positions
Collateral
A new paradigm for financial services
Continuously Process Data in Real-time
“We need to shift our thinking from everything
at rest, to everything in motion.” —
Real-Time Stream Processing
Orders, Execution,
Algorithms, Pricing
Models
Firm-wide Risk
Management

“We look at events as running
our business. Business people
within our organization want to
be able to react to events—and
oftentimes it's a combination of
events.”
VP of Streaming Data Engineering

Hall of Innovation
CTO Innovation
Award Winner
2019
Enterprise Technology
Innovation
AWARDS
...Confluent is the Only Company
Focused on Data in Motion
Vision
● Original creators of
Kafka
● Data in Motion
pioneers
Category Leadership
● 80% of Kafka commits
● 1M+ hours of Kafka
technical experience
● Operate 5K+ clusters
Value
● Remove risk
● Deploy at scale
● Accelerate time-to-
market
Product
● Extends Kafka to be
secure and
enterprise-ready
● Software or cloud-
native service

...
Device
Logs ... ...
...
Data Stores Logs 3rd Party Apps Custom Apps / Microservices
Real-time
Customer 360
Financial Fraud
Detection
Real-time
Risk Analytics
Real-time
Payments
Machine
Learning
Models
...
Real-time Applications
Universal Event Pipeline
Amazon
S3
SaaS
apps
Confluent: Central Nervous System For
Enterprise

Confluent Enables Endless Financial Services Use Cases
Hybrid & Multi-Cloud
Messaging & Mainframe
Modernization
Streaming
Analytics
Event Driven
Microservices
CDC Patterns from
Systems Of Records
Corporate & Investment
Banking, Capital Markets
Trade Processing (Equities,
FICC, Derivatives...)
Real Time Payments and
Payments Tracking
Risk Analytics
Market, Reference, & Security
Master Data Distribution
Trading System Integrations
& Automation
CTO - Technology Modernization
Finance, Risk,
Compliance, IT, Cyber
Credit & Market Risk (CCAR,
BCBS 239, FRTB )
OATS / CAT reporting
Operational Log Hub
IT Observability
Cyber Security | SIEM
Modernization
Retail Banking, Wealth &
Asset Management
Fraud Detection
Open Banking
Customer 360 (omni channel
banking, alerts & notifications)
Client Advisor Workstations
Data and Analytics
for Asset Managers

Everywhere
Be everywhere
our customers
want to be
Cloud-Native
Re-imagined
Kafka experience
for the Cloud
Complete
Enable developers
to reliably &
securely build next-
gen apps faster
The Confluent Product Advantage

Confluent runs everywhere
18
SELF-MANAGED SOFTWARE
Confluent Platform
The Enterprise Distribution of Apache Kafka
In the datacenter
VM
FULLY-MANAGED SOFTWARE
Confluent Cloud
Apache Kafka Re-Engineered for the Cloud
In the cloud

Federated streaming, hybrid
and multi-cloud.
Data syndication and replication
across and between clouds and
on-premises, with self-service APIs,
data governance, and visual
tooling.
Reliable & real-time data streams
between all customer sites, so you
can run always-on streaming
analytics on the data of the entire
enterprise, despite regional or
cloud provider outages.
Everywhere:
Cluster Linking Global Central Nervous System

Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Augment Messaging and Mainframe Systems
and Migrate Over Time with our Support
20
1. Current middleware communication
2. Decouple from consumer app
3. Make your events available for downstream systems
Customer Payment
Jay $10
Sue $15
Grace $5
... ...
Application
producing data
Traditional Messaging
and Mainframe
Systems
Consumer
application
1
2
3
Downstream
data store

Accelerate modernization from on-prem to AWS
Redshift Sink
Lambda Sink
AWS Direct
Connect
Replicator
LEGACY EDW
MAINFRAME
LEGACY DB
JDBC / CDC
connectors
Connect
Leverage +100 Confluent pre-built connectors to
continuously bring valuable data from existing
services on-prem including enterprise data
warehouse, databases and mainframes
Modernize
Increase agility in getting applications to market
and reduce TCO when freeing up resources to
focus on value generating activities and not in
managing servers
On-prem AWS Cloud
Bridge
Hybrid cloud streaming
with consistent, event-
driven architecture for
modern apps
On-prem to AWS modernization
Amazon Athena
AWS Glue
SageMaker
Lake Formation
Amazon
DynamoDB
Amazon
Aurora
S3 Sink
Data Streams
Apps
ksqlDB

Powering real-time analytics for credit
scoring, fraud detection, and merchant
assessment services
Challenge
Drive a digital transformation at the largest bank in Indonesia to
improve the bank’s market position and increase financial inclusion
across the country.
Solution
Use Confluent to deploy an event-driven microservices architecture
that powers big data analytics for real-time credit scoring, fraud
detection, and merchant assessment services.
Results
● Fraud detection performed in real-time
● Loan disbursement times cut from two weeks to two minutes
● ISO-certified open API created
● Loan defaults predicted proactively; NPL at 0%
“Confluent Platform and Apache Kafka, by enabling us to build and
deploy real-time event-driven systems for credit scoring, have
helped BRI become the most profitable bank in Indonesia.”
— Kaspar Situmorang, Executive Vice President

Build Real-time Dashboard And Reports For
Compliance, Audit And Monitoring
Messaging
Systems
Mainframe
Databases
Big Data / Analytics Systems
SQL & NoSQL
Databases
Visualization
Tools
Regulatory
reporting
Real-time
dashboards
In-flight stream
processing

Stateful Stateless
Two types of ESP

Kafka clients Kafka Streams ksqlDB
ConsumerRecords<String, String> records = consumer.poll(100);
Map<String, Integer> counts = new DefaultMap<String, Integer>();
for (ConsumerRecord<String, Integer> record : records) {
String key = record.key();
int c = counts.get(key)
c += record.value()
counts.put(key, c)
}
for (Map.Entry<String, Integer> entry : counts.entrySet()) {
int stateCount;
int attempts;
while (attempts++ < MAX_RETRIES) {
try {
stateCount = stateStore.getValue(entry.getKey())
stateStore.setValue(entry.getKey(), entry.getValue() +
stateCount)
break;
} catch (StateStoreException e) {
RetryUtils.backoff(attempts);
}
}
}
builder
.stream("input-stream",
Consumed.with(Serdes.String(), Serdes.String()))
.groupBy((key, value) -> value)
.count()
.toStream()
.to("counts", Produced.with(Serdes.String(), Serdes.Long()));
SELECT x, count(*) FROM stream GROUP BY x EMIT CHANGES;
Flexibility Simplicity
3 modalities of stream processing with
Confluent

ksqlDB at a Glance
What is it?
ksqlDB is an event streaming
database for working with streams
and tables of data.
All the key features of a modern
streaming solution.
Aggregations Joins
Windowing
Event-Time
Dual Query
Support
Exactly-Once
Semantics
Out-of-Order
Handling
User-Defined
Functions
Compute Storage
CREATE TABLE activePromotions AS
SELECT rideId,
qualifyPromotion(distanceToDst) AS promotion
FROM locations
GROUP BY rideId
EMIT CHANGES
How does it work?
It separates compute from storage, and scales
elastically in a fault-tolerant manner.
It remains highly available during disruption,
even in the face of failure to a quorum of its
servers.
ksqlDB Kafka

Reduce Data Indexing, Analysis and Storage
Costs
Application Logs
Network Logs
Database logs
OS Logs
Collect all data sources into
Confluent
Filter events streams and only send
priority events to SIEM
Shorten SIEM retention window; store
in cheaper storage for long-term
retention
Offload fast query
and search
Send high
priority data
to SIEM
Send all data to
S3/HDFS for
cold storage
Open up data access
to new use cases
Preserve data in low-cost storage
Tier and route data to cloud storage
infrastructure for long term
retention with pre-built connectors
Save on ingest and indexing costs
Moderate ingest and index volume
by aggregating, and filtering events
to reduce license spend
Lower overall operating expenses
Reduce the need for proprietary
forwarders like Splunk Heavy
Forwarder with Splunk S2S
connector

Serverless integration
Connect existing and apps & data stores in a repeatable way without
having to manage- Apache Kafka, Schema Registry to maintain
app compatibility, ksqlDB to develop real-time apps with SQL syntax
and Connect for effortless integrations with Lambda & data stores
AWS serverless platform
Stop provisioning, maintaining or administering servers for
backend components such as compute, databases and
storage so that you can focus on increasing agility and
innovation for your developer teams
Increase developer agility & speed of innovation
Apps
Microservices
ksqlDB
Schema
Registry
COMPUTE
AWS
Lambda
Data stores
REST Proxy
& Clients
Source
Connectors
Lambda
Sink
DATA STORES
Amazon
DynamoDB
Amazon
Aurora
STORAGE
Amazon
S3
S3 Sink
ANALYTICS
Amazon
Athena
Amazon
Redshift
Serverless app integration

Stateless Serverless Stream
Processing

No provisioning,
no management
Pay for value
Automatic
scaling
Highly available
and secure
What is serverless?

No servers to manage
Only pay for stream consumption when
processing messages
Automatically scales consumers
Benefits of Serverless stream processing
Write less code

Serverless processing Server-based
processing
✔ Stream polling logic is separate from
application logic
✔ Stream polling logic is baked into your
application code
✔ Event driven processing ✔ Consumer must be running to poll the kafka clusters
✔ Scaling is handled automatically ✔ Scaling is done using consumer
groups.
✔ Poller: Open source APIs/libraries( KafkaStreams
javalibrary, kafka-python )
• Poller: Lambda ESM
Confluent Lambda Sink
Connector

Apache Kafka – Writes to
partitions

Apache Kafka – Reads from partitions

Lambda consumer options
Event
Source
Mapping
Lambda Service
Confluent Kafka Sink Connector

Confluent Lambda Sink connector
• Sink connector polls Kafka partitions and calls your function
• Lambda can be called synchronously or asynchronously.
• At least once semantics
• Provides a dead letter queue (DLQ) for any failed invocations

Confluent Lambda Sink connector – Scaling
and Error Handling
• Sink connector scales upto a soft maximum of 10 connectors.
• Error handling semantics similar to sync and async lambda invocations.
• Async: Lambda service retries twice (three total attempts)
• Sync: By default, fails and stop processing for that partition.
Option to log to another kafka topic and continue processing
• Option to batch records. Configured through aws.lambda.batch.size

Lambda ESM consumer for Kafka
Lambda
Function
instance
Poller
• Starts with one concurrent poller and
customer function
• Lambda service polls the Kafka partitions
and invokes your lambda function
synchronously

Lambda ESM consumer for Kafka
– Scaling and Batching
Lambd
a
Function
instance
Polle
r
Function
instance
Function
instance
• Scaling:
• Lambda service checks every 3 mins if
scaling is needed.
• Starts with 1 poller and scales upto <=
#partitions
• Batching: Batch records based on a BatchSize
or Batchwindow.

Capture and log exceptions
data
producer
Lambda
service
function A
(instance 1)
batch size =
200
300 records
✔
function A
(instance 1)
✔
Catch exceptions and log
to CloudWatch Logs
CloudWatch
Logs
Return successfully from
Lambda function
• Ensure processing moves forward by catching exceptions and returning successfully
!

Optimize batch-size/batch-
window to lower cost
Lambda
Function
instance
Poller
• Lambda’s maximum execution time is 15 minutes
• Adjust the batch size (max 10,000) to ensure
execution time is optimal
• For sparse topics, consider batch window to
aggregate over a time period

Kafka Producer in Lambda (create
once, use many)
• Create Producer in the constructor
• Producer will be re-used across executions for the life of
the Lambda instance
• Reduce strain on brokers by minimizing
connections and producer clients
Producer Producers!!!!
This Not this

Consider using ksqlDB for state
• A powerful combination of ksqlDB and Lambda provides a stateful -> stateless ->
stateful pattern

Enrich Transaction events for
Fraud scoring
Customer
Transactio
n
Jay $10
ksqlDB

Fraud scoring
Customer Transaction Avg 7 days Num trans 10m
Jay $10 $8.5 1

Fraud scoring
Customer Transaction Avg 7 days Num trans 10m
Jay $10 $8.5 1
Amazon
SageMaker
AWS
Lambda
ksqlDB

Next Steps
How did we do? Enter your feedback
<CSAT link & QR Code>
Schedule an executive briefing
Schedule a briefing for your business
and technology leadership team
Join or schedule a workshop
● Join our FSI workshop on July 27th
<link/QR code to registration>
● Schedule a workshop for your team
Learn more about serverless
Visit Serverlessland.com for self guided
workshops, videos, and resources
1
2
3

Thank
You!
Don’t forget to fill
out the survey!

Building Serverless EDA w_ AWS Lambda (1).pptx

Recommended

Recommended

More Related Content

Similar to Building Serverless EDA w_ AWS Lambda (1).pptx

Similar to Building Serverless EDA w_ AWS Lambda (1).pptx (20)

Recently uploaded

Recently uploaded (20)

Building Serverless EDA w_ AWS Lambda (1).pptx