Stefano Pampaloni, Maria Pina Di Cataldo - Meetup #AperiTech di Roma Apache Kafka - Codemotion Rome 2019

#AperiTech Kafka
Stefano Pampaloni
Maria Pina Di Cataldo
Rome | March 22 - 23, 2019

What’s Apache Kafka?
• Pub/Sub Logic - Messaging, Done Right
• STORE - Hadoop, Made Fast
• ETL & Data Integration as a Platform
• Stream processing

Industry Trends… and why
Apache Kafka matters!
1. From ‘big data’ (batch) to ‘fast data’ (stream processing)
2. Internet of Things (IoT) and sensor data 
3. Microservices and asynchronous communication
(coordination messages and data streams) between
loosely coupled and fine-grained services

Overview
"Kafka is a “publish-subscribe messaging
rethought as a distributed commit log”
"Fast
"Scalable
"Durable
"Distributed

Kafka adoption and use cases
"LinkedIn: activity streams, operational metrics, data bus 400
nodes, 18k topics, 220B msg/day (peak 3.2M msg/s), May 2014
"Netflix: real-time monitoring and event processing
"Twitter: as part of their Storm real-time data pipelines
"Spotify: log delivery (from 4h down to 10s), Hadoop
"Loggly: log collection and processing
"Mozilla: telemetry data
"Airbnb, Cisco, Gnip, InfoChimps, Ooyala, Square, Uber, …

Over 35% of Fortune 500’s are using
Apache Kafka™
6 of top 10
Travel
7 of top 10
Global banks
8 of top 10
Insurance
9 of top 10
Telecom

From Big Data to Fast Data
5
From big data to fast data
Stream data is
The faster the better
Stream data can be
big or fast (Lambda)
Stream data will be
big AND fast (Kappa)
Apache Kafka is the enabling technology of this transition
Big data was
The more the better
ValueofData
Volume of Data
ValueofData
Age of Data Speed Table Batch Table
DB
Streams Hadoop
Job 1 Job 2
Streams
Table 1 Table 2
DB

Confluent Platform
13
Confluent Platform
Open Source ExternalEnterprise
Confluent Platform
Monitoring
Analytics
Custom Apps
Transformations
Real-time
Applications
…
CRM
Data Warehouse
Database
Hadoop
Data 
Integration
…
Control Center
Auto-data 
Balancing
Multi-Data  
Center Replication
24/7 Support
Supported
Connectors
Clients
Schema  
Registry
REST  
Proxy
Apache Kafka
Kafka 
Connect
Kafka 
Streams
Kafka 
Core
Database Changes Log Events loT Data Web Events …

ATM Fraud Detection
@rmoff
with Apache Kafka and KSQL
Photoby FreddieCollins on Unsplash

ATM Fraud Detection with Apache Kafka and KSQL
@rmoff
Spotting fraud in realtime
hotoby MirzaBabic on Unsplash

Photoby LasayeHommes on Unsplash

@rmoff
• Account id
• Location
• Amount
•
Inbound stream of ATM data
https://github.com/rmoff/gess

@rmoff
Demo!

@rmoff
Spot patterns within this stream
Ac. ID Transaction ID Time ATM
A42 xxx116d91d6-ef17 11:56:58 Midland
A42 116d91d6-ef17 11:58:19 Halifax
A42 09c2f660-ef17 19:31:11 Lloyds

@rmoff
A42 116d91d6-ef17 11:58:19 Halifax
A42 09c2f660-ef17 19:31:11 Lloyds
Legit
Legit

@rmoff
A42 116d91d6-ef17 11:58:19 Halifax
A42 09c2f660-ef17 19:31:11 Lloyds
Legit
Dodgy!
Legit

@rmoff
KSQL : Stream Processing with SQL
TXN_ID, ATM,
CUSTOMER_NAME,
CUSTOMER_PHONE
ATM_POSSIBLE_FRAUD;
SELECT
FROM

@rmoff

@rmoff
Customer
details
ATM fraud txns
with customer
details
Elasticsearch
Notification
service
1. Spot fraud in stream of
transactions
2.Enrich transaction events
with customer data

@rmoff
KSQLis the
Streaming
SQL Engine
for
Apache Kafka

@rmoff
KSQL for Real-Time Monitoring
• Log data monitoring, tracking and alerting
• syslog data
• Sensor / IoT data
CREATE STREAM SYSLOG_INVALID_USERS AS
SELECT HOST, MESSAGE
FROM SYSLOG
WHERE MESSAGE LIKE '%Invalid user%';
http://cnfl.io/syslogs-filtering / http://cnfl.io/syslog-alerting

@rmoff
KSQL for Streaming ETL
CREATE STREAM vip_actions AS  
SELECT userid, page, action
FROM clickstream c
LEFT JOIN users u
ON c.userid = u.user_id  
WHERE u.level = 'Platinum';
Joining, filtering, and aggregating streams of event data

@rmoff
KSQL for Anomaly Detection
CREATE TABLE possible_fraud AS 
SELECT card_number, count(*) 
FROM authorization_attempts  
WINDOW TUMBLING (SIZE 5 SECONDS) 
GROUP BY card_number 
HAVING count(*) > 3;
Identifying patterns or anomalies in real-time data,
surfaced in milliseconds

@rmoff
CREATE STREAM pageviews
WITH (PARTITIONS=4,
VALUE_FORMAT='AVRO') AS  
SELECT * FROM pageviews_json;
KSQL for Data Transformation
Make simple derivations of existing topics from the command line

@rmoff
KSQL in Development and Production
Interactive KSQL 
for development and testing
Headless KSQL 
for Production
Desired KSQL queries
have been identified
REST
“Hmm, let me try 
out this idea...”

@rmoff
Stream Stream joins
Orders
Shipments
Which orders
haven't shipped?
order.id = shipment.order_id
Leadtime
shipment_ts - order_ts

@rmoff
Stream Stream joins
ATM transactions

@rmoff
Self-Join (Cartesian product)
A42 116d91d6-ef17 11:58:19 Halifax
A42 09c2f660-ef17 19:31:11 Lloyds
T
A42 116d91d6-ef17 11:58:19 Halifax
A42 09c2f660-ef17 19:31:11 Lloyds
A42 116d91d6-ef17 11:58:19 Halifax
A42 09c2f660-ef17 19:31:11 Lloyds
T1 T2

@rmoff
A42 116d91d6-ef17 11:58:19 Halifax
A42 09c2f660-ef17 19:31:11 Lloyds
A42 116d91d6-ef17 11:58:19 Halifax
A42 09c2f660-ef17 19:31:11 Lloyds
T1 T2
ATM_TXNS T1
INNER JOIN ATM_TXNS T2
ON T1.ACCOUNT_ID = T2.ACCOUNT_ID

@rmoff
FROM ATM_TXNS T1
INNER JOIN ATM_TXNS T2
WITHIN 10 MINUTES
ON T1.ACCOUNT_ID = T2.ACCOUNT_ID
A42 116d91d6-ef17 11:58:19 Halifax
A42 09c2f660-ef17 19:31:11 Lloyds
A42 116d91d6-ef17 11:58:19 Halifax
A42 09c2f660-ef17 19:31:11 Lloyds
T1 T2

@rmoff
Self-Join
T1 Txn ID T2 Txn ID T1 Time T2 Time T1 ATM T2 ATM
xxx116d91d6-ef17 116d91d6-ef17 11:56:58 11:58:19 Midland Halifax
116d91d6-ef17 xxx116d91d6-ef17 11:58:19 11:56:58 Halifax Midland
xxx116d91d6-ef17 xxx116d91d6-ef17 11:56:58 11:56:58 Midland Midland
116d91d6-ef17 116d91d6-ef17 11:58:19 11:58:19 Halifax Halifax

@rmoff
Self-Join
xxx116d91d6-ef17 xxx116d91d6-ef17 11:56:58 11:56:58 Midland Midland
116d91d6-ef17 116d91d6-ef17 11:58:19 11:58:19 Halifax Halifax
Self join on same txn IDs

@rmoff
Exclude joins on the same txn
WHERE T1.TRANSACTION_ID !=
T2.TRANSACTION_ID

@rmoff
Exclude joins on the same txn
Duplicate results (A:B / B:A)

@rmoff
Join Windows
A42 116d91d6-ef17 11:58:19 Halifax
A42 09c2f660-ef17 19:31:11 Lloyds
A42 116d91d6-ef17 11:58:19 Halifax
A42 09c2f660-ef17 19:31:11 Lloyds
T1 T2
WITHIN 10 MINUTES
T2.TRANSACTION_ID

@rmoff
Only join forward
A42 116d91d6-ef17 11:58:19 Halifax
A42 09c2f660-ef17 19:31:11 Lloyds
A42 116d91d6-ef17 11:58:19 Halifax
A42 09c2f660-ef17 19:31:11 Lloyds
T1 T2
WITHIN (0 MINUTES, 10 MINUTES)
T2.TRANSACTION_ID

@rmoff
Only join forward
WITHIN (0 MINUTES, 10 MINUTES)
Ignore events in the right-hand
stream prior to those in the left

@rmoff
Only join forward
Legit Dodgy!

@rmoff
Photoby EstebanLopez on Unsplash

@rmoff
Calcuate distance between ATMs
GEO_DISTANCE(TX1.location->lat, TX1.location->lon,
TX2.location->lat, TX2.location->lon,
'KM')
TX1
TX2

@rmoff
Calculate time between transactions
TX2.ROWTIME - TX1.ROWTIME AS
MILLISECONDS_DIFFERENCE
(TX2.ROWTIME - TX1.ROWTIME)
/ 1000 / 60 / 60 AS HOURS_DIFFERENCE

@rmoff
Photoby EstebanLopez on Unsplash
GEO_DISTANCE(…) / HOURS_DIFFERENCE
AS KMH_REQUIRED

@rmoff
So speaking of time…
ksql> PRINT 'atm_txns_gess' ;
Format:JSON
{
"ROWTIME": 1544116309152,
"ROWKEY": "null",
"account_id": "a218",
"timestamp": "2018-12-06 17:09:58 +0000",
"atm": "HSBC",
…}
Kafka message
timestamp
2018-12-06 17:11:49
Event time

ksql> PRINT 'atm_txns_gess' ;
Format:JSON
{
"ROWTIME": 1544116309152,
"ROWKEY": "null",
"account_id": "a218",
"timestamp": "2018-12-06 17:09:58 +0000",
CREATE STREAM ATM_TXNS_GESS
(account_id VARCHAR, timestamp VARCHAR, …
WITH (KAFKA_TOPIC='atm_txns_gess',
TIMESTAMP='timestamp',
TIMESTAMP_FORMAT=
'yyyy-MM-dd HH:mm:ss X');
"timestamp": "2018-12-06 17:09:58 +0000",

@rmoff
But what about the account holder?
!

@rmoff
Photoby SamuelZeller on Unsplash

@rmoff
Customer
details
ATM fraud txns
with customer
details
Elasticsearch
Notification
service
1. Enrich transaction events
with customer data

@rmoff
Streaming Integration with Kafka Connect
Kafka Brokers
Kafka Connect
Tasks Workers
Sourcessyslog
flat file
CSV
JSON
MQTT

@rmoff
Kafka Brokers
Kafka Connect
Tasks Workers
Sinks
Amazon S3
MQTT

@rmoff
Kafka Brokers
Kafka Connect
Tasks Workers
Sources Sinks
Amazon S3
MQTT
syslog
flat file
CSV
JSON
MQTT

@rmoff
Confluent Hub
hub.confluent.io
• One-stop place to discover and
download :
• Connectors
• Transformations
• Converters

@rmoff
Demo Time!
Customer
details
Kafka Connect
Debezium

@rmoff
Do you think that’s a table
you are querying?

@rmoff
The Table Stream Duality
Account ID Balance
12345 €50
Account ID Amount
12345 + €50
12345 + €25
12345 -€60
Account ID Balance
12345 €75
Account ID Balance
12345 €15
Time
Stream Table

@rmoff
The truth is the log.
The database is a cache
of a subset of the log.
—Pat Helland
Immutability Changes Everything
http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf
Photo by Bobby Burch on Unsplash

@rmoff
Ac. ID T1 Time ATM T2 Time ATM
A42 11:56:58 Midland 11:58:19 Halifax
Suspect Transactions
Dodgy!

@rmoff
Name Phone Ac. ID T1 Time ATM T2 Time ATM
Robin M 1234 567 A42 11:56:58 Midland 11:58:19 Halifax
Suspect Transactions

@rmoff
Customer
details
ATM fraud txns
with customer
details
Elasticsearch
Notification
service
1. Spot fraud in stream of
transactions
2.Enrich transaction events
with customer data
ATM_POSSIBLE_FRAUD_ENRICHED
atm_txns_gess
accounts

@rmoff
What can we do with it?
Photoby JoshuaRodriguez on Unsplash

@rmoff
Realtime Operations View & Analysis

@rmoff
Push notification to the customer

@rmoff
Confluent Community Components
Apache Kafka with a bunch of cool stuff! For free!
Database Changes Log Events loT Data Web Events …
CRM
Data Warehouse
Database
Hadoop
Data 
Integration
…
Monitoring
Analytics
Custom Apps
Transformations
Real-time Applications
…
Confluent Platform
Confluent Platform
Apache Kafka®
Core | Connect API | Streams API
Data Compatibility
Schema Registry
Monitoring & Administration
Confluent Control Center | Security
Operations
Replicator | Auto Data Balancing
Development and Connectivity
Clients | Connectors | REST Proxy | CLI
SQL Stream Processing
KSQL
Datacenter Public Cloud Confluent Cloud
CONFLUENT FULLY-MANAGEDCUSTOMER SELF-MANAGED

@rmoff
Free Books!
https://www.confluent.io/apache-kafka-stream-processing-book-bundle

@rmoff
robin@confluent.io
https://www.confluent.io/ksql
http://cnfl.io/slack https://cnfl.io/demo-scene

@rmoff
• CDC Spreadsheet
• Blog: No More Silos: How to Integrate your Databases with Apache Kafka and CDC
• #partner-engineering on Slack for questions
• BD team (#partners / partners@confluent.io) can help with introductions on a given sales op
Resources
#EOF

Stefano Pampaloni, Maria Pina Di Cataldo - Meetup #AperiTech di Roma Apache Kafka - Codemotion Rome 2019

Recommended

Recommended

More Related Content

Similar to Stefano Pampaloni, Maria Pina Di Cataldo - Meetup #AperiTech di Roma Apache Kafka - Codemotion Rome 2019

Similar to Stefano Pampaloni, Maria Pina Di Cataldo - Meetup #AperiTech di Roma Apache Kafka - Codemotion Rome 2019 (20)

More from Codemotion

More from Codemotion (20)

Recently uploaded

Recently uploaded (20)

Stefano Pampaloni, Maria Pina Di Cataldo - Meetup #AperiTech di Roma Apache Kafka - Codemotion Rome 2019