What is (data) streaming? What is it good for? (use cases) How to build a streaming solution? Showing which building blocks are needed, using both an AWS example and a Kafka example, plus some other considerations.

1. Photo by Leo Rivas on Unsplash
OfirManor

3. Whatis(data)streaming
Dataintegrationof backendsystems
“realtimeETL”
Oppositeofbatchprocessing
Continuous,incrementalcomputation
Low-latency
“humanrealtime”

5. Improveoperationaldecisions
by combining historical and real-time data
Retail
real-timepersonalization
real-timesupplychainandlogisticsoptimizations
IOT/ OnlineGaming/Cyber
Healthcare
collect,analyzeandrespondtoreadingsfrom
multiplepatientdevices(biometrics)or
multiplehospitaldevices(Smarthospitals)
FinancialServices
real-timefrauddetection
real-timemobilenotifications
Ad Tech
real-timeusertargeting
IT
EnterpriseServiceBus

7. STREAMINGPIPELINESOURCES TARGETS
Oracle
DB/2
SQLServer
SAP
Files
Salesforce
EDW
DataLake
(Hadoop)
DataMarts
DataLake(S3)
FileServer
PERSIST
TheStreamingPipelineBuildingBlocks
enterpriseoperationalsourcesanalyticsenvironments
PROCESS
Stateless/Stateful
Code / SQL / UI
CAPTURE
Events
WRITE
Bulk Write

8. STREAMINGPIPELINESOURCES
Oracle
DB/2
SQLServer
SAP
Files
Salesforce
TARGETS
EDW
DataLake
(Hadoop)
DataMarts
DataLake(S3)
FileServer
PERSIST
PROCESS
CAPTURE
WRITE
Database
MigrationService
Others
Kinesis
DataAnalytics
Kinesis
DataFirehose
Kinesis
DataStreams
EMR/Glue
Example: AWS-centricarchitecture

9. STREAMINGPIPELINESOURCES
Oracle
DB/2
SQLServer
SAP
Files
Salesforce
TARGETS
EDW
DataLake
(Hadoop)
DataMarts
DataLake(S3)
FileServer
PERSIST
PROCESS
CAPTURE
WRITE
Example: Kafka-centricarchitecture
KafkaStreams
KSQL(Confluent)
KafkaConnect
KafkaConnect
Debezium
KafkaCore

10. STREAMINGPIPELINESOURCES
Oracle
DB/2
SQLServer
SAP
Files
Salesforce
TARGETS
EDW
DataLake
(Hadoop)
DataMarts
DataLake(S3)
FileServer
PERSIST
PROCESS
CAPTURE
Example: Kafka-centricvariant
KafkaConnect
Debezium
KafkaCore
ApacheSpark
ApacheFlink
WRITE
KafkaConnect

11. Justpick yourtools
GoogleCloudDataflow
Cloud
services
Open
source
technologies
Commercial
products

13. Summary
What?
Continuouslymovingdatabetweensystemswhileapplyingcomputations
Why?
Improveoperationaldecisionsbycombininghistoricalandreal-timedata
How?
CAPTURE  PERSIST  PROCESS  WRITE
Developer/Analyst/ ETLuser ↔ code/SQL/ drag-and-dropUI
Cloudservices↔Products Assemble↔Distribution Specialsources/targets

14. Photo by Leo Rivas on Unsplash
OfirManor
@ofirm
slideshare.net/ofirm