Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Go Against the Flow
Databases and Stream
Processing
N E H A N A R K H E D E
Businesses are streams of events
DB
Old World
DB
DB
DB
DWH
Operational Databases Relational Data Warehouse
Reporting
Analytics
App
App
App
New World
Streaming First
• DB/DWH + Many more distributed
data systems
• Monolith -> Microservices
• Batch -> Real-time
App
01
Databases
A Swiss-army Knife
C H A L L E N G E 0 1
Shared state is unsuitable
for microservices
App
03
App
02
App
01
Databases
Mutable state hurts
forward compatibility
App
03
App
02
App
01
Databases
C H A L L E N G E 0 2
App App
App
Query
Inefficient for
streaming data
C H A L L E N G E 0 3
Databases
Turning the database inside out
for a streaming-first world
Storage
What would the core storage abstraction
for streaming ...
T U R N I N G T H E D A T A B A S E I N S I D E O U T
Storage in Databases
The log is an implementation detail
Log
T U R N I N G T H E D A T A B A S E I N S I D E O U T
Storage for Streams
The log as a first class citizen
Log• Suitable f...
T U R N I N G T H E D A T A B A S E I N S I D E O U T
Query
Processing in Databases
One-time short-lived queries
T U R N I N G T H E D A T A B A S E I N S I D E O U T
Processing on Streams
Continuous queries
Stream
Table
Stream 01:
Str...
Continuous queries core abstractions
Streams and Tables
Stream
Table
Stream 01:
Stream 02:
Continuous
Query Derived Table
...
Query
Insert data
Source Tables Materialized View
Create via a query
Select ⭑ FROM ORDERS
Where Region – ‘USA’
T U R N I N...
Streaming Materialized Views
In Kafka
Stream
Table
Stream 01:
Stream 02:
Continuous
Query
Streaming
Materialized View
QueryQuery
What is Stream Processing?
Stream 01:
Stream 02:
Continuous
Query
Stream
Table
Processing streams of data to cr...
Stream Processing is approachable
only to those of us who can write code
Introducing KSQL
Open source Streaming SQL for Apache Kafka
The first completely interactive SQL interface for Kafka
KSQL ...
N O C O D I N G R E Q U I R E D
KSQL
A look inside
• You can submit queries using an
interactive SQL command line client
• Several continuous queries run ...
KS Q L DE M O
Real-time Anomaly Detection:
Malicious Web Users
KSQL in practice
Use Cases
A big step towards a streaming-first world:
• Real-time monitoring and analytics
• Streaming ET...
KSQL
Streaming SQL for Apache Kafka™
github.com/confluentinc/ksql
slackpass.io/confluentcommunity -- #ksql
confluent.io/ks...
Kafka Summit SF 2017 - Keynote - Go Against the Flow: Databases and Stream Processing
Kafka Summit SF 2017 - Keynote - Go Against the Flow: Databases and Stream Processing
Upcoming SlideShare
Loading in …5
×

Kafka Summit SF 2017 - Keynote - Go Against the Flow: Databases and Stream Processing

3,327 views

Published on

Keynote Presented at Kafka Summit SF 2017 by Neha Narkhede, CTO and Co-founder, Confluent

Published in: Software
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Kafka Summit SF 2017 - Keynote - Go Against the Flow: Databases and Stream Processing

  1. 1. Go Against the Flow Databases and Stream Processing N E H A N A R K H E D E
  2. 2. Businesses are streams of events
  3. 3. DB Old World DB DB DB DWH Operational Databases Relational Data Warehouse Reporting Analytics App App App
  4. 4. New World Streaming First • DB/DWH + Many more distributed data systems • Monolith -> Microservices • Batch -> Real-time
  5. 5. App 01 Databases A Swiss-army Knife
  6. 6. C H A L L E N G E 0 1 Shared state is unsuitable for microservices App 03 App 02 App 01 Databases
  7. 7. Mutable state hurts forward compatibility App 03 App 02 App 01 Databases C H A L L E N G E 0 2 App App App
  8. 8. Query Inefficient for streaming data C H A L L E N G E 0 3 Databases
  9. 9. Turning the database inside out for a streaming-first world Storage What would the core storage abstraction for streaming data look like? Processing What would queries on streaming data look like? Materialized Views How can materialized views be constructed on streaming data?
  10. 10. T U R N I N G T H E D A T A B A S E I N S I D E O U T Storage in Databases The log is an implementation detail Log
  11. 11. T U R N I N G T H E D A T A B A S E I N S I D E O U T Storage for Streams The log as a first class citizen Log• Suitable for streaming data • Built around immutability as a core construct
  12. 12. T U R N I N G T H E D A T A B A S E I N S I D E O U T Query Processing in Databases One-time short-lived queries
  13. 13. T U R N I N G T H E D A T A B A S E I N S I D E O U T Processing on Streams Continuous queries Stream Table Stream 01: Stream 02: Continuous Query
  14. 14. Continuous queries core abstractions Streams and Tables Stream Table Stream 01: Stream 02: Continuous Query Derived Table Source of Truth Stream
  15. 15. Query Insert data Source Tables Materialized View Create via a query Select ⭑ FROM ORDERS Where Region – ‘USA’ T U R N I N G T H E D A T A B A S E I N S I D E O U T Materialized Views In relational databases
  16. 16. Streaming Materialized Views In Kafka Stream Table Stream 01: Stream 02: Continuous Query Streaming Materialized View
  17. 17. QueryQuery What is Stream Processing? Stream 01: Stream 02: Continuous Query Stream Table Processing streams of data to create more streams or tables
  18. 18. Stream Processing is approachable only to those of us who can write code
  19. 19. Introducing KSQL Open source Streaming SQL for Apache Kafka The first completely interactive SQL interface for Kafka KSQL supports a variety of powerful stream processing operations Continuous window aggregations Stream-table joins Filters, projections Sessionization
  20. 20. N O C O D I N G R E Q U I R E D
  21. 21. KSQL A look inside • You can submit queries using an interactive SQL command line client • Several continuous queries run in parallel on a KSQL cluster • Adding more server processes scales a KSQL cluster
  22. 22. KS Q L DE M O Real-time Anomaly Detection: Malicious Web Users
  23. 23. KSQL in practice Use Cases A big step towards a streaming-first world: • Real-time monitoring and analytics • Streaming ETL, not Batch ETL • Application development
  24. 24. KSQL Streaming SQL for Apache Kafka™ github.com/confluentinc/ksql slackpass.io/confluentcommunity -- #ksql confluent.io/ksql

×