The database is half done

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Databases Are Only Half-Done
Dr. Michael Noll
Vienna, 15. Juni 2020
miguno
Office of the CTO, Confluent

DATABASES ARE
ONLY HALF-DONE

Marc Andreessen:
Software is Eating the World

AI Mobile IoTCloud

Strong Form
Companies are
BECOMING SOFTWARE
Weak Form
Companies are
USING MORE SOFTWARE

Using Software vs. Becoming Software

Loan Application Using Software
BORROWER
1
CREDIT
OFFICER
3
LOAN
OFFICER
5
RISK
OFFICER
4
APPROVE
DENY
6
APPLICATION
FORM
2

Loan Application Using Software
BORROWER
1
CREDIT
OFFICER
3
LOAN
OFFICER
5
RISK
OFFICER
4
APPROVE
DENY
6
APPLICATION
FORM
...

Loan Application in Software
BORROWER
1
APPROVE
DENY
3
LOAN APP UI
CREDIT
SERVICE
RISK
SERVICE
CRM
SERVICE
$ !
2

Loan Application in Software
BORROWER
1
APPROVE
DENY
3
LOAN APP UI
CREDIT
SERVICE
RISK
SERVICE
CRM
SERVICE
$ !
2
SOFTWARE ...

Using Software:
Classic Three-Tier Architecture
USER UI SERVICE DATABASE

Becoming Software:
Services Talking To Each Other With APIs
SERVICESERVICESERVICE SERVICE

GEOSPATIAL
MATCHING
ROUTE
RE-PLANNING
BUSINESS
EVENTS
BUSINESS
EVENTS
DRIVERCUSTOMER
REQUESTING
A RIDE

IS MORE
SOFTWARE
THE USER OF
THE SOFTWARE

What does this mean for databases?

We have hundreds
of databases...

We have hundreds
of databases...
FUNDAMENTAL ASSUMPTION:
DATA IS PASSIVE

Fundamental Assumption: Data is Passive
QUERY
UPDATE

Good for Building:
CRUD
Applications
Synchronous
Services

Databases are designed to help you!

Unless there is a user and UI waiting,
why should it be synchronous?

The Solution: Event Streams

APP
APP APP
APP
APP APP
APP
APP
DB
DB
DB
APP
APP
APP
APP
DB
DB
DB
APP
APP
EVENT
STREAMING
PLATFORM

Kafka is the Foundational ‘Filesystem’ for Event Streams
0 1 2 3 4 5 6 77
STREAM
STORAGE
(LOG)
READS
WRITES
DESTINATION
SY...

Solution: Event Stream Processing
INPUT STREAMS OUTPUT STREAMS
STREAM
PROCESSOR

Loan Application With Event Streams
RISK
SERVICE
LOAN
SERVICE
CREDIT
SERVICE

ksqlDB
The Event Streaming Database

TRADITIONAL
DATABASE
SELECT * FROM
DB_TABLE
Active Query Passive Data
DB Table
EVENT STREAMING
DATABASE
CREATE TABLE T AS
...

TABLES STREAMS
V2
V1
USER
JAY
SUE
FRED
CREDIT_SCORE
695
430
710V3
PAYMENTS
42
18
65
...
USER
JAY
SUE
FRED
...

STREAMS
CREATE STREAM payments (user VARCHAR, amount INT)
WITH (kafka_topic = 'all_payments',
key = 'user',
value_format =...

TABLES
CREATE TABLE credit_scores AS
SELECT user, updateScore(p.amount) AS credit_score
FROM payments AS p
GROUP BY user
E...

The stream is aggregated & materialized into a table,
which is emitted as a stream
Payments Stream
Credit Score Stream
CRE...

But wait, some things are still
synchronous...

PULL
What is Jay’s
credit score now?
695
APP
PUSH
APP
Jay’s credit score is 670
Jay’s credit score is 710
Jay’s credit sco...

PULL PUSH

Today: Integrating all these systems is up to you
CONNECTOR
CONNECTOR
APP
DB
APP
DB
STREAM
PROCESSING
CONNECTOR APPDB

STREAM
PROCESSING
Today: Integrating all these systems is up to you
CONNECTOR
CONNECTOR
APP
DB
APP
DB
CONNECTOR APPDB
EXTR...

CAN WE MAKE IT SIMPLER?
EASY ⇔ MAINSTREAM

ksqlDB: Streaming import & export of external data
through Kafka Connect integration
CREATE SINK CONNECTOR elastic_connect...

SELECT user, credit_score
FROM credit_scores
WHERE ROWKEY = 'Jay';
ksqlDB: Pull Queries for data lookups

PULL
SELECT user, credit_score
FROM credit_scores
WHERE ROWKEY = 'Jay';
PUSH
SELECT user, credit_score
FROM credit_scores
...

All-in-One Package: Extract, Stream, Summarize, Materialize, Notify, Query
DB
ksqlDB
Payments
Stream
APP
Query
Credit Scor...

Now we can move from this architecture to…
CONNECTOR
CONNECTOR
APP
DB
APP
DB
STREAM
PROCESSING
CONNECTOR APPDB

A Solution with fewer moving parts
APP
DB
APP
DB
APP
ksqlDB

This is a natural generalization
of databases.

Architecture of a Distributed Database
Storage
SERVING
NODES
SQL
SQL
SQL

Architecture of a Distributed Stream Processing System
STORAGE
STREAM
PROCESSORS
SQL
SQL
SQL
SERVING
NODES
SQL
SQL
SQL

STREAM
PROCESSORS
SQL
SQL
SQL
SERVING
NODES
SQL
SQL
SQL
ksqlDB
ksqlDB: Event Streaming Database

We still need all
of these!

In fact, ksqlDB makes it easier to leverage them...
CREATE SINK CONNECTOR elastic_connector WITH (
'connector.class' = 'ES...

As Software Is Eating the World

IS MORE
SOFTWARE
THE USER OF
THE SOFTWARE

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1011447 views
Storytelling For The Web: Integrate... by Chiara Aliotta 919244 views
Artificial Intelligence, Data and C... by OECD Directorate ... 820562 views
How to Leverage AI to Boost Employe... by SocialHRCamp 342424 views
2024 State of Marketing Report – by... by Marius Sescu 195380 views
Everything You Need To Know About C... by Expeed Software 218594 views

View on Slideshare

1 of 52 Ad

View on Slideshare

1 of 52 Ad