Passen traditionelle Datenbankarchitekturen noch in unsere Welt? Was wäre, wenn wir den Fluß all dessen, was in einem Unternehmen geschieht, als "Daten" behandeln und kontinuierliche "Abfragen" zulassen, die diese verarbeiten, beantworten und darauf reagieren? Dies führt zu einem grundlegend anderen Rahmen dessen, was eine Datenbank sein kann. Freuen Sie sich auf einen spannenden Vortrag von Dr. Michael Noll aus dem Office of the CTO bei Confluent.

1. Databases Are Only Half-Done
Dr. Michael Noll
Vienna, 15. Juni 2020
miguno
Office of the CTO, Confluent

2. DATABASES ARE
ONLY HALF-DONE

3. Marc Andreessen:
Software is Eating the World

4. AI Mobile IoTCloud

5. Strong Form
Companies are
BECOMING SOFTWARE
Weak Form
Companies are
USING MORE SOFTWARE

6. Using Software vs. Becoming Software

7. Loan Application Using Software
BORROWER
1
CREDIT
OFFICER
3
LOAN
OFFICER
5
RISK
OFFICER
4
APPROVE
DENY
6
APPLICATION
FORM
2

8. Loan Application Using Software
BORROWER
1
CREDIT
OFFICER
3
LOAN
OFFICER
5
RISK
OFFICER
4
APPROVE
DENY
6
APPLICATION
FORM
2
HUMAN CENTRIC
1-2 WEEKS

9. Loan Application in Software
BORROWER
1
APPROVE
DENY
3
LOAN APP UI
CREDIT
SERVICE
RISK
SERVICE
CRM
SERVICE
$ !
2

10. Loan Application in Software
BORROWER
1
APPROVE
DENY
3
LOAN APP UI
CREDIT
SERVICE
RISK
SERVICE
CRM
SERVICE
$ !
2
SOFTWARE CENTRIC
SECONDS

11. Using Software:
Classic Three-Tier Architecture
USER UI SERVICE DATABASE

12. Becoming Software:
Services Talking To Each Other With APIs
SERVICESERVICESERVICE SERVICE

13. GEOSPATIAL
MATCHING
ROUTE
RE-PLANNING
BUSINESS
EVENTS
BUSINESS
EVENTS
DRIVERCUSTOMER
REQUESTING
A RIDE

14. IS MORE
SOFTWARE
THE USER OF
THE SOFTWARE

15. What does this mean for databases?

16. We have hundreds
of databases...

17. We have hundreds
of databases...
FUNDAMENTAL ASSUMPTION:
DATA IS PASSIVE

18. Fundamental Assumption: Data is Passive
QUERY
UPDATE

19. Good for Building:
CRUD
Applications
Synchronous
Services

20. Databases are designed to help you!

21. Unless there is a user and UI waiting,
why should it be synchronous?

22. The Solution: Event Streams

23. APP
APP APP
APP
APP APP
APP
APP
DB
DB
DB
APP
APP
APP
APP
DB
DB
DB
APP
APP
EVENT
STREAMING
PLATFORM

24. Kafka is the Foundational ‘Filesystem’ for Event Streams
0 1 2 3 4 5 6 77
STREAM
STORAGE
(LOG)
READS
WRITES
DESTINATION
SYSTEM A
DESTINATION
SYSTEM B

25. Solution: Event Stream Processing
INPUT STREAMS OUTPUT STREAMS
STREAM
PROCESSOR

26. Loan Application With Event Streams
RISK
SERVICE
LOAN
SERVICE
CREDIT
SERVICE

27. ksqlDB
The Event Streaming Database

28. TRADITIONAL
DATABASE
SELECT * FROM
DB_TABLE
Active Query Passive Data
DB Table
EVENT STREAMING
DATABASE
CREATE TABLE T AS
SELECT * FROM
EVENT_STREAM
EMIT CHANGES
Active Data Passive Query
Event Stream

29. TABLES STREAMS
V2
V1
USER
JAY
SUE
FRED
CREDIT_SCORE
695
430
710V3
PAYMENTS
42
18
65
...
USER
JAY
SUE
FRED
...

30. STREAMS
CREATE STREAM payments (user VARCHAR, amount INT)
WITH (kafka_topic = 'all_payments',
key = 'user',
value_format = 'avro');

31. TABLES
CREATE TABLE credit_scores AS
SELECT user, updateScore(p.amount) AS credit_score
FROM payments AS p
GROUP BY user
EMIT CHANGES;

32. The stream is aggregated & materialized into a table,
which is emitted as a stream
Payments Stream
Credit Score Stream
CREATE TABLE credit_scores AS
SELECT user, updateScore(p.amount) ...
EMIT CHANGES;
Credit Score Table

33. But wait, some things are still
synchronous...

34. PULL
What is Jay’s
credit score now?
695
APP
PUSH
APP
Jay’s credit score is 670
Jay’s credit score is 710
Jay’s credit score is 695
...

35. PULL PUSH

36. Today: Integrating all these systems is up to you
CONNECTOR
CONNECTOR
APP
DB
APP
DB
STREAM
PROCESSING
CONNECTOR APPDB

37. STREAM
PROCESSING
Today: Integrating all these systems is up to you
CONNECTOR
CONNECTOR
APP
DB
APP
DB
CONNECTOR APPDB
EXTRACT STORE
LOAD STORE
TRANSFORM

38. CAN WE MAKE IT SIMPLER?
EASY ⇔ MAINSTREAM

39. ksqlDB: Streaming import & export of external data
through Kafka Connect integration
CREATE SINK CONNECTOR elastic_connector WITH (
'connector.class' = 'ElasticsearchSinkConnector',
'connection.url' = 'https://elastic-server:9200',
'topics' = 'CREDIT_SCORES',
'type.name’ = 'kafka-connect', ...);

40. SELECT user, credit_score
FROM credit_scores
WHERE ROWKEY = 'Jay';
ksqlDB: Pull Queries for data lookups

41. PULL
SELECT user, credit_score
FROM credit_scores
WHERE ROWKEY = 'Jay';
PUSH
SELECT user, credit_score
FROM credit_scores
WHERE ROWKEY = 'Jay'
EMIT CHANGES;
What is Jay’s
credit score now?
695
APP APP
Jay’s credit score is 670
Jay’s credit score is 710
Jay’s credit score is 695
...

42. All-in-One Package: Extract, Stream, Summarize, Materialize, Notify, Query
DB
ksqlDB
Payments
Stream
APP
Query
Credit Scores
Stream
Credit Scores
Payments
DB
Compute & Materialize
Credit Scores

43. Now we can move from this architecture to…
CONNECTOR
CONNECTOR
APP
DB
APP
DB
STREAM
PROCESSING
CONNECTOR APPDB

44. A Solution with fewer moving parts
APP
DB
APP
DB
APP
ksqlDB

45. This is a natural generalization
of databases.

46. Architecture of a Distributed Database
Storage
SERVING
NODES
SQL
SQL
SQL

47. Architecture of a Distributed Stream Processing System
STORAGE
STREAM
PROCESSORS
SQL
SQL
SQL
SERVING
NODES
SQL
SQL
SQL

48. STREAM
PROCESSORS
SQL
SQL
SQL
SERVING
NODES
SQL
SQL
SQL
ksqlDB
ksqlDB: Event Streaming Database

49. We still need all
of these!

50. In fact, ksqlDB makes it easier to leverage them...
CREATE SINK CONNECTOR elastic_connector WITH (
'connector.class' = 'ESSinkConnector',
'connection.url' = 'https://elastic:9200',
'topics' = 'CREDIT_SCORES',
'type.name' = 'kafka-connect', ...);
APP
ksqlDB APPCONNECTOR

51. As Software Is Eating the World

52. IS MORE
SOFTWARE
THE USER OF
THE SOFTWARE