This document discusses the history and evolution of database concepts like commit logs, replication, and partitioning into the modern streaming data architecture. It traces how technologies like Kafka evolved from basic concepts like commit logs for high availability and scalability to power stream processing, data integration, and real-time analytics. The core idea of storing a commit log of all changes has remained fundamental to powering new use cases with streaming data.

1. 1
Back to database fundamentals
(… aka the origin of the streaming platform)
Florent Ramiere florent@confluent.io

2. 2

3. 3

4. 44
What happened in 1978 in
our industry ?

5. 5

6. 6

7. 7

8. 8

9. 9
Commit log
Redo log
Transaction log
Bin log
WAL

10. 10
Materialize events into an opinionated structure

11. 11
So you get the power of SQL

12. 1212
What if you need to scale ?

13. 13
Master
handles the writes
Replication
Replica
handles the reads

14. 14
Master
Replication
Replication
Slave

15. 15
Leader
Replication
Replication
Follower

16. 1616
Replication leveraged
the events too!

17. 1717
But what if you want to
leverage these events
yourself ?

18. 18
SQL on tables
CDC
SQL on events

19. 19
Load events and
impact another
system
CDC

20. 20
ETL
CDC

21. 21
… You just added another line!

22. 2222
Could we simplify ?

23. 23
CDC

24. 24
CDC
CDC

25. 25
CDC
CDC
CDC

26. 2626
What about transformation ?

27. 27

28. 28

29. 29
?!

30. 30
ETL?

31. 31
… You just added another line!

32. 32

33. 33

34. 34

35. 3535
What was standardized in
1998 ?

36. 36
CREATE TABLE sales (
prod_id NUMBER(6),
cust_id NUMBER,
quantity_sold NUMBER(3),
amount_sold NUMBER(10, 2)
)
CREATE TRIGGER sales_deletion
AFTER DELETE ON sales
REFERENCING OLD ROW AS Old
FOR EACH ROW
INSERT INTO sales_deleted_Log
VALUES (Old.title);
CREATE TRIGGER sales_update
AFTER UPDATE ON sales
REFERENCING OLD ROW AS Old, NEW ROW AS New
FOR EACH ROW
BEGIN ATOMIC
UPDATE customer SET
amount = amount
+ (New.amount * New.quantity)
- (Old.amount * Old.quantity)
WHERE
id=New.cust_id
END;
CREATE TRIGGER sales_creation
AFTER INSERT ON sales
REFERENCING NEW ROW AS New
BEGIN ATOMIC
UPDATE customer SET
amount = amount
+ (New.amount * New.quantity)
WHERE
id=New.cust_id
END;

37. 3737
Why can’t use use a
big shared global database
that does it all ?

38. 38
Like this one ...

39. 39
Shared database
Application refactoring
Direct schema coupling
Scaling challenge
Single point of failure

40. 4040
File Transfer
Latent Data
Direct database load
Consistency Challenges
E
X
P
O
R
T
I
M
P
O
R
T

41. 41
RPC/REST/SOA
Direct coupling
Application refactoring
Availability concerns
Scaling Concerns
Function call
answer

42. 4242
Pub/Sub messaging
Loosely coupled
High scalability
High availability
Real-time
E
V
E
N
T

43. 43
Database
Database Replication
CDC & Consommation
Transformation

44. 44
Central nervous system
Focus on the essence of our systems

45. 4545
Back to Database

46. 4646
What problem was solved in
1998 ?

47. 47

48. 48

49. 49

50. 50

51. 51
CREATE TABLE sales (
prod_id NUMBER(6),
cust_id NUMBER,
time_id DATE,
channel_id CHAR(1),
promo_id NUMBER(6),
quantity_sold NUMBER(3),
amount_sold NUMBER(10, 2)
) PARTITION BY RANGE (time_id) INTERVAL ( NUMTOYMINTERVAL(1,'MONTH'))
CREATE TABLE order_items (
order_id NUMBER(12) NOT NULL,
line_item_id NUMBER(3) NOT NULL,
product_id NUMBER(6) NOT NULL,
unit_price NUMBER(8, 2),
quantity NUMBER(8),
CONSTRAINT order_items_fk FOREIGN KEY(order_id) REFERENCES orders(order_id)
) PARTITION BY REFERENCE(order_items_fk);
CREATE TABLE dept (
deptno NUMBER,
deptname VARCHAR(32)
) PARTITION BY HASH(deptno) PARTITIONS 2;

52. 52
Logical
Physical

53. 53
Table
Partition

54. 54

55. 55

56. 5656
How do we query ?

57. 57
SELECT * FROM dept WHERE deptno=47;

58. 58
SELECT * FROM dept WHERE deptno=47;
HASH(deptno) PARTITIONS 2

59. 59
SELECT * FROM dept WHERE deptno=47;
47 % 2

60. 60
SELECT * FROM dept WHERE deptno=47;
47 % 2
SELECT * FROM dept WHERE deptno=47;

61. 61
SELECT * FROM dept WHERE deptno=47;
SELECT * FROM dept WHERE deptno=47;

62. 62
SELECT * FROM dept WHERE deptno=47;

63. 6363
We are querying the state
produced by the processor
executing the commit log

64. 6464
We just re-created
materialized views

65. 6565
To express an always up to
date business view

66. 66

67. 67

68. 6868
We are querying the state
produced by the commit log
processor

69. 6969
We are querying the state
produced by the commit log
processorBusiness
Events
Output

70. 7070
We are querying the state
produced by the commit log
processorProducer/Consumer
Kafka Stream
KSQL
...
Events
State store

71. 71

72. 7272
We get to choose
the right tool !

73. 7373
This is streaming

74. 7474
The foundational component
has already been serving you
for years!

75. 75

76. 7676
Confluent
is the streaming platform
with Kafka at its core
as the commit log

77. 77
THANK YOU
@framiere
florent@confluent.io
cnfl.io/blog