Several different frameworks have been developed to draw data from Kafka and maintain standard SQL over continually changing data. This provides an easy way to query and transform data - now accessible by orders of magnitude more users. At the same time, using Standard SQL against changing data is a new pattern for many engineers and analysts. While the language hasn’t changed, we’re still in the early stages of understanding the power of SQL over Kafka - and in some interesting ways, this new pattern introduces some exciting new idioms. In this session, we’ll start with some basic use cases of how Standard SQL can be effectively used over events in Kafka- including how these SQL engines can help teams that are brand new to streaming data get started. From there, we’ll cover a series of more advanced functions and their implications, including: - WHERE clauses that contain time change the validity intervals of your data; you can programmatically introduce and retract records based on their payloads! - LATERAL joins turn streams of query arguments into query results; they will automatically share their query plans and resources! - GROUP BY aggregations can be applied to ever-growing data collections; reduce data that wouldn't even fit in a database in the first place. We'll review in-production examples where each of these cases make unmodified Standard SQL, run and maintain over data streams in Kafka, and provide the functionality of bespoke stream processors.

1. Frank McSherry, Chief Scientist
How to use Standard SQL over Kafka
From the basics to advanced use cases

2. You can write SQL and make
streaming infrastructure

3. A brief history of SQL systems
(spoiler: it ends up with streams)

4. OLTP
Reads
Writes
Transactions
Analytics
Dashboards
Monitoring

5. OLTP
Reads
Writes
Transactions
Analytics
Dashboards
Monitoring

6. OLTP OLAP
Reads
Writes
Transactions
Analytics
Dashboards
Monitoring

7. OLTP OLAP
Reads
Writes
Transactions
Analytics
Dashboards
Monitoring
(row-based) (columnar)
different designs

8. OLTP OLAP
Reads
Writes
Transactions
Analytics
Dashboards
Monitoring
(row-based) (columnar)

9. OLTP OLVM
Reads
Writes
Transactions
Analytics
Dashboards
Monitoring
(pull) (push)
different designs

10. Analytics
Dashboards
Monitoring

11. OLTP
Kafka
OLTP
Kafka
Analytics
Dashboards
Monitoring

14. Stream connection 1/3:
SQL systems can have
streaming internals

16. Stream connection 2/3:
SQL systems can consume
streams as inputs

18. Stream connection 3/3:
SQL systems can produce
streams as outputs

20. sensors
readings
max_temp_by_region

21. Bonus Features 1/3:
SQL over unbounded data

22. This could be unbounded
This is all we need to store

23. Bonus Features 2/3:
Focus on recent events.

24. Like now() but for "event time"

25. Bonus Features 3/3:
Event-driven SQL queries

27. For each query in queries ...
... restrict by the query argument ...
... keep the top three regions.

28. You can write SQL and make
streaming infrastructure

29. Streams and SQL combine as
inputs flow to systems flow to outputs
Streams and SQL lead to new idioms
working with unbounded data,
focusing on recent events,
event-driven SQL queries,
... and more. (seriously)

30. https://materialize.com/kafka
Head to
for free shirts, cloud sign-up, and a guide to Kafka.
- @frankmcsherry
"best DB shirt released in 2020
(great design, soft material)"
@andy_pavlo