The document discusses various options for streaming data pipelines, including Kafka Streams, streaming SQL solutions, and cloud-based data warehouses and data lakes. It highlights the strengths and weaknesses of each technology, focusing on real-time OLAP databases for low latency analytics and the role of Kafka in integrating diverse data streams. Additionally, it mentions the challenges of using data lakes and provides an overview of several tools and frameworks for building streaming analytics applications.

1. How
Stream
a
Query
to
Tim Berglund
@tlberglund

2. Simpler times

3. Simpler analytics

4. DHW->Hadoop

5. Streaming data pipelines

7. OLAP vs. OLTP in Streams

8. Our Options
• Connect/Relational DB
• Kafka Streams
• Streaming SQL
• Cloud Data Warehouse
• Data Lake
• Real-Time OLAP Database

9. Our Options
• Connect/Relational DB
• Kafka Streams
• Streaming SQL
• Cloud Data Warehouse
• Data Lake
• Real-Time OLAP Database

11. Connect/RDBMS
• Suitable for smaller data
• Transactional
• Familiar
Kafka Connect
Broker
Broker
Broker
Cluster
Kafka Connect Data Sink

13. Kafka Streams
• Ingests directly from a topic
• KTable
• Forms an in-memory key/value store
suitable for querying by topic key
• Scalable across members of a
consumer group
• Readable through Interactive Queries
(transactional)

14. Kafka Streams
final KStream<String, String> stream =
builder.stream(inputTopic, Consumed.with(stringSerde, stringSerde));
final KTable<String, String> convertedTable =
stream.toTable(Materialized.as("stream-converted-to-table"));
(transactional)

15. Kafka Streams
• Full-featured Java stream processing
API
• Arbitrary streaming computation
• Can emit new streams (not this talk)
• KTables queryable by key
• Every read pattern requires its own
topology
• Interactive Queries again
(analytical)

16. Kafka Streams
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> textLines =
builder.stream("TextLinesTopic");
KTable<String, Long> wordCounts = textLines
.flatMapValues(textLine ->
Arrays.asList(textLine.toLowerCase().split("W+")))
.groupBy((key, word) -> word)
.count(Materialized.<String, Long, KeyValueStore<Bytes, byte[]>>as("counts-store"));
wordCounts.toStream().to("WordsWithCountsTopic",
Produced.with(Serdes.String(), Serdes.Long()));
(analytical)

18. Streaming SQL
• Materialize
• DeltaStream
• RisingWave
• ksqlDB

19. Why not Flink?
I love you,
little squirrel.

20. Why not Flink?
But this talk is
not about you.

21. Materialize
• Replacement data warehouse
• Integrates with Kafka, Postgres, dbt
• The Materialized View is the central
abstraction
• Views are persistent and queryable
• Postgres wire-compatible
• Positioned as an analytics solution

22. Delta Stream
• Cloud-native streaming SQL
• Serverless, BYOC
• Kafka, Kinesis integration
• Materialized views and streaming
pipelines
• Positioned as streaming database
and streaming analytics

23. Rising Wave
• Distributed SQL Streaming database
• Cloud and OSS versions
• Based on a cleanroom
implementation of Flink in C++
• Kafka, Pulsar, Kinesis integrations
• Flink+persistent views
• Postgres wire-compatible

24. ksqlDB
• “Streaming Database”
• Provides persistent TABLE
abstraction
• Pull and Push queries
• Like KafkaStreams, but in SQL

26. Cloud Data Warehouses

27. Cloud Data Warehouses
• The cloud-based heir of legacy data
warehousing
• Ingest from batch and streaming
sources
• Biased towards structured data and
batch access

29. Data
Lake
We’ll figure
this out
Anything
else

30. Data Lakes
• Started as the HDFS cluster
• Became S3
• That didn’t help…
• ELT vs. ETL
• Iceberg/Hudi/DeltaLake

32. Data Lakes
• Storage and compute are radically
decoupled
• Structure is relatively less
important
• Reads are slow
• Streaming is historically difficult

33. Data Lakes
• Storage and compute are radically
decoupled
• Structure is relatively less
important
• Reads are slow
• Streaming is historically difficult

35. Real-Time OLAP
• Designed for high concurrency, low
latency queries
• Ingests from streaming and batch
sources
• Intimate integration with Kafka
• Conventional tables and SQL
interface

36. Real-Time OLAP
• Analytics shaped like real-time
data
• Analytics when users are decision
makers

37. Technology Selection

38. Go With What You Know

39. Performance

40. Community/Adoption

41. Area of
Exploration
Differentiated Application Code
Kafka

43. Tim Berglund
@tlberglund
https://dev.startree.ai
https://rtasummit.com
For more resources on Apache Pinot:
To register for Real-Time Analytics Summit: