The document outlines the use of Apache Gradle to build and automate KSQL and Apache Kafka Streams, focusing on data integration and the creation of pipelines for handling streaming data. It provides a primer on KSQL, including examples of queries and the management of dependencies within and across pipelines. The document also introduces Gradle as a tool for continuous delivery and highlights relevant plugins for KSQL and Kafka Streams.

1. Use Apache Gradle to Build
and Automate KSQL and
Apache Kafka Streams

2. twitter: @stewartbryson
medium: @stewartbryson
linkedin: stewartbryson
Owner & CEO
Red Pill Analytics

3. @redpilla
What We Do
Data Warehouse Analytics
ANALYTICS
Data-engineering & ETL

4. @redpilla
Project Overview
KSQL
KSQL User Defined Functions
(demo-only)
Kafka Streams (demo-only)
Agenda

5. @redpilla

6. @redpilla
Project Inspiration

7. @redpilla
Data Integration: Streaming In

8. @redpilla
Data Integration: Streaming Out

9. @redpilla
Packaging of Payloads for OWMC

10. @redpilla
KSQL Primer

11. @redpilla
KSQL Registration Queries
CREATE STREAM clickstream
( _time bigint,
time varchar,
ip varchar,
request varchar,
status int,
userid int,
bytes bigint,
agent varchar )
with
( kafka_topic = 'clickstream',
value_format = 'json' );

12. @redpilla
KSQL Persistent Queries
CREATE TABLE events_per_min AS
SELECT userid,
count(*) AS events
FROM clickstream window
TUMBLING (size 60 second)
GROUP BY userid;

13. @redpilla
A pipeline is a group of SQL
statements that work together to
define an end-to-end process.

14. @redpilla
KSQL Dependencies

15. @redpilla
KSQL Pipelines
clickstream
clickstream_codes
Streams and TablesSQL Scripts
enriched_error_codes
enriched_error_codes_count
customer_clickstream
user_clickstream
web_users
click_user_sessions

16. @redpilla
KSQL Pipelines
clickstream
clickstream_codes
Streams and TablesSQL Scripts
enriched_error_codes
enriched_error_codes_count
customer_clickstream
user_clickstream
web_users
click_user_sessions
Dependencies
within a pipeline

17. @redpilla
KSQL Pipelines
clickstream
clickstream_codes
Streams and TablesSQL Scripts
enriched_error_codes
enriched_error_codes_count
customer_clickstream
user_clickstream
web_users
click_user_sessions
Dependencies
across pipelines

18. @redpilla
KSQL Pipelines
clickstream
clickstream_codes
Streams and TablesSQL Scripts
enriched_error_codes
enriched_error_codes_count
customer_clickstream
user_clickstream
web_users
click_user_sessions
Manage all KSQL
dependencies in
one place

19. @redpilla
Multiple statements per pipeline
Statements are in script files
Script files are in Git
Developers need to iterate

20. @redpilla
What is Apache Gradle?
Gradle is an open-source build automation tool for the Age
of Continuous Delivery (CD). Building software is no longer
just about compiling, linking and packaging.

21. @redpilla
What is Apache Gradle?

22. @redpilla
gradle-confluent for KSQL
pipelines
shadow for KSQL user defined
functions
application for Kafka Streams
Gradle Plugins

23. @redpilla
Plugin

24. @redpilla
Plugin
github.com/RedPillAnalytics/gradle-confluent

25. @redpilla
Unfortunately…
I only have 40 minutes

26. @redpilla
Unfortunately…
I only have 33 minutes

27. @redpilla
Demo

28. @redpilla
Demo
...using a Jupyter Notebook