Kubernetes is turning into a preferred platform for real-time analytic app that crunch billions of events per day and return insights in seconds. In this talk we'll introduce the standard analytic app design pattern of fast event streams coupled with low-latency data warehouses, using open source projects. We'll then walk through deploying the pipeline on Kubernetes from ingest to end user access. We'll touch on use of operators, scaling, monitoring, upgrade, security, and approaches to adding custom components. Attendees can expect to leave with concrete lessons about how to stand up low-latency analytics quickly on Kubernetes. This talk was given by Robert Hodges for DoK Day Europe @ KubeCon 2022.

1. © 2022 Altinity, Inc.
Growing Up Fast:
Kubernetes and Real-Time
Analytic Applications
Robert Hodges, Altinity
KubeCon Data On Kubernetes Day 2022
16 May 2022
1

2. © 2022 Altinity, Inc.
Very quick introductions!
ClickHouse support and services including Altinity.Cloud
Authors of Altinity Kubernetes Operator for ClickHouse
and other open source projects
Robert Hodges
Database geek with 30+ years
on DBMS systems. Day job:
Altinity CEO
Altinity Engineering
Database geeks with centuries
of experience in DBMS and
applications
2

3. © 2022 Altinity, Inc.
Why real-time analytics?
3
eCommerce
Transactions
Data
Warehouse
Event
Stream
Web Server
Event Logs
Fraud
Detection
Apps
Operational
Dashboards
“How can I boost the rate of return on
special offers for products?”
“Is this a fraudulent transaction?”

4. © 2022 Altinity, Inc.
Building pipelines in Kubernetes
4
Direction of data flow
DBMS for
eCommerce
transactions
Change data
capture from
DBMS log
Kafka-
compatible
event stream
Low latency data
warehouse
Dashboards for
business analysts

5. © 2022 Altinity, Inc.
Bring up components one by one…
5
ClickHouse
CRD
ClickHouse
Operator
ZooKeeper
Deployment
Install ZooKeeper Deployment from script
1
Install ClickHouse Operator from script
2
Apply CRD using kubectl
3
clickhouse
Namespace

6. © 2022 Altinity, Inc.
Sample CRD for the data warehouse
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
name: "first"
spec:
configuration:
clusters:
- name: "first"
layout:
shardsCount: 2
replicasCount: 2
templates:
podTemplate: clickhouse-stable
volumeClaimTemplate: pd-default
zookeeper:
nodes:
- host: zookeeper.zoo1ns
port: 2181 6
Plus another couple
pages to define pods
and storage…

7. © 2022 Altinity, Inc.
Here’s what the full pipeline looks like
7
MySQL
CRD
MySQL
Operator
MySQL Helm
Chart
Redpanda CRD
Redpanda Operator
Cert-Manager Deployment
Redpanda and Jetpack Helm
Chart
ClickHouse
CRD
ClickHouse
Operator
ZooKeeper
Deployment
Grafana
Deployment
Grafana Helm
Chart
clickhouse
mysql debezium redpanda grafana
Debezium
Deployment
Schema Registry
Deployment

8. © 2022 Altinity, Inc.
A key detail: Getting from event stream to warehouse
8
(Or any Kafka-
compatible event stream)
Custom
Application
Direct transfer via
ClickHouse Kafka Engine
What
else?

9. © 2022 Altinity, Inc.
ClickHouse
Kafka Connect solves producing/consuming at both ends
9
(Or any Kafka-
compatible event stream)
Kafka Connect
source connector
MySQL
Kafka Connect sink
connector
Data warehouse

10. © 2022 Altinity, Inc.
Current practice: Kubernetes and Managed Services
10
Managed Kubernetes
Altinity.Cloud
Managed
ClickHouse
Custom
Application
Custom
Application
Custom
Application

11. © 2022 Altinity, Inc.
Where can I get more information?
11
Altinity Sink Connector for ClickHouse
https://github.com/Altinity/clickhouse-sink-connector

12. © 2022 Altinity, Inc.
Thank you!
https://altinity.com
rhodges at altinity.com
12
Altinity.Cloud
Altinity
Kubernetes
Operator for
ClickHouse
We’re hiring!