Building an Analytic Extension to MySQL with ClickHouse and Open Source In this webinar Percona and Altinity offer suggestions and tips on how to recognize when MySQL is overburdened with analytics and can benefit from ClickHouse’s unique capabilities. Also, they will walk you through important patterns for integrating MySQL and ClickHouse which will enable the building of powerful and cost-efficient applications that leverage the strengths of both databases.

1. Building an Analytic extension to MySQL with
ClickHouse
1
Vadim Tkachenko(Percona) and Kanthi Subramanian(Altinity)
2 March 2023

2. Who we are
Vadim Tkachenko
CTO Percona
Kanthi Subramanian
Open source contributor/Data
Engineer/Developer Advocate
2

3. ©2023 Percona
MySQL
Strengths
- OLTP Database (Operational)
Handles up to 1mln transactions per second
- Thousands of concurrent transactions
3

4. ©2023 Percona
MySQL is
good for
- 1. ACID transactions.
- 2. Excellent concurrency.
- 3. Very fast point lookups and short
transactions.
- 4. Excellent tooling for building OLTP
applications.
- It's very good for running interactive online
properties:
- - e-commerce
- - online gaming
- - social networks
4

5. ©2023 Percona
Analytics
with MySQL
- Only for small data sets.
- Aggregation queries (GROUP BY) can be problematic
(slow) on 10mln+ rows
In summary: analyzing data over millions of small
transactions is not good use case for MySQL
Some examples (next slides):
5

6. ©2023 Percona 6
Query comparison (MySQL/ClickHouse)
The number of flights delayed by more than 10 minutes,
grouped by the day of the week, for 2000-2008
SELECT DayOfWeek, count(*) AS c
FROM ontime_snapshot
WHERE DepDel15>10 AND Year>=2000 AND
Year<=2008
GROUP BY DayOfWeek
ORDER BY c DESC;
176mln rows to process
MySQL ClickHouse
573 Seconds (9
minutes 7 seconds)
0.5 seconds

7. ©2023 Percona 7
7
Query comparison(MySQL/ClickHouse)
7
The number of flights delayed by more than 10 minutes,
grouped by the day of the week, for 2000-2008
SELECT Year, avg(DepDelay>10)*100
FROM ontime
GROUP BY Year
ORDER BY Year;
176mln rows to process
MySQL ClickHouse
240 Seconds (4
minutes)
0.674 seconds

8. ©2023 Percona
What gives
such
difference ?
8
MySQL features:
storing data in rows
single-threaded queries,
optimization for high concurrency
are exactly the opposite of those needed to run analytic queries that compute
aggregates on large datasets.
ClickHouse is designed for analytic processing:
- stores data in columns
- has optimizations to minimize I/O
- computes aggregates very efficiently
- parallelized query processing

9. ©2023 Percona 9
Why choose ClickHouse as a complement to
MySQL?
The number of flights delayed by more than 10 minutes,
grouped by the day of the week, for 2000-2008
Read all columns in row (MySQL) Read only selected columns
(ClickHouse)

10. ©2023 Percona
Signs that MySQL needs
Analytic Help
10
Read all
columns
59 GB
(100%)
MySQL, hypothetical query

11. ©2023 Percona
Signs that MySQL needs
Analytic Help
11
21 MB (.035%)
2.6 MB
(.0044%)
1.7 GB
(3%)
Read 3
columns Read 3
compressed
columns
Read 3
compressed
columns
over 8
threads
21 MB (.035%)
ClickHouse, the same query

12. ©2023 Percona
Why is MySQL
a natural
complement
to
ClickHouse?
12
MySQL
Transactional processing
Fast single row updates
High Concurrency. MySQL
support large amount of
concurrent queries
ClickHouse
Does not support ACID
transactions
Updating single row is
problematic. ClickHouse will
need to read and updated a
lot of data
ClickHouse can use a lot of
resources for a single query.
Not good use case for
concurrent access

13. 13
Leveraging Analytical
Benefits of
ClickHouse
● Identify Databases/Tables in
MySQL to be replicated
● Create schema/Databases in
ClickHouse
● Transfer Data from MySQL to
ClickHouse
https://github.com/Altinity/clickhouse-sink-connector

14. Fully wired, continuous replication
14
Table Engine(s)
Initial Dump/Load
MySQL ClickHouse
OLTP App Analytic App
MySQL
Binlog
Debezium
Altinity Sink
Connector
Kafka*
Event
Stream
*Including Pulsar and RedPanda
ReplacingMergeTree

15. Replication Setup
Validate Data
Setup CDC Replication
Initial Dump/Load
1
2
3

16. 1. Initial Dump/Load
Why do we need custom load/dump tools?
● Data Types limits and Data Types are not the same for
MySQL and ClickHouse
Date Max MySQL(9999-12-31), Date CH(2299-12-31)
● Translate/Read MySQL schema and create ClickHouse
schema. (Identify PK, partition and translate to ORDER BY
in CH(RMT))
● Faster transfer, leverage existing MySQL and ClickHouse
tools.

17. 1. Initial Dump/Load (MySQL Shell)
https://dev.mysql.com/blog-archive/mysql-shell-8-0-21-
speeding-up-the-dump-process/
https://blogs.oracle.com/mysql/post/mysql-shell-dump-load-
and-compression

18. 1. Initial Dump/Load
MySQL Shell: Multi-Threaded, Split large tables to smaller chunks, Compression,
Speeds(upto 3GB/s).
Clickhouse Client: Multi-Threaded, read compressed data.

19. 1. Initial Dump/Load
Install mysql-shell (JS)
mysqlsh -uroot -proot -hlocalhost -e "util.dump_tables('test', ['employees'],
'/tmp/employees_12');" --verbose
python db_load/clickhouse_loader.py --clickhouse_host localhost --
clickhouse_database $DATABASE --dump_dir $HOME/dbdumps/$DATABASE --
clickhouse_user root --clickhouse_password root --threads 4 --
mysql_source_database $DATABASE --mysqlshell

20. 1. Initial Dump/Load
CREATE TABLE IF NOT EXISTS `employees_predated` (
`emp_no` int NOT NULL,
`birth_date` Date32 NOT NULL,
`first_name` varchar(14) NOT NULL,
`last_name` varchar(16) NOT NULL,
`gender` enum('M','F') NOT NULL,
`hire_date` Date32 NOT NULL,
`salary` bigint unsigned DEFAULT NULL,
`num_years` tinyint unsigned DEFAULT NULL,
`bonus` mediumint unsigned DEFAULT NULL,
`small_value` smallint unsigned DEFAULT NULL,
`int_value` int unsigned DEFAULT NULL,
`discount` bigint DEFAULT NULL,
`num_years_signed` tinyint DEFAULT NULL,
`bonus_signed` mediumint DEFAULT NULL,
`small_value_signed` smallint DEFAULT NULL,
`int_value_signed` int DEFAULT NULL,
`last_modified_date_time` DateTime64(0) DEFAULT NULL,
`last_access_time` String DEFAULT NULL,
`married_status` char(1) DEFAULT NULL,
`perDiemRate` decimal(30,12) DEFAULT NULL,
`hourlyRate` double DEFAULT NULL,
`jobDescription` text DEFAULT NULL,
`updated_time` String NULL ,
`bytes_date` longblob DEFAULT NULL,
`binary_test_column` varbinary(255) DEFAULT NULL,
`blob_med` mediumblob DEFAULT NULL,
`blob_new` blob DEFAULT NULL,
`_sign` Int8 DEFAULT 1,
`_version` UInt64 DEFAULT 0,
) ENGINE = ReplacingMergeTree(_version) ORDER BY (`emp_no`)
SETTINGS index_granularity = 8192;
CREATE TABLE `employees_predated` (
`emp_no` int NOT NULL,
`birth_date` date NOT NULL,
`first_name` varchar(14) NOT NULL,
`last_name` varchar(16) NOT NULL,
`gender` enum('M','F') NOT NULL,
`hire_date` date NOT NULL,
`salary` bigint unsigned DEFAULT NULL,
`num_years` tinyint unsigned DEFAULT NULL,
`bonus` mediumint unsigned DEFAULT NULL,
`small_value` smallint unsigned DEFAULT NULL,
`int_value` int unsigned DEFAULT NULL,
`discount` bigint DEFAULT NULL,
`num_years_signed` tinyint DEFAULT NULL,
`bonus_signed` mediumint DEFAULT NULL,
`small_value_signed` smallint DEFAULT NULL,
`int_value_signed` int DEFAULT NULL,
`last_modified_date_time` datetime DEFAULT NULL,
`last_access_time` time DEFAULT NULL,
`married_status` char(1) DEFAULT NULL,
`perDiemRate` decimal(30,12) DEFAULT NULL,
`hourlyRate` double DEFAULT NULL,
`jobDescription` text,
`updated_time` timestamp NULL DEFAULT NULL,
`bytes_date` longblob,
`binary_test_column` varbinary(255) DEFAULT NULL,
`blob_med` mediumblob,
`blob_new` blob,
PRIMARY KEY (`emp_no`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
COLLATE=utf8mb4_0900_ai_ci
/*!50100 PARTITION BY RANGE (`emp_no`)
(PARTITION p1 VALUES LESS THAN (1000) ENGINE = InnoDB,
PARTITION p2 VALUES LESS THAN MAXVALUE ENGINE = InnoDB)
*/ |
MySQL
ClickHouse

21. 2. Validate Data
Why is a basic count check not enough?
● Essential to validate the values, example decimal/floating
precision and datatype limits.
● Data Types are different between MySQL and ClickHouse.
Solution: md5 checksum of column data (Courtesy:
Sisense)
1. Take the MD5 of each column. Use a space for
NULL values.
2. Concatenate those results, and MD5 this result.
3. Split into 4 8-character hex strings.
4. Convert into 32-bit integers and sum.
python
db_compare/mysql_table_check
sum.py --mysql_host localhost --
mysql_user root --mysql_password
root --mysql_database menagerie
--tables_regex "^pet" --
debug_output
python
db_compare/clickhouse_table_c
hecksum.py --clickhouse_host
localhost --clickhouse_user root --
clickhouse_password root --
clickhouse_database menagerie --
tables_regex "^pet" --debug_output
diff out.pet.ch.txt out.pet.mysql.txt
| grep "<|>"
Credits: Arnaud

22. 3. Setup CDC Replication
MySQL
binlog file: mysql.bin.00001
binlog position: 100002
Or
Gtid: 1233:223232323
Debezium
Altinity Sink
Connector
Kafka*
Event
Stream
ClickHouse
Setup Debezium to start from binlog file/position or Gtid
https://github.com/Altinity/clickhouse-sink-connector/blob/develop/doc/debezium_setup.md

23. Final step - Deploy
● Docker Compose (Debezium Strimzi, Sink Strimzi)
https://hub.docker.com/repository/docker/altinity/clickhouse-sink-connector
● Kubernetes (Docker images)
● JAR file

24. Simplified Architecture
MySQL
binlog file: mysql.bin.00001
binlog position: 100002
Or
Gtid: 1233:223232323
ClickHouse
Debezium
Altinity Sink
Connector
One executable
One service

25. Final step - Monitor
● Monitor Lag
● Connector Status
● Kafka monitoring
● CPU/Memory Stats

26. Challenges
- MySQL Master failover
- Schema Changes(DDL)

27. MySQL Master Replication

28. MySQL Master Failover

29. MySQL Master Failover - Snowflake ID
binlog timestamp

30. Alter Table support
30
ADD Column <col_name> varchar(1000)
NULL
ADD Column <col_name> Nullable(String)
ADD index type btree ADD index type minmax
MySQL ClickHouse

31. 31
Replicating Schema Changes

32. 32
Replicating Schema Changes
● Debezium does not provide events for all DDL Changes
● Complete DDL is only available in a separate topic(Not a
SinkRecord)
● Parallel Kafka workers might process messages out of order.

33. 33
Replicating Schema Changes

34. Where can I get more information?
34
Altinity Sink Connector for ClickHouse
https://github.com/Altinity/clickhouse-sink-connector
https://github.com/ClickHouse/ClickHouse
https://github.com/mydumper/mydumper

35. 35
Project roadmap and next Steps
- PostgreSQL, Mongo, SQL server support
- CH shards/replicas support
- Support Transactions.

36. 36
Thank you!
Questions?
https://altinity.com https://percona.com