Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 2024 , Vietnam FOSSASIA '24.pptx.pdf

Unleashing Real-time Insights with ClickHouse:
Navigating the Landscape in 2024
ALKIN TEZUYSAL
FOSSASIA , Hanoi, Vietnam - Apr 2024
@ask_dba
@ChistaDATA Inc. 2024

Let’s get connected with Alkin first
Alkin Tezuysal - EVP - Global Services @chistadata
● Linkedin : https://www.linkedin.com/in/askdba/
Open Source Database Evangelist
● Previously PlanetScale, Percona and Pythian as Technical Manager, SRE, DBA
● Previously Enterprise DBA , Informix, Oracle, DB2 , SQL Server
@ask_dba

About ChistaDATA Inc.
Founded in 2021 by Shiv Iyer - CEO and Principal
Strong lineage, backed by leading investors
Focusing on ClickHouse infrastructure engineering and performance operations
What’s ClickHouse anyway?
Services and Products around dedicated DBaaS, Managed Services, Support and Consulting
www.chistadata.io www.chistadata.com
@ask_dba

● Most Influential in Database Community 2022 - The Redgate 100
● MySQL Cookbook, 4th Edition 2022 - O'Reilly Media, Inc.
● MySQL Rockstar 2023 - Oracle (MySQL Community)
● Database Design and Modeling with PostgreSQL and MySQL 2024 - <Packt>
Recognitions
@ask_dba

Maritime Trivia
@ask_dba
What is the term for the process of turning a sailing vessel away from
the wind, allowing the sails to fill and propel the boat forward?

Tacking…
@ask_dba
Gybing…
Tremola?

What is ClickHouse?
ClickHouse is;
● Open-source Apache 2.0
● Column-oriented
● Database management system that is engineered for high-speed analytics.
● Its columnar storage model and advanced compression enable real-time
analysis on large data volumes.
@ask_dba

Row oriented
mysql> select customer_id, customer_zip_code_prefix, customer_city , customer_state from customers limit 5;
+----------------------------------+--------------------------+---------------+----------------+
| customer_id | customer_zip_code_prefix | customer_city | customer_state |
+----------------------------------+--------------------------+---------------+----------------+
| 00012a2ce6f8dcda20d059ce98491703 | 6273 | osasco | SP |
| 000161a058600d5901f007fab4c27140 | 35550 | itapecerica | MG |
| 0001fd6190edaaf884bcaf3d49edf079 | 29830 | nova venecia | ES |
| 0002414f95344307404f0ace7a26f1d5 | 39664 | mendonca | MG |
| 000379cdec625522490c315e70c7a9fb | 4841 | sao paulo | SP |
+----------------------------------+--------------------------+---------------+----------------+
@ask_dba

Each column in separate file with same row offset.
@ask_dba

Column-oriented?
Query id: e8312155-2b9a-4ced-8af4-e05a2a977842
┌─customer_id──────────────────────┬─customer_zip_code_prefix─┬─customer_city─┬─customer_state─┐
1. │ 00012a2ce6f8dcda20d059ce98491703 │ 6273 │ osasco │ SP │
2. │ 000161a058600d5901f007fab4c27140 │ 35550 │ itapecerica │ MG │
3. │ 0001fd6190edaaf884bcaf3d49edf079 │ 29830 │ nova venecia │ ES │
4. │ 0002414f95344307404f0ace7a26f1d5 │ 39664 │ mendonca │ MG │
5. │ 000379cdec625522490c315e70c7a9fb │ 4841 │ sao paulo │ SP │
└──────────────────────────────────┴──────────────────────────┴───────────────┴─────────────
───┘
5 rows in set. Elapsed: 0.010 sec.
@ask_dba

The importance of real-time analytics
● Helps deliver on strategic imperatives
● Competitive advantage
● Improve efficiencies
● Enhance customer experience
● Increase revenues
@ask_dba

ClickHouse Highlights
● Efficient compression
○ Supports multiple compression codecs, such as LZ4 and ZSTD
● Vectorized Query Execution
○ Vectorized query execution processes data in batches, operating on multiple data
points with a single CPU instruction.
● CPU Efficiency
○ Full use of modern CPUs' capabilities, including SIMD (Single Instruction, Multiple
Data) instructions
● Scalability
○ Built-in horizontal sharding and replication.
● Rich Function Library
○ Built-in functions and operators for data transformation, filtering, and aggregation
● Geospatial Support, Materialized Views, Support for SQL Syntax
@ask_dba

Benchmarks
@ask_dba

ClickHouse Engine Family
● MergeTree The most universal and functional table engines for high-load
tasks.
● Log Lightweight engines with minimum functionality.
● Integration Engines Engines for communicating with other data storage and
processing systems.
@ask_dba

ClickHouse Integration Engine Family
@ask_dba

Sample integration with MySQL
statement: CREATE TABLE olist.mysql_data
(
`customer_id` String,
`customer_unique_id` String,
`customer_zip_code_prefix` Nullable(Int32) DEFAULT NULL,
`customer_city` Nullable(String) DEFAULT NULL,
`customer_state` Nullable(String) DEFAULT NULL
)
ENGINE = MySQL('127.0.0.1:3306', 'olist', 'customers', 'root', '[HIDDEN]')
1 row in set. Elapsed: 0.001 sec.
@ask_dba

Load data to ClickHouse
:) INSERT INTO customers SELECT *
FROM mysql_data
Query id: f4e154ad-c6dd-497d-988e-d0d019319a53
Ok.
@ask_dba

Transferred table in ClickHouse
:) select count(*) from customers;
SELECT count(*)
FROM customers
Query id: dffacd95-a0ae-4027-b12b-dfa17d780e79
┌─count()─┐
1. │ 192016 │
└─────────┘
1 row in set. Elapsed: 0.008 sec.
@ask_dba

ClickHouse default compression
+------------------------------------+------------+
| Table | Size in MB |
+------------------------------------+------------+
| geolocation | 54.58 |
| customers | 34.39 |
| order_reviews | 20.58 |
| orders | 20.58 |
| order_items | 14.56 |
| order_payments | 11.55 |
| products | 5.52 |
| sellers | 0.33 |
| product_cateegory_name_translation | 0.02 |
+------------------------------------+------------+
Query id: e18d6938-9213-462d-a0d5-12a297ffa1ef
┌─table_name─┬─size──────┬─total_rows─┐
1. │ customers │ 13.64 MiB │ 192016 │
2. │ mysql_data │ ᴺᵁᴸᴸ │ ᴺᵁᴸᴸ │
└────────────┴───────────┴─────────
───┘
2 rows in set. Elapsed: 0.001 sec.
@ask_dba

Use Case Ideas
● Analytics on denormalized tables
● Star Schema migration
● Time Series data ingestion via streaming
● Log data
● OLTP data archive
● Data Lake and Fabric solutions
● Observibility
@ask_dba

Streaming Data to Real Time Analytics
@ask_dba

@ask_dba
@ask_dba

Get started with clickhouse-local
$ curl https://clickhouse.com/ | sh
$ ./clickhouse local -q "SELECT * FROM 'customers.tsv'"
@ask_dba

Get started with brew on MacOS
$ brew install --cask clickhouse
$ clickhouse
ClickHouse local version 24.3.1.2672 (official build).
macbook-pro-4.local :) SELECT
name AS table_name,
formatReadableSize(total_bytes) AS size,
total_rows
FROM system.tables
WHERE database = 'olist'
ORDER BY total_bytes DESC;
SELECT
name AS table_name,
@ask_dba

Born to Sail, Forced to Work!
Catching winds
@svrubato
How to contribute
to community?
@ask_dba

@ask_dba

THANK YOU
Q&A
@ask_dba

References & Credits
● Blog - ChistaDATA Inc.
● Knowledge Base Archive - ChistaDATA Inc.
● What Is ClickHouse? | ClickHouse Docs
● ClickHouse Quick Start
● Benchmarking Opentelemetry
● clickhouse-benchmark | ClickHouse Docs
@ask_dba

Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 2024 , Vietnam FOSSASIA '24.pptx.pdf

Recommended

Recommended

More Related Content

Similar to Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 2024 , Vietnam FOSSASIA '24.pptx.pdf

Similar to Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 2024 , Vietnam FOSSASIA '24.pptx.pdf (20)

More from Alkin Tezuysal

More from Alkin Tezuysal (20)

Recently uploaded

Recently uploaded (20)

Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 2024 , Vietnam FOSSASIA '24.pptx.pdf