SlideShare a Scribd company logo
Webinar: Secrets of ClickHouse Query Performance, by Robert Hodges
Brief Intros
www.altinity.com
Leading software and services
provider for ClickHouse
Major committer and community
sponsor in US and Western Europe
Robert Hodges - Altinity CEO
30+ years on DBMS plus
virtualization and security.
ClickHouse is DBMS #20
Goals of the talk
● Understand single node MergeTree structure
● Optimize queries without changing data
● Get bigger performance gains by changing data layout
● Introduce tools for performance monitoring
Non-Goals:
● Boost performance of sharded/replicated clusters
● Teach advanced ClickHouse performance management
ClickHouse &
MergeTree Intro
Introduction to ClickHouse
Understands SQL
Runs on bare metal to cloud
Shared nothing architecture
Stores data in columns
Parallel and vectorized execution
Scales to many petabytes
Is Open source (Apache 2.0)
a b c d
a b c d
a b c d
a b c d
And it’s really fast!
Introducing the MergeTree table engine
CREATE TABLE ontime (
Year UInt16,
Quarter UInt8,
Month UInt8,
...
) ENGINE = MergeTree()
PARTITION BY toYYYYMM(FlightDate)
ORDER BY (Carrier, FlightDate)
Table engine type
How to break data
into parts
How to index and
sort data in each part
Basic MergeTree data layout
Table
Part
Index Columns
Sparse Index
Columns
sorted on
ORDER BY
columns
Rows match
PARTITION BY
expression
Part
Index Columns
Part
MergeTree layout within a single part
/var/lib/clickhouse/data/airline/ontime_reordered
2017-01-01 AA
2017-01-01 EV
2018-01-01 UA
2018-01-02 AA
...
primary.idx
||||
.mrk .bin
20170701_20170731_355_355_2/
(FlightDate, Carrier...) ActualElapsedTime Airline AirlineID...
||||
.mrk .bin
||||
.mrk .bin
Granule Compressed
Block
Mark
Basic Query
Tuning
ClickHouse performance tuning is different...
The bad news…
● No query optimizer
● No EXPLAIN PLAN
● May need to move [a lot
of] data for performance
The good news…
● No query optimizer!
● System log is great
● System tables are too
● Performance drivers are
simple: I/O and CPU
Your friend: the ClickHouse query log
clickhouse-client --send_logs_level=trace
select * from system.text_log
sudo less 
/var/log/clickhouse-server/clickhouse-server.log
Return messages to
clickhouse-client
View all log
messages on server
Must enable in
config.xml
(Log messages)
Limit
Expression
MergeSorting
PartialSorting
Expression
ParallelAggregating
Expression × 8
MergeTreeThread
Use system log to find out query details
SELECT toYear(FlightDate) year,
sum(Cancelled)/count(*) cancelled,
sum(DepDel15)/count(*) delayed_15
FROM airline.ontime
GROUP BY year ORDER BY year LIMIT 10
8 parallel threads
to read table
Query pipeline in log
Speed up query executing by adding threads
SELECT toYear(FlightDate) year,
sum(Cancelled)/count(*) cancelled,
sum(DepDel15)/count(*) delayed_15
FROM airline.ontime
GROUP BY year ORDER BY year LIMIT 10
SET max_threads = 2
SET max_threads = 4
. . .
max_threads defaults to half the
number of physical CPU cores
(Log messages)
Selected 355 parts by date,
355 parts by key,
21393 marks to read from 355
ranges
Speed up queries by reducing reads
SELECT toYear(FlightDate) year,
sum(Cancelled)/count(*) cancelled,
sum(DepDel15)/count(*) delayed_15
FROM airline.ontime
GROUP BY year ORDER BY year LIMIT 10
(Log messages)
Selected 12 parts by date,
12 parts by key,
692 marks to read from 12
ranges
SELECT toYear(FlightDate) year,
sum(Cancelled)/count(*) cancelled,
sum(DepDel15)/count(*) delayed_15
FROM airline.ontime
WHERE year =
toYear(toDate('2016-01-01'))
GROUP BY year ORDER BY year LIMIT 10
(Log messages)
Selected 2 parts by date,
2 parts by key,
73 marks to read from 2 ranges
Query execution tends to scale with I/O
SELECT
FlightDate,
count(*) AS total_flights,
sum(Cancelled) / count(*) AS cancelled,
sum(DepDel15) / count(*) AS delayed_15
FROM airline.ontime
WHERE (FlightDate >= toDate('2016-01-01'))
AND (FlightDate <= toDate('2016-02-10'))
GROUP BY FlightDate
(PREWHERE Log messages)
Elapsed: 0.591 sec.
Processed 173.82 million rows,
2.09 GB (294.34 million rows/s.,
3.53 GB/s.)
Use PREWHERE to help filter unindexed data
SELECT
Year, count(*) AS total_flights,
count(distinct Dest) as destinations,
count(distinct Carrier) as carriers,
sum(Cancelled) / count(*) AS cancelled,
sum(DepDel15) / count(*) AS delayed_15
FROM airline.ontime [PRE]WHERE Dest = 'BLI' GROUP BY Year
(WHERE Log messages)
Elapsed: 0.816 sec.
Processed 173.82 million rows,
5.74 GB (213.03 million rows/s.,
7.03 GB/s.)
But PREWHERE can kick in automatically
SET optimize_move_to_prewhere = 1
SELECT
Year, count(*) AS total_flights,
count(distinct Dest) as destinations,
count(distinct Carrier) as carriers,
sum(Cancelled) / count(*) AS cancelled,
sum(DepDel15) / count(*) AS delayed_15
FROM airline.ontime
WHERE Dest = 'BLI' GROUP BY Year (Log messages)
InterpreterSelectQuery:
MergeTreeWhereOptimizer: condition
"Dest = 'BLI'" moved to PREWHERE
This is the default value
Restructure joins to reduce data scanning
SELECT
Dest d, Name n, count(*) c, avg(ArrDelayMinutes)
FROM ontime
JOIN airports ON (airports.IATA = ontime.Dest)
GROUP BY d, n HAVING c > 100000 ORDER BY d DESC
LIMIT 10
SELECT dest, Name n, c AS flights, ad FROM (
SELECT Dest dest, count(*) c, avg(ArrDelayMinutes) ad
FROM ontime
GROUP BY dest HAVING c > 100000
ORDER BY ad DESC
) LEFT JOIN airports ON airports.IATA = dest LIMIT 10
Faster
3.878
seconds
1.177
seconds
(Log messages)
ParallelAggregatingBlockInputStream
: Total aggregated. 173790727 rows
(from 10199.035 MiB) in 3.844 sec.
(45214666.568 rows/sec., 2653.455
MiB/sec.)
The log tells the story
(Log messages)
ParallelAggregatingBlockInputStream
: Total aggregated. 173815409 rows
(from 2652.213 MiB) in 1.142 sec.
(152149486.717 rows/sec., 2321.617
MiB/sec.)
Join during
MergeTree scan
Join after
MergeTree scan
More ways to find out about queries
SET log_queries = 1
Run a query
SELECT version()
SET log_queries = 0
SELECT * FROM system.query_log
WHERE query='SELECT version()'
SHOW PROCESSLIST
Start query logging
Stop query logging
Show currently
executing queries
Optimizing Data
Layout
Restructure data for big performance gains
● Ensure optimal number of parts
● Optimize primary key index and ordering to reduce data size and index
selectivity
● Use skip indexes to avoid unnecessary I/O
● Use encodings to reduce data size before compression
● Use materialized views to transform data outside of source table
● Plus many other tricks
CREATE TABLE ontime ...
ENGINE=MergeTree()
PARTITION BY
toYYYYMM(FlightDate)
CREATE TABLE ontime_many_parts
...
ENGINE=MergeTree()
PARTITION BY FlightDate
How do partition keys affect performance?
Is there a
practical
difference?
Keep parts in the hundreds, not thousands
Table Rows Parts
ontime 174M 355
ontime_many_parts (after
OPTIMIZE)
174M 10,085
ontime_many_parts (before
OPTIMIZE)
174M 14,635
CREATE TABLE ontime ...
ENGINE=MergeTree()
PARTITION BY
toYYYYMM(FlightDate)
CREATE TABLE ontime_many_parts
...
ENGINE=MergeTree()
PARTITION BY FlightDate
Think about primary key index structure
CREATE TABLE ontime_reordered (
Year UInt16,
Quarter` UInt8,
. . .)
ENGINE = MergeTree()
PARTITION BY toYYYYMM(FlightDate)
ORDER BY (Carrier, Origin, FlightDate)
SETTINGS index_granularity=8192
Hashing large values
allows index to fit in
memory more easily
Large granularity
makes index smaller
Small granularity can make
skip indexes more selective
Table ORDER BY is key to performance
CREATE TABLE ontime_reordered (
Year UInt16,
Quarter` UInt8,
. . .)
ENGINE = MergeTree()
PARTITION BY toYYYYMM(FlightDate)
ORDER BY (Carrier, Origin, FlightDate)
SETTINGS index_granularity=8192
Choose order to make
dependent non-key
values less random
Benefits:
➔ Higher compression
➔ Better index selectivity
➔ Better PREWHERE
performance
SET allow_experimental_data_skipping_indices=1;
ALTER TABLE ontime ADD INDEX
dest_name Dest TYPE ngrambf_v1(3, 512, 2, 0) GRANULARITY 1
ALTER TABLE ontime ADD INDEX
cname Carrier TYPE set(100) GRANULARITY 1
OPTIMIZE TABLE ontime FINAL
-- In future releases:
ALTER TABLE ontime
UPDATE Dest=Dest, Carrier=Carrier
WHERE 1=1
Skip indexes cut down on I/O
Indexes & PREWHERE remove granules
(Log messages)
InterpreterSelectQuery: MergeTreeWhereOptimizer:
condition "Dest = 'PPG'" moved to PREWHERE
. . .
(SelectExecutor): Index `dest_name` has dropped 55
granules.
(SelectExecutor): Index `dest_name` has dropped 52
granules.
Apply PREWHERE
on Dest predicate
Use index to remove
granules from scan
Effectiveness depends on data distribution
SELECT
Year, count(*) AS flights,
sum(Cancelled) / flights AS cancelled,
sum(DepDel15) / flights AS delayed_15
FROM airline.ontime WHERE [Column] = [Value] GROUP BY Year
Column Value Index Count Rows Processed Query Response
Dest PPG ngrambf_v1 525 4.30M 0.053
Dest ATL ngrambf_v1 9,360,581 166.81M 0.622
Carrier ML set 70,622 3.39M 0.090
Carrier WN set 25,918,402 166.24M 0.566
Current index types
Name What it tracks
minmax High and low range of data; good for numbers with strong
locality like timestamps
set Unique values
ngrambf_v1 Presence of character ngrams, works with =, LIKE, search
predicates; good for long strings
tokenbf_v1 Like ngram but for whitespace-separated strings; good for
searching on tags
bloomfilter Presence of value in column
Encodings reduce data size
CREATE TABLE test_codecs ( a String,
a_lc LowCardinality(String) DEFAULT a,
b UInt32,
b_delta UInt32 DEFAULT b Codec(Delta),
b_delta_lz4 UInt32 DEFAULT b Codec(Delta, LZ4),
b_dd UInt32 DEFAULT b Codec(DoubleDelta),
b_dd_lz4 UInt32 DEFAULT b Codec(DoubleDelta, LZ4)
)
Engine = MergeTree
PARTITION BY tuple() ORDER BY tuple();
Differences
between
values
Differences
between change
of value
Values with
dictionary
encoding
Relative sizes of column data
But wait, there’s more! Encodings are faster
SELECT a AS a, count(*) AS c FROM test_codecs
GROUP BY a ORDER BY c ASC LIMIT 10
. . .
10 rows in set. Elapsed: 0.681 sec. Processed 100.00 million
rows, 2.69 GB (146.81 million rows/s., 3.95 GB/s.)
SELECT a_lc AS a, count(*) AS c FROM test_codecs
GROUP BY a ORDER BY c ASC LIMIT 10
. . .
10 rows in set. Elapsed: 0.148 sec. Processed 100.00 million
rows, 241.16 MB (675.55 million rows/s., 1.63 GB/s.)
Faster
Overview of encodings
Name Best for
LowCardinality Strings with fewer than 10K values
Delta Time series
Double Delta Increasing counters
Gorilla Gauge data (bounces around mean)
T64 Integers other than random hashes
Compression may vary across ZSTD and LZ4
Use mat views to boost performance further
CREATE MATERIALIZED VIEW ontime_daily_cancelled_mv
ENGINE = SummingMergeTree
PARTITION BY tuple() ORDER BY (FlightDate, Carrier)
POPULATE
AS SELECT
FlightDate, Carrier, count(*) AS flights,
sum(Cancelled) / count(*) AS cancelled,
sum(DepDel15) / count(*) AS delayed_15
FROM ontime
GROUP BY FlightDate, Carrier
Returns cancelled/late
flights where Carrier =
‘WN’ in 0.007 seconds
More things to think about
Use smaller datatypes wherever possible
Use ZSTD compression (slower but better ratio)
Use dictionaries instead of joins
Use sampling when approximate answers are acceptable
Shard/replicate data across a cluster for large datasets
We’ll cover performance of distributed
clusters in a future webinar
Metrics and
Monitoring
Use system.parts to track partition content
SELECT
database, table,
count(*) AS parts,
uniq(partition) AS partitions,
sum(marks) AS marks,
sum(rows) AS rows,
formatReadableSize(sum(data_compressed_bytes)) AS compressed,
formatReadableSize(sum(data_uncompressed_bytes)) AS uncompressed,
round(sum(data_compressed_bytes) / sum(data_uncompressed_bytes) * 100.0, 2)
AS percentage
FROM system.parts
WHERE active and database = currentDatabase()
GROUP BY database, table
ORDER BY database ASC, table ASC
Use system.columns to check data size
SELECT table,
formatReadableSize(sum(data_compressed_bytes)) tc,
formatReadableSize(sum(data_uncompressed_bytes)) tu,
sum(data_compressed_bytes) / sum(data_uncompressed_bytes) as ratio
FROM system.columns
WHERE database = currentDatabase()
GROUP BY table ORDER BY table
ClickHouse has great operational metrics
Some of our favorite tables in the system database…
query_log
(Query history)
part_log
(Part merges)
text_log
(System log
messages)
asynchronous
_events
(Background
metrics)
events
(Cumulative event
counters)
metrics
(Current counters)
It’s easy to dump values and graph them
Open files during
OPTIMIZE TABLE
command on
174M row table
(system.metrics
OpenFileForRead
counter)
And don’t forget all the great OS utilities!
● top and htop -- CPU and memory
● dstat -- I/O and network consumption
● iostat -- I/O by device
● iotop -- I/O by process
● iftop -- Network consumption by host
● perf top -- CPU utilization by system function
For a full description see Performance Analysis of ClickHouse Queries by Alexey
Milovidov
Wrap-up
Takeaways on ClickHouse Performance
● ClickHouse performance drivers are CPU and I/O
● The system query log is key to understanding performance
● Query optimization can improve response substantially
● Restructure tables and add indexes/mat views for biggest
gains
● Use the system database tables and OS tools for deeper
analysis of performance
Further resources
● Altinity Blog
● ClickHouse and the Magic of Materialized Views (Webinar)
● Performance Analysis of ClickHouse Queries by Alexey Milovidov
● ClickHouse Telegram Channel
Thank you!
Special Offer:
Contact us for a
1-hour consultation!
Contacts:
info@altinity.com
Visit us at:
https://www.altinity.com
Free Consultation:
https://blog.altinity.com/offer

More Related Content

What's hot

ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEOClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
Altinity Ltd
 
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEO
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEOClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEO
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEO
Altinity Ltd
 
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdf
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdfDeep Dive on ClickHouse Sharding and Replication-2202-09-22.pdf
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdf
Altinity Ltd
 
ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...
ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...
ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...
Altinity Ltd
 
ClickHouse Deep Dive, by Aleksei Milovidov
ClickHouse Deep Dive, by Aleksei MilovidovClickHouse Deep Dive, by Aleksei Milovidov
ClickHouse Deep Dive, by Aleksei Milovidov
Altinity Ltd
 
Altinity Quickstart for ClickHouse
Altinity Quickstart for ClickHouseAltinity Quickstart for ClickHouse
Altinity Quickstart for ClickHouse
Altinity Ltd
 
All about Zookeeper and ClickHouse Keeper.pdf
All about Zookeeper and ClickHouse Keeper.pdfAll about Zookeeper and ClickHouse Keeper.pdf
All about Zookeeper and ClickHouse Keeper.pdf
Altinity Ltd
 
10 Good Reasons to Use ClickHouse
10 Good Reasons to Use ClickHouse10 Good Reasons to Use ClickHouse
10 Good Reasons to Use ClickHouse
rpolat
 
All About JSON and ClickHouse - Tips, Tricks and New Features-2022-07-26-FINA...
All About JSON and ClickHouse - Tips, Tricks and New Features-2022-07-26-FINA...All About JSON and ClickHouse - Tips, Tricks and New Features-2022-07-26-FINA...
All About JSON and ClickHouse - Tips, Tricks and New Features-2022-07-26-FINA...
Altinity Ltd
 
Webinar: Strength in Numbers: Introduction to ClickHouse Cluster Performance
Webinar: Strength in Numbers: Introduction to ClickHouse Cluster PerformanceWebinar: Strength in Numbers: Introduction to ClickHouse Cluster Performance
Webinar: Strength in Numbers: Introduction to ClickHouse Cluster Performance
Altinity Ltd
 
Clickhouse Capacity Planning for OLAP Workloads, Mik Kocikowski of CloudFlare
Clickhouse Capacity Planning for OLAP Workloads, Mik Kocikowski of CloudFlareClickhouse Capacity Planning for OLAP Workloads, Mik Kocikowski of CloudFlare
Clickhouse Capacity Planning for OLAP Workloads, Mik Kocikowski of CloudFlare
Altinity Ltd
 
High Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouseHigh Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouse
Altinity Ltd
 
Better than you think: Handling JSON data in ClickHouse
Better than you think: Handling JSON data in ClickHouseBetter than you think: Handling JSON data in ClickHouse
Better than you think: Handling JSON data in ClickHouse
Altinity Ltd
 
A Practical Introduction to Handling Log Data in ClickHouse, by Robert Hodges...
A Practical Introduction to Handling Log Data in ClickHouse, by Robert Hodges...A Practical Introduction to Handling Log Data in ClickHouse, by Robert Hodges...
A Practical Introduction to Handling Log Data in ClickHouse, by Robert Hodges...
Altinity Ltd
 
A day in the life of a click house query
A day in the life of a click house queryA day in the life of a click house query
A day in the life of a click house query
CristinaMunteanu43
 
ClickHouse Monitoring 101: What to monitor and how
ClickHouse Monitoring 101: What to monitor and howClickHouse Monitoring 101: What to monitor and how
ClickHouse Monitoring 101: What to monitor and how
Altinity Ltd
 
Analytics at Speed: Introduction to ClickHouse and Common Use Cases. By Mikha...
Analytics at Speed: Introduction to ClickHouse and Common Use Cases. By Mikha...Analytics at Speed: Introduction to ClickHouse and Common Use Cases. By Mikha...
Analytics at Speed: Introduction to ClickHouse and Common Use Cases. By Mikha...
Altinity Ltd
 
ClickHouse Features for Advanced Users, by Aleksei Milovidov
ClickHouse Features for Advanced Users, by Aleksei MilovidovClickHouse Features for Advanced Users, by Aleksei Milovidov
ClickHouse Features for Advanced Users, by Aleksei Milovidov
Altinity Ltd
 
Data Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UI
Data Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UIData Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UI
Data Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UI
Altinity Ltd
 
Migration to ClickHouse. Practical guide, by Alexander Zaitsev
Migration to ClickHouse. Practical guide, by Alexander ZaitsevMigration to ClickHouse. Practical guide, by Alexander Zaitsev
Migration to ClickHouse. Practical guide, by Alexander Zaitsev
Altinity Ltd
 

What's hot (20)

ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEOClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
 
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEO
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEOClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEO
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEO
 
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdf
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdfDeep Dive on ClickHouse Sharding and Replication-2202-09-22.pdf
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdf
 
ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...
ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...
ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...
 
ClickHouse Deep Dive, by Aleksei Milovidov
ClickHouse Deep Dive, by Aleksei MilovidovClickHouse Deep Dive, by Aleksei Milovidov
ClickHouse Deep Dive, by Aleksei Milovidov
 
Altinity Quickstart for ClickHouse
Altinity Quickstart for ClickHouseAltinity Quickstart for ClickHouse
Altinity Quickstart for ClickHouse
 
All about Zookeeper and ClickHouse Keeper.pdf
All about Zookeeper and ClickHouse Keeper.pdfAll about Zookeeper and ClickHouse Keeper.pdf
All about Zookeeper and ClickHouse Keeper.pdf
 
10 Good Reasons to Use ClickHouse
10 Good Reasons to Use ClickHouse10 Good Reasons to Use ClickHouse
10 Good Reasons to Use ClickHouse
 
All About JSON and ClickHouse - Tips, Tricks and New Features-2022-07-26-FINA...
All About JSON and ClickHouse - Tips, Tricks and New Features-2022-07-26-FINA...All About JSON and ClickHouse - Tips, Tricks and New Features-2022-07-26-FINA...
All About JSON and ClickHouse - Tips, Tricks and New Features-2022-07-26-FINA...
 
Webinar: Strength in Numbers: Introduction to ClickHouse Cluster Performance
Webinar: Strength in Numbers: Introduction to ClickHouse Cluster PerformanceWebinar: Strength in Numbers: Introduction to ClickHouse Cluster Performance
Webinar: Strength in Numbers: Introduction to ClickHouse Cluster Performance
 
Clickhouse Capacity Planning for OLAP Workloads, Mik Kocikowski of CloudFlare
Clickhouse Capacity Planning for OLAP Workloads, Mik Kocikowski of CloudFlareClickhouse Capacity Planning for OLAP Workloads, Mik Kocikowski of CloudFlare
Clickhouse Capacity Planning for OLAP Workloads, Mik Kocikowski of CloudFlare
 
High Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouseHigh Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouse
 
Better than you think: Handling JSON data in ClickHouse
Better than you think: Handling JSON data in ClickHouseBetter than you think: Handling JSON data in ClickHouse
Better than you think: Handling JSON data in ClickHouse
 
A Practical Introduction to Handling Log Data in ClickHouse, by Robert Hodges...
A Practical Introduction to Handling Log Data in ClickHouse, by Robert Hodges...A Practical Introduction to Handling Log Data in ClickHouse, by Robert Hodges...
A Practical Introduction to Handling Log Data in ClickHouse, by Robert Hodges...
 
A day in the life of a click house query
A day in the life of a click house queryA day in the life of a click house query
A day in the life of a click house query
 
ClickHouse Monitoring 101: What to monitor and how
ClickHouse Monitoring 101: What to monitor and howClickHouse Monitoring 101: What to monitor and how
ClickHouse Monitoring 101: What to monitor and how
 
Analytics at Speed: Introduction to ClickHouse and Common Use Cases. By Mikha...
Analytics at Speed: Introduction to ClickHouse and Common Use Cases. By Mikha...Analytics at Speed: Introduction to ClickHouse and Common Use Cases. By Mikha...
Analytics at Speed: Introduction to ClickHouse and Common Use Cases. By Mikha...
 
ClickHouse Features for Advanced Users, by Aleksei Milovidov
ClickHouse Features for Advanced Users, by Aleksei MilovidovClickHouse Features for Advanced Users, by Aleksei Milovidov
ClickHouse Features for Advanced Users, by Aleksei Milovidov
 
Data Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UI
Data Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UIData Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UI
Data Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UI
 
Migration to ClickHouse. Practical guide, by Alexander Zaitsev
Migration to ClickHouse. Practical guide, by Alexander ZaitsevMigration to ClickHouse. Practical guide, by Alexander Zaitsev
Migration to ClickHouse. Practical guide, by Alexander Zaitsev
 

Similar to Webinar: Secrets of ClickHouse Query Performance, by Robert Hodges

ClickHouse materialized views - a secret weapon for high performance analytic...
ClickHouse materialized views - a secret weapon for high performance analytic...ClickHouse materialized views - a secret weapon for high performance analytic...
ClickHouse materialized views - a secret weapon for high performance analytic...
Altinity Ltd
 
Flink Forward Berlin 2018: Dawid Wysakowicz - "Detecting Patterns in Event St...
Flink Forward Berlin 2018: Dawid Wysakowicz - "Detecting Patterns in Event St...Flink Forward Berlin 2018: Dawid Wysakowicz - "Detecting Patterns in Event St...
Flink Forward Berlin 2018: Dawid Wysakowicz - "Detecting Patterns in Event St...
Flink Forward
 
Data warehouse or conventional database: Which is right for you?
Data warehouse or conventional database: Which is right for you?Data warehouse or conventional database: Which is right for you?
Data warehouse or conventional database: Which is right for you?
Data Con LA
 
Simplifying SQL with CTE's and windowing functions
Simplifying SQL with CTE's and windowing functionsSimplifying SQL with CTE's and windowing functions
Simplifying SQL with CTE's and windowing functions
Clayton Groom
 
Oracle tips and tricks
Oracle tips and tricksOracle tips and tricks
Oracle tips and tricks
Yanli Liu
 
PostgreSQL 8.4 TriLUG 2009-11-12
PostgreSQL 8.4 TriLUG 2009-11-12PostgreSQL 8.4 TriLUG 2009-11-12
PostgreSQL 8.4 TriLUG 2009-11-12
Andrew Dunstan
 
Database Foundation Training
Database Foundation TrainingDatabase Foundation Training
Database Foundation Training
Franky Lao
 
Less08 Schema
Less08 SchemaLess08 Schema
Less08 Schema
vivaankumar
 
PHP tips by a MYSQL DBA
PHP tips by a MYSQL DBAPHP tips by a MYSQL DBA
PHP tips by a MYSQL DBA
Amit Kumar Singh
 
Tactical data engineering
Tactical data engineeringTactical data engineering
Tactical data engineering
Julian Hyde
 
Fun with click house window functions webinar slides 2021-08-19
Fun with click house window functions webinar slides  2021-08-19Fun with click house window functions webinar slides  2021-08-19
Fun with click house window functions webinar slides 2021-08-19
Altinity Ltd
 
Fun with ClickHouse Window Functions-2021-08-19.pdf
Fun with ClickHouse Window Functions-2021-08-19.pdfFun with ClickHouse Window Functions-2021-08-19.pdf
Fun with ClickHouse Window Functions-2021-08-19.pdf
Altinity Ltd
 
Performance Enhancements In Postgre Sql 8.4
Performance Enhancements In Postgre Sql 8.4Performance Enhancements In Postgre Sql 8.4
Performance Enhancements In Postgre Sql 8.4
HighLoad2009
 
User Defined Partitioning on PlazmaDB
User Defined Partitioning on PlazmaDBUser Defined Partitioning on PlazmaDB
User Defined Partitioning on PlazmaDB
Kai Sasaki
 
How to tune a query - ODTUG 2012
How to tune a query - ODTUG 2012How to tune a query - ODTUG 2012
How to tune a query - ODTUG 2012
Connor McDonald
 
CS 542 -- Query Execution
CS 542 -- Query ExecutionCS 542 -- Query Execution
CS 542 -- Query Execution
J Singh
 
nter-pod Revolutions: Connected Enterprise Solution in Oracle EPM Cloud
nter-pod Revolutions: Connected Enterprise Solution in Oracle EPM Cloud nter-pod Revolutions: Connected Enterprise Solution in Oracle EPM Cloud
nter-pod Revolutions: Connected Enterprise Solution in Oracle EPM Cloud
Alithya
 
Monitoring InfluxEnterprise
Monitoring InfluxEnterpriseMonitoring InfluxEnterprise
Monitoring InfluxEnterprise
InfluxData
 
Eff Plsql
Eff PlsqlEff Plsql
Eff Plsql
afa reg
 
Pumps, Compressors and Turbine Fault Frequency Analysis
Pumps, Compressors and Turbine Fault Frequency AnalysisPumps, Compressors and Turbine Fault Frequency Analysis
Pumps, Compressors and Turbine Fault Frequency Analysis
University of Illinois,Chicago
 

Similar to Webinar: Secrets of ClickHouse Query Performance, by Robert Hodges (20)

ClickHouse materialized views - a secret weapon for high performance analytic...
ClickHouse materialized views - a secret weapon for high performance analytic...ClickHouse materialized views - a secret weapon for high performance analytic...
ClickHouse materialized views - a secret weapon for high performance analytic...
 
Flink Forward Berlin 2018: Dawid Wysakowicz - "Detecting Patterns in Event St...
Flink Forward Berlin 2018: Dawid Wysakowicz - "Detecting Patterns in Event St...Flink Forward Berlin 2018: Dawid Wysakowicz - "Detecting Patterns in Event St...
Flink Forward Berlin 2018: Dawid Wysakowicz - "Detecting Patterns in Event St...
 
Data warehouse or conventional database: Which is right for you?
Data warehouse or conventional database: Which is right for you?Data warehouse or conventional database: Which is right for you?
Data warehouse or conventional database: Which is right for you?
 
Simplifying SQL with CTE's and windowing functions
Simplifying SQL with CTE's and windowing functionsSimplifying SQL with CTE's and windowing functions
Simplifying SQL with CTE's and windowing functions
 
Oracle tips and tricks
Oracle tips and tricksOracle tips and tricks
Oracle tips and tricks
 
PostgreSQL 8.4 TriLUG 2009-11-12
PostgreSQL 8.4 TriLUG 2009-11-12PostgreSQL 8.4 TriLUG 2009-11-12
PostgreSQL 8.4 TriLUG 2009-11-12
 
Database Foundation Training
Database Foundation TrainingDatabase Foundation Training
Database Foundation Training
 
Less08 Schema
Less08 SchemaLess08 Schema
Less08 Schema
 
PHP tips by a MYSQL DBA
PHP tips by a MYSQL DBAPHP tips by a MYSQL DBA
PHP tips by a MYSQL DBA
 
Tactical data engineering
Tactical data engineeringTactical data engineering
Tactical data engineering
 
Fun with click house window functions webinar slides 2021-08-19
Fun with click house window functions webinar slides  2021-08-19Fun with click house window functions webinar slides  2021-08-19
Fun with click house window functions webinar slides 2021-08-19
 
Fun with ClickHouse Window Functions-2021-08-19.pdf
Fun with ClickHouse Window Functions-2021-08-19.pdfFun with ClickHouse Window Functions-2021-08-19.pdf
Fun with ClickHouse Window Functions-2021-08-19.pdf
 
Performance Enhancements In Postgre Sql 8.4
Performance Enhancements In Postgre Sql 8.4Performance Enhancements In Postgre Sql 8.4
Performance Enhancements In Postgre Sql 8.4
 
User Defined Partitioning on PlazmaDB
User Defined Partitioning on PlazmaDBUser Defined Partitioning on PlazmaDB
User Defined Partitioning on PlazmaDB
 
How to tune a query - ODTUG 2012
How to tune a query - ODTUG 2012How to tune a query - ODTUG 2012
How to tune a query - ODTUG 2012
 
CS 542 -- Query Execution
CS 542 -- Query ExecutionCS 542 -- Query Execution
CS 542 -- Query Execution
 
nter-pod Revolutions: Connected Enterprise Solution in Oracle EPM Cloud
nter-pod Revolutions: Connected Enterprise Solution in Oracle EPM Cloud nter-pod Revolutions: Connected Enterprise Solution in Oracle EPM Cloud
nter-pod Revolutions: Connected Enterprise Solution in Oracle EPM Cloud
 
Monitoring InfluxEnterprise
Monitoring InfluxEnterpriseMonitoring InfluxEnterprise
Monitoring InfluxEnterprise
 
Eff Plsql
Eff PlsqlEff Plsql
Eff Plsql
 
Pumps, Compressors and Turbine Fault Frequency Analysis
Pumps, Compressors and Turbine Fault Frequency AnalysisPumps, Compressors and Turbine Fault Frequency Analysis
Pumps, Compressors and Turbine Fault Frequency Analysis
 

More from Altinity Ltd

Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptxBuilding an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
Altinity Ltd
 
Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...
Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...
Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...
Altinity Ltd
 
Building an Analytic Extension to MySQL with ClickHouse and Open Source
Building an Analytic Extension to MySQL with ClickHouse and Open SourceBuilding an Analytic Extension to MySQL with ClickHouse and Open Source
Building an Analytic Extension to MySQL with ClickHouse and Open Source
Altinity Ltd
 
Cloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdf
Cloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdfCloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdf
Cloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdf
Altinity Ltd
 
Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...
Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...
Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...
Altinity Ltd
 
Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...
Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...
Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...
Altinity Ltd
 
Own your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdf
Own your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdfOwn your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdf
Own your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdf
Altinity Ltd
 
ClickHouse ReplacingMergeTree in Telecom Apps
ClickHouse ReplacingMergeTree in Telecom AppsClickHouse ReplacingMergeTree in Telecom Apps
ClickHouse ReplacingMergeTree in Telecom Apps
Altinity Ltd
 
Adventures with the ClickHouse ReplacingMergeTree Engine
Adventures with the ClickHouse ReplacingMergeTree EngineAdventures with the ClickHouse ReplacingMergeTree Engine
Adventures with the ClickHouse ReplacingMergeTree Engine
Altinity Ltd
 
Building a Real-Time Analytics Application with Apache Pulsar and Apache Pinot
Building a Real-Time Analytics Application with  Apache Pulsar and Apache PinotBuilding a Real-Time Analytics Application with  Apache Pulsar and Apache Pinot
Building a Real-Time Analytics Application with Apache Pulsar and Apache Pinot
Altinity Ltd
 
Altinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdf
Altinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdfAltinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdf
Altinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdf
Altinity Ltd
 
OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...
OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...
OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...
Altinity Ltd
 
OSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdf
OSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdfOSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdf
OSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdf
Altinity Ltd
 
OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...
OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...
OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...
Altinity Ltd
 
OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...
OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...
OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...
Altinity Ltd
 
OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...
OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...
OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...
Altinity Ltd
 
OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...
OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...
OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...
Altinity Ltd
 
OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...
OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...
OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...
Altinity Ltd
 
OSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdf
OSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdfOSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdf
OSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdf
Altinity Ltd
 
OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...
OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...
OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...
Altinity Ltd
 

More from Altinity Ltd (20)

Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptxBuilding an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
 
Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...
Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...
Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...
 
Building an Analytic Extension to MySQL with ClickHouse and Open Source
Building an Analytic Extension to MySQL with ClickHouse and Open SourceBuilding an Analytic Extension to MySQL with ClickHouse and Open Source
Building an Analytic Extension to MySQL with ClickHouse and Open Source
 
Cloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdf
Cloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdfCloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdf
Cloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdf
 
Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...
Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...
Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...
 
Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...
Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...
Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...
 
Own your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdf
Own your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdfOwn your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdf
Own your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdf
 
ClickHouse ReplacingMergeTree in Telecom Apps
ClickHouse ReplacingMergeTree in Telecom AppsClickHouse ReplacingMergeTree in Telecom Apps
ClickHouse ReplacingMergeTree in Telecom Apps
 
Adventures with the ClickHouse ReplacingMergeTree Engine
Adventures with the ClickHouse ReplacingMergeTree EngineAdventures with the ClickHouse ReplacingMergeTree Engine
Adventures with the ClickHouse ReplacingMergeTree Engine
 
Building a Real-Time Analytics Application with Apache Pulsar and Apache Pinot
Building a Real-Time Analytics Application with  Apache Pulsar and Apache PinotBuilding a Real-Time Analytics Application with  Apache Pulsar and Apache Pinot
Building a Real-Time Analytics Application with Apache Pulsar and Apache Pinot
 
Altinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdf
Altinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdfAltinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdf
Altinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdf
 
OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...
OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...
OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...
 
OSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdf
OSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdfOSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdf
OSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdf
 
OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...
OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...
OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...
 
OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...
OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...
OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...
 
OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...
OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...
OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...
 
OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...
OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...
OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...
 
OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...
OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...
OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...
 
OSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdf
OSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdfOSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdf
OSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdf
 
OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...
OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...
OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...
 

Recently uploaded

Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
shanihomely
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
bhumivarma35300
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
Priyanka Aash
 
Integrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecaseIntegrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecase
shyamraj55
 
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Nicolás Lopéz
 
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Torry Harris
 
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
bellared2
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
Google Developer Group - Harare
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
Safe Software
 
Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
ldtexsolbl
 
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptxDublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Kunal Gupta
 
Acumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptxAcumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptx
BrainSell Technologies
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
alexjohnson7307
 
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
aslasdfmkhan4750
 
Semantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software DevelopmentSemantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software Development
Baishakhi Ray
 
Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
Steven Carlson
 
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes..."Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
Anant Gupta
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
313mohammedarshad
 
Mastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for SuccessMastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for Success
David Wilson
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc
 

Recently uploaded (20)

Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
 
Integrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecaseIntegrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecase
 
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024
 
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
 
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
 
Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
 
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptxDublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
 
Acumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptxAcumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptx
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
 
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
 
Semantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software DevelopmentSemantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software Development
 
Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
 
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes..."Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
 
Mastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for SuccessMastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for Success
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
 

Webinar: Secrets of ClickHouse Query Performance, by Robert Hodges

  • 2. Brief Intros www.altinity.com Leading software and services provider for ClickHouse Major committer and community sponsor in US and Western Europe Robert Hodges - Altinity CEO 30+ years on DBMS plus virtualization and security. ClickHouse is DBMS #20
  • 3. Goals of the talk ● Understand single node MergeTree structure ● Optimize queries without changing data ● Get bigger performance gains by changing data layout ● Introduce tools for performance monitoring Non-Goals: ● Boost performance of sharded/replicated clusters ● Teach advanced ClickHouse performance management
  • 5. Introduction to ClickHouse Understands SQL Runs on bare metal to cloud Shared nothing architecture Stores data in columns Parallel and vectorized execution Scales to many petabytes Is Open source (Apache 2.0) a b c d a b c d a b c d a b c d And it’s really fast!
  • 6. Introducing the MergeTree table engine CREATE TABLE ontime ( Year UInt16, Quarter UInt8, Month UInt8, ... ) ENGINE = MergeTree() PARTITION BY toYYYYMM(FlightDate) ORDER BY (Carrier, FlightDate) Table engine type How to break data into parts How to index and sort data in each part
  • 7. Basic MergeTree data layout Table Part Index Columns Sparse Index Columns sorted on ORDER BY columns Rows match PARTITION BY expression Part Index Columns Part
  • 8. MergeTree layout within a single part /var/lib/clickhouse/data/airline/ontime_reordered 2017-01-01 AA 2017-01-01 EV 2018-01-01 UA 2018-01-02 AA ... primary.idx |||| .mrk .bin 20170701_20170731_355_355_2/ (FlightDate, Carrier...) ActualElapsedTime Airline AirlineID... |||| .mrk .bin |||| .mrk .bin Granule Compressed Block Mark
  • 10. ClickHouse performance tuning is different... The bad news… ● No query optimizer ● No EXPLAIN PLAN ● May need to move [a lot of] data for performance The good news… ● No query optimizer! ● System log is great ● System tables are too ● Performance drivers are simple: I/O and CPU
  • 11. Your friend: the ClickHouse query log clickhouse-client --send_logs_level=trace select * from system.text_log sudo less /var/log/clickhouse-server/clickhouse-server.log Return messages to clickhouse-client View all log messages on server Must enable in config.xml
  • 12. (Log messages) Limit Expression MergeSorting PartialSorting Expression ParallelAggregating Expression × 8 MergeTreeThread Use system log to find out query details SELECT toYear(FlightDate) year, sum(Cancelled)/count(*) cancelled, sum(DepDel15)/count(*) delayed_15 FROM airline.ontime GROUP BY year ORDER BY year LIMIT 10 8 parallel threads to read table Query pipeline in log
  • 13. Speed up query executing by adding threads SELECT toYear(FlightDate) year, sum(Cancelled)/count(*) cancelled, sum(DepDel15)/count(*) delayed_15 FROM airline.ontime GROUP BY year ORDER BY year LIMIT 10 SET max_threads = 2 SET max_threads = 4 . . . max_threads defaults to half the number of physical CPU cores
  • 14. (Log messages) Selected 355 parts by date, 355 parts by key, 21393 marks to read from 355 ranges Speed up queries by reducing reads SELECT toYear(FlightDate) year, sum(Cancelled)/count(*) cancelled, sum(DepDel15)/count(*) delayed_15 FROM airline.ontime GROUP BY year ORDER BY year LIMIT 10 (Log messages) Selected 12 parts by date, 12 parts by key, 692 marks to read from 12 ranges SELECT toYear(FlightDate) year, sum(Cancelled)/count(*) cancelled, sum(DepDel15)/count(*) delayed_15 FROM airline.ontime WHERE year = toYear(toDate('2016-01-01')) GROUP BY year ORDER BY year LIMIT 10
  • 15. (Log messages) Selected 2 parts by date, 2 parts by key, 73 marks to read from 2 ranges Query execution tends to scale with I/O SELECT FlightDate, count(*) AS total_flights, sum(Cancelled) / count(*) AS cancelled, sum(DepDel15) / count(*) AS delayed_15 FROM airline.ontime WHERE (FlightDate >= toDate('2016-01-01')) AND (FlightDate <= toDate('2016-02-10')) GROUP BY FlightDate
  • 16. (PREWHERE Log messages) Elapsed: 0.591 sec. Processed 173.82 million rows, 2.09 GB (294.34 million rows/s., 3.53 GB/s.) Use PREWHERE to help filter unindexed data SELECT Year, count(*) AS total_flights, count(distinct Dest) as destinations, count(distinct Carrier) as carriers, sum(Cancelled) / count(*) AS cancelled, sum(DepDel15) / count(*) AS delayed_15 FROM airline.ontime [PRE]WHERE Dest = 'BLI' GROUP BY Year (WHERE Log messages) Elapsed: 0.816 sec. Processed 173.82 million rows, 5.74 GB (213.03 million rows/s., 7.03 GB/s.)
  • 17. But PREWHERE can kick in automatically SET optimize_move_to_prewhere = 1 SELECT Year, count(*) AS total_flights, count(distinct Dest) as destinations, count(distinct Carrier) as carriers, sum(Cancelled) / count(*) AS cancelled, sum(DepDel15) / count(*) AS delayed_15 FROM airline.ontime WHERE Dest = 'BLI' GROUP BY Year (Log messages) InterpreterSelectQuery: MergeTreeWhereOptimizer: condition "Dest = 'BLI'" moved to PREWHERE This is the default value
  • 18. Restructure joins to reduce data scanning SELECT Dest d, Name n, count(*) c, avg(ArrDelayMinutes) FROM ontime JOIN airports ON (airports.IATA = ontime.Dest) GROUP BY d, n HAVING c > 100000 ORDER BY d DESC LIMIT 10 SELECT dest, Name n, c AS flights, ad FROM ( SELECT Dest dest, count(*) c, avg(ArrDelayMinutes) ad FROM ontime GROUP BY dest HAVING c > 100000 ORDER BY ad DESC ) LEFT JOIN airports ON airports.IATA = dest LIMIT 10 Faster 3.878 seconds 1.177 seconds
  • 19. (Log messages) ParallelAggregatingBlockInputStream : Total aggregated. 173790727 rows (from 10199.035 MiB) in 3.844 sec. (45214666.568 rows/sec., 2653.455 MiB/sec.) The log tells the story (Log messages) ParallelAggregatingBlockInputStream : Total aggregated. 173815409 rows (from 2652.213 MiB) in 1.142 sec. (152149486.717 rows/sec., 2321.617 MiB/sec.) Join during MergeTree scan Join after MergeTree scan
  • 20. More ways to find out about queries SET log_queries = 1 Run a query SELECT version() SET log_queries = 0 SELECT * FROM system.query_log WHERE query='SELECT version()' SHOW PROCESSLIST Start query logging Stop query logging Show currently executing queries
  • 22. Restructure data for big performance gains ● Ensure optimal number of parts ● Optimize primary key index and ordering to reduce data size and index selectivity ● Use skip indexes to avoid unnecessary I/O ● Use encodings to reduce data size before compression ● Use materialized views to transform data outside of source table ● Plus many other tricks
  • 23. CREATE TABLE ontime ... ENGINE=MergeTree() PARTITION BY toYYYYMM(FlightDate) CREATE TABLE ontime_many_parts ... ENGINE=MergeTree() PARTITION BY FlightDate How do partition keys affect performance? Is there a practical difference?
  • 24. Keep parts in the hundreds, not thousands Table Rows Parts ontime 174M 355 ontime_many_parts (after OPTIMIZE) 174M 10,085 ontime_many_parts (before OPTIMIZE) 174M 14,635 CREATE TABLE ontime ... ENGINE=MergeTree() PARTITION BY toYYYYMM(FlightDate) CREATE TABLE ontime_many_parts ... ENGINE=MergeTree() PARTITION BY FlightDate
  • 25. Think about primary key index structure CREATE TABLE ontime_reordered ( Year UInt16, Quarter` UInt8, . . .) ENGINE = MergeTree() PARTITION BY toYYYYMM(FlightDate) ORDER BY (Carrier, Origin, FlightDate) SETTINGS index_granularity=8192 Hashing large values allows index to fit in memory more easily Large granularity makes index smaller Small granularity can make skip indexes more selective
  • 26. Table ORDER BY is key to performance CREATE TABLE ontime_reordered ( Year UInt16, Quarter` UInt8, . . .) ENGINE = MergeTree() PARTITION BY toYYYYMM(FlightDate) ORDER BY (Carrier, Origin, FlightDate) SETTINGS index_granularity=8192 Choose order to make dependent non-key values less random Benefits: ➔ Higher compression ➔ Better index selectivity ➔ Better PREWHERE performance
  • 27. SET allow_experimental_data_skipping_indices=1; ALTER TABLE ontime ADD INDEX dest_name Dest TYPE ngrambf_v1(3, 512, 2, 0) GRANULARITY 1 ALTER TABLE ontime ADD INDEX cname Carrier TYPE set(100) GRANULARITY 1 OPTIMIZE TABLE ontime FINAL -- In future releases: ALTER TABLE ontime UPDATE Dest=Dest, Carrier=Carrier WHERE 1=1 Skip indexes cut down on I/O
  • 28. Indexes & PREWHERE remove granules (Log messages) InterpreterSelectQuery: MergeTreeWhereOptimizer: condition "Dest = 'PPG'" moved to PREWHERE . . . (SelectExecutor): Index `dest_name` has dropped 55 granules. (SelectExecutor): Index `dest_name` has dropped 52 granules. Apply PREWHERE on Dest predicate Use index to remove granules from scan
  • 29. Effectiveness depends on data distribution SELECT Year, count(*) AS flights, sum(Cancelled) / flights AS cancelled, sum(DepDel15) / flights AS delayed_15 FROM airline.ontime WHERE [Column] = [Value] GROUP BY Year Column Value Index Count Rows Processed Query Response Dest PPG ngrambf_v1 525 4.30M 0.053 Dest ATL ngrambf_v1 9,360,581 166.81M 0.622 Carrier ML set 70,622 3.39M 0.090 Carrier WN set 25,918,402 166.24M 0.566
  • 30. Current index types Name What it tracks minmax High and low range of data; good for numbers with strong locality like timestamps set Unique values ngrambf_v1 Presence of character ngrams, works with =, LIKE, search predicates; good for long strings tokenbf_v1 Like ngram but for whitespace-separated strings; good for searching on tags bloomfilter Presence of value in column
  • 31. Encodings reduce data size CREATE TABLE test_codecs ( a String, a_lc LowCardinality(String) DEFAULT a, b UInt32, b_delta UInt32 DEFAULT b Codec(Delta), b_delta_lz4 UInt32 DEFAULT b Codec(Delta, LZ4), b_dd UInt32 DEFAULT b Codec(DoubleDelta), b_dd_lz4 UInt32 DEFAULT b Codec(DoubleDelta, LZ4) ) Engine = MergeTree PARTITION BY tuple() ORDER BY tuple(); Differences between values Differences between change of value Values with dictionary encoding
  • 32. Relative sizes of column data
  • 33. But wait, there’s more! Encodings are faster SELECT a AS a, count(*) AS c FROM test_codecs GROUP BY a ORDER BY c ASC LIMIT 10 . . . 10 rows in set. Elapsed: 0.681 sec. Processed 100.00 million rows, 2.69 GB (146.81 million rows/s., 3.95 GB/s.) SELECT a_lc AS a, count(*) AS c FROM test_codecs GROUP BY a ORDER BY c ASC LIMIT 10 . . . 10 rows in set. Elapsed: 0.148 sec. Processed 100.00 million rows, 241.16 MB (675.55 million rows/s., 1.63 GB/s.) Faster
  • 34. Overview of encodings Name Best for LowCardinality Strings with fewer than 10K values Delta Time series Double Delta Increasing counters Gorilla Gauge data (bounces around mean) T64 Integers other than random hashes Compression may vary across ZSTD and LZ4
  • 35. Use mat views to boost performance further CREATE MATERIALIZED VIEW ontime_daily_cancelled_mv ENGINE = SummingMergeTree PARTITION BY tuple() ORDER BY (FlightDate, Carrier) POPULATE AS SELECT FlightDate, Carrier, count(*) AS flights, sum(Cancelled) / count(*) AS cancelled, sum(DepDel15) / count(*) AS delayed_15 FROM ontime GROUP BY FlightDate, Carrier Returns cancelled/late flights where Carrier = ‘WN’ in 0.007 seconds
  • 36. More things to think about Use smaller datatypes wherever possible Use ZSTD compression (slower but better ratio) Use dictionaries instead of joins Use sampling when approximate answers are acceptable Shard/replicate data across a cluster for large datasets We’ll cover performance of distributed clusters in a future webinar
  • 38. Use system.parts to track partition content SELECT database, table, count(*) AS parts, uniq(partition) AS partitions, sum(marks) AS marks, sum(rows) AS rows, formatReadableSize(sum(data_compressed_bytes)) AS compressed, formatReadableSize(sum(data_uncompressed_bytes)) AS uncompressed, round(sum(data_compressed_bytes) / sum(data_uncompressed_bytes) * 100.0, 2) AS percentage FROM system.parts WHERE active and database = currentDatabase() GROUP BY database, table ORDER BY database ASC, table ASC
  • 39. Use system.columns to check data size SELECT table, formatReadableSize(sum(data_compressed_bytes)) tc, formatReadableSize(sum(data_uncompressed_bytes)) tu, sum(data_compressed_bytes) / sum(data_uncompressed_bytes) as ratio FROM system.columns WHERE database = currentDatabase() GROUP BY table ORDER BY table
  • 40. ClickHouse has great operational metrics Some of our favorite tables in the system database… query_log (Query history) part_log (Part merges) text_log (System log messages) asynchronous _events (Background metrics) events (Cumulative event counters) metrics (Current counters)
  • 41. It’s easy to dump values and graph them Open files during OPTIMIZE TABLE command on 174M row table (system.metrics OpenFileForRead counter)
  • 42. And don’t forget all the great OS utilities! ● top and htop -- CPU and memory ● dstat -- I/O and network consumption ● iostat -- I/O by device ● iotop -- I/O by process ● iftop -- Network consumption by host ● perf top -- CPU utilization by system function For a full description see Performance Analysis of ClickHouse Queries by Alexey Milovidov
  • 44. Takeaways on ClickHouse Performance ● ClickHouse performance drivers are CPU and I/O ● The system query log is key to understanding performance ● Query optimization can improve response substantially ● Restructure tables and add indexes/mat views for biggest gains ● Use the system database tables and OS tools for deeper analysis of performance
  • 45. Further resources ● Altinity Blog ● ClickHouse and the Magic of Materialized Views (Webinar) ● Performance Analysis of ClickHouse Queries by Alexey Milovidov ● ClickHouse Telegram Channel
  • 46. Thank you! Special Offer: Contact us for a 1-hour consultation! Contacts: info@altinity.com Visit us at: https://www.altinity.com Free Consultation: https://blog.altinity.com/offer