SlideShare a Scribd company logo
Brief Intros
www.altinity.com
Leading software and services
provider for ClickHouse
Major committer and community
sponsor in US and Western Europe
Robert Hodges - Altinity CEO
30+ years on DBMS plus
virtualization and security.
ClickHouse is DBMS #20
Goals of the talk
● Understand single node MergeTree structure
● Optimize queries without changing data
● Get bigger performance gains by changing data layout
● Introduce tools for performance monitoring
Non-Goals:
● Boost performance of sharded/replicated clusters
● Teach advanced ClickHouse performance management
ClickHouse &
MergeTree Intro
Introduction to ClickHouse
Understands SQL
Runs on bare metal to cloud
Shared nothing architecture
Stores data in columns
Parallel and vectorized execution
Scales to many petabytes
Is Open source (Apache 2.0)
a b c d
a b c d
a b c d
a b c d
And it’s really fast!
Introducing the MergeTree table engine
CREATE TABLE ontime (
Year UInt16,
Quarter UInt8,
Month UInt8,
...
) ENGINE = MergeTree()
PARTITION BY toYYYYMM(FlightDate)
ORDER BY (Carrier, FlightDate)
Table engine type
How to break data
into parts
How to index and
sort data in each part
Basic MergeTree data layout
Table
Part
Index Columns
Sparse Index
Columns
sorted on
ORDER BY
columns
Rows match
PARTITION BY
expression
Part
Index Columns
Part
MergeTree layout within a single part
/var/lib/clickhouse/data/airline/ontime_reordered
2017-01-01 AA
2017-01-01 EV
2018-01-01 UA
2018-01-02 AA
...
primary.idx
||||
.mrk .bin
20170701_20170731_355_355_2/
(FlightDate, Carrier...) ActualElapsedTime Airline AirlineID...
||||
.mrk .bin
||||
.mrk .bin
Granule Compressed
Block
Mark
Basic Query
Tuning
ClickHouse performance tuning is different...
The bad news…
● No query optimizer
● No EXPLAIN PLAN
● May need to move [a lot
of] data for performance
The good news…
● No query optimizer!
● System log is great
● System tables are too
● Performance drivers are
simple: I/O and CPU
Your friend: the ClickHouse query log
clickhouse-client --send_logs_level=trace
select * from system.text_log
sudo less 
/var/log/clickhouse-server/clickhouse-server.log
Return messages to
clickhouse-client
View all log
messages on server
Must enable in
config.xml
(Log messages)
Limit
Expression
MergeSorting
PartialSorting
Expression
ParallelAggregating
Expression × 8
MergeTreeThread
Use system log to find out query details
SELECT toYear(FlightDate) year,
sum(Cancelled)/count(*) cancelled,
sum(DepDel15)/count(*) delayed_15
FROM airline.ontime
GROUP BY year ORDER BY year LIMIT 10
8 parallel threads
to read table
Query pipeline in log
Speed up query executing by adding threads
SELECT toYear(FlightDate) year,
sum(Cancelled)/count(*) cancelled,
sum(DepDel15)/count(*) delayed_15
FROM airline.ontime
GROUP BY year ORDER BY year LIMIT 10
SET max_threads = 2
SET max_threads = 4
. . .
max_threads defaults to half the
number of physical CPU cores
(Log messages)
Selected 355 parts by date,
355 parts by key,
21393 marks to read from 355
ranges
Speed up queries by reducing reads
SELECT toYear(FlightDate) year,
sum(Cancelled)/count(*) cancelled,
sum(DepDel15)/count(*) delayed_15
FROM airline.ontime
GROUP BY year ORDER BY year LIMIT 10
(Log messages)
Selected 12 parts by date,
12 parts by key,
692 marks to read from 12
ranges
SELECT toYear(FlightDate) year,
sum(Cancelled)/count(*) cancelled,
sum(DepDel15)/count(*) delayed_15
FROM airline.ontime
WHERE year =
toYear(toDate('2016-01-01'))
GROUP BY year ORDER BY year LIMIT 10
(Log messages)
Selected 2 parts by date,
2 parts by key,
73 marks to read from 2 ranges
Query execution tends to scale with I/O
SELECT
FlightDate,
count(*) AS total_flights,
sum(Cancelled) / count(*) AS cancelled,
sum(DepDel15) / count(*) AS delayed_15
FROM airline.ontime
WHERE (FlightDate >= toDate('2016-01-01'))
AND (FlightDate <= toDate('2016-02-10'))
GROUP BY FlightDate
(PREWHERE Log messages)
Elapsed: 0.591 sec.
Processed 173.82 million rows,
2.09 GB (294.34 million rows/s.,
3.53 GB/s.)
Use PREWHERE to help filter unindexed data
SELECT
Year, count(*) AS total_flights,
count(distinct Dest) as destinations,
count(distinct Carrier) as carriers,
sum(Cancelled) / count(*) AS cancelled,
sum(DepDel15) / count(*) AS delayed_15
FROM airline.ontime [PRE]WHERE Dest = 'BLI' GROUP BY Year
(WHERE Log messages)
Elapsed: 0.816 sec.
Processed 173.82 million rows,
5.74 GB (213.03 million rows/s.,
7.03 GB/s.)
But PREWHERE can kick in automatically
SET optimize_move_to_prewhere = 1
SELECT
Year, count(*) AS total_flights,
count(distinct Dest) as destinations,
count(distinct Carrier) as carriers,
sum(Cancelled) / count(*) AS cancelled,
sum(DepDel15) / count(*) AS delayed_15
FROM airline.ontime
WHERE Dest = 'BLI' GROUP BY Year (Log messages)
InterpreterSelectQuery:
MergeTreeWhereOptimizer: condition
"Dest = 'BLI'" moved to PREWHERE
This is the default value
Restructure joins to reduce data scanning
SELECT
Dest d, Name n, count(*) c, avg(ArrDelayMinutes)
FROM ontime
JOIN airports ON (airports.IATA = ontime.Dest)
GROUP BY d, n HAVING c > 100000 ORDER BY d DESC
LIMIT 10
SELECT dest, Name n, c AS flights, ad FROM (
SELECT Dest dest, count(*) c, avg(ArrDelayMinutes) ad
FROM ontime
GROUP BY dest HAVING c > 100000
ORDER BY ad DESC
) LEFT JOIN airports ON airports.IATA = dest LIMIT 10
Faster
3.878
seconds
1.177
seconds
(Log messages)
ParallelAggregatingBlockInputStream
: Total aggregated. 173790727 rows
(from 10199.035 MiB) in 3.844 sec.
(45214666.568 rows/sec., 2653.455
MiB/sec.)
The log tells the story
(Log messages)
ParallelAggregatingBlockInputStream
: Total aggregated. 173815409 rows
(from 2652.213 MiB) in 1.142 sec.
(152149486.717 rows/sec., 2321.617
MiB/sec.)
Join during
MergeTree scan
Join after
MergeTree scan
More ways to find out about queries
SET log_queries = 1
Run a query
SELECT version()
SET log_queries = 0
SELECT * FROM system.query_log
WHERE query='SELECT version()'
SHOW PROCESSLIST
Start query logging
Stop query logging
Show currently
executing queries
Optimizing Data
Layout
Restructure data for big performance gains
● Ensure optimal number of parts
● Optimize primary key index and ordering to reduce data size and index
selectivity
● Use skip indexes to avoid unnecessary I/O
● Use encodings to reduce data size before compression
● Use materialized views to transform data outside of source table
● Plus many other tricks
CREATE TABLE ontime ...
ENGINE=MergeTree()
PARTITION BY
toYYYYMM(FlightDate)
CREATE TABLE ontime_many_parts
...
ENGINE=MergeTree()
PARTITION BY FlightDate
How do partition keys affect performance?
Is there a
practical
difference?
Keep parts in the hundreds, not thousands
Table Rows Parts
ontime 174M 355
ontime_many_parts (after
OPTIMIZE)
174M 10,085
ontime_many_parts (before
OPTIMIZE)
174M 14,635
CREATE TABLE ontime ...
ENGINE=MergeTree()
PARTITION BY
toYYYYMM(FlightDate)
CREATE TABLE ontime_many_parts
...
ENGINE=MergeTree()
PARTITION BY FlightDate
Think about primary key index structure
CREATE TABLE ontime_reordered (
Year UInt16,
Quarter` UInt8,
. . .)
ENGINE = MergeTree()
PARTITION BY toYYYYMM(FlightDate)
ORDER BY (Carrier, Origin, FlightDate)
SETTINGS index_granularity=8192
Hashing large values
allows index to fit in
memory more easily
Large granularity
makes index smaller
Small granularity can make
skip indexes more selective
Table ORDER BY is key to performance
CREATE TABLE ontime_reordered (
Year UInt16,
Quarter` UInt8,
. . .)
ENGINE = MergeTree()
PARTITION BY toYYYYMM(FlightDate)
ORDER BY (Carrier, Origin, FlightDate)
SETTINGS index_granularity=8192
Choose order to make
dependent non-key
values less random
Benefits:
➔ Higher compression
➔ Better index selectivity
➔ Better PREWHERE
performance
SET allow_experimental_data_skipping_indices=1;
ALTER TABLE ontime ADD INDEX
dest_name Dest TYPE ngrambf_v1(3, 512, 2, 0) GRANULARITY 1
ALTER TABLE ontime ADD INDEX
cname Carrier TYPE set(100) GRANULARITY 1
OPTIMIZE TABLE ontime FINAL
-- In future releases:
ALTER TABLE ontime
UPDATE Dest=Dest, Carrier=Carrier
WHERE 1=1
Skip indexes cut down on I/O
Indexes & PREWHERE remove granules
(Log messages)
InterpreterSelectQuery: MergeTreeWhereOptimizer:
condition "Dest = 'PPG'" moved to PREWHERE
. . .
(SelectExecutor): Index `dest_name` has dropped 55
granules.
(SelectExecutor): Index `dest_name` has dropped 52
granules.
Apply PREWHERE
on Dest predicate
Use index to remove
granules from scan
Effectiveness depends on data distribution
SELECT
Year, count(*) AS flights,
sum(Cancelled) / flights AS cancelled,
sum(DepDel15) / flights AS delayed_15
FROM airline.ontime WHERE [Column] = [Value] GROUP BY Year
Column Value Index Count Rows Processed Query Response
Dest PPG ngrambf_v1 525 4.30M 0.053
Dest ATL ngrambf_v1 9,360,581 166.81M 0.622
Carrier ML set 70,622 3.39M 0.090
Carrier WN set 25,918,402 166.24M 0.566
Current index types
Name What it tracks
minmax High and low range of data; good for numbers with strong
locality like timestamps
set Unique values
ngrambf_v1 Presence of character ngrams, works with =, LIKE, search
predicates; good for long strings
tokenbf_v1 Like ngram but for whitespace-separated strings; good for
searching on tags
bloomfilter Presence of value in column
Encodings reduce data size
CREATE TABLE test_codecs ( a String,
a_lc LowCardinality(String) DEFAULT a,
b UInt32,
b_delta UInt32 DEFAULT b Codec(Delta),
b_delta_lz4 UInt32 DEFAULT b Codec(Delta, LZ4),
b_dd UInt32 DEFAULT b Codec(DoubleDelta),
b_dd_lz4 UInt32 DEFAULT b Codec(DoubleDelta, LZ4)
)
Engine = MergeTree
PARTITION BY tuple() ORDER BY tuple();
Differences
between
values
Differences
between change
of value
Values with
dictionary
encoding
Relative sizes of column data
But wait, there’s more! Encodings are faster
SELECT a AS a, count(*) AS c FROM test_codecs
GROUP BY a ORDER BY c ASC LIMIT 10
. . .
10 rows in set. Elapsed: 0.681 sec. Processed 100.00 million
rows, 2.69 GB (146.81 million rows/s., 3.95 GB/s.)
SELECT a_lc AS a, count(*) AS c FROM test_codecs
GROUP BY a ORDER BY c ASC LIMIT 10
. . .
10 rows in set. Elapsed: 0.148 sec. Processed 100.00 million
rows, 241.16 MB (675.55 million rows/s., 1.63 GB/s.)
Faster
Overview of encodings
Name Best for
LowCardinality Strings with fewer than 10K values
Delta Time series
Double Delta Increasing counters
Gorilla Gauge data (bounces around mean)
T64 Integers other than random hashes
Compression may vary across ZSTD and LZ4
Use mat views to boost performance further
CREATE MATERIALIZED VIEW ontime_daily_cancelled_mv
ENGINE = SummingMergeTree
PARTITION BY tuple() ORDER BY (FlightDate, Carrier)
POPULATE
AS SELECT
FlightDate, Carrier, count(*) AS flights,
sum(Cancelled) / count(*) AS cancelled,
sum(DepDel15) / count(*) AS delayed_15
FROM ontime
GROUP BY FlightDate, Carrier
Returns cancelled/late
flights where Carrier =
‘WN’ in 0.007 seconds
More things to think about
Use smaller datatypes wherever possible
Use ZSTD compression (slower but better ratio)
Use dictionaries instead of joins
Use sampling when approximate answers are acceptable
Shard/replicate data across a cluster for large datasets
We’ll cover performance of distributed
clusters in a future webinar
Metrics and
Monitoring
Use system.parts to track partition content
SELECT
database, table,
count(*) AS parts,
uniq(partition) AS partitions,
sum(marks) AS marks,
sum(rows) AS rows,
formatReadableSize(sum(data_compressed_bytes)) AS compressed,
formatReadableSize(sum(data_uncompressed_bytes)) AS uncompressed,
round(sum(data_compressed_bytes) / sum(data_uncompressed_bytes) * 100.0, 2)
AS percentage
FROM system.parts
WHERE active and database = currentDatabase()
GROUP BY database, table
ORDER BY database ASC, table ASC
Use system.columns to check data size
SELECT table,
formatReadableSize(sum(data_compressed_bytes)) tc,
formatReadableSize(sum(data_uncompressed_bytes)) tu,
sum(data_compressed_bytes) / sum(data_uncompressed_bytes) as ratio
FROM system.columns
WHERE database = currentDatabase()
GROUP BY table ORDER BY table
ClickHouse has great operational metrics
Some of our favorite tables in the system database…
query_log
(Query history)
part_log
(Part merges)
text_log
(System log
messages)
asynchronous
_events
(Background
metrics)
events
(Cumulative event
counters)
metrics
(Current counters)
It’s easy to dump values and graph them
Open files during
OPTIMIZE TABLE
command on
174M row table
(system.metrics
OpenFileForRead
counter)
And don’t forget all the great OS utilities!
● top and htop -- CPU and memory
● dstat -- I/O and network consumption
● iostat -- I/O by device
● iotop -- I/O by process
● iftop -- Network consumption by host
● perf top -- CPU utilization by system function
For a full description see Performance Analysis of ClickHouse Queries by Alexey
Milovidov
Wrap-up
Takeaways on ClickHouse Performance
● ClickHouse performance drivers are CPU and I/O
● The system query log is key to understanding performance
● Query optimization can improve response substantially
● Restructure tables and add indexes/mat views for biggest
gains
● Use the system database tables and OS tools for deeper
analysis of performance
Further resources
● Altinity Blog
● ClickHouse and the Magic of Materialized Views (Webinar)
● Performance Analysis of ClickHouse Queries by Alexey Milovidov
● ClickHouse Telegram Channel
Thank you!
Special Offer:
Contact us for a
1-hour consultation!
Contacts:
info@altinity.com
Visit us at:
https://www.altinity.com
Free Consultation:
https://blog.altinity.com/offer

More Related Content

What's hot

Clickhouse Capacity Planning for OLAP Workloads, Mik Kocikowski of CloudFlare
Clickhouse Capacity Planning for OLAP Workloads, Mik Kocikowski of CloudFlareClickhouse Capacity Planning for OLAP Workloads, Mik Kocikowski of CloudFlare
Clickhouse Capacity Planning for OLAP Workloads, Mik Kocikowski of CloudFlare
Altinity Ltd
 
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdf
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdfDeep Dive on ClickHouse Sharding and Replication-2202-09-22.pdf
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdf
Altinity Ltd
 
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEOClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
Altinity Ltd
 
Introduction to the Mysteries of ClickHouse Replication, By Robert Hodges and...
Introduction to the Mysteries of ClickHouse Replication, By Robert Hodges and...Introduction to the Mysteries of ClickHouse Replication, By Robert Hodges and...
Introduction to the Mysteries of ClickHouse Replication, By Robert Hodges and...
Altinity Ltd
 
Altinity Quickstart for ClickHouse
Altinity Quickstart for ClickHouseAltinity Quickstart for ClickHouse
Altinity Quickstart for ClickHouse
Altinity Ltd
 
Size Matters-Best Practices for Trillion Row Datasets on ClickHouse-2202-08-1...
Size Matters-Best Practices for Trillion Row Datasets on ClickHouse-2202-08-1...Size Matters-Best Practices for Trillion Row Datasets on ClickHouse-2202-08-1...
Size Matters-Best Practices for Trillion Row Datasets on ClickHouse-2202-08-1...
Altinity Ltd
 
ClickHouse Materialized Views: The Magic Continues
ClickHouse Materialized Views: The Magic ContinuesClickHouse Materialized Views: The Magic Continues
ClickHouse Materialized Views: The Magic Continues
Altinity Ltd
 
Webinar slides: MORE secrets of ClickHouse Query Performance. By Robert Hodge...
Webinar slides: MORE secrets of ClickHouse Query Performance. By Robert Hodge...Webinar slides: MORE secrets of ClickHouse Query Performance. By Robert Hodge...
Webinar slides: MORE secrets of ClickHouse Query Performance. By Robert Hodge...
Altinity Ltd
 
ClickHouse Monitoring 101: What to monitor and how
ClickHouse Monitoring 101: What to monitor and howClickHouse Monitoring 101: What to monitor and how
ClickHouse Monitoring 101: What to monitor and how
Altinity Ltd
 
ClickHouse Features for Advanced Users, by Aleksei Milovidov
ClickHouse Features for Advanced Users, by Aleksei MilovidovClickHouse Features for Advanced Users, by Aleksei Milovidov
ClickHouse Features for Advanced Users, by Aleksei Milovidov
Altinity Ltd
 
Migration to ClickHouse. Practical guide, by Alexander Zaitsev
Migration to ClickHouse. Practical guide, by Alexander ZaitsevMigration to ClickHouse. Practical guide, by Alexander Zaitsev
Migration to ClickHouse. Practical guide, by Alexander Zaitsev
Altinity Ltd
 
ClickHouse Deep Dive, by Aleksei Milovidov
ClickHouse Deep Dive, by Aleksei MilovidovClickHouse Deep Dive, by Aleksei Milovidov
ClickHouse Deep Dive, by Aleksei Milovidov
Altinity Ltd
 
Materialize: a platform for changing data
Materialize: a platform for changing dataMaterialize: a platform for changing data
Materialize: a platform for changing data
Altinity Ltd
 
ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...
ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...
ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...
Altinity Ltd
 
All about Zookeeper and ClickHouse Keeper.pdf
All about Zookeeper and ClickHouse Keeper.pdfAll about Zookeeper and ClickHouse Keeper.pdf
All about Zookeeper and ClickHouse Keeper.pdf
Altinity Ltd
 
ClickHouse Keeper
ClickHouse KeeperClickHouse Keeper
ClickHouse Keeper
Altinity Ltd
 
Query Compilation in Impala
Query Compilation in ImpalaQuery Compilation in Impala
Query Compilation in Impala
Cloudera, Inc.
 
Data Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UI
Data Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UIData Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UI
Data Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UI
Altinity Ltd
 
ClickHouse Mark Cache, by Mik Kocikowski, Cloudflare
ClickHouse Mark Cache, by Mik Kocikowski, CloudflareClickHouse Mark Cache, by Mik Kocikowski, Cloudflare
ClickHouse Mark Cache, by Mik Kocikowski, Cloudflare
Altinity Ltd
 
Creating Beautiful Dashboards with Grafana and ClickHouse
Creating Beautiful Dashboards with Grafana and ClickHouseCreating Beautiful Dashboards with Grafana and ClickHouse
Creating Beautiful Dashboards with Grafana and ClickHouse
Altinity Ltd
 

What's hot (20)

Clickhouse Capacity Planning for OLAP Workloads, Mik Kocikowski of CloudFlare
Clickhouse Capacity Planning for OLAP Workloads, Mik Kocikowski of CloudFlareClickhouse Capacity Planning for OLAP Workloads, Mik Kocikowski of CloudFlare
Clickhouse Capacity Planning for OLAP Workloads, Mik Kocikowski of CloudFlare
 
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdf
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdfDeep Dive on ClickHouse Sharding and Replication-2202-09-22.pdf
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdf
 
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEOClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
 
Introduction to the Mysteries of ClickHouse Replication, By Robert Hodges and...
Introduction to the Mysteries of ClickHouse Replication, By Robert Hodges and...Introduction to the Mysteries of ClickHouse Replication, By Robert Hodges and...
Introduction to the Mysteries of ClickHouse Replication, By Robert Hodges and...
 
Altinity Quickstart for ClickHouse
Altinity Quickstart for ClickHouseAltinity Quickstart for ClickHouse
Altinity Quickstart for ClickHouse
 
Size Matters-Best Practices for Trillion Row Datasets on ClickHouse-2202-08-1...
Size Matters-Best Practices for Trillion Row Datasets on ClickHouse-2202-08-1...Size Matters-Best Practices for Trillion Row Datasets on ClickHouse-2202-08-1...
Size Matters-Best Practices for Trillion Row Datasets on ClickHouse-2202-08-1...
 
ClickHouse Materialized Views: The Magic Continues
ClickHouse Materialized Views: The Magic ContinuesClickHouse Materialized Views: The Magic Continues
ClickHouse Materialized Views: The Magic Continues
 
Webinar slides: MORE secrets of ClickHouse Query Performance. By Robert Hodge...
Webinar slides: MORE secrets of ClickHouse Query Performance. By Robert Hodge...Webinar slides: MORE secrets of ClickHouse Query Performance. By Robert Hodge...
Webinar slides: MORE secrets of ClickHouse Query Performance. By Robert Hodge...
 
ClickHouse Monitoring 101: What to monitor and how
ClickHouse Monitoring 101: What to monitor and howClickHouse Monitoring 101: What to monitor and how
ClickHouse Monitoring 101: What to monitor and how
 
ClickHouse Features for Advanced Users, by Aleksei Milovidov
ClickHouse Features for Advanced Users, by Aleksei MilovidovClickHouse Features for Advanced Users, by Aleksei Milovidov
ClickHouse Features for Advanced Users, by Aleksei Milovidov
 
Migration to ClickHouse. Practical guide, by Alexander Zaitsev
Migration to ClickHouse. Practical guide, by Alexander ZaitsevMigration to ClickHouse. Practical guide, by Alexander Zaitsev
Migration to ClickHouse. Practical guide, by Alexander Zaitsev
 
ClickHouse Deep Dive, by Aleksei Milovidov
ClickHouse Deep Dive, by Aleksei MilovidovClickHouse Deep Dive, by Aleksei Milovidov
ClickHouse Deep Dive, by Aleksei Milovidov
 
Materialize: a platform for changing data
Materialize: a platform for changing dataMaterialize: a platform for changing data
Materialize: a platform for changing data
 
ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...
ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...
ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...
 
All about Zookeeper and ClickHouse Keeper.pdf
All about Zookeeper and ClickHouse Keeper.pdfAll about Zookeeper and ClickHouse Keeper.pdf
All about Zookeeper and ClickHouse Keeper.pdf
 
ClickHouse Keeper
ClickHouse KeeperClickHouse Keeper
ClickHouse Keeper
 
Query Compilation in Impala
Query Compilation in ImpalaQuery Compilation in Impala
Query Compilation in Impala
 
Data Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UI
Data Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UIData Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UI
Data Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UI
 
ClickHouse Mark Cache, by Mik Kocikowski, Cloudflare
ClickHouse Mark Cache, by Mik Kocikowski, CloudflareClickHouse Mark Cache, by Mik Kocikowski, Cloudflare
ClickHouse Mark Cache, by Mik Kocikowski, Cloudflare
 
Creating Beautiful Dashboards with Grafana and ClickHouse
Creating Beautiful Dashboards with Grafana and ClickHouseCreating Beautiful Dashboards with Grafana and ClickHouse
Creating Beautiful Dashboards with Grafana and ClickHouse
 

Similar to Webinar: Secrets of ClickHouse Query Performance, by Robert Hodges

ClickHouse materialized views - a secret weapon for high performance analytic...
ClickHouse materialized views - a secret weapon for high performance analytic...ClickHouse materialized views - a secret weapon for high performance analytic...
ClickHouse materialized views - a secret weapon for high performance analytic...
Altinity Ltd
 
A day in the life of a click house query
A day in the life of a click house queryA day in the life of a click house query
A day in the life of a click house query
CristinaMunteanu43
 
Flink Forward Berlin 2018: Dawid Wysakowicz - "Detecting Patterns in Event St...
Flink Forward Berlin 2018: Dawid Wysakowicz - "Detecting Patterns in Event St...Flink Forward Berlin 2018: Dawid Wysakowicz - "Detecting Patterns in Event St...
Flink Forward Berlin 2018: Dawid Wysakowicz - "Detecting Patterns in Event St...
Flink Forward
 
Data warehouse or conventional database: Which is right for you?
Data warehouse or conventional database: Which is right for you?Data warehouse or conventional database: Which is right for you?
Data warehouse or conventional database: Which is right for you?
Data Con LA
 
Webinar: Strength in Numbers: Introduction to ClickHouse Cluster Performance
Webinar: Strength in Numbers: Introduction to ClickHouse Cluster PerformanceWebinar: Strength in Numbers: Introduction to ClickHouse Cluster Performance
Webinar: Strength in Numbers: Introduction to ClickHouse Cluster Performance
Altinity Ltd
 
Simplifying SQL with CTE's and windowing functions
Simplifying SQL with CTE's and windowing functionsSimplifying SQL with CTE's and windowing functions
Simplifying SQL with CTE's and windowing functions
Clayton Groom
 
Oracle tips and tricks
Oracle tips and tricksOracle tips and tricks
Oracle tips and tricks
Yanli Liu
 
PostgreSQL 8.4 TriLUG 2009-11-12
PostgreSQL 8.4 TriLUG 2009-11-12PostgreSQL 8.4 TriLUG 2009-11-12
PostgreSQL 8.4 TriLUG 2009-11-12
Andrew Dunstan
 
Database Foundation Training
Database Foundation TrainingDatabase Foundation Training
Database Foundation Training
Franky Lao
 
Less08 Schema
Less08 SchemaLess08 Schema
Less08 Schema
vivaankumar
 
PHP tips by a MYSQL DBA
PHP tips by a MYSQL DBAPHP tips by a MYSQL DBA
PHP tips by a MYSQL DBA
Amit Kumar Singh
 
Tactical data engineering
Tactical data engineeringTactical data engineering
Tactical data engineering
Julian Hyde
 
Fun with click house window functions webinar slides 2021-08-19
Fun with click house window functions webinar slides  2021-08-19Fun with click house window functions webinar slides  2021-08-19
Fun with click house window functions webinar slides 2021-08-19
Altinity Ltd
 
Fun with ClickHouse Window Functions-2021-08-19.pdf
Fun with ClickHouse Window Functions-2021-08-19.pdfFun with ClickHouse Window Functions-2021-08-19.pdf
Fun with ClickHouse Window Functions-2021-08-19.pdf
Altinity Ltd
 
Performance Enhancements In Postgre Sql 8.4
Performance Enhancements In Postgre Sql 8.4Performance Enhancements In Postgre Sql 8.4
Performance Enhancements In Postgre Sql 8.4
HighLoad2009
 
User Defined Partitioning on PlazmaDB
User Defined Partitioning on PlazmaDBUser Defined Partitioning on PlazmaDB
User Defined Partitioning on PlazmaDB
Kai Sasaki
 
How to tune a query - ODTUG 2012
How to tune a query - ODTUG 2012How to tune a query - ODTUG 2012
How to tune a query - ODTUG 2012
Connor McDonald
 
CS 542 -- Query Execution
CS 542 -- Query ExecutionCS 542 -- Query Execution
CS 542 -- Query Execution
J Singh
 
nter-pod Revolutions: Connected Enterprise Solution in Oracle EPM Cloud
nter-pod Revolutions: Connected Enterprise Solution in Oracle EPM Cloud nter-pod Revolutions: Connected Enterprise Solution in Oracle EPM Cloud
nter-pod Revolutions: Connected Enterprise Solution in Oracle EPM Cloud
Alithya
 
Monitoring InfluxEnterprise
Monitoring InfluxEnterpriseMonitoring InfluxEnterprise
Monitoring InfluxEnterprise
InfluxData
 

Similar to Webinar: Secrets of ClickHouse Query Performance, by Robert Hodges (20)

ClickHouse materialized views - a secret weapon for high performance analytic...
ClickHouse materialized views - a secret weapon for high performance analytic...ClickHouse materialized views - a secret weapon for high performance analytic...
ClickHouse materialized views - a secret weapon for high performance analytic...
 
A day in the life of a click house query
A day in the life of a click house queryA day in the life of a click house query
A day in the life of a click house query
 
Flink Forward Berlin 2018: Dawid Wysakowicz - "Detecting Patterns in Event St...
Flink Forward Berlin 2018: Dawid Wysakowicz - "Detecting Patterns in Event St...Flink Forward Berlin 2018: Dawid Wysakowicz - "Detecting Patterns in Event St...
Flink Forward Berlin 2018: Dawid Wysakowicz - "Detecting Patterns in Event St...
 
Data warehouse or conventional database: Which is right for you?
Data warehouse or conventional database: Which is right for you?Data warehouse or conventional database: Which is right for you?
Data warehouse or conventional database: Which is right for you?
 
Webinar: Strength in Numbers: Introduction to ClickHouse Cluster Performance
Webinar: Strength in Numbers: Introduction to ClickHouse Cluster PerformanceWebinar: Strength in Numbers: Introduction to ClickHouse Cluster Performance
Webinar: Strength in Numbers: Introduction to ClickHouse Cluster Performance
 
Simplifying SQL with CTE's and windowing functions
Simplifying SQL with CTE's and windowing functionsSimplifying SQL with CTE's and windowing functions
Simplifying SQL with CTE's and windowing functions
 
Oracle tips and tricks
Oracle tips and tricksOracle tips and tricks
Oracle tips and tricks
 
PostgreSQL 8.4 TriLUG 2009-11-12
PostgreSQL 8.4 TriLUG 2009-11-12PostgreSQL 8.4 TriLUG 2009-11-12
PostgreSQL 8.4 TriLUG 2009-11-12
 
Database Foundation Training
Database Foundation TrainingDatabase Foundation Training
Database Foundation Training
 
Less08 Schema
Less08 SchemaLess08 Schema
Less08 Schema
 
PHP tips by a MYSQL DBA
PHP tips by a MYSQL DBAPHP tips by a MYSQL DBA
PHP tips by a MYSQL DBA
 
Tactical data engineering
Tactical data engineeringTactical data engineering
Tactical data engineering
 
Fun with click house window functions webinar slides 2021-08-19
Fun with click house window functions webinar slides  2021-08-19Fun with click house window functions webinar slides  2021-08-19
Fun with click house window functions webinar slides 2021-08-19
 
Fun with ClickHouse Window Functions-2021-08-19.pdf
Fun with ClickHouse Window Functions-2021-08-19.pdfFun with ClickHouse Window Functions-2021-08-19.pdf
Fun with ClickHouse Window Functions-2021-08-19.pdf
 
Performance Enhancements In Postgre Sql 8.4
Performance Enhancements In Postgre Sql 8.4Performance Enhancements In Postgre Sql 8.4
Performance Enhancements In Postgre Sql 8.4
 
User Defined Partitioning on PlazmaDB
User Defined Partitioning on PlazmaDBUser Defined Partitioning on PlazmaDB
User Defined Partitioning on PlazmaDB
 
How to tune a query - ODTUG 2012
How to tune a query - ODTUG 2012How to tune a query - ODTUG 2012
How to tune a query - ODTUG 2012
 
CS 542 -- Query Execution
CS 542 -- Query ExecutionCS 542 -- Query Execution
CS 542 -- Query Execution
 
nter-pod Revolutions: Connected Enterprise Solution in Oracle EPM Cloud
nter-pod Revolutions: Connected Enterprise Solution in Oracle EPM Cloud nter-pod Revolutions: Connected Enterprise Solution in Oracle EPM Cloud
nter-pod Revolutions: Connected Enterprise Solution in Oracle EPM Cloud
 
Monitoring InfluxEnterprise
Monitoring InfluxEnterpriseMonitoring InfluxEnterprise
Monitoring InfluxEnterprise
 

More from Altinity Ltd

Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptxBuilding an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
Altinity Ltd
 
Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...
Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...
Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...
Altinity Ltd
 
Building an Analytic Extension to MySQL with ClickHouse and Open Source
Building an Analytic Extension to MySQL with ClickHouse and Open SourceBuilding an Analytic Extension to MySQL with ClickHouse and Open Source
Building an Analytic Extension to MySQL with ClickHouse and Open Source
Altinity Ltd
 
Cloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdf
Cloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdfCloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdf
Cloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdf
Altinity Ltd
 
Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...
Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...
Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...
Altinity Ltd
 
Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...
Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...
Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...
Altinity Ltd
 
Own your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdf
Own your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdfOwn your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdf
Own your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdf
Altinity Ltd
 
ClickHouse ReplacingMergeTree in Telecom Apps
ClickHouse ReplacingMergeTree in Telecom AppsClickHouse ReplacingMergeTree in Telecom Apps
ClickHouse ReplacingMergeTree in Telecom Apps
Altinity Ltd
 
Building a Real-Time Analytics Application with Apache Pulsar and Apache Pinot
Building a Real-Time Analytics Application with  Apache Pulsar and Apache PinotBuilding a Real-Time Analytics Application with  Apache Pulsar and Apache Pinot
Building a Real-Time Analytics Application with Apache Pulsar and Apache Pinot
Altinity Ltd
 
Altinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdf
Altinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdfAltinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdf
Altinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdf
Altinity Ltd
 
OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...
OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...
OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...
Altinity Ltd
 
OSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdf
OSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdfOSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdf
OSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdf
Altinity Ltd
 
OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...
OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...
OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...
Altinity Ltd
 
OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...
OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...
OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...
Altinity Ltd
 
OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...
OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...
OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...
Altinity Ltd
 
OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...
OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...
OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...
Altinity Ltd
 
OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...
OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...
OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...
Altinity Ltd
 
OSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdf
OSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdfOSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdf
OSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdf
Altinity Ltd
 
OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...
OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...
OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...
Altinity Ltd
 
OSA Con 2022 - Signal Correlation, the Ho11y Grail - Michael Hausenblas - AWS...
OSA Con 2022 - Signal Correlation, the Ho11y Grail - Michael Hausenblas - AWS...OSA Con 2022 - Signal Correlation, the Ho11y Grail - Michael Hausenblas - AWS...
OSA Con 2022 - Signal Correlation, the Ho11y Grail - Michael Hausenblas - AWS...
Altinity Ltd
 

More from Altinity Ltd (20)

Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptxBuilding an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
 
Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...
Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...
Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...
 
Building an Analytic Extension to MySQL with ClickHouse and Open Source
Building an Analytic Extension to MySQL with ClickHouse and Open SourceBuilding an Analytic Extension to MySQL with ClickHouse and Open Source
Building an Analytic Extension to MySQL with ClickHouse and Open Source
 
Cloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdf
Cloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdfCloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdf
Cloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdf
 
Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...
Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...
Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...
 
Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...
Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...
Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...
 
Own your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdf
Own your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdfOwn your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdf
Own your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdf
 
ClickHouse ReplacingMergeTree in Telecom Apps
ClickHouse ReplacingMergeTree in Telecom AppsClickHouse ReplacingMergeTree in Telecom Apps
ClickHouse ReplacingMergeTree in Telecom Apps
 
Building a Real-Time Analytics Application with Apache Pulsar and Apache Pinot
Building a Real-Time Analytics Application with  Apache Pulsar and Apache PinotBuilding a Real-Time Analytics Application with  Apache Pulsar and Apache Pinot
Building a Real-Time Analytics Application with Apache Pulsar and Apache Pinot
 
Altinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdf
Altinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdfAltinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdf
Altinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdf
 
OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...
OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...
OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...
 
OSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdf
OSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdfOSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdf
OSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdf
 
OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...
OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...
OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...
 
OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...
OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...
OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...
 
OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...
OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...
OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...
 
OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...
OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...
OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...
 
OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...
OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...
OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...
 
OSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdf
OSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdfOSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdf
OSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdf
 
OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...
OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...
OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...
 
OSA Con 2022 - Signal Correlation, the Ho11y Grail - Michael Hausenblas - AWS...
OSA Con 2022 - Signal Correlation, the Ho11y Grail - Michael Hausenblas - AWS...OSA Con 2022 - Signal Correlation, the Ho11y Grail - Michael Hausenblas - AWS...
OSA Con 2022 - Signal Correlation, the Ho11y Grail - Michael Hausenblas - AWS...
 

Recently uploaded

Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Safe Software
 
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Albert Hoitingh
 
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
名前 です男
 
Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1
DianaGray10
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
danishmna97
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
innovationoecd
 
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
SOFTTECHHUB
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
Matthew Sinclair
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Speck&Tech
 
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
SOFTTECHHUB
 
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
DianaGray10
 
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Neo4j
 
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR EventsMonitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
Ana-Maria Mihalceanu
 
RESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for studentsRESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for students
KAMESHS29
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
Matthew Sinclair
 
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Neo4j
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
Kari Kakkonen
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
DianaGray10
 
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial ApplicationsLarge Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
Rohit Gautam
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
Kumud Singh
 

Recently uploaded (20)

Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
 
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
 
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
 
Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
 
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
 
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
 
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
 
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
 
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR EventsMonitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
 
RESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for studentsRESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for students
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
 
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
 
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial ApplicationsLarge Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
 

Webinar: Secrets of ClickHouse Query Performance, by Robert Hodges

  • 1.
  • 2. Brief Intros www.altinity.com Leading software and services provider for ClickHouse Major committer and community sponsor in US and Western Europe Robert Hodges - Altinity CEO 30+ years on DBMS plus virtualization and security. ClickHouse is DBMS #20
  • 3. Goals of the talk ● Understand single node MergeTree structure ● Optimize queries without changing data ● Get bigger performance gains by changing data layout ● Introduce tools for performance monitoring Non-Goals: ● Boost performance of sharded/replicated clusters ● Teach advanced ClickHouse performance management
  • 5. Introduction to ClickHouse Understands SQL Runs on bare metal to cloud Shared nothing architecture Stores data in columns Parallel and vectorized execution Scales to many petabytes Is Open source (Apache 2.0) a b c d a b c d a b c d a b c d And it’s really fast!
  • 6. Introducing the MergeTree table engine CREATE TABLE ontime ( Year UInt16, Quarter UInt8, Month UInt8, ... ) ENGINE = MergeTree() PARTITION BY toYYYYMM(FlightDate) ORDER BY (Carrier, FlightDate) Table engine type How to break data into parts How to index and sort data in each part
  • 7. Basic MergeTree data layout Table Part Index Columns Sparse Index Columns sorted on ORDER BY columns Rows match PARTITION BY expression Part Index Columns Part
  • 8. MergeTree layout within a single part /var/lib/clickhouse/data/airline/ontime_reordered 2017-01-01 AA 2017-01-01 EV 2018-01-01 UA 2018-01-02 AA ... primary.idx |||| .mrk .bin 20170701_20170731_355_355_2/ (FlightDate, Carrier...) ActualElapsedTime Airline AirlineID... |||| .mrk .bin |||| .mrk .bin Granule Compressed Block Mark
  • 10. ClickHouse performance tuning is different... The bad news… ● No query optimizer ● No EXPLAIN PLAN ● May need to move [a lot of] data for performance The good news… ● No query optimizer! ● System log is great ● System tables are too ● Performance drivers are simple: I/O and CPU
  • 11. Your friend: the ClickHouse query log clickhouse-client --send_logs_level=trace select * from system.text_log sudo less /var/log/clickhouse-server/clickhouse-server.log Return messages to clickhouse-client View all log messages on server Must enable in config.xml
  • 12. (Log messages) Limit Expression MergeSorting PartialSorting Expression ParallelAggregating Expression × 8 MergeTreeThread Use system log to find out query details SELECT toYear(FlightDate) year, sum(Cancelled)/count(*) cancelled, sum(DepDel15)/count(*) delayed_15 FROM airline.ontime GROUP BY year ORDER BY year LIMIT 10 8 parallel threads to read table Query pipeline in log
  • 13. Speed up query executing by adding threads SELECT toYear(FlightDate) year, sum(Cancelled)/count(*) cancelled, sum(DepDel15)/count(*) delayed_15 FROM airline.ontime GROUP BY year ORDER BY year LIMIT 10 SET max_threads = 2 SET max_threads = 4 . . . max_threads defaults to half the number of physical CPU cores
  • 14. (Log messages) Selected 355 parts by date, 355 parts by key, 21393 marks to read from 355 ranges Speed up queries by reducing reads SELECT toYear(FlightDate) year, sum(Cancelled)/count(*) cancelled, sum(DepDel15)/count(*) delayed_15 FROM airline.ontime GROUP BY year ORDER BY year LIMIT 10 (Log messages) Selected 12 parts by date, 12 parts by key, 692 marks to read from 12 ranges SELECT toYear(FlightDate) year, sum(Cancelled)/count(*) cancelled, sum(DepDel15)/count(*) delayed_15 FROM airline.ontime WHERE year = toYear(toDate('2016-01-01')) GROUP BY year ORDER BY year LIMIT 10
  • 15. (Log messages) Selected 2 parts by date, 2 parts by key, 73 marks to read from 2 ranges Query execution tends to scale with I/O SELECT FlightDate, count(*) AS total_flights, sum(Cancelled) / count(*) AS cancelled, sum(DepDel15) / count(*) AS delayed_15 FROM airline.ontime WHERE (FlightDate >= toDate('2016-01-01')) AND (FlightDate <= toDate('2016-02-10')) GROUP BY FlightDate
  • 16. (PREWHERE Log messages) Elapsed: 0.591 sec. Processed 173.82 million rows, 2.09 GB (294.34 million rows/s., 3.53 GB/s.) Use PREWHERE to help filter unindexed data SELECT Year, count(*) AS total_flights, count(distinct Dest) as destinations, count(distinct Carrier) as carriers, sum(Cancelled) / count(*) AS cancelled, sum(DepDel15) / count(*) AS delayed_15 FROM airline.ontime [PRE]WHERE Dest = 'BLI' GROUP BY Year (WHERE Log messages) Elapsed: 0.816 sec. Processed 173.82 million rows, 5.74 GB (213.03 million rows/s., 7.03 GB/s.)
  • 17. But PREWHERE can kick in automatically SET optimize_move_to_prewhere = 1 SELECT Year, count(*) AS total_flights, count(distinct Dest) as destinations, count(distinct Carrier) as carriers, sum(Cancelled) / count(*) AS cancelled, sum(DepDel15) / count(*) AS delayed_15 FROM airline.ontime WHERE Dest = 'BLI' GROUP BY Year (Log messages) InterpreterSelectQuery: MergeTreeWhereOptimizer: condition "Dest = 'BLI'" moved to PREWHERE This is the default value
  • 18. Restructure joins to reduce data scanning SELECT Dest d, Name n, count(*) c, avg(ArrDelayMinutes) FROM ontime JOIN airports ON (airports.IATA = ontime.Dest) GROUP BY d, n HAVING c > 100000 ORDER BY d DESC LIMIT 10 SELECT dest, Name n, c AS flights, ad FROM ( SELECT Dest dest, count(*) c, avg(ArrDelayMinutes) ad FROM ontime GROUP BY dest HAVING c > 100000 ORDER BY ad DESC ) LEFT JOIN airports ON airports.IATA = dest LIMIT 10 Faster 3.878 seconds 1.177 seconds
  • 19. (Log messages) ParallelAggregatingBlockInputStream : Total aggregated. 173790727 rows (from 10199.035 MiB) in 3.844 sec. (45214666.568 rows/sec., 2653.455 MiB/sec.) The log tells the story (Log messages) ParallelAggregatingBlockInputStream : Total aggregated. 173815409 rows (from 2652.213 MiB) in 1.142 sec. (152149486.717 rows/sec., 2321.617 MiB/sec.) Join during MergeTree scan Join after MergeTree scan
  • 20. More ways to find out about queries SET log_queries = 1 Run a query SELECT version() SET log_queries = 0 SELECT * FROM system.query_log WHERE query='SELECT version()' SHOW PROCESSLIST Start query logging Stop query logging Show currently executing queries
  • 22. Restructure data for big performance gains ● Ensure optimal number of parts ● Optimize primary key index and ordering to reduce data size and index selectivity ● Use skip indexes to avoid unnecessary I/O ● Use encodings to reduce data size before compression ● Use materialized views to transform data outside of source table ● Plus many other tricks
  • 23. CREATE TABLE ontime ... ENGINE=MergeTree() PARTITION BY toYYYYMM(FlightDate) CREATE TABLE ontime_many_parts ... ENGINE=MergeTree() PARTITION BY FlightDate How do partition keys affect performance? Is there a practical difference?
  • 24. Keep parts in the hundreds, not thousands Table Rows Parts ontime 174M 355 ontime_many_parts (after OPTIMIZE) 174M 10,085 ontime_many_parts (before OPTIMIZE) 174M 14,635 CREATE TABLE ontime ... ENGINE=MergeTree() PARTITION BY toYYYYMM(FlightDate) CREATE TABLE ontime_many_parts ... ENGINE=MergeTree() PARTITION BY FlightDate
  • 25. Think about primary key index structure CREATE TABLE ontime_reordered ( Year UInt16, Quarter` UInt8, . . .) ENGINE = MergeTree() PARTITION BY toYYYYMM(FlightDate) ORDER BY (Carrier, Origin, FlightDate) SETTINGS index_granularity=8192 Hashing large values allows index to fit in memory more easily Large granularity makes index smaller Small granularity can make skip indexes more selective
  • 26. Table ORDER BY is key to performance CREATE TABLE ontime_reordered ( Year UInt16, Quarter` UInt8, . . .) ENGINE = MergeTree() PARTITION BY toYYYYMM(FlightDate) ORDER BY (Carrier, Origin, FlightDate) SETTINGS index_granularity=8192 Choose order to make dependent non-key values less random Benefits: ➔ Higher compression ➔ Better index selectivity ➔ Better PREWHERE performance
  • 27. SET allow_experimental_data_skipping_indices=1; ALTER TABLE ontime ADD INDEX dest_name Dest TYPE ngrambf_v1(3, 512, 2, 0) GRANULARITY 1 ALTER TABLE ontime ADD INDEX cname Carrier TYPE set(100) GRANULARITY 1 OPTIMIZE TABLE ontime FINAL -- In future releases: ALTER TABLE ontime UPDATE Dest=Dest, Carrier=Carrier WHERE 1=1 Skip indexes cut down on I/O
  • 28. Indexes & PREWHERE remove granules (Log messages) InterpreterSelectQuery: MergeTreeWhereOptimizer: condition "Dest = 'PPG'" moved to PREWHERE . . . (SelectExecutor): Index `dest_name` has dropped 55 granules. (SelectExecutor): Index `dest_name` has dropped 52 granules. Apply PREWHERE on Dest predicate Use index to remove granules from scan
  • 29. Effectiveness depends on data distribution SELECT Year, count(*) AS flights, sum(Cancelled) / flights AS cancelled, sum(DepDel15) / flights AS delayed_15 FROM airline.ontime WHERE [Column] = [Value] GROUP BY Year Column Value Index Count Rows Processed Query Response Dest PPG ngrambf_v1 525 4.30M 0.053 Dest ATL ngrambf_v1 9,360,581 166.81M 0.622 Carrier ML set 70,622 3.39M 0.090 Carrier WN set 25,918,402 166.24M 0.566
  • 30. Current index types Name What it tracks minmax High and low range of data; good for numbers with strong locality like timestamps set Unique values ngrambf_v1 Presence of character ngrams, works with =, LIKE, search predicates; good for long strings tokenbf_v1 Like ngram but for whitespace-separated strings; good for searching on tags bloomfilter Presence of value in column
  • 31. Encodings reduce data size CREATE TABLE test_codecs ( a String, a_lc LowCardinality(String) DEFAULT a, b UInt32, b_delta UInt32 DEFAULT b Codec(Delta), b_delta_lz4 UInt32 DEFAULT b Codec(Delta, LZ4), b_dd UInt32 DEFAULT b Codec(DoubleDelta), b_dd_lz4 UInt32 DEFAULT b Codec(DoubleDelta, LZ4) ) Engine = MergeTree PARTITION BY tuple() ORDER BY tuple(); Differences between values Differences between change of value Values with dictionary encoding
  • 32. Relative sizes of column data
  • 33. But wait, there’s more! Encodings are faster SELECT a AS a, count(*) AS c FROM test_codecs GROUP BY a ORDER BY c ASC LIMIT 10 . . . 10 rows in set. Elapsed: 0.681 sec. Processed 100.00 million rows, 2.69 GB (146.81 million rows/s., 3.95 GB/s.) SELECT a_lc AS a, count(*) AS c FROM test_codecs GROUP BY a ORDER BY c ASC LIMIT 10 . . . 10 rows in set. Elapsed: 0.148 sec. Processed 100.00 million rows, 241.16 MB (675.55 million rows/s., 1.63 GB/s.) Faster
  • 34. Overview of encodings Name Best for LowCardinality Strings with fewer than 10K values Delta Time series Double Delta Increasing counters Gorilla Gauge data (bounces around mean) T64 Integers other than random hashes Compression may vary across ZSTD and LZ4
  • 35. Use mat views to boost performance further CREATE MATERIALIZED VIEW ontime_daily_cancelled_mv ENGINE = SummingMergeTree PARTITION BY tuple() ORDER BY (FlightDate, Carrier) POPULATE AS SELECT FlightDate, Carrier, count(*) AS flights, sum(Cancelled) / count(*) AS cancelled, sum(DepDel15) / count(*) AS delayed_15 FROM ontime GROUP BY FlightDate, Carrier Returns cancelled/late flights where Carrier = ‘WN’ in 0.007 seconds
  • 36. More things to think about Use smaller datatypes wherever possible Use ZSTD compression (slower but better ratio) Use dictionaries instead of joins Use sampling when approximate answers are acceptable Shard/replicate data across a cluster for large datasets We’ll cover performance of distributed clusters in a future webinar
  • 38. Use system.parts to track partition content SELECT database, table, count(*) AS parts, uniq(partition) AS partitions, sum(marks) AS marks, sum(rows) AS rows, formatReadableSize(sum(data_compressed_bytes)) AS compressed, formatReadableSize(sum(data_uncompressed_bytes)) AS uncompressed, round(sum(data_compressed_bytes) / sum(data_uncompressed_bytes) * 100.0, 2) AS percentage FROM system.parts WHERE active and database = currentDatabase() GROUP BY database, table ORDER BY database ASC, table ASC
  • 39. Use system.columns to check data size SELECT table, formatReadableSize(sum(data_compressed_bytes)) tc, formatReadableSize(sum(data_uncompressed_bytes)) tu, sum(data_compressed_bytes) / sum(data_uncompressed_bytes) as ratio FROM system.columns WHERE database = currentDatabase() GROUP BY table ORDER BY table
  • 40. ClickHouse has great operational metrics Some of our favorite tables in the system database… query_log (Query history) part_log (Part merges) text_log (System log messages) asynchronous _events (Background metrics) events (Cumulative event counters) metrics (Current counters)
  • 41. It’s easy to dump values and graph them Open files during OPTIMIZE TABLE command on 174M row table (system.metrics OpenFileForRead counter)
  • 42. And don’t forget all the great OS utilities! ● top and htop -- CPU and memory ● dstat -- I/O and network consumption ● iostat -- I/O by device ● iotop -- I/O by process ● iftop -- Network consumption by host ● perf top -- CPU utilization by system function For a full description see Performance Analysis of ClickHouse Queries by Alexey Milovidov
  • 44. Takeaways on ClickHouse Performance ● ClickHouse performance drivers are CPU and I/O ● The system query log is key to understanding performance ● Query optimization can improve response substantially ● Restructure tables and add indexes/mat views for biggest gains ● Use the system database tables and OS tools for deeper analysis of performance
  • 45. Further resources ● Altinity Blog ● ClickHouse and the Magic of Materialized Views (Webinar) ● Performance Analysis of ClickHouse Queries by Alexey Milovidov ● ClickHouse Telegram Channel
  • 46. Thank you! Special Offer: Contact us for a 1-hour consultation! Contacts: info@altinity.com Visit us at: https://www.altinity.com Free Consultation: https://blog.altinity.com/offer