SlideShare a Scribd company logo
1 of 45
Download to read offline
© 2022 Altinity, Inc.
All about ZooKeeper
(and ClickHouse
Keeper, too!)
Robert Hodges and Altinity Engineering
30 March 2022
1
Copyright © Altinity Inc 2022
© 2022 Altinity, Inc.
Let’s make some introductions
ClickHouse support and services including Altinity.Cloud
Authors of Altinity Kubernetes Operator for ClickHouse
and other open source projects
Robert Hodges
Database geek with 30+ years
on DBMS systems. Day job:
Altinity CEO
Altinity Engineering
Database geeks with centuries
of experience in DBMS and
applications
2
© 2022 Altinity, Inc.
Why does ClickHouse need ZooKeeper?
3
© 2022 Altinity, Inc.
Horizontal scaling is a key to ClickHouse performance
4
clickhouse-0
events
_local
Sharded,
replicated
table
clickhouse-1
events
_local
clickhouse-2
events
_local
clickhouse-3
events
_local
© 2022 Altinity, Inc.
Let’s create the table and try it out!
5
CREATE TABLE IF NOT EXISTS `events_local` ON CLUSTER '{cluster}' (
EventDate DateTime, CounterID UInt32, Value String
)
Engine=ReplicatedMergeTree(
'/clickhouse/{cluster}/tables/{shard}/{database}/events_local',
'{replica}')
PARTITION BY toYYYYMM(EventDate)
ORDER BY (CounterID, EventDate, intHash32(UserID))
INSERT INTO events_local(EventDate, EventID, Value) VALUES
(now(), 1, 'In-Progress'), (now(), 2, 'OK')
. . .
© 2022 Altinity, Inc.
What could possibly go wrong?
6
ON CLUSTER
command
failed on one
node!
Two replicas
merge
overlapping
parts!
Node offline for
maintenance;
missed the
memo!
Two replicas
delete
overlapping
parts!
clickhouse-0
events
_local
Sharded,
replicated
table
clickhouse-1
events
_local
clickhouse-2
events
_local
clickhouse-3
events
_local
© 2022 Altinity, Inc.
ZooKeeper solves the distributed consistency problem
7
zookeeper-0 zookeeper-2
zookeeper-1
clickhouse-0
events
_local
Sharded,
replicated
table
clickhouse-1
events
_local
clickhouse-2
events
_local
clickhouse-3
events
_local
Consistent ON
CLUSTER commands,
replicated table parts
© 2022 Altinity, Inc.
How ZooKeeper Works
8
© 2022 Altinity, Inc.
ZooKeeper Architecture
9
ZooKeeper Ensemble
zookeeper-0 zookeeper-2
zookeeper-1
ClickHouse
Server
ClickHouse
Server
ClickHouse
Server
ClickHouse
Server
ZooKeeper
Clients
ZooKeeper
Servers on
Port :2181
ZooKeeper Atomic Broadcast (ZAB)
© 2022 Altinity, Inc.
ZooKeeper leaders and followers
10
zookeeper-0
zookeeper-2
zookeeper-1
Leader coordinates writes
Follower Follower
Leader
Followers handle reads and
delegate writes to leader
Write request
Read request
Leader must maintain a
quorum of followers
© 2022 Altinity, Inc.
ZooKeeper directory structure for ClickHouse
11
/
zookeeper
clickhouse
first
tables
task_queue
0
default
log
events_local
quorum
replicas
query
query-0000
000000
DROP TABLE IF EXISTS
default.events_local ON
CLUSTER first
Znode
. . .
© 2022 Altinity, Inc.
What kind of ClickHouse information is stored in znodes?
● Tasks
○ Pending and completed ON CLUSTER DDL commands
● Table information
○ Schema information
○ Replicas
○ Leader elections used to control merges and mutations
○ Log of operations on the table (insert, merge, delete partition, etc.)
○ Parts contained in each replica
○ Last N blocks inserts so we can deduplicate data
○ Data to ensure quorum on writes
12
© 2022 Altinity, Inc.
Installing and configuring ZooKeeper
13
© 2022 Altinity, Inc.
Installing a ZooKeeper on Ubuntu
14
sudo apt update
sudo apt install zookeeper netcat
(edit /etc/sysconfig/config/zoo.cfg to set configuration)
Install Zookeeper
3.4.9 or greater
© 2022 Altinity, Inc.
Ensuring ZooKeeper maximum speed and availability
Host recommendations
● Dedicated host for ZooKeepers - don’t share with other applications
● Put ZooKeeper log on dedicated SSD
● Low network latency between ZooKeeper nodes
● At least 4GiB of RAM
● Disable swap (remove entry from /etc/fstab)
● Tune the Java heap to use as much RAM as possible
○ E.g., 3GiB out of 4GiB available RAM
15
© 2022 Altinity, Inc.
ZooKeeper moving parts
16
ZooKeeper
Java
Process
dataLogDir
zoo.cfg
dataDir
Configuration file
Logs with changes to data
In-RAM
copy of
Znode
Tree
Snapshots of data
myid Server instance ID (1-255)
© 2022 Altinity, Inc.
Editing important zoo.cfg settings
17
…
autopurge.purgeInterval=1
autopurge.snapRetainCount=5
…
server.1=zookeeper1:2888:3888
server.2=zookeeper2:2888:3888
server.3=zookeeper3:2888:3888
…
dataDir=/var/lib/zookeeper
…
dataLogDir=/ssd/zookeeper/logs
Must be added; prevents
snapshots from accumulating
Servers in ensemble; must be
identical everywhere
Location for snapshots
Put logs on fast storage
© 2022 Altinity, Inc.
Starting ZooKeeper and ensuring it’s up
18
sudo -u zookeeper /usr/share/zookeeper/bin/zkServer.sh
ZooKeeper JMX enabled by default
Using config: /etc/zookeeper/conf/zoo.cfg
Starting zookeeper ... STARTED
echo ruok | nc localhost 2181
imok
echo mntr | nc localhost 2181
zk_version 3.4.10-3--1, built on Sat, 03 Feb 2018 14:58:02 -0800
. . .
echo stat | nc localhost 2181
zk_version 3.4.10-3--1, built on Sat, 03 Feb 2018 14:58:02 -0800
. . .
© 2022 Altinity, Inc.
Tell ClickHouse where ZooKeeper lives
19
<yandex>
<zookeeper>
<node>
<host>zookeeper.zoo1ns</host>
<port>2181</port>
</node>
</zookeeper>
<distributed_ddl>
<path>/clickhouse/first/task_queue/ddl</path>
</distributed_ddl>
</yandex>
© 2022 Altinity, Inc.
Add macros so ON CLUSTER commands can run
20
<yandex>
<macros>
<installation>first</installation>
<all-sharded-shard>0</all-sharded-shard>
<cluster>first</cluster>
<shard>0</shard>
<replica>chi-first-first-0-0</replica>
</macros>
</yandex>
© 2022 Altinity, Inc.
Practical Administration Tips
21
© 2022 Altinity, Inc.
How many ZooKeepers are enough?
22
1 2 5
4
3
OK for dev
Less available
than 1 replica!
Best for prod OK but writes may be slower
Less available
than 3 replicas!
© 2022 Altinity, Inc.
What’s in ZooKeeper? The system.zookeeper table knows!
23
SELECT * FROM system.zookeeper WHERE path = '/'
ORDER BY name FORMAT Vertical
Row 1:
──────
name: clickhouse
value:
czxid: 4294967298
mzxid: 4294967298
ctime: 2021-12-08 01:54:50
mtime: 2021-12-08 01:54:50
. . .
path: /
Path value is required!
If this query works, ClickHouse
can see ZooKeeper!
© 2022 Altinity, Inc.
Printing znode values from system.zookeeper
24
SELECT name, value FROM system.zookeeper
WHERE path = '/clickhouse/first/task_queue/ddl/'
FORMAT Vertical
Row 1:
──────
name: query-0000000009
value: version: 1
query: CREATE TABLE IF NOT EXISTS default.events_local UUID
'2a8ed83e-a6ef-48b4-aa8e-d83ea6efa8b4' ON CLUSTER first (`EventDate`
DateTime, `EventID` UInt32, `Value` String) ENGINE =
ReplicatedMergeTree('/clickhouse/{cluster}/tables/{shard}/{database}/even
ts_local', '{replica}') PARTITION BY toYYYYMM(EventDate) ORDER BY
(CounterID, EventDate, intHash32(UserID))
hosts:
. . .
Prints values for znodes
under this path
© 2022 Altinity, Inc.
Using the zkCli utility to talk to ZooKeeper directly
25
(Connect to ZooKeeper host)
$ zkCli.sh
Connecting to localhost:2181
. . .
[zk: localhost:2181(CONNECTED) 0] ls /
[clickhouse, zookeeper]
[zk: localhost:2181(CONNECTED) 1] get
/clickhouse/first/task_queue/ddl/query-0000000009
version: 1
query: CREATE TABLE IF NOT EXISTS default.events_local UUID
'2a8ed83e-a6ef-48b4-aa8e-d83ea6efa8b4' ON CLUSTER first . . .
© 2022 Altinity, Inc.
ZooKeeper four letter word commands
26
Example: echo ruok | nc localhost 2181 → imok
Command What it does
ruok Check server liveness
conf Print server config
cons Print connections
mntr Dump monitoring information
srvr Dump server information
There are more
commands! Check
the docs.
© 2022 Altinity, Inc.
ZooKeeper Monitoring
Older approach for Nagios and Icinga[2]
● Use check_zookeeper.pl
Newer approach: Use Prometheus + AlertManager + Grafana
● ZooKeeper by Prometheus Dashboard for Grafana
The Altinity Knowledge Base has a page on ZooKeeper Monitoring
27
© 2022 Altinity, Inc.
The dreaded read-only table error
28
INSERT INTO events2_local (EventDate, EventID, Value)
VALUES (now(), 1, 'In-Progress'), (now(), 2, 'OK')
Received exception from server (version 21.8.10):
Code: 242. DB::Exception: Received from 34.83.194.130:9000.
DB::Exception: Table is in readonly mode (zookeeper path:
/clickhouse/first/tables/0/default/events2_local).
(TABLE_IS_READ_ONLY)
Must be added; prevents snapshots from accumulating
ZooKeeper is offline!
© 2022 Altinity, Inc.
Steps to address read-only tables
29
ClickHouse ZooKeeper
SELECT * FROM
system.zookeeper
WHERE path = '/'
echo ruok | nc localhost 2181
Log
Loss of quorum?
ZXID Rollover?
Other errors?
/var/log/zookeeper/zookeeper.log
Can ClickHouse
see ZooKeeper?
Is ZooKeeper alive?
ZooKeeper Connection
© 2022 Altinity, Inc.
ZooKeeper “Session Expired” errors
If ClickHouse loses its connection to ZooKeeper, pending INSERTs or ON
CLUSTER commands may fail with a Session Expired error.
1. Occasional failure is normal in distributed systems. Retry the operation!!
2. If the problem happens commonly, you may have a ZooKeeper problem.
a. Check ZooKeeper logs for errors
b. This could be an ZXID overflow due to too many transactions on ZooKeeper.
Check that only ClickHouse is using ZooKeeper!
c. Too many parts in the table? (> 5000)
d. Jute.maxbuffer seting on ZooKeeper is too low.
30
© 2022 Altinity, Inc.
Recovering from failures
Loss of a single ZooKeeper node
1. Create fresh node with same ZooKeeper instance ID as lost node
2. Ensure new host name is correct in all zoo.cfg files
3. Start new node
Loss of entire ZooKeeper ensemble
1. Briefly consider taking an immediate vacation
2. Bring up new ZooKeeper ensemble
3. Use SYSTEM RESTORE REPLICA command to restore metadata from
ClickHouse server(s)
31
© 2022 Altinity, Inc.
ClickHouse Keeper
32
© 2022 Altinity, Inc.
So…What is ClickHouse Keeper?
33
It’s a from-scratch reimplementation of ZooKeeper
● Mimics ZooKeeper API and admin commands
● Uses Raft protocol instead of ZAB for consensus
● Is written in C++
● Is part of ClickHouse
No extra installation required!
© 2022 Altinity, Inc.
Why replace ZooKeeper?
34
● ClickHouse should contain everything it needs to run
● Old, not very actively developed
● Java executable adds dependencies and requires tuning
● Many people find it hard to operate
● Problems like ZXID rollover, uncompressed logs, etc.
© 2022 Altinity, Inc.
ClickHouse Keeper can be a drop-in ZK replacement…
35
ch-keeper-0 ch-keeper-2
ch-keeper-1
clickhouse-0
events
_local
Sharded,
replicated
table
clickhouse-1
events
_local
clickhouse-2
events
_local
clickhouse-3
events
_local
Consistent ON
CLUSTER commands,
replicated table parts
© 2022 Altinity, Inc.
Or it can run directly in ClickHouse itself!
36
“Sharded”,
replicated
table
clickhouse-0
events
_local
keeper-1
clickhouse-0
events
_local
keeper-1
clickhouse-1
events
_local
keeper-2
clickhouse-2
events
_local
keeper-3
© 2022 Altinity, Inc.
ClickHouse Keeper single node configuration
37
<yandex>
<keeper_server incl="keeper_server">
<server_id>1</server_id>
<tcp_port>2181</tcp_port>
<coordination_settings>
<raft_logs_level>debug</raft_logs_level>
</coordination_settings>
<raft_configuration>
<server>
<id>1</id>
<hostname>logos3</hostname><port>9444</port>
</server>
</raft_configuration>
</keeper_server> </yandex>
© 2022 Altinity, Inc.
ClickHouse Keeper moving parts for single node install
38
ClickHouse
Keeper
logs
keeper-config.xml
snapshots
Configuration file
Transaction logs
In-RAM
copy of
Znode
Tree
Snapshots
/etc/clickhouse-server/config.d
/var/lib/clickhouse/coordination
ClickHouse
© 2022 Altinity, Inc.
ClickHouse Keeper “just works”
1. ON CLUSTER commands and replication work exactly as before
2. System.zookeeper table shows directory structure
3. ZooKeeper four letter commands work
4. You can use zkCli.sh (and other tools) to navigate the directory structure
39
© 2022 Altinity, Inc.
How to tell you are using ClickHouse Keeper
40
$ echo srvr |netcat logos3 2181
ClickHouse Keeper version:
v22.3.2.1-prestable-92ab33f560e638d1989c5ca543021ab53d110f5c
Latency min/avg/max: 0/0/12
Received: 1456
Sent : 1457
Connections: 1
Outstanding: 0
Zxid: 405
Mode: standalone
Node count: 54
Must be added; prevents snapshots from accumulating
© 2022 Altinity, Inc.
How do I migrate from ZooKeeper to ClickHouse Keeper?
Clickhouse-keeper-converter converts ZooKeeper logs and snapshots.
Procedure for migration:
1. Stop ZooKeeper ensemble.
2. Restart the ZooKeeper leader node to create a consistent snapshot.
3. Run clickhouse-keeper-converter
4. Copy to ClickHouse Keeper snapshot directory and start ClickHouse Keeper
Test the procedure carefully before applying to production systems.
41
© 2022 Altinity, Inc.
Is ClickHouse Keeper ready for prime time?
42
It’s getting there.
ClickHouse Keeper is much more convenient for developers
It fixes a number of known problems like ZKID overflow
There will be glitches but our experience is ‘so far, so good’
ClickHouse Keeper is ready for prod use on 22.3
© 2022 Altinity, Inc.
References
43
© 2022 Altinity, Inc.
List of references for more information
ZooKeeper Docs: https://zookeeper.apache.org/
ClickHouse Docs: https://clickhouse.com/docs/
Altinity Knowledge Base: https://kb.altinity.com/
Altinity Docs: https://docs.altinity.com
Alexander Sapin ClickHouse Keeper talk:
https://www.slideshare.net/Altinity/clickhouse-keeper
44
© 2022 Altinity, Inc.
Thank you!
Questions?
https://altinity.com
45
Copyright © Altinity Inc 2022

More Related Content

What's hot

High Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouseHigh Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouseAltinity Ltd
 
A Day in the Life of a ClickHouse Query Webinar Slides
A Day in the Life of a ClickHouse Query Webinar Slides A Day in the Life of a ClickHouse Query Webinar Slides
A Day in the Life of a ClickHouse Query Webinar Slides Altinity Ltd
 
ClickHouse Monitoring 101: What to monitor and how
ClickHouse Monitoring 101: What to monitor and howClickHouse Monitoring 101: What to monitor and how
ClickHouse Monitoring 101: What to monitor and howAltinity Ltd
 
Better than you think: Handling JSON data in ClickHouse
Better than you think: Handling JSON data in ClickHouseBetter than you think: Handling JSON data in ClickHouse
Better than you think: Handling JSON data in ClickHouseAltinity Ltd
 
ClickHouse on Kubernetes, by Alexander Zaitsev, Altinity CTO
ClickHouse on Kubernetes, by Alexander Zaitsev, Altinity CTOClickHouse on Kubernetes, by Alexander Zaitsev, Altinity CTO
ClickHouse on Kubernetes, by Alexander Zaitsev, Altinity CTOAltinity Ltd
 
Altinity Quickstart for ClickHouse
Altinity Quickstart for ClickHouseAltinity Quickstart for ClickHouse
Altinity Quickstart for ClickHouseAltinity Ltd
 
ClickHouse Deep Dive, by Aleksei Milovidov
ClickHouse Deep Dive, by Aleksei MilovidovClickHouse Deep Dive, by Aleksei Milovidov
ClickHouse Deep Dive, by Aleksei MilovidovAltinity Ltd
 
Analytics at Speed: Introduction to ClickHouse and Common Use Cases. By Mikha...
Analytics at Speed: Introduction to ClickHouse and Common Use Cases. By Mikha...Analytics at Speed: Introduction to ClickHouse and Common Use Cases. By Mikha...
Analytics at Speed: Introduction to ClickHouse and Common Use Cases. By Mikha...Altinity Ltd
 
ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...
ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...
ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...Altinity Ltd
 
Creating Beautiful Dashboards with Grafana and ClickHouse
Creating Beautiful Dashboards with Grafana and ClickHouseCreating Beautiful Dashboards with Grafana and ClickHouse
Creating Beautiful Dashboards with Grafana and ClickHouseAltinity Ltd
 
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEO
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEOClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEO
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEOAltinity Ltd
 
Evolution of MongoDB Replicaset and Its Best Practices
Evolution of MongoDB Replicaset and Its Best PracticesEvolution of MongoDB Replicaset and Its Best Practices
Evolution of MongoDB Replicaset and Its Best PracticesMydbops
 
Real-time Data Ingestion from Kafka to ClickHouse with Deterministic Re-tries...
Real-time Data Ingestion from Kafka to ClickHouse with Deterministic Re-tries...Real-time Data Ingestion from Kafka to ClickHouse with Deterministic Re-tries...
Real-time Data Ingestion from Kafka to ClickHouse with Deterministic Re-tries...HostedbyConfluent
 
Your first ClickHouse data warehouse
Your first ClickHouse data warehouseYour first ClickHouse data warehouse
Your first ClickHouse data warehouseAltinity Ltd
 
ClickHouse in Real Life. Case Studies and Best Practices, by Alexander Zaitsev
ClickHouse in Real Life. Case Studies and Best Practices, by Alexander ZaitsevClickHouse in Real Life. Case Studies and Best Practices, by Alexander Zaitsev
ClickHouse in Real Life. Case Studies and Best Practices, by Alexander ZaitsevAltinity Ltd
 
HTTP Analytics for 6M requests per second using ClickHouse, by Alexander Boc...
HTTP Analytics for 6M requests per second using ClickHouse, by  Alexander Boc...HTTP Analytics for 6M requests per second using ClickHouse, by  Alexander Boc...
HTTP Analytics for 6M requests per second using ClickHouse, by Alexander Boc...Altinity Ltd
 
ClickHouse Introduction by Alexander Zaitsev, Altinity CTO
ClickHouse Introduction by Alexander Zaitsev, Altinity CTOClickHouse Introduction by Alexander Zaitsev, Altinity CTO
ClickHouse Introduction by Alexander Zaitsev, Altinity CTOAltinity Ltd
 
Data Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UI
Data Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UIData Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UI
Data Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UIAltinity Ltd
 
Altinity Quickstart for ClickHouse-2202-09-15.pdf
Altinity Quickstart for ClickHouse-2202-09-15.pdfAltinity Quickstart for ClickHouse-2202-09-15.pdf
Altinity Quickstart for ClickHouse-2202-09-15.pdfAltinity Ltd
 
Evening out the uneven: dealing with skew in Flink
Evening out the uneven: dealing with skew in FlinkEvening out the uneven: dealing with skew in Flink
Evening out the uneven: dealing with skew in FlinkFlink Forward
 

What's hot (20)

High Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouseHigh Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouse
 
A Day in the Life of a ClickHouse Query Webinar Slides
A Day in the Life of a ClickHouse Query Webinar Slides A Day in the Life of a ClickHouse Query Webinar Slides
A Day in the Life of a ClickHouse Query Webinar Slides
 
ClickHouse Monitoring 101: What to monitor and how
ClickHouse Monitoring 101: What to monitor and howClickHouse Monitoring 101: What to monitor and how
ClickHouse Monitoring 101: What to monitor and how
 
Better than you think: Handling JSON data in ClickHouse
Better than you think: Handling JSON data in ClickHouseBetter than you think: Handling JSON data in ClickHouse
Better than you think: Handling JSON data in ClickHouse
 
ClickHouse on Kubernetes, by Alexander Zaitsev, Altinity CTO
ClickHouse on Kubernetes, by Alexander Zaitsev, Altinity CTOClickHouse on Kubernetes, by Alexander Zaitsev, Altinity CTO
ClickHouse on Kubernetes, by Alexander Zaitsev, Altinity CTO
 
Altinity Quickstart for ClickHouse
Altinity Quickstart for ClickHouseAltinity Quickstart for ClickHouse
Altinity Quickstart for ClickHouse
 
ClickHouse Deep Dive, by Aleksei Milovidov
ClickHouse Deep Dive, by Aleksei MilovidovClickHouse Deep Dive, by Aleksei Milovidov
ClickHouse Deep Dive, by Aleksei Milovidov
 
Analytics at Speed: Introduction to ClickHouse and Common Use Cases. By Mikha...
Analytics at Speed: Introduction to ClickHouse and Common Use Cases. By Mikha...Analytics at Speed: Introduction to ClickHouse and Common Use Cases. By Mikha...
Analytics at Speed: Introduction to ClickHouse and Common Use Cases. By Mikha...
 
ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...
ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...
ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...
 
Creating Beautiful Dashboards with Grafana and ClickHouse
Creating Beautiful Dashboards with Grafana and ClickHouseCreating Beautiful Dashboards with Grafana and ClickHouse
Creating Beautiful Dashboards with Grafana and ClickHouse
 
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEO
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEOClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEO
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEO
 
Evolution of MongoDB Replicaset and Its Best Practices
Evolution of MongoDB Replicaset and Its Best PracticesEvolution of MongoDB Replicaset and Its Best Practices
Evolution of MongoDB Replicaset and Its Best Practices
 
Real-time Data Ingestion from Kafka to ClickHouse with Deterministic Re-tries...
Real-time Data Ingestion from Kafka to ClickHouse with Deterministic Re-tries...Real-time Data Ingestion from Kafka to ClickHouse with Deterministic Re-tries...
Real-time Data Ingestion from Kafka to ClickHouse with Deterministic Re-tries...
 
Your first ClickHouse data warehouse
Your first ClickHouse data warehouseYour first ClickHouse data warehouse
Your first ClickHouse data warehouse
 
ClickHouse in Real Life. Case Studies and Best Practices, by Alexander Zaitsev
ClickHouse in Real Life. Case Studies and Best Practices, by Alexander ZaitsevClickHouse in Real Life. Case Studies and Best Practices, by Alexander Zaitsev
ClickHouse in Real Life. Case Studies and Best Practices, by Alexander Zaitsev
 
HTTP Analytics for 6M requests per second using ClickHouse, by Alexander Boc...
HTTP Analytics for 6M requests per second using ClickHouse, by  Alexander Boc...HTTP Analytics for 6M requests per second using ClickHouse, by  Alexander Boc...
HTTP Analytics for 6M requests per second using ClickHouse, by Alexander Boc...
 
ClickHouse Introduction by Alexander Zaitsev, Altinity CTO
ClickHouse Introduction by Alexander Zaitsev, Altinity CTOClickHouse Introduction by Alexander Zaitsev, Altinity CTO
ClickHouse Introduction by Alexander Zaitsev, Altinity CTO
 
Data Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UI
Data Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UIData Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UI
Data Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UI
 
Altinity Quickstart for ClickHouse-2202-09-15.pdf
Altinity Quickstart for ClickHouse-2202-09-15.pdfAltinity Quickstart for ClickHouse-2202-09-15.pdf
Altinity Quickstart for ClickHouse-2202-09-15.pdf
 
Evening out the uneven: dealing with skew in Flink
Evening out the uneven: dealing with skew in FlinkEvening out the uneven: dealing with skew in Flink
Evening out the uneven: dealing with skew in Flink
 

Similar to Everything You Need to Know About ZooKeeper and ClickHouse Keeper

Presto best practices for Cluster admins, data engineers and analysts
Presto best practices for Cluster admins, data engineers and analystsPresto best practices for Cluster admins, data engineers and analysts
Presto best practices for Cluster admins, data engineers and analystsShubham Tagra
 
Embedded Recipes 2019 - RT is about to make it to mainline. Now what?
Embedded Recipes 2019 - RT is about to make it to mainline. Now what?Embedded Recipes 2019 - RT is about to make it to mainline. Now what?
Embedded Recipes 2019 - RT is about to make it to mainline. Now what?Anne Nicolas
 
Kubecon seattle 2018 workshop slides
Kubecon seattle 2018 workshop slidesKubecon seattle 2018 workshop slides
Kubecon seattle 2018 workshop slidesWeaveworks
 
cloudera Apache Kudu Updatable Analytical Storage for Modern Data Platform
cloudera Apache Kudu Updatable Analytical Storage for Modern Data Platformcloudera Apache Kudu Updatable Analytical Storage for Modern Data Platform
cloudera Apache Kudu Updatable Analytical Storage for Modern Data PlatformRakuten Group, Inc.
 
Docker Logging and analysing with Elastic Stack - Jakub Hajek
Docker Logging and analysing with Elastic Stack - Jakub Hajek Docker Logging and analysing with Elastic Stack - Jakub Hajek
Docker Logging and analysing with Elastic Stack - Jakub Hajek PROIDEA
 
Docker Logging and analysing with Elastic Stack
Docker Logging and analysing with Elastic StackDocker Logging and analysing with Elastic Stack
Docker Logging and analysing with Elastic StackJakub Hajek
 
GLOC 2014 NEOOUG - Oracle Database 12c New Features
GLOC 2014 NEOOUG - Oracle Database 12c New FeaturesGLOC 2014 NEOOUG - Oracle Database 12c New Features
GLOC 2014 NEOOUG - Oracle Database 12c New FeaturesBiju Thomas
 
Containers with systemd-nspawn
Containers with systemd-nspawnContainers with systemd-nspawn
Containers with systemd-nspawnGábor Nyers
 
MySQL Database Monitoring: Must, Good and Nice to Have
MySQL Database Monitoring: Must, Good and Nice to HaveMySQL Database Monitoring: Must, Good and Nice to Have
MySQL Database Monitoring: Must, Good and Nice to HaveSveta Smirnova
 
Taming the Tiger: Tips and Tricks for Using Telegraf
Taming the Tiger: Tips and Tricks for Using TelegrafTaming the Tiger: Tips and Tricks for Using Telegraf
Taming the Tiger: Tips and Tricks for Using TelegrafInfluxData
 
Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...
Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...
Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...Altinity Ltd
 
Best practices for optimizing Red Hat platforms for large scale datacenter de...
Best practices for optimizing Red Hat platforms for large scale datacenter de...Best practices for optimizing Red Hat platforms for large scale datacenter de...
Best practices for optimizing Red Hat platforms for large scale datacenter de...Jeremy Eder
 
Deep Learning and Gene Computing Acceleration with Alluxio in Kubernetes
Deep Learning and Gene Computing Acceleration with Alluxio in KubernetesDeep Learning and Gene Computing Acceleration with Alluxio in Kubernetes
Deep Learning and Gene Computing Acceleration with Alluxio in KubernetesAlluxio, Inc.
 
LIMS_DOCUMENTATION
LIMS_DOCUMENTATIONLIMS_DOCUMENTATION
LIMS_DOCUMENTATIONRAHUL KUMAR
 
Setup oracle golden gate 11g replication
Setup oracle golden gate 11g replicationSetup oracle golden gate 11g replication
Setup oracle golden gate 11g replicationKanwar Batra
 
Apache Kudu: Technical Deep Dive


Apache Kudu: Technical Deep Dive

Apache Kudu: Technical Deep Dive


Apache Kudu: Technical Deep Dive

Cloudera, Inc.
 
Zoo keeper in the wild
Zoo keeper in the wildZoo keeper in the wild
Zoo keeper in the wilddatamantra
 
CoreOS, or How I Learned to Stop Worrying and Love Systemd
CoreOS, or How I Learned to Stop Worrying and Love SystemdCoreOS, or How I Learned to Stop Worrying and Love Systemd
CoreOS, or How I Learned to Stop Worrying and Love SystemdRichard Lister
 
Percona xtra db cluster(pxc) non blocking operations, what you need to know t...
Percona xtra db cluster(pxc) non blocking operations, what you need to know t...Percona xtra db cluster(pxc) non blocking operations, what you need to know t...
Percona xtra db cluster(pxc) non blocking operations, what you need to know t...Marco Tusa
 

Similar to Everything You Need to Know About ZooKeeper and ClickHouse Keeper (20)

Presto best practices for Cluster admins, data engineers and analysts
Presto best practices for Cluster admins, data engineers and analystsPresto best practices for Cluster admins, data engineers and analysts
Presto best practices for Cluster admins, data engineers and analysts
 
Embedded Recipes 2019 - RT is about to make it to mainline. Now what?
Embedded Recipes 2019 - RT is about to make it to mainline. Now what?Embedded Recipes 2019 - RT is about to make it to mainline. Now what?
Embedded Recipes 2019 - RT is about to make it to mainline. Now what?
 
Kubecon seattle 2018 workshop slides
Kubecon seattle 2018 workshop slidesKubecon seattle 2018 workshop slides
Kubecon seattle 2018 workshop slides
 
cloudera Apache Kudu Updatable Analytical Storage for Modern Data Platform
cloudera Apache Kudu Updatable Analytical Storage for Modern Data Platformcloudera Apache Kudu Updatable Analytical Storage for Modern Data Platform
cloudera Apache Kudu Updatable Analytical Storage for Modern Data Platform
 
Docker Logging and analysing with Elastic Stack - Jakub Hajek
Docker Logging and analysing with Elastic Stack - Jakub Hajek Docker Logging and analysing with Elastic Stack - Jakub Hajek
Docker Logging and analysing with Elastic Stack - Jakub Hajek
 
Docker Logging and analysing with Elastic Stack
Docker Logging and analysing with Elastic StackDocker Logging and analysing with Elastic Stack
Docker Logging and analysing with Elastic Stack
 
GLOC 2014 NEOOUG - Oracle Database 12c New Features
GLOC 2014 NEOOUG - Oracle Database 12c New FeaturesGLOC 2014 NEOOUG - Oracle Database 12c New Features
GLOC 2014 NEOOUG - Oracle Database 12c New Features
 
Containers with systemd-nspawn
Containers with systemd-nspawnContainers with systemd-nspawn
Containers with systemd-nspawn
 
MySQL Database Monitoring: Must, Good and Nice to Have
MySQL Database Monitoring: Must, Good and Nice to HaveMySQL Database Monitoring: Must, Good and Nice to Have
MySQL Database Monitoring: Must, Good and Nice to Have
 
Taming the Tiger: Tips and Tricks for Using Telegraf
Taming the Tiger: Tips and Tricks for Using TelegrafTaming the Tiger: Tips and Tricks for Using Telegraf
Taming the Tiger: Tips and Tricks for Using Telegraf
 
Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...
Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...
Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...
 
Best practices for optimizing Red Hat platforms for large scale datacenter de...
Best practices for optimizing Red Hat platforms for large scale datacenter de...Best practices for optimizing Red Hat platforms for large scale datacenter de...
Best practices for optimizing Red Hat platforms for large scale datacenter de...
 
Deep Learning and Gene Computing Acceleration with Alluxio in Kubernetes
Deep Learning and Gene Computing Acceleration with Alluxio in KubernetesDeep Learning and Gene Computing Acceleration with Alluxio in Kubernetes
Deep Learning and Gene Computing Acceleration with Alluxio in Kubernetes
 
LIMS_DOCUMENTATION
LIMS_DOCUMENTATIONLIMS_DOCUMENTATION
LIMS_DOCUMENTATION
 
Setup oracle golden gate 11g replication
Setup oracle golden gate 11g replicationSetup oracle golden gate 11g replication
Setup oracle golden gate 11g replication
 
Apache Kudu: Technical Deep Dive


Apache Kudu: Technical Deep Dive

Apache Kudu: Technical Deep Dive


Apache Kudu: Technical Deep Dive


 
Perf Tuning Short
Perf Tuning ShortPerf Tuning Short
Perf Tuning Short
 
Zoo keeper in the wild
Zoo keeper in the wildZoo keeper in the wild
Zoo keeper in the wild
 
CoreOS, or How I Learned to Stop Worrying and Love Systemd
CoreOS, or How I Learned to Stop Worrying and Love SystemdCoreOS, or How I Learned to Stop Worrying and Love Systemd
CoreOS, or How I Learned to Stop Worrying and Love Systemd
 
Percona xtra db cluster(pxc) non blocking operations, what you need to know t...
Percona xtra db cluster(pxc) non blocking operations, what you need to know t...Percona xtra db cluster(pxc) non blocking operations, what you need to know t...
Percona xtra db cluster(pxc) non blocking operations, what you need to know t...
 

More from Altinity Ltd

Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptxBuilding an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptxAltinity Ltd
 
Building an Analytic Extension to MySQL with ClickHouse and Open Source
Building an Analytic Extension to MySQL with ClickHouse and Open SourceBuilding an Analytic Extension to MySQL with ClickHouse and Open Source
Building an Analytic Extension to MySQL with ClickHouse and Open SourceAltinity Ltd
 
Fun with ClickHouse Window Functions-2021-08-19.pdf
Fun with ClickHouse Window Functions-2021-08-19.pdfFun with ClickHouse Window Functions-2021-08-19.pdf
Fun with ClickHouse Window Functions-2021-08-19.pdfAltinity Ltd
 
Cloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdf
Cloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdfCloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdf
Cloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdfAltinity Ltd
 
Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...
Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...
Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...Altinity Ltd
 
Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...
Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...
Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...Altinity Ltd
 
Own your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdf
Own your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdfOwn your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdf
Own your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdfAltinity Ltd
 
ClickHouse ReplacingMergeTree in Telecom Apps
ClickHouse ReplacingMergeTree in Telecom AppsClickHouse ReplacingMergeTree in Telecom Apps
ClickHouse ReplacingMergeTree in Telecom AppsAltinity Ltd
 
Adventures with the ClickHouse ReplacingMergeTree Engine
Adventures with the ClickHouse ReplacingMergeTree EngineAdventures with the ClickHouse ReplacingMergeTree Engine
Adventures with the ClickHouse ReplacingMergeTree EngineAltinity Ltd
 
Building a Real-Time Analytics Application with Apache Pulsar and Apache Pinot
Building a Real-Time Analytics Application with  Apache Pulsar and Apache PinotBuilding a Real-Time Analytics Application with  Apache Pulsar and Apache Pinot
Building a Real-Time Analytics Application with Apache Pulsar and Apache PinotAltinity Ltd
 
Altinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdf
Altinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdfAltinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdf
Altinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdfAltinity Ltd
 
OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...
OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...
OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...Altinity Ltd
 
OSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdf
OSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdfOSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdf
OSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdfAltinity Ltd
 
OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...
OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...
OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...Altinity Ltd
 
OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...
OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...
OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...Altinity Ltd
 
OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...
OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...
OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...Altinity Ltd
 
OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...
OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...
OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...Altinity Ltd
 
OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...
OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...
OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...Altinity Ltd
 
OSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdf
OSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdfOSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdf
OSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdfAltinity Ltd
 
OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...
OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...
OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...Altinity Ltd
 

More from Altinity Ltd (20)

Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptxBuilding an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
 
Building an Analytic Extension to MySQL with ClickHouse and Open Source
Building an Analytic Extension to MySQL with ClickHouse and Open SourceBuilding an Analytic Extension to MySQL with ClickHouse and Open Source
Building an Analytic Extension to MySQL with ClickHouse and Open Source
 
Fun with ClickHouse Window Functions-2021-08-19.pdf
Fun with ClickHouse Window Functions-2021-08-19.pdfFun with ClickHouse Window Functions-2021-08-19.pdf
Fun with ClickHouse Window Functions-2021-08-19.pdf
 
Cloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdf
Cloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdfCloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdf
Cloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdf
 
Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...
Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...
Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...
 
Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...
Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...
Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...
 
Own your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdf
Own your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdfOwn your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdf
Own your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdf
 
ClickHouse ReplacingMergeTree in Telecom Apps
ClickHouse ReplacingMergeTree in Telecom AppsClickHouse ReplacingMergeTree in Telecom Apps
ClickHouse ReplacingMergeTree in Telecom Apps
 
Adventures with the ClickHouse ReplacingMergeTree Engine
Adventures with the ClickHouse ReplacingMergeTree EngineAdventures with the ClickHouse ReplacingMergeTree Engine
Adventures with the ClickHouse ReplacingMergeTree Engine
 
Building a Real-Time Analytics Application with Apache Pulsar and Apache Pinot
Building a Real-Time Analytics Application with  Apache Pulsar and Apache PinotBuilding a Real-Time Analytics Application with  Apache Pulsar and Apache Pinot
Building a Real-Time Analytics Application with Apache Pulsar and Apache Pinot
 
Altinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdf
Altinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdfAltinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdf
Altinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdf
 
OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...
OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...
OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...
 
OSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdf
OSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdfOSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdf
OSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdf
 
OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...
OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...
OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...
 
OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...
OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...
OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...
 
OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...
OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...
OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...
 
OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...
OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...
OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...
 
OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...
OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...
OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...
 
OSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdf
OSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdfOSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdf
OSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdf
 
OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...
OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...
OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...
 

Recently uploaded

why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...Jack Cole
 
Digital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing worksDigital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing worksdeepakthakur548787
 
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...Boston Institute of Analytics
 
DATA ANALYSIS using various data sets like shoping data set etc
DATA ANALYSIS using various data sets like shoping data set etcDATA ANALYSIS using various data sets like shoping data set etc
DATA ANALYSIS using various data sets like shoping data set etclalithasri22
 
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdfEnglish-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdfblazblazml
 
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis model
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis modelDecoding Movie Sentiments: Analyzing Reviews with Data Analysis model
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis modelBoston Institute of Analytics
 
Role of Consumer Insights in business transformation
Role of Consumer Insights in business transformationRole of Consumer Insights in business transformation
Role of Consumer Insights in business transformationAnnie Melnic
 
Presentation of project of business person who are success
Presentation of project of business person who are successPresentation of project of business person who are success
Presentation of project of business person who are successPratikSingh115843
 
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...Dr Arash Najmaei ( Phd., MBA, BSc)
 
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBoston Institute of Analytics
 
Statistics For Management by Richard I. Levin 8ed.pdf
Statistics For Management by Richard I. Levin 8ed.pdfStatistics For Management by Richard I. Levin 8ed.pdf
Statistics For Management by Richard I. Levin 8ed.pdfnikeshsingh56
 
IBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaIBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaManalVerma4
 
Non Text Magic Studio Magic Design for Presentations L&P.pdf
Non Text Magic Studio Magic Design for Presentations L&P.pdfNon Text Magic Studio Magic Design for Presentations L&P.pdf
Non Text Magic Studio Magic Design for Presentations L&P.pdfPratikPatil591646
 
Digital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdfDigital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdfNicoChristianSunaryo
 

Recently uploaded (17)

Insurance Churn Prediction Data Analysis Project
Insurance Churn Prediction Data Analysis ProjectInsurance Churn Prediction Data Analysis Project
Insurance Churn Prediction Data Analysis Project
 
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
 
Digital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing worksDigital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing works
 
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
 
DATA ANALYSIS using various data sets like shoping data set etc
DATA ANALYSIS using various data sets like shoping data set etcDATA ANALYSIS using various data sets like shoping data set etc
DATA ANALYSIS using various data sets like shoping data set etc
 
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdfEnglish-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
 
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis model
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis modelDecoding Movie Sentiments: Analyzing Reviews with Data Analysis model
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis model
 
Role of Consumer Insights in business transformation
Role of Consumer Insights in business transformationRole of Consumer Insights in business transformation
Role of Consumer Insights in business transformation
 
Presentation of project of business person who are success
Presentation of project of business person who are successPresentation of project of business person who are success
Presentation of project of business person who are success
 
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...
 
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
 
Statistics For Management by Richard I. Levin 8ed.pdf
Statistics For Management by Richard I. Levin 8ed.pdfStatistics For Management by Richard I. Levin 8ed.pdf
Statistics For Management by Richard I. Levin 8ed.pdf
 
Data Analysis Project: Stroke Prediction
Data Analysis Project: Stroke PredictionData Analysis Project: Stroke Prediction
Data Analysis Project: Stroke Prediction
 
IBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaIBEF report on the Insurance market in India
IBEF report on the Insurance market in India
 
Non Text Magic Studio Magic Design for Presentations L&P.pdf
Non Text Magic Studio Magic Design for Presentations L&P.pdfNon Text Magic Studio Magic Design for Presentations L&P.pdf
Non Text Magic Studio Magic Design for Presentations L&P.pdf
 
Digital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdfDigital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdf
 
2023 Survey Shows Dip in High School E-Cigarette Use
2023 Survey Shows Dip in High School E-Cigarette Use2023 Survey Shows Dip in High School E-Cigarette Use
2023 Survey Shows Dip in High School E-Cigarette Use
 

Everything You Need to Know About ZooKeeper and ClickHouse Keeper

  • 1. © 2022 Altinity, Inc. All about ZooKeeper (and ClickHouse Keeper, too!) Robert Hodges and Altinity Engineering 30 March 2022 1 Copyright © Altinity Inc 2022
  • 2. © 2022 Altinity, Inc. Let’s make some introductions ClickHouse support and services including Altinity.Cloud Authors of Altinity Kubernetes Operator for ClickHouse and other open source projects Robert Hodges Database geek with 30+ years on DBMS systems. Day job: Altinity CEO Altinity Engineering Database geeks with centuries of experience in DBMS and applications 2
  • 3. © 2022 Altinity, Inc. Why does ClickHouse need ZooKeeper? 3
  • 4. © 2022 Altinity, Inc. Horizontal scaling is a key to ClickHouse performance 4 clickhouse-0 events _local Sharded, replicated table clickhouse-1 events _local clickhouse-2 events _local clickhouse-3 events _local
  • 5. © 2022 Altinity, Inc. Let’s create the table and try it out! 5 CREATE TABLE IF NOT EXISTS `events_local` ON CLUSTER '{cluster}' ( EventDate DateTime, CounterID UInt32, Value String ) Engine=ReplicatedMergeTree( '/clickhouse/{cluster}/tables/{shard}/{database}/events_local', '{replica}') PARTITION BY toYYYYMM(EventDate) ORDER BY (CounterID, EventDate, intHash32(UserID)) INSERT INTO events_local(EventDate, EventID, Value) VALUES (now(), 1, 'In-Progress'), (now(), 2, 'OK') . . .
  • 6. © 2022 Altinity, Inc. What could possibly go wrong? 6 ON CLUSTER command failed on one node! Two replicas merge overlapping parts! Node offline for maintenance; missed the memo! Two replicas delete overlapping parts! clickhouse-0 events _local Sharded, replicated table clickhouse-1 events _local clickhouse-2 events _local clickhouse-3 events _local
  • 7. © 2022 Altinity, Inc. ZooKeeper solves the distributed consistency problem 7 zookeeper-0 zookeeper-2 zookeeper-1 clickhouse-0 events _local Sharded, replicated table clickhouse-1 events _local clickhouse-2 events _local clickhouse-3 events _local Consistent ON CLUSTER commands, replicated table parts
  • 8. © 2022 Altinity, Inc. How ZooKeeper Works 8
  • 9. © 2022 Altinity, Inc. ZooKeeper Architecture 9 ZooKeeper Ensemble zookeeper-0 zookeeper-2 zookeeper-1 ClickHouse Server ClickHouse Server ClickHouse Server ClickHouse Server ZooKeeper Clients ZooKeeper Servers on Port :2181 ZooKeeper Atomic Broadcast (ZAB)
  • 10. © 2022 Altinity, Inc. ZooKeeper leaders and followers 10 zookeeper-0 zookeeper-2 zookeeper-1 Leader coordinates writes Follower Follower Leader Followers handle reads and delegate writes to leader Write request Read request Leader must maintain a quorum of followers
  • 11. © 2022 Altinity, Inc. ZooKeeper directory structure for ClickHouse 11 / zookeeper clickhouse first tables task_queue 0 default log events_local quorum replicas query query-0000 000000 DROP TABLE IF EXISTS default.events_local ON CLUSTER first Znode . . .
  • 12. © 2022 Altinity, Inc. What kind of ClickHouse information is stored in znodes? ● Tasks ○ Pending and completed ON CLUSTER DDL commands ● Table information ○ Schema information ○ Replicas ○ Leader elections used to control merges and mutations ○ Log of operations on the table (insert, merge, delete partition, etc.) ○ Parts contained in each replica ○ Last N blocks inserts so we can deduplicate data ○ Data to ensure quorum on writes 12
  • 13. © 2022 Altinity, Inc. Installing and configuring ZooKeeper 13
  • 14. © 2022 Altinity, Inc. Installing a ZooKeeper on Ubuntu 14 sudo apt update sudo apt install zookeeper netcat (edit /etc/sysconfig/config/zoo.cfg to set configuration) Install Zookeeper 3.4.9 or greater
  • 15. © 2022 Altinity, Inc. Ensuring ZooKeeper maximum speed and availability Host recommendations ● Dedicated host for ZooKeepers - don’t share with other applications ● Put ZooKeeper log on dedicated SSD ● Low network latency between ZooKeeper nodes ● At least 4GiB of RAM ● Disable swap (remove entry from /etc/fstab) ● Tune the Java heap to use as much RAM as possible ○ E.g., 3GiB out of 4GiB available RAM 15
  • 16. © 2022 Altinity, Inc. ZooKeeper moving parts 16 ZooKeeper Java Process dataLogDir zoo.cfg dataDir Configuration file Logs with changes to data In-RAM copy of Znode Tree Snapshots of data myid Server instance ID (1-255)
  • 17. © 2022 Altinity, Inc. Editing important zoo.cfg settings 17 … autopurge.purgeInterval=1 autopurge.snapRetainCount=5 … server.1=zookeeper1:2888:3888 server.2=zookeeper2:2888:3888 server.3=zookeeper3:2888:3888 … dataDir=/var/lib/zookeeper … dataLogDir=/ssd/zookeeper/logs Must be added; prevents snapshots from accumulating Servers in ensemble; must be identical everywhere Location for snapshots Put logs on fast storage
  • 18. © 2022 Altinity, Inc. Starting ZooKeeper and ensuring it’s up 18 sudo -u zookeeper /usr/share/zookeeper/bin/zkServer.sh ZooKeeper JMX enabled by default Using config: /etc/zookeeper/conf/zoo.cfg Starting zookeeper ... STARTED echo ruok | nc localhost 2181 imok echo mntr | nc localhost 2181 zk_version 3.4.10-3--1, built on Sat, 03 Feb 2018 14:58:02 -0800 . . . echo stat | nc localhost 2181 zk_version 3.4.10-3--1, built on Sat, 03 Feb 2018 14:58:02 -0800 . . .
  • 19. © 2022 Altinity, Inc. Tell ClickHouse where ZooKeeper lives 19 <yandex> <zookeeper> <node> <host>zookeeper.zoo1ns</host> <port>2181</port> </node> </zookeeper> <distributed_ddl> <path>/clickhouse/first/task_queue/ddl</path> </distributed_ddl> </yandex>
  • 20. © 2022 Altinity, Inc. Add macros so ON CLUSTER commands can run 20 <yandex> <macros> <installation>first</installation> <all-sharded-shard>0</all-sharded-shard> <cluster>first</cluster> <shard>0</shard> <replica>chi-first-first-0-0</replica> </macros> </yandex>
  • 21. © 2022 Altinity, Inc. Practical Administration Tips 21
  • 22. © 2022 Altinity, Inc. How many ZooKeepers are enough? 22 1 2 5 4 3 OK for dev Less available than 1 replica! Best for prod OK but writes may be slower Less available than 3 replicas!
  • 23. © 2022 Altinity, Inc. What’s in ZooKeeper? The system.zookeeper table knows! 23 SELECT * FROM system.zookeeper WHERE path = '/' ORDER BY name FORMAT Vertical Row 1: ────── name: clickhouse value: czxid: 4294967298 mzxid: 4294967298 ctime: 2021-12-08 01:54:50 mtime: 2021-12-08 01:54:50 . . . path: / Path value is required! If this query works, ClickHouse can see ZooKeeper!
  • 24. © 2022 Altinity, Inc. Printing znode values from system.zookeeper 24 SELECT name, value FROM system.zookeeper WHERE path = '/clickhouse/first/task_queue/ddl/' FORMAT Vertical Row 1: ────── name: query-0000000009 value: version: 1 query: CREATE TABLE IF NOT EXISTS default.events_local UUID '2a8ed83e-a6ef-48b4-aa8e-d83ea6efa8b4' ON CLUSTER first (`EventDate` DateTime, `EventID` UInt32, `Value` String) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{shard}/{database}/even ts_local', '{replica}') PARTITION BY toYYYYMM(EventDate) ORDER BY (CounterID, EventDate, intHash32(UserID)) hosts: . . . Prints values for znodes under this path
  • 25. © 2022 Altinity, Inc. Using the zkCli utility to talk to ZooKeeper directly 25 (Connect to ZooKeeper host) $ zkCli.sh Connecting to localhost:2181 . . . [zk: localhost:2181(CONNECTED) 0] ls / [clickhouse, zookeeper] [zk: localhost:2181(CONNECTED) 1] get /clickhouse/first/task_queue/ddl/query-0000000009 version: 1 query: CREATE TABLE IF NOT EXISTS default.events_local UUID '2a8ed83e-a6ef-48b4-aa8e-d83ea6efa8b4' ON CLUSTER first . . .
  • 26. © 2022 Altinity, Inc. ZooKeeper four letter word commands 26 Example: echo ruok | nc localhost 2181 → imok Command What it does ruok Check server liveness conf Print server config cons Print connections mntr Dump monitoring information srvr Dump server information There are more commands! Check the docs.
  • 27. © 2022 Altinity, Inc. ZooKeeper Monitoring Older approach for Nagios and Icinga[2] ● Use check_zookeeper.pl Newer approach: Use Prometheus + AlertManager + Grafana ● ZooKeeper by Prometheus Dashboard for Grafana The Altinity Knowledge Base has a page on ZooKeeper Monitoring 27
  • 28. © 2022 Altinity, Inc. The dreaded read-only table error 28 INSERT INTO events2_local (EventDate, EventID, Value) VALUES (now(), 1, 'In-Progress'), (now(), 2, 'OK') Received exception from server (version 21.8.10): Code: 242. DB::Exception: Received from 34.83.194.130:9000. DB::Exception: Table is in readonly mode (zookeeper path: /clickhouse/first/tables/0/default/events2_local). (TABLE_IS_READ_ONLY) Must be added; prevents snapshots from accumulating ZooKeeper is offline!
  • 29. © 2022 Altinity, Inc. Steps to address read-only tables 29 ClickHouse ZooKeeper SELECT * FROM system.zookeeper WHERE path = '/' echo ruok | nc localhost 2181 Log Loss of quorum? ZXID Rollover? Other errors? /var/log/zookeeper/zookeeper.log Can ClickHouse see ZooKeeper? Is ZooKeeper alive? ZooKeeper Connection
  • 30. © 2022 Altinity, Inc. ZooKeeper “Session Expired” errors If ClickHouse loses its connection to ZooKeeper, pending INSERTs or ON CLUSTER commands may fail with a Session Expired error. 1. Occasional failure is normal in distributed systems. Retry the operation!! 2. If the problem happens commonly, you may have a ZooKeeper problem. a. Check ZooKeeper logs for errors b. This could be an ZXID overflow due to too many transactions on ZooKeeper. Check that only ClickHouse is using ZooKeeper! c. Too many parts in the table? (> 5000) d. Jute.maxbuffer seting on ZooKeeper is too low. 30
  • 31. © 2022 Altinity, Inc. Recovering from failures Loss of a single ZooKeeper node 1. Create fresh node with same ZooKeeper instance ID as lost node 2. Ensure new host name is correct in all zoo.cfg files 3. Start new node Loss of entire ZooKeeper ensemble 1. Briefly consider taking an immediate vacation 2. Bring up new ZooKeeper ensemble 3. Use SYSTEM RESTORE REPLICA command to restore metadata from ClickHouse server(s) 31
  • 32. © 2022 Altinity, Inc. ClickHouse Keeper 32
  • 33. © 2022 Altinity, Inc. So…What is ClickHouse Keeper? 33 It’s a from-scratch reimplementation of ZooKeeper ● Mimics ZooKeeper API and admin commands ● Uses Raft protocol instead of ZAB for consensus ● Is written in C++ ● Is part of ClickHouse No extra installation required!
  • 34. © 2022 Altinity, Inc. Why replace ZooKeeper? 34 ● ClickHouse should contain everything it needs to run ● Old, not very actively developed ● Java executable adds dependencies and requires tuning ● Many people find it hard to operate ● Problems like ZXID rollover, uncompressed logs, etc.
  • 35. © 2022 Altinity, Inc. ClickHouse Keeper can be a drop-in ZK replacement… 35 ch-keeper-0 ch-keeper-2 ch-keeper-1 clickhouse-0 events _local Sharded, replicated table clickhouse-1 events _local clickhouse-2 events _local clickhouse-3 events _local Consistent ON CLUSTER commands, replicated table parts
  • 36. © 2022 Altinity, Inc. Or it can run directly in ClickHouse itself! 36 “Sharded”, replicated table clickhouse-0 events _local keeper-1 clickhouse-0 events _local keeper-1 clickhouse-1 events _local keeper-2 clickhouse-2 events _local keeper-3
  • 37. © 2022 Altinity, Inc. ClickHouse Keeper single node configuration 37 <yandex> <keeper_server incl="keeper_server"> <server_id>1</server_id> <tcp_port>2181</tcp_port> <coordination_settings> <raft_logs_level>debug</raft_logs_level> </coordination_settings> <raft_configuration> <server> <id>1</id> <hostname>logos3</hostname><port>9444</port> </server> </raft_configuration> </keeper_server> </yandex>
  • 38. © 2022 Altinity, Inc. ClickHouse Keeper moving parts for single node install 38 ClickHouse Keeper logs keeper-config.xml snapshots Configuration file Transaction logs In-RAM copy of Znode Tree Snapshots /etc/clickhouse-server/config.d /var/lib/clickhouse/coordination ClickHouse
  • 39. © 2022 Altinity, Inc. ClickHouse Keeper “just works” 1. ON CLUSTER commands and replication work exactly as before 2. System.zookeeper table shows directory structure 3. ZooKeeper four letter commands work 4. You can use zkCli.sh (and other tools) to navigate the directory structure 39
  • 40. © 2022 Altinity, Inc. How to tell you are using ClickHouse Keeper 40 $ echo srvr |netcat logos3 2181 ClickHouse Keeper version: v22.3.2.1-prestable-92ab33f560e638d1989c5ca543021ab53d110f5c Latency min/avg/max: 0/0/12 Received: 1456 Sent : 1457 Connections: 1 Outstanding: 0 Zxid: 405 Mode: standalone Node count: 54 Must be added; prevents snapshots from accumulating
  • 41. © 2022 Altinity, Inc. How do I migrate from ZooKeeper to ClickHouse Keeper? Clickhouse-keeper-converter converts ZooKeeper logs and snapshots. Procedure for migration: 1. Stop ZooKeeper ensemble. 2. Restart the ZooKeeper leader node to create a consistent snapshot. 3. Run clickhouse-keeper-converter 4. Copy to ClickHouse Keeper snapshot directory and start ClickHouse Keeper Test the procedure carefully before applying to production systems. 41
  • 42. © 2022 Altinity, Inc. Is ClickHouse Keeper ready for prime time? 42 It’s getting there. ClickHouse Keeper is much more convenient for developers It fixes a number of known problems like ZKID overflow There will be glitches but our experience is ‘so far, so good’ ClickHouse Keeper is ready for prod use on 22.3
  • 43. © 2022 Altinity, Inc. References 43
  • 44. © 2022 Altinity, Inc. List of references for more information ZooKeeper Docs: https://zookeeper.apache.org/ ClickHouse Docs: https://clickhouse.com/docs/ Altinity Knowledge Base: https://kb.altinity.com/ Altinity Docs: https://docs.altinity.com Alexander Sapin ClickHouse Keeper talk: https://www.slideshare.net/Altinity/clickhouse-keeper 44
  • 45. © 2022 Altinity, Inc. Thank you! Questions? https://altinity.com 45 Copyright © Altinity Inc 2022