Building ClickHouse and Making Your First Contribution: A Tutorial_06.10.2021

ClickHouse 101
Building ClickHouse and Making Your First
Contribution: A Tutorial
Vasily Nemkov @ Altinity 06.10.2021

About me
Server team lead @ Altinity
working on CH for ~3 years now
~70 merged PRs in CH
IPv4 and IPv6 datatype
Gorilla and DoubleDelta codecs
DateTime64 datatype + extended range for date/time*
system.session_log
+ many others
Building ClickHouse and Making Your First Contribution: A Tutorial by Vasily Nemkov @ Altinity 06.10.2021
2

About Altinity
The #1 enterprise ClickHouse provider

Now offering Altinity.Cloud

Major committer and community sponsor for ClickHouse in US/EU
3

Overview
What CH is
How to
Clone
Make changes
Build
Run tests
Submit a PR
4

What ClickHouse is
an OLAP DBMS [0][1]
Open source software [2]
fastest growing open source project now
highly optimized modern C++
~0.5M LOC (excluding third-party)
80-ish third-party libraries
Lots of tests
Docs are in the repo [3]
5

Why contribute
Fastest way to have a bug fixed or feature implemented
Learn a lot in the process
Fame
system.contributors
ClickHouse merged contributors badge
Arctic Code Vault Contributor [4]
6

Ways to contribute
Benchmarks
Docs
Tests
Code [5]
7

The Flow
Review Release
Development
8

Development
Review Release
Development
clone
change
build
test
commit & push
9

Clone
Review Release
Development
clone
change
build
test
commit & push
10

Clone
Make a fork (if you are going to contribute changes to the community)
Clone repo and fetch submodules
$ cd ~
$ git clone https://github.com/ClickHouse/ClickHouse.git ClickHouse
$ cd ClickHouse
$ git submodule update --init --recursive
11

Make changes
Review Release
Development
clone
change
build
test
commit & push
12

Make changes - complexity hierarchy
[7]
Functions: regular, aggregate,
table-function
Data format
Table Engine
Codec ( None , LZ4 , DoubleDelta )
SQL feature: clause ( UNION ALL ),
section ( LIMIT BY ), request type (
SYSTEM )
DB engine
Dictionaries: layout, source
Data type ( UUID )
Client/server protocol
13

Make changes
We are going to add a helloWorld() function
SELECT helloWorld();
"Hello World!"
Follow guides [8]
$ vim ~/ClickHouse/src/Functions/helloWorld.cpp
14

Make changes
#include <DataTypes/DataTypeString.h>
#include <Functions/IFunction.h>
#include <Functions/FunctionFactory.h>
#include <Core/Field.h>
namespace DB
{
class FunctionHelloWorld : public IFunction
{
public:
static constexpr auto name = "helloWorld";
static FunctionPtr create(ContextPtr) { return std::make_shared<FunctionHelloWorld>(); }
String getName() const override { return name; }
bool useDefaultImplementationForConstants() const override { return true; }
bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return false; }
size_t getNumberOfArguments() const override { return 0; }
DataTypePtr getReturnTypeImpl(const DataTypes &) const override { return std::make_shared<DataTypeString>(); }
ColumnPtr executeImpl(const ColumnsWithTypeAndName &, const DataTypePtr & result_type, size_t input_rows_count) const override
{
return result_type->createColumnConst(input_rows_count, "Hello World!");
}
};
void registerFunctionHelloWrold(FunctionFactory & factory)
{
factory.registerFunction<FunctionHelloWorld>();
}
}
15

Make changes
diff --git a/src/Functions/registerFunctions.cpp b/src/Functions/registerFunctions.cpp
index 9b1a7faebb..0451ff0ead 100644
--- a/src/Functions/registerFunctions.cpp
+++ b/src/Functions/registerFunctions.cpp
@@ -68,2 +68,5 @@ void registerFunctionAESDecryptMysql(FunctionFactory & factory);
+void registerFunctionHelloWorld(FunctionFactory & factory);
+
void registerFunctions()
@@ -132,2 +135,3 @@ void registerFunctions()
registerFunctionLogTrace(factory);
+ registerFunctionHelloWorld(factory);
}
16

Build
Review Release
Development
clone
change
build
test
commit & push
17

Build
Build in Docker
Can produce .deb , .rpm and .tar.gz
Easy to set up - no prerequisites, only docker
any OS
Build on Host
More suitable for development process (i.e. developing CH core)
Ubuntu, Prerequisites
faster (~2x)
18

Build - Hardware Requirements
16GB of RAM (32 is better)
multiple cores (4+)
20-50GB on disk
19

Build in Docker
$ docker run
--network=host
--rm
--volume=$(realpath ./PACKAGES):/output
--volume=$(realpath .):/build
-e DEB_CC=clang-11
-e DEB_CXX=clang++-11
-e ALIEN_PKGS='--rpm --tgz'
-e CMAKE_FLAGS="$CMAKE_FLAGS -DADD_GDB_INDEX_FOR_GOLD=1"
clickhouse/clickhouse-deb-builder
20

Build in Docker
21

Build in Docker
$ cd ~/ClickHouse/PACKAGE
$ ls *.deb
clickhouse-client_21.11.1_all.deb clickhouse-common-static-dbg_21.11.1_amd64.deb
clickhouse-common-static_21.11.1_amd64.deb clickhouse-server_21.11.1_all.deb
clickhouse-test_21.11.1_all.deb
$ ls *.rpm
clickhouse-client-21.11.1-2.noarch.rpm clickhouse-common-static-21.11.1-2.x86_64.rpm
clickhouse-common-static-dbg-21.11.1-2.x86_64.rpm clickhouse-server-21.11.1-2.noarch.rpm
clickhouse-test-21.11.1-2.noarch.rpm
$ ls *.tgz
clickhouse-client-21.11.1.tgz clickhouse-common-static-21.11.1.tgz
clickhouse-common-static-dbg-21.11.1.tgz clickhouse-server-21.11.1.tgz
clickhouse-test-21.11.1.tgz
$ ls *.buildinfo *.changes
clickhouse_21.11.1_amd64.buildinfo clickhouse_21.11.1_amd64.changes
22

Build on Host - Prerequisites
for Ubuntu 20.04:
$ apt-get install
clang-11
clang-tidy-11
cmake
lld-11
llvm-11
llvm-11-dev
ninja-build
tzdata
--yes --no-install-recommends
23

Build on Host - Building
$ mkdir ~/build
$ cd ~/build
$ CC=clang-11 CXX=clang++-11 cmake ~/ClickHouse -GNinja
$ ninja clickhouse
24

~80 mins
25

$ ls ~/build/programs/
CMakeFiles benchmark cmake_install.cmake
extract-from-config install library-bridge
odbc-bridge CTestTestfile.cmake **clickhouse**
compressor format keeper
local server bash-completion
client copier git-import
keeper-converter obfuscator static-files-disk-uploader
26

Test
Review Release
Development
clone
change
build
test
commit & push
27

Test - Smoke-test
$ cd ~/ClickHouse/programs/server
$ ~/build/programs/clickhouse server
$ ~/build/programs/clickhouse client
ClickHouse client version 21.11.1.1.
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 21.11.1 revision 54449.
Warnings:
* Server was built in debug mode. It will work slowly.
:)
28

Test - Smoke-test
:) SELECT version()
SELECT version()
Query id: 6f8b1cd8-f5d9-4f49-87ce-125ea27e08f8
┌─version()─┐
│ 21.11.1.1 │
└───────────┘
1 rows in set. Elapsed: 0.008 sec.
29

Tests
Executed on CI, but can also be run locally

There are multiple ways to validate a build [6]
Unit-tests - src/.../tests/gtest_*.cpp ~ 81 files ~ 10K test (4 min)
Stateless* - tests/queries/0_stateless ~ 3424 tests (45min)
Integration* - tests/integration/test_x/test.py ~ 276 tests
Performance* - tests/performance/*.xml ~ 220 tests
Lots of others

* either of those must be present in PR
30

Test - add test
$ cat ~/ClickHouse/tests/queries/0_stateless/02030_hello_world.sql
SELECT helloWorld() as r, toTypeName(r);
$ cat ~/ClickHouse/tests/queries/0_stateless/02030_hello_world.reference
Hello World! String
31

Test - run test
$ cd ~/ClickHouse/tests
$ # Requires server accessible via port 9000
$ ./clickhouse-test -b ~/build/programs/server/clickhouse 02030_hello_world
WARNING: jinja2 not installed! Template tests will be skipped.
Using queries from 'queries' directory
Using ~/ClickHouse/programs/server/clickhouse as client program (expecting split build)
Connecting to ClickHouse server... OK
Running 1 stateless tests (MainProcess).
02030_hello_world: [ OK ]
1 tests passed. 0 tests skipped. 1.85 s elapsed (MainProcess).
Won't run stateful tests because test data wasn't loaded.
All tests have finished.
32

Commit & Push
Review Release
Development
clone
change
build
test
commit & push
33

Commit & Push
$ git checkout -b hello_world
$ git add src/Functions/helloWorld.cpp
src/Functions/registerFunctions.cpp
tests/queries/0_stateless/02030_hello_world.sql
tests/queries/0_stateless/02030_hello_world.reference
$ git commit
$ git push enmk HEAD #*
* enmk - name of the remote, you can add a remote after cloning a repo
with git remote add
34

Review
Review Release
Development
description
CLA
fix checks
fix requests
merge to master
35

Review
Pushed a review:
https://github.com/ClickHouse/ClickHouse/pull/29800
36

Review - Description
Review Release
Development
description
CLA
fix checks
fix requests
merge to master
37

Changelog category (leave one):
- New Feature
- Improvement
- Bug Fix (user-visible misbehaviour in official stable or prestable release)
- Performance Improvement
- Backward Incompatible Change
- Build/Testing/Packaging Improvement
- Documentation (changelog entry is not required)
- Not for changelog (changelog entry is not required)
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
...
Detailed description / Documentation draft:
...
38

Changelog category (leave one):
- New Feature
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Added `helloWorld` function that outputs a "Hello World!"
39

Review - CLA
Review Release
Development
description
CLA
fix checks
fix requests
merge to master
40

Review - CLA
41

Review - checks
Review Release
Development
description
CLA
fix checks
fix requests
merge to master
42

Review - checks
There are ~50 checks [9], roughly grouped as:
style checks
fasttest
builds
stateless
Unit tests
statefull
integration
sanitizers
performance
AST Fuzzer
Stress tests
TestFlows tests
PVS Checks
43

Review - checks
44

Review - check page - Fasttest
45

Review - check page - Stateless
46

Review - check page - Integration
47

Review - fix change requests
Review Release
Development
description
CLA
fix checks
fix requests
merge to master
48

Review - Merge
Review Release
Development
description
CLA
fix checks
fix requests
merge to master
49

Review - Merge
50

Release
Review Release
Development
new release
backported
51

References
[0]: clickhouse.com

[1]: altinity.com

[2]: github.com/ClickHouse/ClickHouse

[3]: clickhouse.com/docs
[4]: archiveprogram.github.com

[5]: clickhouse.com/docs/en/development/developer-instruction

[6]: clickhouse.com/docs/en/development/tests

[7]: youtu.be/g5CDi8Nbut4?t=3669
[8]: clickhouse.com/docs/en/development/style/

[9]: clickhouse.com/docs/en/development/continuous-integration/
52

BTW, we are hiring!
altinity.com/careers
altinity.com/job/clickhouse-server-engineer
53

Questions?
54

!!! BONUS PAGES !!!
Examples of how to the execute most important tests on your machine
Unit-tests
Stateless
Integration
Performance
55

Tests - Unit-tests
src/.../tests/gtest_*.cpp ~ 81 files
cover specific isolated thing
fast to execute
GTest-based
easy to debug
not so easy to write
56

Tests - Unit-tests - run on Host
Need to build a test binary first
$ cd ~/build
$ ninja unit_tests_dbms
Run the tests
$ ~/build/src/unit_tests_dbms --gtest_filter="*GTEST-FILTER-CLAUSE*"
# $ ~/build/src/unit_tests_dbms --gtest_filter="*TokenExtractorTest*"
57

Tests - Stateless
tests/queries/0_stateless ~ 3424 tests
functional test
require running ClickHouse server
executed by tests/clickhouse-test
easy to write, can be written in either SQL , bash , or Python
require .reference file
simple to run
58

Tests - Stateless - run in Docker
$ docker run
--rm
-v $(realpath ~/ClickHouse/PACKAGES):/package_folder
-v $(realpath ./test_output):/test_output
-v $(realpath ./ch_server_logs):/var/log/clickhouse-server/
clickhouse/stateless-test
# image can be built locally
$ cd ~/ClickHouse/docker/test/stateless
$ docker build . -t clickhouse/stateless-test
59

Tests - Stateless - run on Host
Needs a running server
$ cd ~/ClickHouse/programs/server
$ ~/build/programs/clickhouse server
Execute test driver
$ cd ~/ClickHouse/tests
$ ./clickhouse-test -b ~/build/programs/clickhouse
60

Tests - Integration
tests/integration/test_x/test.py ~ 276 tests
check how CH cooperates with various external entities
Distributed (replicas, distributed queries, Zookeeper)
External storages (MySQL, PostgreSQL, Kafka, S3, etc)
Python-based + Docker-compose
61

Tests - Integration - run in Docker
Requires docker in docker, hence sudo
$ cd ~/ClickHouse/tests/integration
$ sudo ./runner
--binary ~/build/programs/clickhouse
--odbc-bridge-binary ~/build/programs/clickhouse-odbc-bridge
--base-configs-dir ../../programs/server/
'test_odbc_interaction -ss'
# image can be built locally
$ cd ~/ClickHouse/docker/test/integration/runner
$ docker build -t clickhouse/integration-tests-runner .
62

Tests - Performance
tests/performance/*.xml ~ 220 tests
XML-based
preconditions: <preconditions>
parameters: <substitution>
setup: <settings> , <create_query> , <fill_query>
run: <query>
tear down: <drop_query>
63

Tests - Performance - run in Docker
Usefull for measuring performance of the
Requires a running server, reachable on port 9000
$ docker run
-it
--network=host
-v $(realpath ~/ClickHouse/tests/performance/encrypt_decrypt.xml):/perf.xml
clickhouse/clickhouse-performance-comparison
bash -c "./perf.py /perf.xml"
64

Building ClickHouse and Making Your First Contribution: A Tutorial_06.10.2021

More Related Content

What's hot

Similar to Building ClickHouse and Making Your First Contribution: A Tutorial_06.10.2021

More from Altinity Ltd

Recently uploaded

Building ClickHouse and Making Your First Contribution: A Tutorial_06.10.2021