Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Dangerous on ClickHouse in 30 minutes, by Robert Hodges, Altinity CEO

766 views

Published on

Presented at ClickHouse Meetup in Cloudflare, February 19, 2019
Dangerous on ClickHouse in 30 minutes, by Robert Hodges, Altinity CEO

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Dangerous on ClickHouse in 30 minutes, by Robert Hodges, Altinity CEO

  1. 1. Become dangerous on ClickHouse in 30 minutes Robert Hodges SF ClickHouse Meetup - 19 February 2019
  2. 2. Presenter Background ● CEO of Altinity, a ClickHouse commercial provider ● Experience in databases: ○ M204... ○ Followed by Sybase, Cloudscape, Oracle, MySQL, …, ClickHouse ● Qualifications for working in data management: ○ A sick interest in math ○ Insatiable curiosity
  3. 3. So...What is this ClickHouse? Understands SQL Portable from bare metal to cloud Stores data in columns Parallelizes query execution Scales to many petabytes Is open source (Apache 2.0) Is WAY fast! Id a b c d Id a b c d Id a b c d Id a b c d
  4. 4. What does “WAY fast” mean? SELECT Dest d, count(*) c, avg(DepDelayMinutes) dd FROM ontime GROUP BY d HAVING c > 100000 ORDER BY c DESC limit 5 ┌─d───┬───────c─┬─────────────────dd─┐ │ ATL │ 8281604 │ 10.328921909330608 │ │ ORD │ 8040484 │ 11.893720825761235 │ │ DFW │ 6928489 │ 8.954579418398442 │ │ LAX │ 5164096 │ 10.109661594207388 │ │ DEN │ 4641382 │ 9.664600112638865 │ └─────┴─────────┴────────────────────┘ 5 rows in set. Elapsed: 0.938 sec. Processed 153.53 million rows, 2.46 GB (163.67 million rows/s., 2.62 GB/s.) TEST BED 40€/month Intel Core i7-6700 32GB Seagate 4TB 7200RPM 128MB cache
  5. 5. Getting started is easy $ docker run -d --name ch-s yandex/clickhouse-server $ docker exec -it ch-s clickhouse client ... 11e99303c78e :) select version() SELECT version() ┌─version()─┐ │ 19.3.3 │ └───────────┘ 1 rows in set. Elapsed: 0.001 sec.
  6. 6. Let’s do some technical due diligence CREATE TABLE sdata ( DevId Int32, Type String, MDate Date, MDatetime DateTime, Value Float64 ) ENGINE = MergeTree() PARTITION BY toYYYYMM(MDate) ORDER BY (DevId, MDatetime) INSERT INTO sdata VALUES (15, 'TEMP', '2018-01-01', '2018-01-01 23:29:55', 18.0), (15, 'TEMP', '2018-01-01', '2018-01-01 23:30:56', 18.7) INSERT INTO sdata VALUES (15, 'TEMP', '2018-01-01', '2018-01-01 23:31:53', 18.1), (2, 'TEMP', '2018-01-01', '2018-01-01 23:31:55', 7.9)
  7. 7. Select results can be surprising! SELECT * FROM sdata ┌─DevId─┬─Type─┬──────MDate─┬───────────MDatetime─┬─Value─┐ │ 15 │ TEMP │ 2018-01-01 │ 2018-01-01 23:29:55 │ 18 │ │ 15 │ TEMP │ 2018-01-01 │ 2018-01-01 23:30:56 │ 18.7 │ └───────┴──────┴────────────┴─────────────────────┴───────┘ ┌─DevId─┬─Type─┬──────MDate─┬───────────MDatetime─┬─Value─┐ │ 2 │ TEMP │ 2018-01-01 │ 2018-01-01 23:31:55 │ 7.9 │ │ 15 │ TEMP │ 2018-01-01 │ 2018-01-01 23:31:53 │ 18.1 │ └───────┴──────┴────────────┴─────────────────────┴───────┘ ┌─DevId─┬─Type─┬──────MDate─┬───────────MDatetime─┬─Value─┐ │ 2 │ TEMP │ 2018-01-01 │ 2018-01-01 23:31:55 │ 7.9 │ │ 15 │ TEMP │ 2018-01-01 │ 2018-01-01 23:29:55 │ 18 │ │ 15 │ TEMP │ 2018-01-01 │ 2018-01-01 23:30:56 │ 18.7 │ │ 15 │ TEMP │ 2018-01-01 │ 2018-01-01 23:31:53 │ 18.1 │ └───────┴──────┴────────────┴─────────────────────┴───────┘ Result right after INSERT: Result somewhat later:
  8. 8. Time for some research into table engines CREATE TABLE sdata ( DevId Int32, Type String, MDate Date, MDatetime DateTime, Value Float64 ) ENGINE = MergeTree() PARTITION BY toYYYYMM(MDate) ORDER BY (DevId, MDatetime) How to manage data and handle queries How to break table into parts How to index and sort data in each part
  9. 9. MergeTree writes parts quickly and merges them offline /var/lib/clickhouse/data/default/sdata 201801_1_1_0/ 201801_2_2_0/ Multiple parts after initial insertion ( => very fast writes) 201801_1_2_1/ Single part after merge ( => very fast reads)
  10. 10. Rows are indexed and sorted inside each part /var/lib/clickhouse/data/default/sdata ... ... 956 2018-01-01 15:22:37 575 2018-01-01 23:31:53 1300 2018-01-02 05:14:47 ... ... primary.idx |||| .mrk .bin |||| .mrk .bin |||| .mrk .bin |||| .mrk .bin 201802_1_1_0/ (DevId, MDateTime) DevId Type MDate MDatetime... primary.idx .mrk .bin .mrk .bin .mrk .bin .mrk .bin 201801_1_2_1/ (DevId, MDateTime) DevId Type MDate MDatetime...
  11. 11. ClickHouse Now we can follow how query works on a single server SELECT DevId, Type, avg(Value) FROM sdata WHERE MDate = '2018-01-01' GROUP BY DevId, Type Identify parts to search Query in parallel Aggregate results Result Set
  12. 12. What if you have a lot of data? CREATE TABLE sense.sdata_dist AS sense.sdata ENGINE = Distributed(scluster, sense, sdata, DevId) Engine for distributed tables Cluster, database, and table names Distribute data on this value Copy this schema
  13. 13. What’s a Cluster? (Hint: it’s a file.) <yandex> <remote_servers> <scluster> <shard> <internal_replication>true</internal_replication> <replica> <host>ch-d7dc980i1-0.d7dc980i1s</host> <port>9000</port> </replica> </shard> <shard> <internal_replication>true</internal_replication> <replica> <host>ch-d7dc980i2-0.d7dc980i2s</host> <port>9000</port> </replica> </shard> ... </scluster> </remote_servers> </yandex>
  14. 14. Distributed engine spreads queries across shards SELECT ... FROM sdata_dist ClickHouse sdata_dist (Distributed) sdata (MergeTable) ClickHouse sdata_dist sdata ClickHouse sdata_dist sdata Result Set
  15. 15. What if you have a lot of data and you don’t want to lose it? CREATE TABLE sense.sdata ( DevId Int32, Type String, MDate Date, MDatetime DateTime, Value Float64 ) engine=ReplicatedMergeTree( '/clickhouse/{installation}/{scluster}/tables/{scluster- shard}/sense/sdata', '{replica}', MDate, (DevId, MDatetime), 8192); Engine for replicated tables MergeTree parameters Cluster identification Replica #
  16. 16. Now we distribute across shards and replicas ClickHouse sdata_dist sdata ReplicatedMergeTree Engine ClickHouse sdata_dist sdata ClickHouse sdata_dist sdata ClickHouse sdata_dist sdata ClickHouse sdata_dist sdata ClickHouse sdata_dist sdata SELECT ... FROM sdata_dist Result Set Zookeeper Zookeeper Zookeeper
  17. 17. SELECT Dest, count(*) c, avg(DepDelayMinutes) FROM ontime GROUP BY Dest HAVING c > 100000 ORDER BY c DESC limit 5 SELECT Dest, count(*) c, avg(DepDelayMinutes) FROM ontime WHERE toYear(FlightDate) = toYear(toDate('2016-01-01')) GROUP BY Dest HAVING c > 100000 ORDER BY c DESC limit 5 With basic engine knowledge you can now tune queries Scans 355 table parts in parallel; does not use index Scans 12 parts (3% of data) because FlightDate is partition key Hint: clickhouse-server.log has the query plan Faster
  18. 18. SELECT Dest d, Name n, count(*) c, avg(ArrDelayMinutes) FROM ontime JOIN airports ON (airports.IATA = ontime.Dest) GROUP BY d, n HAVING c > 100000 ORDER BY ad DESC SELECT dest, Name n, c AS flights, ad FROM ( SELECT Dest dest, count(*) c, avg(ArrDelayMinutes) ad FROM ontime GROUP BY dest HAVING c > 100000 ORDER BY ad DESC ) LEFT JOIN airports ON airports.IATA = dest You can also optimize joins Subquery minimizes data scanned in parallel; joins on GROUP BY results Joins on data before GROUP BY, increased amount to scan Faster
  19. 19. ClickHouse has a wealth of features to help queries go fast Dictionaries Materialized Views Arrays Specialized functions and SQL extensions Lots more table engines
  20. 20. Parting Question: What’s ClickHouse good for? ● Fast, scalable data warehouse for online services (SaaS and in-house apps ● Built-in data warehouse for installed analytic applications ● Exploration -- throw in a bunch of data and go crazy!
  21. 21. Thank you! Comments to: rhodges at altinity.com Visit us at: https://www.altinity.com

×