A Benchmark Journey at ContentSquare. How to Migrate from a legacy backend to A blazing fast Column Oriented Analytics Database

1. 1
ClickHouse
at ContentSquare

2. 2
Summary
What are we doing at ContentSquare?
What have we done so far with ClickHouse?
What do we think about ClickHouse?

3. 3
Who are we
Christophe Kalenzaga
Senior Data Engineer
Vianney Foucault
Senior Platform Engineer

4. 4
ContentSquare in a few words
● Web analytic Company
○ 600GB collected per day
○ 13 months retention (~230 TBytes)
○ websites and mobile applications
● 100+ clients Worldwide
○ Automotive | BFSI | Retail | Grocery | Luxury
○ B2B | Energy | Travel | Telco | Gaming
● Collected data doubles every year

5. 5
One of the many challenges of ContentSquare

6. 6
Our workload
● 30k queries per day
● Skewed load
● CPU intensive queries
● 20% of cacheable queries
● Can’t precompute queries

7. 7
current backend infrastructure
● Many ElasticSearch clusters
● Storage cost
● Compute cost
● Can’t handle clients with too much data
Scala Microservices
Data Pipeline Components
(Scala/Spark)

8. 8
to sum up

9. 9
what we’ve done so far
Aug 2018Jun 2018Mar 2018Jan 2018Dec 2017
Selection of a few technologies
to replace ElasticSearch
Start mobile analytics
solution on ClickHouse
One-week POC on ClickHouse on a
specific feature of the web analytics
solution First release of the mobile analytics
solution on clickhouse.
insert data from the web analytics
solution into ClickHouse check if
there are no side effects
Start migrating the web analytic
solution to ClickHouse
Benchmarks to find the
right data model and the
right configuration
the mobile analytics solution is
released
Looking for a new technology
POC of multiple technologies

10. 10
How to benchmark?

11. 11
Benchmark methodology
● Define different types of query
● Get the statistical distribution of each type of query in production
● Create a dataset (10 TBytes)
● Create queries with the same statistical distribution as in production
● Be deterministic

12. 12
Benchmark Constraints for a #Platfomer
Deployment
CPU’s
Memory
Configuration
Management
Deterministic
Automation Cost Optimisation
Monitoring
Data Load

13. 13
Benchmark timeline
{ Time consuming Manual tasks }
Iteration
Setup
infrastructure
Deploy
Middlewares
~ 4 hours ~ 12 hours
Load test Data Benchmark
{ Time consuming }
Cleanup

14. 14
Platformer’s Words
DevOps Motto:
Industrialise As Soon As Possible

15. 15
Fully Automated Benchmark timeline
~ 4 hours ~ 12 hours5 Minutes 1 Minute
Iteration
Setup
infrastructure
Deploy
Middlewares
Load test Data Benchmark Cleanup
DevOps Approved

16. 16
Platformer’s Words
DevOps Leitmotiv:
Monitor As Soon As Possible

17. 17
Benchmark iterations on operating systems
http://openbenchmarking.org/result/1704225-TR-AMAZON38819

18. 18
Benchmark iterations on different operating systems
ResponseTime
Percentiles
+60%
+60%

19. 19
Benchmark iterations on Instances
Instance Type vCPU Mem (GiB) Networking Perf. $ Per Hour Perf Index
i3.large 2 15,25 Up to 10 Gigabit 0.172 1
i3.xlarge 4 30,5 Up to 10 Gigabit 0.344 1
i3.2xlarge 8 61 Up to 10 Gigabit 0.688 1
i3.4xlarge 16 122 Up to 10 Gigabit 1.376 1
i3.8xlarge 32 244 10 Gigabit 2.752 1
i3.16xlarge 64 488 25 Gigabit 5.504 1
i3.metal 72 512 25 Gigabit 5.504 1.3

20. 20
Benchmark iterations on different Amazon servers
All the cluster configurations have the same price!
ResponseTime
Percentiles
Blue To Red +40%

21. 21
Numa Architecture
QPI Bus
ClickHouse Instance 1
MEMORY CPU
I/O Bridge
MEMORYCPU
I/O Bridge
Linux OS
NUMA Node0 NUMA Node1
ClickHouse Instance 2

22. 22
Benchmark iterations on scalability
ResponseTime
Percentiles
+85%
+55%

23. 23
Benchmark iterations on parallelism
ResponseTime
Percentiles

24. 24
New Backend Infrastructure (Zoom in)
1 EC2 (i3.metal) instance hosts 2 clickhouse servers
1
1’
2
2’
3
3’
4
4’
5
5’
6
6’
ClickhouseCluster
AZ-1aAZ-1b
Replication

25. 25
New Backend Infrastructure (Big picture)
ClickHouse Cluster
Data Pipeline
M
assiveInserts
Scala
Microservices
Reads
CHProxy Servers
With local Cache
Monitoring: Prometheus / Grafana
Load Balancers

26. 26
Next Steps
Cluster Extension
Major Disaster
Staging Refresh
Major ClickHouse
upgrade
Full InHouse Automation

27. 27
What do we like with ClickHouse
Native sampling
Stability
Easy to understand
Fast
Active developments

28. 28
What parts of ClickHouse still need to be improved
Difficult to master
Lack of tooling
Random stability of a new version
No query optimizer

29. 29
Do we recommend ClickHouse?

30. 30
Any question?
Thank you
We’re hiring!