ClickHouse Monitoring 101: What to monitor and how
•
7 likes•2,563 views
Webinar. Presented by Robert Hodges and Ned McClain, April 1, 2020 You are about to deploy ClickHouse into production. Congratulations! But what about monitoring? In this webinar we will introduce how to track the health of individual ClickHouse nodes as well as clusters. We'll describe available monitoring data, how to collect and store measurements, and graphical display using Grafana. We'll demo techniques and share sample Grafana dashboards that you can use for your own clusters.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to ClickHouse Monitoring 101: What to monitor and how
Similar to ClickHouse Monitoring 101: What to monitor and how (20)
More from Altinity Ltd
More from Altinity Ltd (20)
Recently uploaded
Recently uploaded (20)
ClickHouse Monitoring 101: What to monitor and how
- 2. Presenting the Presenters Ned McClain Data Warehouse Engineer 20+ years architecting Internet-scale infrastructure with an emphasis on performance and security. Robert Hodges CEO 30+ years on DBMS including 20 different DBMS types. Working on Kubernetes since 2018.
- 3. ...And Altinity Leading software and services provider for ClickHouse #2 committer on ClickHouse Lead maintainer for Kubernetes, Grafana, ODBC, ... Sponsor of US and EU ClickHouse events
- 4. What to Monitor on ClickHouse
- 5. Useful information from monitoring Determining current state Analyzing long-term trends Alerts and reporting Comparing over time or experiment groups Conducting ad hoc retrospective analysis (i.e., debugging) Security incident detection and response Cost modeling and analysis
- 6. Case #1: When INSERTs go bad Too many parts (655). Merges are processing significantly slower than inserts
- 7. How does ClickHouse add or change data? ClickHouse Server Table: events Part: 201801_1_1_0 Part: 201801_2_2_0 Part: 201801_1_2_1 INSERT SELECT Merge ALTER TABLE
- 8. Which metrics can reveal root causes? ClickHouse Server Table: events Part: 201801_1_1_0 Part: 201801_2_2_0 Part: 201801_1_2_1 INSERT SELECT Merge ALTER TABLE events.InsertQuery events.InsertRows events.RejectedInserts events.DelayedInserts asychronous_events.MaxPartCountForPartition events.MergedRows metrics.Merge metrics.BackgroundPoolTask Total number of parts [per table] metrics.PartMutations
- 9. ClickHouse has great operational metrics Always-available system tables query_log (Query history) part_log (Part merges) text_log (System log messages) async… _metrics (Background metrics) events (Cumulative event counters) metrics (Current counters) parts (All table parts) More system tables you can enable replicas (Replicated tables) metric_log (Query history)
- 10. Case #2: Query memory questions kernel: Out of memory: Kill process 16959 (AsyncBlockInput) score 947 or sacrifice child
- 11. How does ClickHouse process queries? ClickHouse Server Table: events Parts primary.idx .mrk & .bin Table: devices Parts primary.idx .mrk & .bin SELECT … FROM events JOIN devices ON ...
- 12. Which metrics show query performance? ClickHouse Server Table: events Parts primary.idx .mrk & .bin Table: devices Parts primary.idx .mrk & .bin SELECT … FROM events JOIN devices ON ... events. Query SelectQuery asynchronous_events.MarkCacheFiles events. SelectedMarks/SelectedParts MarkCacheHits/MarkCacheMisses UncompressedCacheHits UncompressedCacheMisses CompressedReadBufferBytes ReadCompressedBytes metrics. MemoryTracking
- 13. Case #3: Replication regrets <Error> . . . Table is in readonly mode
- 14. Zookeeper ZNodes Zookeeper ZNodes How does ClickHouse replicate data? INSERT Zookeeper ZNodes ClickHouse Table: events ClickHouse Table: events
- 15. Zookeeper ZNodes Zookeeper ZNodes Which metrics diagnose replication issues? INSERT Zookeeper ZNodes ClickHouse Table: events ClickHouse Table: events asynchronous_events. ReplicasMaxAbsoluteDelay ReplicasMaxInsertsInQueue ReplicasMaxMergesInQueue metrics. LeaderReplica ReadonlyReplica ZookeeperTransactions ZookeeperHardwareExceptions zookeeper_NumAliveConnections zookeeper_Max[Avg]RequestLatency zookeeper_Max[Min]ClientResponseSize zookeeper_OutstandingRequests zookeeper_PacketsReceived[Sent] zookeeper_ElectionTimeTaken zookeeper_Log[Data]DirSize zookeeper_InMemoryDataTree_NodeCount
- 17. Standard ClickHouse monitoring platforms Sematext / Datadog / Instana Nagios/Icinga/Zabbix InfluxDB / Graphite… Prometheus + Grafana Prometheus + ClickHouse Kubernetes Operator + Grafana
- 18. Three high-level approaches to monitoring 1. Proprietary Agents 2. Application-aware Polling 3. Metrics Exposition
- 19. Metrics collection with Prometheus Alerts to: messaging, ticketing, and webhooks. Dashboards and trend analysis. Prometheus Alertmanager Prometheus Exposed Metrics
- 20. Monitoring a ClickHouse node Prometheus Exposed MetricsClickHouse
- 21. Monitoring a ClickHouse node $ curl clickhouse:9363/metrics
- 22. Monitoring a ClickHouse node /etc/clickhouse-server/config.xml
- 23. Monitoring a ClickHouse Cluster Prometheus ClickHouse Replica ClickHouse Replica ClickHouse Shard ClickHouse Replica ClickHouse Replica ClickHouse Shard
- 24. Monitoring Linux and cloud dependencies ● node_exporter: operating system metrics ○ CPU, memory, load, I/O, filesystem, network, and more. ● cloudwatch_exporter: AWS metrics ○ EC2 instances, ELBs, hosted databases, and more. ● stackdriver_exporter: GCP metrics ● azure_metrics_exporter: Azure metrics
- 27. Google's Four Golden Signals ● Latency ● Traffic ● Errors ● Saturation
- 29. Demo Time! ClickHouse Monitoring on Linux
- 31. ClickHouse operator exports to Prometheus ClickHouse Operator your-favorite namespace ClickHouse Resource Definition z z K8s API Monitoring data Monitoringdata Prometheus Grafana
- 32. System dynamism complicates monitoring Load Balancer Service Shard 1 Replica 1 Stateful Set Pod Persistent Volume Claim Persistent Volume Per-replica Config Map Replica Service ClickHouse Operator Monitoring data from a single replica
- 33. Operator exports new pods automatically Load Balancer Service Shard 1 Replica 1 Stateful Set Pod Persistent Volume Claim Persistent Volume Per-replica Config Map Shard 1 Replica 2 Replica Service Replica Service ClickHouse Operator Monitoring data from two replicas
- 34. Demo Time! ClickHouse Monitoring on Kubernetes with Grafana Dashboard
- 36. Summary ● ClickHouse has a wide range of accessible monitoring data ● Export relevant measurements from system database tables ○ Metrics, events, and asynchronous events ○ Query_log ● Prometheus and Grafana offer a great open source solution ○ But there are many other tools as well ● ClickHouse Operator exports monitoring data automatically to Prometheus ● Build monitoring for the whole application, not just ClickHouse
- 37. Where to look next... ● Google SRE Book: Monitoring Distributed Systems ○ https://landing.google.com/sre/sre-book ● ClickHouse monitoring documentation ○ https://clickhouse.tech/docs/en/operations/monitoring/ ● Grafana Labs Dashboards ○ https://grafana.com/grafana/dashboards ● Altinity Blog ○ https://www.altinity.com/blog ● ClickHouse Kubernetes Operator ○ https://github.com/Altinity/clickhouse-operator
- 38. Thank you! Special Offer: Contact us for a free 1-hour consultation on monitoring Contacts: info@altinity.com Visit us at: https://www.altinity.com Free Consultation: https://blog.altinity.com/offer