Download as PDF, PPTX
![try to tune!
remaining := whisper.Size() - whisper.MetadataSize()
whisper.file.Seek(int64(remaining-1), 0)
whisper.file.Write([]byte{0})
chunkSize := 16384
zeros := make([]byte, chunkSize)
for remaining > chunkSize {
// if _, err = whisper.file.Write(zeros); err != nil {
// return nil, err
// }
remaining -= chunkSize
}
if _, err = whisper.file.Write(zeros[:remaining]); err != nil {
return nil, err
}](https://image.slidesharecdn.com/2-160603121007/85/rootconf-2016-47-320.jpg)
Uploaded byVsevolod Polyakov
Мониторинг. Опять, rootconf 2016
This document discusses metrics monitoring systems. It provides an overview of the Graphite monitoring system architecture, including the carbon-cache and carbon-relay components. It then evaluates various open source alternatives for handling high volumes of metrics data, finding that a combination of go-carbon and carbon-c-relay can process over 1.4 million requests per second. The document also discusses tagging metrics with metadata and time series databases that support tags.
More Related Content
Similar to Мониторинг. Опять, rootconf 2016
Мониторинг. Опять, rootconf 2016
- 1.
- 2. Platform Engineer .Grammarly ctrlok.com
- 3.
- 6.
- 7.
- 8.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 17.
- 18.
- 19. what? • RRD-like (gram.ly/gfsx) •so.it.is.my.metric → /so/it/is/my/metric.wsp • Fixed retention (by namepattern) • Fixed size (actually no)
- 20. Retention and size •1s:1d → 1 036 828 bytes • 10s:10d → 1 036 828 bytes • 1s:365d → 378 432 028 bytes (1 TB ~ 3 000) • 10s:365d → 37 843 228 bytes (1 TB ~ 30 000) whisper calc
- 21. Retention and size •10s:30d,1m:120d,10m:365d → 4 564 864 bytes • 240 864 metrics in 1 TB • aggregation: average, sum, min, max, and last. • can be assign per metric
- 22. How • terraform (https://www.terraform.io/) •docker (https://www.docker.com/) • ansible (https://www.ansible.com/) • rocker (https://github.com/grammarly/rocker) • rocker-compose (https://github.com/grammarly/rocker-compose)
- 23.
- 24. carbon-cache.py • single-core • manyoptions in config file • default link
- 25.
- 26. Start load testing •m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2) • retentions = 1s:1d • MAX_CACHE_SIZE, MAX_UPDATES_PER_SECOND, MAX_CREATES_PER_MINUTE = inf • defaults • almost 1.5h to get limit :(
- 27. carbon-cache.py cache size→ 75k ms
- 30. results • 75 000ms max • 60 000 ms flagman speed • IO :(
- 31. Try to tune! •WHISPER_SPARSE_CREATE = true (don’t allocate space on creation) non-linear IO load. • CACHE_WRITE_STRATEGY = sorted (default)
- 32. cache size 1k→ 195k ms
- 33. results • 120 000ms flagman speed • cache flush problem :(
- 34. Try to tune! •CACHE_WRITE_STRATEGY = max will give a strong flush preference to frequently updated metrics and will also reduce random file-io.
- 35. from 1k to150k
- 36. results • 90 000ms flagman speed • cache flush problem :(
- 37. Try to tune! •CACHE_WRITE_STRATEGY = naive just flush. Better with random IO.
- 38. from 45k to135k
- 39. results • 120 000ms flagman speed • still CPU
- 40.
- 41. • Maybe it’sIO EBS limitation? → 512 GB disk. • No.
- 42. go-carbon • multi-core singledaemon • written in golang • not many options to tune :( link
- 43. Start load testing •m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2) • retentions = 1s:1d • max-size = 0 • max-updates-per-second = 0 • almost 1h to get limit :(
- 44. 1k → 130kms ~3k/min
- 46. results • 120 000ms flagman speed • but it’s without sparse. • try to implement
- 47. try to tune! remaining:= whisper.Size() - whisper.MetadataSize() whisper.file.Seek(int64(remaining-1), 0) whisper.file.Write([]byte{0}) chunkSize := 16384 zeros := make([]byte, chunkSize) for remaining > chunkSize { // if _, err = whisper.file.Write(zeros); err != nil { // return nil, err // } remaining -= chunkSize } if _, err = whisper.file.Write(zeros[:remaining]); err != nil { return nil, err }
- 48. Уже есть вgo-carbon
- 49.
- 51. try to tune! •max update operation = 1500
- 52. results • TLDR 210000 - 240 000 ms flagman speed • 31 000 000 cache size!
- 54. try to tune! •max update operation = 0 • input-buffer = 400 000
- 55. results • 270 000ms flagman speed • 10-20kk cache size!
- 57. try to tune! •vm.dirty_background_ratio=40 • vm.dirty_ratio=60
- 58.
- 59. results • 300 000ms flagman speed • 180k+ ms ±without cache
- 60.
- 61.
- 62.
- 63.
- 64.
- 65.
- 66.
- 67. Start load testing •c4.xlarge instance (4 CPU, 7.5 GB ram) • ~1 Gb lan • default parameters • hashing • 10 connections
- 68.
- 69. carbon-relay-ng • golang-based • web-panel •live-updates • aggregators • spooling link
- 70.
- 71. carbon-c-relay • написан наC • advanced cluster management
- 72. from 100 000to 1 600 000 reqs
- 73. 1 400 000flagman speed. Or not?
- 74.
- 80.
- 81.
- 82. Различия • Окружение • Роль •Трек (Модификатор) • IP • Датацентр • Что-угодно
- 83.
- 84. TSDB с тегами •influxDB • openTSDB (hbase) • cyanite (cassandra) • newTS (cassandra) • Prometheus
- 85. (cluster) influx, 130kmetrics увеличить график
- 87. openTSDB single instance +hbase cluster = upto 150k metrics
- 92.
- 94.
- 99.
- 107.
- 108. Zipper • https://github.com/grobian/carbonserver • https://github.com/dgryski/carbonzipper •https://github.com/dgryski/carbonapi
- 111. ALSO • https://github.com/jssjr/carbonate • https://github.com/jjneely/buckytools •https://github.com/dgryski/carbonmem • https://github.com/grobian/carbonwriter
- 112. Планы • Патч statsd→ ES • Патч carbonserver → carbonlink
- 113. feel free toask • Vsevolod Polyakov • ctrlok@gmail.com • skype: ctrlok1987 • github.com/ctrlok • twitter.com/ctrlok • slack: HangOps • Gitter: dev_ua/devops • skype: DevOps from Ukraine • slack.ukrops.club Мы хайрим!