Путь мониторинга 2.0 всё стало другим / Всеволод Поляков (Grammarly)

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

МОНИТОРИНГ.
ОПЯТЬ.
Всеволод Поляков

Platform Engineer . Grammarly
ctrlok.com

Что такое метрики?

Успешность

Количество

Время

Взаимодействие

Внутренние процессы

Системные метрики

Зачем нужны
метрики?

Алерты

Аналитика

Graphite

Default graphite architecture

what?

what?
• RRD-like (gram.ly/gfsx)

what?
• RRD-like (gram.ly/gfsx)
• so.it.is.my.metric → /so/it/is/my/metric.wsp

what?
• RRD-like (gram.ly/gfsx)
• so.it.is.my.metric → /so/it/is/my/metric.wsp
• Fixed retention (by namepattern)

what?
• RRD-like (gram.ly/gfsx)
• so.it.is.my.metric → /so/it/is/my/metric.wsp
• Fixed retention (by namepattern)
• Fixed ...

Retention and size

Retention and size
• 1s:1d → 1 036 828 bytes

Retention and size
• 1s:1d → 1 036 828 bytes
• 10s:10d → 1 036 828 bytes

Retention and size
• 1s:1d → 1 036 828 bytes
• 10s:10d → 1 036 828 bytes
whisper calc

Retention and size
• 1s:1d → 1 036 828 bytes
• 10s:10d → 1 036 828 bytes
• 1s:365d → 378 432 028 bytes (1 TB ~ 3 000)
whis...

Retention and size
• 1s:1d → 1 036 828 bytes
• 10s:10d → 1 036 828 bytes
• 1s:365d → 378 432 028 bytes (1 TB ~ 3 000)
• 10...

Retention and size

Retention and size
• 10s:30d,1m:120d,10m:365d → 4 564 864 bytes

Retention and size
• 10s:30d,1m:120d,10m:365d → 4 564 864 bytes
• 240 864 metrics in 1 TB

Retention and size
• 10s:30d,1m:120d,10m:365d → 4 564 864 bytes
• 240 864 metrics in 1 TB
• aggregation: average, sum, min...

Retention and size
• 10s:30d,1m:120d,10m:365d → 4 564 864 bytes
• 240 864 metrics in 1 TB
• aggregation: average, sum, min...

How
• terraform (https://www.terraform.io/)
• docker (https://www.docker.com/)
• ansible (https://www.ansible.com/)
• rock...

Default graphite architecture

Default graphite architecture

carbon-cache.py
link

carbon-cache.py
• single-core
link

carbon-cache.py
• single-core
• many options in conﬁg ﬁle
link

carbon-cache.py
• single-core
• many options in conﬁg ﬁle
• default
link

architecture
carbon-cache.py

Start load testing

Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)

Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d

Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d
• MAX_CACHE_SIZE, MAX...

Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d
• MAX_CACHE_SIZE, MAX...

Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d
• MAX_CACHE_SIZE, MAX...

carbon-cache.py cache size → 75k ms

updates

upd time

results
• 75 000 ms max
• 60 000 ms ﬂagman speed
• IO :(

Try to tune!
• WHISPER_SPARSE_CREATE = true
(don’t allocate space on creation)
non-linear IO load.
• CACHE_WRITE_STRATEGY ...

cache size 1k → 195k ms

results
• 120 000 ms ﬂagman speed
• cache ﬂush problem :(

Try to tune!
• CACHE_WRITE_STRATEGY = max
will give a strong ﬂush preference to
frequently updated metrics and will
also r...

from 1k to 150k

results
• 90 000 ms ﬂagman speed
• cache ﬂush problem :(

Try to tune!
• CACHE_WRITE_STRATEGY = naive
just ﬂush. Better with random IO.

from 45k to 135k

results
• 120 000 ms ﬂagman speed
• still CPU

sorted
max
naive

• Maybe it’s IO EBS limitation? → 512 GB disk.

• Maybe it’s IO EBS limitation? → 512 GB disk.
• No.

• Maybe it’s IO EBS limitation? → 512 GB disk.
• No.

go-carbon
link

go-carbon
• multi-core single daemon
link

go-carbon
• multi-core single daemon
• written in golang
link

go-carbon
• multi-core single daemon
• written in golang
• not many options to tune :(
link

Start load testing

Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)

Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d

Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d
• max-size = 0

Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d
• max-size = 0
• max-...

Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d
• max-size = 0
• max-...

1k → 130k ms ~3k/min

1k → 130k ms ~3k/min

1k → 130k ms ~3k/min

results

results
• 120 000 ms ﬂagman speed

results
• 120 000 ms ﬂagman speed
• but it’s without sparse.

results
• 120 000 ms ﬂagman speed
• but it’s without sparse.
• try to implement

try to tune!
remaining := whisper.Size() - whisper.MetadataSize()
whisper.ﬁle.Seek(int64(remaining-1), 0)
whisper.ﬁle.Writ...

Уже есть в go-carbon

180 000 ms !

try to tune!
• max update operation = 1500

results
• TLDR 210 000 - 240 000 ms ﬂagman speed
• 31 000 000 cache size!

try to tune!
• max update operation = 0
• input-buffer = 400 000

results
• 270 000 ms ﬂagman speed
• 10-20kk cache size!

try to tune!
• vm.dirty_background_ratio=40
• vm.dirty_ratio=60

300 000 reqs

results
• 300 000 ms ﬂagman speed
• 180k+ ms ±without cache

Re:Lays

Default graphite architecture

Default graphite architecture

arch forward

arch namedregexp

arch hash

arch hash replicafactor: 2

carbon-relay.py
• twisted based
• native

Start load testing

Start load testing
• c4.xlarge instance (4 CPU, 7.5 GB ram)

Start load testing
• c4.xlarge instance (4 CPU, 7.5 GB ram)
• ~1 Gb lan

Start load testing
• c4.xlarge instance (4 CPU, 7.5 GB ram)
• ~1 Gb lan
• default parameters

Start load testing
• c4.xlarge instance (4 CPU, 7.5 GB ram)
• ~1 Gb lan
• default parameters
• hashing

Start load testing
• c4.xlarge instance (4 CPU, 7.5 GB ram)
• ~1 Gb lan
• default parameters
• hashing
• 10 connections

WTF!

carbon-relay-ng
link

carbon-relay-ng
• golang-based
link

carbon-relay-ng
• golang-based
• web-panel
link

carbon-relay-ng
• golang-based
• web-panel
• live-updates
link

carbon-relay-ng
• golang-based
• web-panel
• live-updates
• aggregators
link

carbon-relay-ng
• golang-based
• web-panel
• live-updates
• aggregators
• spooling
link

<150 000 reqs

carbon-c-relay
• написан на C
• advanced cluster management

from 100 000 to 1 600 000 reqs

1 400 000 ﬂagman speed. Or not?

1 400 000 ﬂagman speed. Or not?

1 400 000 ﬂagman speed. Or not?

Итак…
go-carbon + carbon-c-relay = ♡

Контейнеры

Всё перепутано

Различия
• Окружение
• Роль
• Трек (Модификатор)
• IP
• Датацентр
• Что-угодно

Теги

TSDB с тегами
• inﬂuxDB
• openTSDB (hbase)
• cyanite (cassandra)
• newTS (cassandra)
• Prometheus

(cluster) inﬂux, 130k metrics

openTSDB
single instance + hbase cluster = upto 150k metrics

Compaction

Graphite

Найти уникальное

Работает с Grafana

Zipper
• https://github.com/grobian/carbonserver
• https://github.com/dgryski/carbonzipper
• https://github.com/dgryski/ca...

ALSO
• https://github.com/jssjr/carbonate
• https://github.com/jjneely/buckytools
• https://github.com/dgryski/carbonmem
•...

Планы
• Патч statsd → ES
• Патч carbonserver → carbonlink

feel free to ask
• Vsevolod Polyakov
• ctrlok@gmail.com
• skype: ctrlok1987
• github.com/ctrlok
• twitter.com/ctrlok
• sla...

feel free to ask
• Vsevolod Polyakov
• ctrlok@gmail.com
• skype: ctrlok1987
• github.com/ctrlok
• twitter.com/ctrlok
• sla...

Share SlideShare

Facebook
Twitter
LinkedIn

Embed

Size (px)

Show related SlideShares at end

WordPress Shortcode

Link

Share
Email

Тошнит от колец великая битва систе... by Ontico 1364 views

Top clipped slide

1 of 169 Ad

1 of 169 Ad