This document summarizes the author's experience optimizing Gnocchi, an open source time-series database, to store metrics for hundreds of thousands of resources over many months. The author describes improving performance by adding Ceph storage nodes, tuning Ceph configurations, minimizing I/O operations, and improving the storage format. Benchmark results show the new version achieves 50% higher write throughput, 40-60% faster computation times, 30-60% better overall performance, and 30-40% fewer operations. Usage hints are also provided to help optimize for different use cases.

1. HUAWEI CANADA
Gnocchi v3
Monitoring the next million time-series
Gordon Chung, Engineer

2. HISTORY
do you remember the time…

3. built to address storage performance issues
encountered in Ceilometer

4. designed to be used to store time series and
their associated resource metadata
Metric storage
(Ceph)
MetricD
Computation
workers
data
stores aggregated
measurement data
stores metadata
background workers which
aggregate data to minimise
query computations
LoadBalancer
APIAPIAPI
Indexer (SQL)

5. MY USE CASE
tired of you tellin' the story your way…

6. collect usage information for hundreds of
thousands of metrics* over many months for
use in capacity planning recommendations
and scheduling
* data is received in batches every x minutes. not streaming

7. GETTING STARTED
wanna be startin’ something’…

8. HARDWARE
▪ 3 physical hosts
▪ 24 physical core
▪ 256GB memory
▪ a bunch of 10K 1TB disks
▪ 1Gb network

9. SOFTWARE
▪ Gnocchi 2.1.x (June 3rd
2016)
▪ 32 API processes, 1 thread
▪ 3 metricd agents (24 workers each)
▪ PostgreSQL 9.2.15 – single node
▪ Redis 3.0.6 (for coordination) – single node
▪ Ceph 10.2.1 – 3 nodes (20 OSDs, 1 replica)

10. POST ~1000 generic resources with
20 metrics each (20K metrics)
60 measures per metric.
policy rolls up to minute, hour, and day.
8 different aggregations each*.
* min, max, sum, average, median, 95th
percentile, count, stdev

11. METRIC PROCESSING RATE
• rate drops
significantly
after initial
push
• high variance in
processing rate

12. uhhh… wtf?
this doesn’t happen in NFS backend.

13. “LEARNING” HOW TO USE CEPH
everybody's somebody's fool…

14. give it more power!
add another node… and 10 more OSDs…
and more PG groups… and some SSDs for
journals

15. ~65% better POST rate

16. ~27% better aggregation rate

17. METRIC PROCESSING RATE (with more power)
• same drop in
performance

18. ““LEARNING”” HOW TO USE CEPH
this time around…

19. CEPH CONFIGURATIONS
original conf
[osd]
osd journal size = 10000
osd pool default size = 3
osd pool default min size = 2
osd crush chooseleaf type = 1
[osd]
osd journal size = 10000
osd pool default size = 3
osd pool default min size = 2
osd crush chooseleaf type = 1
osd op threads = 36
filestore op threads = 36
filestore queue max ops = 50000
filestore queue committing max
ops = 50000
journal max write entries = 50000
journal queue max ops = 50000
good enough conf
http://ceph.com/pgcalc/ to calculate required # of placement groups

20. METRIC PROCESSING RATE (varying configurations)
shorter the
horizontal length
equals better
performance.
Higher the spikes
equals quicker
rate.

21. IMPROVING GNOCCHI
take a look at yourself, and then make a change…

22. computing and storing ~29
aggregates/worker per second is not bad

23. we can minimise IO

24. MINIMISING IO
- each
aggregation
requires:
1. read
object
2. update
object
3. write
object
- with Ceph, we
can just write
to save.

25. NEW STORAGE FORMAT
V2.x
{‘values’:{<timestamp>: float,
<timestamp>: float,
...
<timestamp>: float}
}
msgpacks serialised
<time><float><time><fl
oat>…<time><float>
binary serialized
and lz4 compressed
V3.x

26. asking questions about code

27. why is this so long?
update existing aggregates
retrieve existing aggregates
why we call this so
much?
writing aggregates

28. BENCHMARK RESULTS
showin' how funky strong is your fight…

29. WRITE THROUGHPUT
- ~970K
measures/s
with 5K
batches
- ~13K
measures/s
with 10
measure
batch
- 50% gains at
higher end

30. READ PERFORMANCE
- Negligible
change in
response
time.
- Majority of
time is client
rendering

31. COMPUTATION TIME
- ~0.12s to
compute 24
aggregates
from 1 point
- ~4.2s to
compute 24
aggregates
from 11.5K
points
- 40%-60%
quicker

32. DISK USAGE
- 16B/point vs
~6.25B/point
(depending on
series length
and
compression
schedule)

33. OUR USE CASE
- Consistent
performance
between
batches
- 30% to 60%
better
performance
- more
performance
gain for larger
series.

34. OUR USE CASE
- 30% to 40%
less
operations
required

35. now computing and storing ~53
aggregates/worker per second.

36. USAGE HINTS
what more can i give…

37. EFFECTS OF AGGREGATES
- 15%-25%
overhead to
compute each
additional
level of
granularity
- percentile
aggregations
requires more
CPU time

38. THREADING
- set `aggregation_workers_number` to the number of aggregates computed
per series

39. metricD agents and Ceph OSDs are
CPU-intensive services

40. EXTRAS
they don’t care about us…

41. ADDITIONAL FUNCTIONALITY
▪ aggregate of aggregates
▪ get max of means, stdev of maxs, etc…
▪ dynamic resources
▪ create and modify resource definitions
▪ aggregate on demand
▪ avoid/minimise background aggregation tasks and
defer until request

42. GRAFANA V3

43. ROADMAP
don’t stop ‘til you get enough…

44. FUTURE FUNCTIONALITY
▪ derived granularity aggregates
▪ compute annual aggregates using monthly/daily/hourly
aggregates
▪ rolling upgrades
▪ fair scheduling

45. thank you