Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Gnocchi v3 brownbag

slides from my brownbag talk.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

Gnocchi v3 brownbag

  1. 1. HUAWEI CANADA Gnocchi v3 Monitoring the next million time-series Gordon Chung, Engineer
  2. 2. HISTORY do you remember the time…
  3. 3. built to address storage performance issues encountered in Ceilometer
  4. 4. designed to be used to store time series and their associated resource metadata Metric storage (Ceph) MetricD Computation workers data stores aggregated measurement data stores metadata background workers which aggregate data to minimise query computations LoadBalancer APIAPIAPI Indexer (SQL)
  5. 5. MY USE CASE tired of you tellin' the story your way…
  6. 6. collect usage information for hundreds of thousands of metrics* over many months for use in capacity planning recommendations and scheduling * data is received in batches every x minutes. not streaming
  7. 7. GETTING STARTED wanna be startin’ something’…
  8. 8. HARDWARE ▪ 3 physical hosts ▪ 24 physical core ▪ 256GB memory ▪ a bunch of 10K 1TB disks ▪ 1Gb network
  9. 9. SOFTWARE ▪ Gnocchi 2.1.x (June 3rd 2016) ▪ 32 API processes, 1 thread ▪ 3 metricd agents (24 workers each) ▪ PostgreSQL 9.2.15 – single node ▪ Redis 3.0.6 (for coordination) – single node ▪ Ceph 10.2.1 – 3 nodes (20 OSDs, 1 replica)
  10. 10. POST ~1000 generic resources with 20 metrics each (20K metrics) 60 measures per metric. policy rolls up to minute, hour, and day. 8 different aggregations each*. * min, max, sum, average, median, 95th percentile, count, stdev
  11. 11. METRIC PROCESSING RATE • rate drops significantly after initial push • high variance in processing rate
  12. 12. uhhh… wtf? this doesn’t happen in NFS backend.
  13. 13. “LEARNING” HOW TO USE CEPH everybody's somebody's fool…
  14. 14. give it more power! add another node… and 10 more OSDs… and more PG groups… and some SSDs for journals
  15. 15. ~65% better POST rate
  16. 16. ~27% better aggregation rate
  17. 17. METRIC PROCESSING RATE (with more power) • same drop in performance
  18. 18. ““LEARNING”” HOW TO USE CEPH this time around…
  19. 19. CEPH CONFIGURATIONS original conf [osd] osd journal size = 10000 osd pool default size = 3 osd pool default min size = 2 osd crush chooseleaf type = 1 [osd] osd journal size = 10000 osd pool default size = 3 osd pool default min size = 2 osd crush chooseleaf type = 1 osd op threads = 36 filestore op threads = 36 filestore queue max ops = 50000 filestore queue committing max ops = 50000 journal max write entries = 50000 journal queue max ops = 50000 good enough conf http://ceph.com/pgcalc/ to calculate required # of placement groups
  20. 20. METRIC PROCESSING RATE (varying configurations) shorter the horizontal length equals better performance. Higher the spikes equals quicker rate.
  21. 21. IMPROVING GNOCCHI take a look at yourself, and then make a change…
  22. 22. computing and storing ~29 aggregates/worker per second is not bad
  23. 23. we can minimise IO
  24. 24. MINIMISING IO - each aggregation requires: 1. read object 2. update object 3. write object - with Ceph, we can just write to save.
  25. 25. NEW STORAGE FORMAT V2.x {‘values’:{<timestamp>: float, <timestamp>: float, ... <timestamp>: float} } msgpacks serialised <time><float><time><fl oat>…<time><float> binary serialized and lz4 compressed V3.x
  26. 26. asking questions about code
  27. 27. why is this so long? update existing aggregates retrieve existing aggregates why we call this so much? writing aggregates
  28. 28. BENCHMARK RESULTS showin' how funky strong is your fight…
  29. 29. WRITE THROUGHPUT - ~970K measures/s with 5K batches - ~13K measures/s with 10 measure batch - 50% gains at higher end
  30. 30. READ PERFORMANCE - Negligible change in response time. - Majority of time is client rendering
  31. 31. COMPUTATION TIME - ~0.12s to compute 24 aggregates from 1 point - ~4.2s to compute 24 aggregates from 11.5K points - 40%-60% quicker
  32. 32. DISK USAGE - 16B/point vs ~6.25B/point (depending on series length and compression schedule)
  33. 33. OUR USE CASE - Consistent performance between batches - 30% to 60% better performance - more performance gain for larger series.
  34. 34. OUR USE CASE - 30% to 40% less operations required
  35. 35. now computing and storing ~53 aggregates/worker per second.
  36. 36. USAGE HINTS what more can i give…
  37. 37. EFFECTS OF AGGREGATES - 15%-25% overhead to compute each additional level of granularity - percentile aggregations requires more CPU time
  38. 38. THREADING - set `aggregation_workers_number` to the number of aggregates computed per series
  39. 39. metricD agents and Ceph OSDs are CPU-intensive services
  40. 40. EXTRAS they don’t care about us…
  41. 41. ADDITIONAL FUNCTIONALITY ▪ aggregate of aggregates ▪ get max of means, stdev of maxs, etc… ▪ dynamic resources ▪ create and modify resource definitions ▪ aggregate on demand ▪ avoid/minimise background aggregation tasks and defer until request
  42. 42. GRAFANA V3
  43. 43. ROADMAP don’t stop ‘til you get enough…
  44. 44. FUTURE FUNCTIONALITY ▪ derived granularity aggregates ▪ compute annual aggregates using monthly/daily/hourly aggregates ▪ rolling upgrades ▪ fair scheduling
  45. 45. thank you

    Be the first to comment

    Login to see the comments

  • williammdavis

    Nov. 30, 2016

slides from my brownbag talk.

Views

Total views

544

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

16

Shares

0

Comments

0

Likes

1

×