Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Gnocchi v3

Benchmarking Gnocchi v3

  • Be the first to comment

  • Be the first to like this

Gnocchi v3

  1. 1. Gnocchi v3
  2. 2. Optimisation opportunities ▹ Improve coordination ▸ A lot of contention for same job among workers resulting in wasted operations ▹ Performance degradation in large series ▸ Larger file == slower read ▸ Larger file == slower write
  3. 3. Coordinated Scheduling Minimising contentions
  4. 4. metricdmetricd worker worker worker worker worker worker Current distribution ▹ Agents unaware of each other ▹ Workers unaware of each other ▹ Each worker queries backlog, grabs a chunk starting from offset based on id + Workers are completely dumb - Workers are completely dumb
  5. 5. metricd scheduler worker worker worker metricd scheduler worker worker worker Proposed distribution ▹ Agents aware of each other ▹ Workers unaware of each other ▹ A central process queries backlog and pipes jobs to common queue for workers ▹ Each agent grabs a chunk starting from offset based on something. + More coordination, less contentions + One query to control them all - More management
  6. 6. Testing methodology (small series) ▹ POST 1000 generic resources spread across 20 workers, 20 metrics each ▸ every 5 minutes ▸ 60 points/metric per POST ▸ 1.2 million points every 5 min ▸ 3 granularities x 8 aggregates ▸ ~500 points in most granular series ▹ 3 metricd services, 24 workers each
  7. 7. Processing time ~4% to ~23% improvement.
  8. 8. Minimising operations Writing with Offsets
  9. 9. v2.x {‘values’:{<timestamp>: float, <timestamp>: float, ... <timestamp>: float}} New serialisation format v3.x Uncompressed: Delimiter+float+delimiter+float+ ...+delimiter+float Compressed: Time+float+time_delta+float+ ...+time_delta+float
  10. 10. Testing methodology (small series) ▹ POST 1000 generic resources spread across 20 workers, 20 metrics each ▸ every 5 minutes ▸ 60 points/metric per POST ▸ 1.2 million points every 5 min ▸ 3 granularities x 8 aggregates ▸ ~500 points in most granular series ▹ 3 metricd services, 24 workers each
  11. 11. Amount of time required to compute 1.2M points into 20K metrics of 24 different aggregations.
  12. 12. Testing methodology (medium series) ▹ POST 500 generic resources spread across 20 workers, 20 metrics each ▸ every 5 minutes ▸ 720 points/metric per POST ▸ 7.2 million points every 5 min ▸ 3 granularities x 8 aggregates ▸ ~7000 points in most granular series ▹ 3 metricd services, 24 workers each
  13. 13. Time to compute 720 data points into 24 different aggregates and store data
  14. 14. Total metrics processed. 10K metrics sent periodically. A higher slope represents faster processing rate
  15. 15. Gnocchi v2.1.4 vs Gnocchi v3.0 Old vs New
  16. 16. Release notes v3 ▹ New storage format for new, back window and aggregated series (msgpacks vs struct serialisation) ▹ Storage compression ▹ No-read append writes (Ceph only) ▹ Dynamic resource configuration ▹ Coordinated task scheduling ▹ Performance related changes to aggregation logic ▹ Grafana 3.0 support
  17. 17. Computation time per metric Amount of time required to compute new measures into 24 different aggregates. ~60% less processing time at lower unprocessed sizes ~40% less processing time at higher unprocessed sizes
  18. 18. Write throughput Data generated using benchmark tool in client. 32 single-threaded api server processes, 4x12threads client. Gnocchi writes to disk but will be enhanced to write to memory (for Ceph)
  19. 19. Average time to write measures
  20. 20. POST per second The number of metrics that can be pushed to 32 API workers.
  21. 21. Response time. Majority of time taken by client side processing and formatting.
  22. 22. Disk size. Live datapoints are 18B in v2.1 and at worst 8B (9B in Ceph) in v3. Compression is applied which additionally lowers size. v2.1 datapoints were serialized using msgpacks. In v3, the storage format is optimised for space efficiency and compression.
  23. 23. Testing methodology (short series) ▹ POST 1000 generic resources spread across 20 workers, 20 metrics each ▸ every 5 minutes ▸ 60 points/metric per POST ▸ 1.2 million points every 5 min ▸ 3 granularities x 8 aggregates ▸ ~500 points in most granular series ▹ 3 metricd services, 24 workers each
  24. 24. Amount of time required to process 1.2M points across 20K metrics into 24 different aggregates per cycle.
  25. 25. Number of metrics processed per 5s. No degradation between batches and more consistent processing rates V2.1.4 averages ~6 metrics (144 aggregates) calculated every 5 seconds per worker. V3.0 averages ~10.88 metrics (261 aggregates) calculated every 5 seconds per worker.
  26. 26. Amount of Ceph IOPs required to process 1.2M points across 20K metrics into 24 different aggregates per cycle. Less operations/s represents lower hardware requirements
  27. 27. Number of times a worker attempts to handle a metric that has already been handled. Less contentions represents better job scheduling.
  28. 28. Time required to POST 1.2M points across 20K metrics under load. 20 workers making 50 POSTs of 20 metrics, 60 points per metric.
  29. 29. Testing methodology (medium series) ▹ POST 500 generic resources spread across 20 workers, 20 metrics each ▸ every 5 minutes ▸ 720 points/metric per POST ▸ 7.2 million points every 5 min ▸ 3 granularities x 8 aggregates ▸ ~5760 points in most granular series ▹ 3 metricd services, 24 workers each
  30. 30. Amount of time required to process 7.2M points across 10K metrics into 24 different aggregates per cycle.
  31. 31. More insights Gnocchi v3
  32. 32. Threading enabled. Python threading only works when I/O heavy (see: GIL). CPU usage has been minimised and now threading works 54 metricd with 8 threads has roughly same performance as 72 single-threaded metricd.
  33. 33. Effect of number of aggregates on processing time. Less aggregates/metric, less time to process. More aggregates/metric, more time. Note: spike at batch 5 is due to compression logic that is triggered by dataset.
  34. 34. Effect of series length on datapoint size. Long, medium, short series contain 83K, 1.4K, and 400 points respectively. Shorter series are more likely to have to-be-deleted stale data as well as less compression opportunities (latter applicable only to Ceph).
  35. 35. Effect of series length on datapoint size. Shorter series tend to have higher granularity and thus larger back window requirements.
  36. 36. Future Dreams
  37. 37. Potential future work ▹ Dynamic re-aggregation ▹ Python3.5 AsyncIO threading support ▹ Rolling upgrades ▹ Ceph asynchronous writes ▹ In-memory storage to improve write performance
  38. 38. Appendix
  39. 39. Test Configuration
  40. 40. Hardware configuration ▹ 4 physical hosts ▸ CentOS 7.2.1511 ▸ 24 physical cores (hyperthreaded), 256 GB memory ▸ 25 - 1TB disks, 10K RPM ▹ 1Gb network
  41. 41. Host configuration ▹ Host1 ▸ OpenStack Controller Node ▸ Ceph Monitoring service ▹ Host2 ▸ OpenStack Compute Node (idle) ▸ Ceph OSD node (10 OSDs + SSD Journal) ▹ Host3 ▸ Ceph OSD node (10 OSDs + SSD Journal) ▸ Gnocchi API (32 workers) ▹ Host4 ▸ OpenStack Compute Node (~idle) ▸ Ceph OSD node (10 OSDs + SSD Journal) ▸ PostgreSQL
  42. 42. Services ▹ PostgreSQL 9.2.15 (single node) ▸ Default everything, except 300 connections vs 100(default) ▹ Ceph 10.2.2 (4 nodes, 1 monitoring, 3 OSD) ▸ 30 OSDs (1 TB disk), Journals share SSD, 2 replica, 2048 placement groups ▹ Gnocchi MetricD ▸ 3 nodes, shared with Ceph OSDs ▸ 24 workers per service

    Be the first to comment

    Login to see the comments

Benchmarking Gnocchi v3

Views

Total views

1,766

On Slideshare

0

From embeds

0

Number of embeds

1

Actions

Downloads

22

Shares

0

Comments

0

Likes

0

×