4. Testing Methodology
- Start 3 metricd services - 24 workers each
- POST 1000 generic resources spread across 20 workers, 20 metrics each.
- POST Every 10 minutes
- 1 minute granularity, 10 points/metric/request
- 20 000 metrics, medium archive policy
- 1 min for a day, 1 hr for a week, 1 day for a year, 8 aggregates each
5. Batch1 metricd details
- POST time (50 posts) - avg=10.8s (-65.5%), stdev=0.79
- Injection time - ~ 144 seconds
- Stats
- Per metric injection - avg=0.462s, min=0.235s, max=1.693s, stdev=0.174
- Average IO time - ~66% of _add_measures()
- Overhead - ~10.8% (~9.89% minus all IO once metric locked)
- Comparison to 20OSD w/ shared journal
- POST - 65.5% quicker
- Injection time - 27% quicker
6. Batch2 metricd details
- POST time (50 posts) - avg=30.6s, stdev=2.72
- Injection time - ~ 400 seconds
- Stats
- Per metric injection - avg=1.316s, min=0.286s, max=5.758s, stdev=0.844
- Average IO time - ~76.0% of _add_measures()
- Overhead - ~9.23% (~6.78% minus all IO once metric locked)
- Comparison to 20OSD w/ shared journal
- POST - 70% quicker
- Injection time - 28.4% quicker
7. Batch3 metricd details
- POST time (50 posts) - avg=30.2s, stdev=2.87
- Injection time - ~ 408 seconds
- Stats
- Per metric injection - avg=1.33s, min=0.285s, max=5.647s, stdev=0.824
- Average IO time - ~74.9% of _add_measures()
- Overhead - ~9.58% (~6.95% minus all IO once metric locked)
- Comparison to 20OSD w/ shared journal
- POST - 65.4% quicker
- Injection time - 26% quicker
23. Optimisation Opportunities
- Gnocchi has a lot of IO
- By default, over 25 reads and 25 writes for every single metric
- Serialising and deserialising each time
- Degradation as number of points grows (up to object split size)
- Needs to read in full object with related points, update, and write full object for each aggregate
even if updating one point out of thousands.
26. Serialisation Format
Existing
{‘values’:{<timestamp>: float,
<timestamp>: float,
...
<timestamp>: float}}
- ~18B/point or ~10B/point (compressed)
- Not appendable
- Msgpack serialisation, super fast
Proposed
delimiter+float+delimiter+float+.
..+delimiter+float
- 9B/point (or much if compressed)
- Appendable
- Delimiter can be used to describe subsequent
bytes
- Timestamp computed by offset
- eg. Position 9 to 17 is data x seconds from
start
- Zero padding required if first point not start of split
- Handles compression much better
28. Looking to 3.x
- Testing larger datasets (a few thousand points/metric)
- Benchmarking new proposed format
- Study effects of alternative storage solutions
- Try to add in support for intermediary storage in memory
